Introduction to Data Mining - Computer Science Assignment Help

Assignment Task:

Assignment 1 

Introduction 
This assignment involves a bioinformatics problem, specifically, unsupervised clustering of gene-expression micro-array data. Two scenarios are considered: 
Clustering different tissue samples based on their gene-expression levels across multiple genes (Activity 1)Clustering genes according to their gene-expression levels across multiple experimental conditions (Activity 2 and Activity 3). Activity 1: Clustering cancerous tissue samples In this first activity, we consider the problem of clustering tissue samples based on their gene-expression levels. This is a relevant problem in bioinformatics as it can help the discovery of different subtypes of cancer. In particular, we will use a dataset from the study in Golub et al. (1999), which contains 72 human samples with leukemia. The expression levels of 1868 selected genes have been measured in all these samples. The dataset thus contains 72 observations (rows) and 1868 variables (columns). It is available in the file golub- 1999-v1_database.arff. 
Important notes: 
An ARFF file is basically a CSV file with some metadata at the beginning. The metadata could be removed manually using a text editor and then the data could be saved as a CSV file, but there is no need for that as this type of file can be straightforwardly read into a data frame in R using, for example, the function read.arff() from the package foreign . Use this option in your assignment. Once you read the file, you will notice in the resulting data frame that the dataset actually contains 1869 (rather than 1868) columns. In fact, there is an additional, rightmost column (column 1869, named Classe ), with class labels. These labels (‘1’ or ‘2’) indicate the subtype of leukemia associated with each sample. This information is available from specific domain knowledge. In particular, it is already known that there are 47 tissue samples of subtype ALL (class ‘1’) and 25 samples of subtype AML (class ‘2’). These class labels will not be used for clustering, but only for external assessment of the results. The goal is to assess to what extent the two subtypes of leukemia can be revealed as clusters in a completely unsupervised way. 
You are asked to: 
1. Read the dataset directly from the ARFF file into a data frame. 
2. Set aside the rightmost column (containing the class labels) from the data, storing it separately from the remaining data frame (with the 1868 predictors). 
3. Use the 72 × 1868 data frame to compute a matrix containing all the pairwise Euclidean distances between observations, that is, a 72 × 72 matrix with distances between tissue samples according to their 1868 expression levels. This matrix must be of type dist , which can be achieved either by using the function dist() from the base R package stats or by coercion using the function as.dist() . 
4. Use the distance matrix as input to call the Single-Linkage clustering algorithm available from the base R package stats and plot the resulting dendrogram. Do not use any class labels to perform this step. 
5. Use the distance matrix as input to call the Complete-Linkage clustering algorithm available from the base R package stats and plot the resulting dendrogram. Do not use any class labels to perform this step. 
6. Use the distance matrix as input to call the Average-Linkage clustering algorithm available from the base R package stats and plot the resulting dendrogram. Do not use any class labels to perform this step. 
7. Use the distance matrix as input to call Ward’s clustering algorithm available from the base R package stats and plot the resulting dendrogram. Do not use any class labels to perform this step. 
8. Compare the dendrograms plotted in Items 4 to 7. Visually, the dendrograms suggest that some clustering algorithm(s) generate more clear clusters than the others. In your opinion, which algorithm(s) may we be referring to and why? In particular, in which aspects do the results produced by this/these algorithm(s) look more clear? Perform Item 9 below only for this/those algorithm(s). 
9. Redraw the dendrogram(s) for the selected algorithm(s) in Item 8, now using the class labels that you stored separately in Item 2 to label the observations (as disposed along the horizontal axis of the dendrogram). Do some prominent clusters in the dendrogram(s) correspond approximately to the classes (that is, the two subtypes of leukemia)? 
10. Repeat the analysis, now using normalised data. The 1868 predictors have not been normalised before computing the distance matrix in Item 3. Normalisation is a non-trivial aspect in unsupervised clustering, as there is no ground truth to assess whether or not it improves performance. On the one hand, it may prevent variables with wider value ranges to dominate distance computations, but on the other hand it may distort clusters by removing natural differences in variance that help characterise them as clusters. Normalisation thus becomes an aspect of Exploratory Data Analysis when it comes to clustering: the analyst will usually generate and try to interpret results both with normalised and non-normalised versions of the data. The type of normalisation depends on the application in hand. Here, we are computing Euclidean distance between rows of the dataset, so the type of normalisation that applies is typically the so-called z-score normalisation of columns, where each column is rescaled to have zero mean and standard deviation of 1. In this item, you are first asked to normalise the data this way before computing the distance matrix in Item 3.Then, repeat Items 4 to 9. Does normalisation improve or worsen the results in this dataset? 

Activity 2: Clustering genes (Part A) 
In this second activity, we consider the problem of clustering genes according to their gene-expression levels across different conditions in a controlled experiment. The goal is to identify genes that show similar expression patterns over a wide range of experimental conditions. This is a relevant problem in bioinformatics as it can, for example, help identify genes that share the same regulatory mechanisms or functions in an organism. In particular, we will use a dataset YeastGalactose from the study in Yeung, Medvedovic, and Bumgarner (2003), which is composed of the gene expression levels of a subset of 205 selected genes of the yeast Saccharomyces cerevisiae from 20 different measurements (experimental conditions). The dataset thus contains 205 observations (rows) and 20 variables (columns). It is available in the file yeast.arff. 

Important note: Once you read the file, you will notice in the resulting data frame that the dataset actually contains 21 (rather than 20) columns. In fact, there is an additional, rightmost column (column 21, named 
Classe ), with class labels. These labels (‘cluster1’, ‘cluster2’, ‘cluster3’ and ‘cluster4’) indicate genes whose expression patterns reflect four functional categories. Thus, there are four known categories of co-regulated genes in the data. This information is available from specific domain knowledge. These labels will not be used for clustering, but only for external assessment of the results. The goal is to assess to what extent the four categories of genes can be revealed as clusters in a completely unsupervised way. 
You are asked to: 
11. Read the dataset directly from the ARFF file into a data frame. 
12. Set aside the rightmost column (containing the class labels) from the data, storing it separately from the remaining data frame (with the 20 predictors). 
13. Use the 205 × 20 data frame to compute a matrix containing all the pairwise Pearson-based dissimilarities between observations, that is, a 205 × 205 
matrix with dissimilarities between genes according to their 20 expression measurements. Important Note: It is well-known that co-regulated genes are better characterised by similar trends in their gene expression profiles, rather than similar expression levels in terms of their absolute values. In other words, the similarity between genes in terms of their expression profiles for different measurements is better captured by a correlation measure, such as Pearson correlation (James, Witten, Hastie, & Tibshirani, 2013), which is the most widely adopted similarity measure for practical applications of gene clustering. For this reason, in this activity we will use Pearson correlation instead of Euclidean distance. However, recall that Pearson is a similarity measure that ranges from −1 (lowest similarity) to +1 (highest similarity). After computing the 205 × 205 Pearson similarity matrix, you have to convert it to a dissimilarity matrix whose values range from 0 (lowest dissimilarity) to +1 (highest dissimilarity). Once you have this Pearson-based dissimilarity matrix, you can coerce it into type dist as required by the hierarchical clustering methods in the base R package stats . 
14. Repeat the clustering analysis in Items 4 to 9 of Activity 1, now using the dissimilarity matrix for the YeastGalactose data computed in Item 13 (and, when applicable, the class labels that you stored separately in Item 12 to label observations as disposed along the horizontal axis of the relevant dendrograms). 

Activity 3: Clustering genes (Part B) 
Unlike the leukemia data in the first activity, which is very high-dimensional, the YeastGalactose dataset has only moderate dimensionality (20 dimensions), so density-based clustering algorithms may work in this scenario. In this activity we will experiment with the HDBSCAN* algorithm. 

Important note: One problem with the HDBSCAN* implementation that we are familiar with, available in the package dbscan , is that the version currently available (when this assignment was prepared) says that “Euclidean distance is required” (see ?dbscan::hdbscan ). So, although the theoretical HDBSCAN* model works with any distance, in principle we should not run HDBSCAN* directly with Pearson using this package. Apart from the possible existence of other R implementations of the algorithm that could be used instead, we will stick with the package dbscan here by using a mathematical workaround. Specifically, it can be shown that there is a relation between Pearson similarity and Euclidean distance when the observations are normalised as unit vectors, that is, when the rows of the data matrix are rescaled so that each row is a vector with magnitude one (i.e., length = 1). Clustering the normalised data with Euclidean distance is expected to provide results that are similar to those that would be obtained by clustering the original data with Pearson similarity. 
You are asked to: 205 
15. Rescale the × 20 data frame in a row-wise fashion so that each rescaled row has magnitude 1. You can achieve this by dividing each element of a row by the magnitude of the row. 
16. Run HDBSCAN* (with Euclidean distance) on the rescaled version of the data frame obtained in Item 
15. You can (optionally) try different values for the parameter MinPts , but MinPts = 5 is required. Plot the resulting HDBSCAN* dendrograms with and without the class labels along the horizontal axis, just like in Items 4–9 (Activity 1) and Item 14 (Activity 2). 
17. Plot a contingency table. By setting MinPts = 5 , the automatic cluster extraction method provided by HDBSCAN* extracts four clusters from the resulting hierarchy. Plot a contingency table of these clusters (labelled ‘0’, ‘1’, ‘2’, ‘3’ and ‘4’, where ‘0’ means objects left unclustered as noise/outliers) against the ground truth class labels that you stored separately in Item 12 (a factor with levels ‘cluster1’, ‘cluster2’, ‘cluster3’, ‘cluster4’). 
18. Interpret the contingency table. In particular: 
(a) What is the best correspondence between the four found clusters and the clusters according to the ground truth, that is, the best association between cluster labels ‘1’, ‘2’, ‘3’ and ‘4’ as named by HDBSCAN* and the four known functional categories ‘cluster1’, ‘cluster2’, ‘cluster3’ and ‘cluster4’ as named in the ground truth? 
(b) What is the functional category for which most genes have been labelled as noise/outliers? 
19. Plot the genes grouped by their class labels (that is, functional categories ‘cluster1’, ‘cluster2’, ‘cluster3’ and ‘cluster4’), in such a way that all the genes belonging to the same class are plotted in a separate sub-figure (four sub-figures in total, each one in a different colour). Plot each gene as a time-series with 20 data points (where each point is connected by lines to its adjacent points in the series). 
20. Plot a figure analogous to the one in Item 19, but now with genes grouped in separate sub-figures according to their cluster as assigned by HDBSCAN* (‘1’, ‘2’, ‘3’ and ‘4’), rather than by class labels. Do not plot genes that were left unclustered as noise by HDBSCAN* (labelled ‘0’). Use the best class-to- cluster association, as in your answer to Item 18, in order to assign each sub-figure of a cluster the same colour used in the sub-figure of the corresponding class in Item 19. For instance, supposing that the best association of class ‘clusterX’ in the ground truth is with HDBSCAN* cluster ‘Y’, according to the contingency table in Item 18, then if the genes belonging to class ‘clusterX’ have been plotted in red in Item 19, then the genes belonging to HDBSCAN* cluster ‘Y’ should also be plotted in red. 
21. Compare the pairs of sub-figures with the same colour in the plots of Items 19 and 20. In particular: 
(a) Visually, do the genes in each cluster found by HDBSCAN* (that is, each sub-figure in Item 20) correspond reasonably well to the associated functional category in the ground truth (that is, the corresponding sub-figure in Item 19)? 
(b) Look at the contingency table for the functional categories that have had genes left unclustered as noise. Now look at the corresponding pairs of sub-figures in Items 19 and 20, noticing that these genes are plotted in Item 19 but not in Item 20. Does the removal of these genes make the most prominent pattern in each cluster visually more clear (which would indicate that those genes labelled as noise by HDBSCAN* are indeed outliers)? 
 

This Computer Science Assignment has been solved by our Computer Science Experts at My Uni Paper. Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our Experts are well trained to follow all marking rubrics & referencing style.

Be it a used or new solution, the quality of the work submitted by our assignment experts remains unhampered. You may continue to expect the same or even better quality with the used and new assignment solution files respectively. There’s one thing to be noticed that you could choose one between the two and acquire an HD either way. You could choose a new assignment solution file to get yourself an exclusive, plagiarism (with free Turnitin file), expert quality assignment or order an old solution file that was considered worthy of the highest distinction.