Malware Identification - PCA and Visualisation - Dummy Variables - Engineering Assignment Help

Assignment Task:

SCENARIO
You have been brought on as part of a data analysis team to help ‘fix’ the problems of the Email Sentry product. The existing code for Email Sentry relied on assumptions made by developers to detect malicious software. These assumptions were guided by experience and ‘gut feel’ and have no statistical basis. Data from the previous Email Sentry product has been extracted (MalwareDataSet.XLS) and provided to the development team. The initial goals of the Email Sentry 2.0 team
are:
Identify whether PCA of the MalwareDataSet could be used to identify and discriminate between emails.
Create a brief report to the rest of the research team that will describe whether PCA could be used to effectively identify malicious emails.

Part 1 – Data analysis using PCA and report on findings
a) You must clean and standardised the data to make it usable in “R”.
- To run PCA on this data set,
(1) First you will need to address the NAs appropriate prior to PCA. That is, you will need to replace the missing values for outside.network and
Verified.as.Malware with the appropriate value.
(2) Then, you will need to convert the category variables into something usable noting that PCA will only accept numerical values. Hint: Google the terms “dummy variables”. Beware of functions that generate the two dummy columns for each variable. You should remove one of them prior to running PCA and be consistent with your removal across all the variables. Note that retaining the redundant dummy column does not affect the final decision from your PCA analysis, it is more that your biplot will not be as readable.
- Briefly report on the data manipulation, i.e. from (1) and (2), that was required to make the data usable in R.
- Export your “cleaned” data as follows.
#Write to a csv file. This will need to be submitted write.csv(mydata,"mydata.csv")
b) Perform PCA on the 9 features (1st 9 columns in mydata only, excluding Verified.asMalware) using prcomp(.) in R.
- Outline the individual and cumulative proportions of variance explained by each of the first 3 components.
- Outline the coefficients (or loadings) for PC1 to PC3.
c) Create a scree plot and together with part b) outline how many principal components do you believe are adequate to explain at least 50% of the variability in your data.
d) Create a biplot with PC1 and PC2 to help visualize the results of your PCA in the first two dimensions. Colour code the points with the variable Verified.as.Malware. Write a short paragraph to explain what your biplot is showing. That is, comment on the PCA plot, the loading plot and then both combined (see Slides 28-29 of Module 3 notes)

Assignment 1 
Describe
(1) which dimension(s) (if any) can assist with the classification of malwares (Hint: project all the points in the PCA plot to PC1, i.e. horizontal axis and
see whether there is good separation between the points for malware and non-malware. Then project to PC2, i.e. vertical axis and see if there is separation between the malware and non-malware, and whether it is better than the projection to PC1).
(2) what are the key features in this dimension that can drive this process (Hint: based on your decision in part e)(1) above, examine the loadings from part b) of your chosen PC and choose those whose absolute loading (i.e. disregard the sign) is greater than 0.3).
f) Based on the features that you have identified in part e),
- Cross-tabulate each of your categorical feature(s) (if any) against the malware variable, i.e. Verified.as.Malware, using the table(.) function (see Workshop 1 notes for examples) and determine the count and percentage, i.e. N (%), relative to each level of the Malware variable.

What to report
1. Submit a single report (not exceeding 5 pages) containing:
a. explanation of data preparation steps
b. your implementation of PCA and interpretation of the contribution from each principal component
c. scree plot and its interpretation
d. biplot and its interpretation
e. your explanation of selection and contribution of the factors with respect to possible malware identification (attachment includes executable, sender outside network, email URL count)
f. cross-table tabulation of relevant features to malware status, with appropriate interpretation of results
g. any additional analysis and any discussion on data issues
2. If you use any references in your analysis or discussion outside of the notes provided in the unit, you must cite your sources.

This Engineering Assignment has been solved by our Engineering Experts at My Uni Paper. Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our experts are well trained to follow all marking rubrics & referencing style.

Be it a used or new solution, the quality of the work submitted by our assignment experts remains unhampered. You may continue to expect the same or even better quality with the used and new assignment solution files respectively. There’s one thing to be noticed that you could choose one between the two and acquire an HD either way. You could choose a new assignment solution file to get yourself an exclusive, plagiarism (with free Turnitin file), expert quality assignment or order an old solution file that was considered worthy of the highest distinction.