Apache Spark: Lending Club Statistics-Hadoop- Big Data Mining Assignment Help

Internal Code: IAH290

Big Data Mining Assignment Help:

Task: Task 1: Apache PIG – Analyzing LogFiles Dataset: RStudio CRAN Log-Files Instructions: Please deliver all commands in your documentation (use a word-document and convert it later into a pdf); Also add screen-shots of the various steps and of results; Use FOREACH- statements to reduce the dataset (keep only variables which are necessary to solve a task). 1. Import RStudio Log Files (package related) from month February 2018 into HDFS a. Download from RStudio CRAN log files page b. Unzip the files c. Import the complete directory into HDFS into a folder RLogFiles 2. Pig Latin: Top-100-packages (by operating system) a. Load log-file of 28st of February 2018 b. Dump the first 10 entries on screen (attach a screen shot into your report) to check if it works or not c. Count the number of occurences of different packages and print out the top-10-packages; d. Count the number of occurences of different packages by operating system; e. Store the results of both operations in HDFS; 3. sqoop, MySQL and R: a. Export the results of both operations (package frequencies and package frequencies by operating systems) via sqoop into MySQL; b. Access the tables by R/RStudio and display the results (Top-10-results in bar charts) 4. Pig Latin: Number of individual users each day a. Load the log-files into HDFS b. Count the number of distinct users each day 5. Pig Latin: Average number of packages downloaded by an individual user each day a. Load the log-files into HDFS b. Calculate the average number of packages download by an individual user each day 6. Pig Latin: Machine Learning with caret and mlr-packages a. Machine Learning is often done using the caret or the newer mlr-packages (check the CRAN webpages) b. We are interested how often both packages are used by R users: count the number of caret and mlr packages (separately) downloaded each day c. Visualize the results in R (line chart) [follow the step in No. 3 or import the results directly into R) 7. Pig Latin: Download volume (in MB) of Top-10-packages a. Use CRAN to find out the package size of the Top-10-packages (use Windows- Package file size) in MB. Round to 1 decimal place. b. Enter this information into a text file together with the name (should be the same as in log-files) c. Import this file into HDFS d. Load the file in Pig (I assume that the RStudio CRAN Log Files are available already) e. Filter out the Top-10-packages in Pig f. Add the size information g. Calculate the download volume of each of the 10 packages by day h. Export the results and display the results in R Task 2: Apache PIG and Apache Mahout – Predicting Loan Success Dataset: Lending Club Dataset (Q3/2017). – Filename: lc.2017q3.EvalExer.csv Instructions: Please deliver all commands in your documentation (use a word-document and convert it later into a pdf); Also add screen-shots of the various steps and of results; Use FOREACH- statements to reduce the dataset (keep only variables which are necessary to solve a task). 1. Import File into HDFS a. Download the file from Moodle – File: lc.2017q3.EvalExer.csv b. Unzip the file c. Check the structure and the variables in the file d. Load the file into HDFS system 2. Load the file into PIG: Data Understanding and Data Preparation a. Load the file into PIG from HDFS b. Check all variables in the file and clean the variables c. Filter the variables, generate new variables, etc. d. Copy the final set of variables into a new file and store it into HDFS 3. Split the file into train- and test-datasets a. Split the file using the provided macro in PIG into a training-file (70% of cases) and a test-file (30% of cases) b. Store both files into HDFS using PIG 4. Conduct a logistic regression using Apache Mahout a. Note: You cannot run a logistic regression in Mahout based on HDFS files. You have to export both files from HDFS first! b. Run a logistic regression on the training data set c. Check the model based on the test dataset d. Report the results Task 3: Apache Spark – Building Interest Rates Calculator Context You are working as Data Scientist in a project for Lending Club (check web page). The company wants to offer potential customers an online tool to predict potential interest rates based on the purpose and other variables. Your job is to set up a model to look for possible influences on interest rates (variable int_rate) and to set up a multivariate model to predict it. In the last step you must prepare a management presentation with core findings. Tasks: • Set up a new jupyter notebook • Use pandas and seaborn for visualization • Report all steps • Clean the dataset first (outside jupyter, use an editor!) • Target variable: interest rate • Import the dataset into hdfs • Apply CRISP DM Steps: • Preliminary Steps o Select the variables shown in the table above • Data Understanding o Analyze the variables in dataset you selected (Schema, First rows, Descriptive Statistics / Frequency Tables, (Charts),... • Data Preparation o Missing Values o Transformation of all categorical variables o Split into Test and Training Dataset • Modeling o Models ? Model 1: Multiple Linear Regression ? Model 2: Integrate polynomials (simply square the metric variables (measurement level scale) and interaction effects for selected variables! ? Model 3: Tune Model 2 (you can use a grid search or just manipulate some hyperparameters like regularization parameter, lambda etc.) o Evaluate all Model Fits ? Core Parameter is Coefficient of Determination R2 ? Always control for overfitting (just compare training and test datasets and reduce the complexity of the models if necessary) ? Check the distribution of the error part o Report a final model which fits best to the data (due to R2 and overfitting). • Management Presentation o Present the core finding on a maximum of 6 slides (only task 3). Summarize core findings. You are addressing General Management!