CISC5950 - Big Data Computing Programming Assignment

Assignment Task

 Topics

1. Set up a 3-node cluster with Hadoop Distributed File System and run examples.

2. On top of HDFS, set up the cluster with MapReduce programming framework.

3. Run examples of MapReduce programs.

4. Scheuling on the Cloud. In this project, we are going to design our own Hadoop MapReduce-based program to analyze the data. The project consist of two parts.

NY Parking Violations

The NYC Department of Finance collects data on every parking ticket issued in NYC ( 10M per year!). This data is made publicly available to aid in ticket resolution and to guide policymakers

The above figure shows several records, where each row represents a parking ticket and the columns are the details of the tickets. To start the project, you have to,

1. Start the 3-node cluster

2. Set up the HDFS

3. Store the data in HDFS

4. Set up the MapReduce framework along with the scheduler for resource management.

By analyzing the data, we need to answer the following

• When are tickets most likely to be issued?
• What are the most common years and types of cars to be ticketed?
• Where are tickets most commonly issued?
• Which color of the vehicle is most likely to get a ticket?

Question

The biggest challenge when using K-Means is to decide on the number of clusters. Having more clusters creates some small classes with very few records, while having less clusters leads to classes that are too general. Based on a K-Means algorithm above, try to answer the following question,

• Given a Black vehicle parking illegally at 34510, 10030, 34050 (street codes). What is the probability that it will get an ticket? (very rough prediction).
• At 10 am, I want to go to Lincoln Center and I just want to walk within 0.5 mile. Where should I park? (Divided into zones).

This IT Computer Science has been solved by our PhD Experts at My Uni Paper.