COMP6210 - Big Data - Macquarie University

Assignment Task

1. MapReduce: Calculate the number of movies released by each production company for each year in the dataset. For example, determine the count of movies released by Warner Bros, Walt Disney Pictures, and Paramount Pictures in the year of 2015, respectively. You can complete this task by following these steps:

1. Data Retrieval: Access the movie - date path to obtain the release date. Extract the last four characters of the release date to obtain the movie’s release year.

2. Company Identification: Access the 'movie  companies' path to identify the production companies associated with the movie. Note that a movie may be jointly produced by multiple companies, but for our task, we only need to count the top three production companies. This means that if a movie is produced by five companies, only the first three will be considered.

3. Data Formatting: Generate a series of pairs for each movie in the following format: This can be achieved by combining the movie's release year (extracted in Step 1) with the names of the top three production companies (identified in Step 2).

4. Data Storage: Store pairs of all movies into a text file. This text file will serve as the input data for the subsequent MapReduce program.

5. MapReduce Implementation: Implement a MapReduce program using mrjob to calculate the frequency of each pair. In other words, determine how many movies were released by each production company for each year.

2. Implement the following sorting algorithms:

i. Merge Sort : Implement the MapReduce program, named  to sort the pairs in  in ascending order. This sorting is based on the count of movies, and the sorted results should be stored in the task2.

ii. Bucket Sort: Implement the MapReduce program, named task2 to sort the pairs in descending order. This sorting is based on the count of movies, and the sorted results should be stored in the task2.

This COMP6210 - IT Computer Science has been solved by our PhD Experts at My Uni Paper.