Convert CSV to Parquet and Schedule Pipeline in ADF Studio Assignment

Assignment Task

1) Create a pipeline in ADF Studio to convert a csv file to parquet format.

• The pipeline should read csv file from ADLS Gen2 and convert it as parquet and load back to ADLS Gen2
• You can use the attached csv file. Create a test folder in ADLS Gen2 container and upload it.
• Write the parquet to the same test folder.

2) Create a trigger and add it to the above created pipeline to trigger the pipeline every day at 10 AM.

3) Create a dataflow to read csv file data with 14 columns from ADLS Gen2 and write(sink) only 9 columns data by removing 5 columns.

• Input columns:
  • Series_reference, Period, Data_value, Suppressed, STATUS, UNITS, Magnitude, Subject, Group, Series title 1, Series_title_2, Series_title_3, Series_title_4, Series_title_5
• Output columns:
  • Series_reference, Period, Data_value, Suppressed, STATUS, UNITS, Magnitude, Subject, Group

This Arts and Humanities has been solved by our PhD Experts at My Uni Paper.