Implementing a character-level Trigram Language Model Assignment

Assignment Task

Corpus Creation

You will first create the corpus suitable for building a language model using bigrams and trigrams. The JSON-encoded corpus is found. here. The compressed file contains around 56500+ JSON files.

Preprocessing

Analyze the corpus carefully. Recommend a set of Preprocessing steps and implement them. Keep the preprocessed file safe. You will need it for other assignments.

Bigram and Trigram Language Models

Build bigram and trigram models. Save the models in a folder. Use Laplacian or add-1 smoothing for computing the probability score.

Predicting the missing text

For the following sentences, find the missing word/words. For every missing word, list the top ten most common words with probability. Use all two models to predict the missing words.

Perplexity

Find the perplexity score for the following sentence 3 it appears that the overall code stroke volume has decreased since the COVID-19- pandemic. half a century ago hypertension was not treatable. Sarah's tv is broadcasting an advert for private healthcare. Use bigram and trigram models to compute the perplexity score.

This IT and Computer Science has been solved by our PHD Experts at My Uni Paper.