Text Analytics Worksheet - Report Writing - Engineering Assignment Help

Assignment Task

Overview
In this coursework, you will develop a text analytics system for an example scenario and evaluate its behaviour. All of the tasks are designed to allow you to solve them in your own way–creative and innovative solutions are encouraged!

Task Specification
Imagine you are a Data Scientist who is tasked with helping scientists understand the ever-growing body of research in their field. Your clients’ problem is that many new papers are published every day and a simple keyword search is unable to pinpoint all the relevant information inside these papers. In trying to understand the literature in their specific field, the scientists often ask questions like:
Where can I find information about a particular scientific question, problem or task?
What kind of methods, techniques and resources have been applied to this task?
How do these methods relate to other methods I’ve heard about?

This assignment will guide you through the steps of prototyping a solution for your clients. You will use text analytics to help scientists discover different types of entities, how they relate to each other, and analyse the information you extract. While the clients in our scenario are scientists, the same kinds of problems apply in many other knowledge-intensive domains.

As you work through the tasks below, you will run several experiments and write a report that explains which methods you tried, why you tried them, and your findings. We recommend that you first get a basic implementation for all parts of the required assignment, then start writing your report with some results for all tasks. You can then gradually improve your implementation and resultsFor each task:
In your report, write a section that motivates your chosen solution by briefly describing the method and its advantages/disadvantages. You should design methods that incorporate ideas from the lectures and, where suitable, discuss concepts covered in the lectures (e.g., the limitations of bag of words, the value of syntactic information, learning vs. knowledge, relational/compositional/contextual meaning, etc.). You may wish to include a system diagram showing your NLP pipeline or deep neural network. If you experiment with different features or hyperparameter settings on the development set, you might want to include a plot or table showing the results.

Dataset:
We will be working with the ScienceIE dataset, which consists of abstracts of scientific articles. The dataset is already split into training, development and testing sets. The dataset directory includes the following:
A Python module, scienceie_loader.py, containing a data loader, which returns a list of tokens, sequence labels and relations for each split of the dataset.
The scienceie_loader.py module also contains a function that can return the raw text, with the sequence labels and relations defined by character offsets. This is needed if your proposed solution requires untokenised text or uses its own tokenizer (e.g., if using the HuggingFace Transformers library).
data_loader_demo.ipynb, a notebook with an example of how to call the scieneie_loader functions.
The original ScienceIE release of the dataset, including readme files.
The original annotation guidelines that clearly define what each type of entity or relation means.
The data was originally released for a competition/shared task and is described in full in this paper:
Augenstein, I., Das, M., Riedel, S., Vikraman, L., & McCallum, A. (2017, August). SemEval 2017 Task 10: ScienceIE-Extracting Keyphrases and Relations from Scientific Publications. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) (pp. 546-555).
You can find out more about the dataset by reviewing the related literature, e.g., by searching for the topic on the ACL Anthology or searching for the above paper on Google Scholar. An example of the kind of annotations in the ScienceIE dataset is shown below: Link to publication

Task 1: Extracting Entities from Scientific Abstracts
The first step is to extract the relevant types of entities from the abstracts. The entity types are:
Task: specific research tasks (e.g., ‘dependency parsing’) and broader research areas (e.g., ‘machine learning’).
Process: methods/techniques/algorithms, physical equipment and software tools.
Material: physical materials, datasets/corpora and other resources used to solve the problems in a scientific paper.

Part 1.1. Implement and train two methods for extracting these three entity types from scientific abstracts. You can refer to the labs, lecture materials and textbook for this course to identify a suitable method. You can use the training set to train a model, and the development set if you need to tune hyperparameters. Choose suitable pre-processing steps for your selected methods.
The two methods you compare should employ different kinds of model (e.g., an HMM and a neural network). If the methods both use machine learning, they should also use two different sets of features (e.g., POS tags + unigrams vs. word embeddings). It is not necessary to test all combinations of feature sets with methods.
Explain your chosen methods and features in your report. Hypothesise how the results of the two methods will differ and use the evaluation in 1.2 to validate your hypothesis.

Part 1.2. Evaluate your method on the test set and include the results as a table or plot. Interpret and discuss your results, showing representative examples of errors from each method to aid your discussion. Imagine that you are building a real system for your clients and carry out this evaluation to help you and your clients decide which approach is most suitable for entity extraction. Some points you may wish to include in your discussion:
Which performance metrics did you choose and what limitations do they have?
How does the performance of your two methods compare?
What is the effect of the different features?
Does either method make any common types of errors?
How could you improve the methods?
Suggested length of report for task: 1-2 pages

Task 2: Extracting Semantic Relations (25 marks)
Now that we have a way to identify entities of interest, we would like to understand how they relate to one another. The ScienceIE dataset defines two types of semantic relation:
Synonym-of: the two entity phrases refer to the same thing. If this relation holds, <entity A> is the same as <entity B>.
Hyponym-of: the first entity is a more specific word then the second. The second entity is a category or more general term to which the first belongs. If this relation holds, <entity A> is an <entity B>, but not vice versa.
Part 2.1. Implement and train one method for predicting these two types of semantic relation between pairs of entities that are annotated in the dataset. It is up to you to choose a suitable method and features (e.g., based on methods presented in the labs and lectures). In the report, explain the method and how you chose it.
For the entity annotations, you may use either your best entity extraction method from task 1 or the gold standard entities in the test set. In your report, explain which source of entity annotations you use and what effect you think that may have on the results.
Part 2.2. Evaluate your results on the test set and include the results as a table or plot. Discuss and interpret your results and try to analyse the errors your system makes, e.g., by showing a confusion matrix or representative examples of common errors. Are there any topics in the dataset where your method works well/badly? What kind of additional features or model improvements do you think could improve performance?
Suggested length of report for task 2: 2 pages

Task 3: Exploring the Data
Now that you have developed methods for entity and relation extraction, the task is to use this information to explore the dataset. The plots you will create in this section should demonstrate how your text analytics system could help your imaginary clients to understand how entities relate to one another.
Choose two example entities from your predictions on the test set to use as queries. Then, for each query entity:
3.1. Visualise a graph of the relations between the query entity and other entities. Include and discuss the visualisation in your report.
3.2. Visualise the entities that most commonly co-occur with the query entity in the same abstractor are most similar to the query entity.
Be as creative as you like with your use of visualisation but remember that clarity is more important than complexity. Hints: co-occurrences/similarities can be visualised as a graph or word cloud.

This Engineering Assignment has been solved by our Engineering experts at My Uni Paper. Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our Experts are well trained to follow all marking rubrics & referencing style.
Be it a used or new solution, the quality of the work submitted by our assignment experts remains unhampered. You may continue to expect the same or even better quality with the used and new assignment solution files respectively. There’s one thing to be noticed that you could choose one between the two and acquire an HD either way. You could choose a new assignment solution file to get yourself an exclusive, plagiarism (with free Turnitin file), expert quality assignment or order an old solution file that was considered worthy of the highest distinction.