COMP30023: Web Crawler - Web Search Engines - IT Assignment Help

Assignment Task:

1 Background 

How do web search engines know what is on the web? They have to look for it the slow way, by following every link on the web. They use sophisticated algorithms to search efficiently. For example, they don't follow each link equally often; content that changes often is followed more often. 

In this project, you will write a simple code to crawl a web site. This will teach you about socket programming, which is fundamental to writing all internet applications, and also about the HTTP application layer protocol. The crawler must be written in C and cannot use any existing HTTP libraries.

2.1 Crawling 

The code should crawl as described in the previous section. 

No page should be fetched more than once. Two pages are considered to be the same page if the URLs indicate that they are the same. That is different from having the same URL, because multiple URLs can point to the same page. For example, in a document at http://www.comp30023.example/a.html, the URLs http://www.comp30023.example/b.html and /b.html refer to the same page. Relative URLs also exist, like b.html. However, pages one/b.html and two/b.html are different pages, despite ending in the same file name. The equivalence of pages is governed by the specification of URLs, given at the end of this document. 

The program should print a log of the URLs it attempts to fetch whether successfully or unsuccessfully. The log should be sent to stdout. Print one URL per line, with no other characters. The program should send no other output to stdout; other output may be sent to stderr, which will be ignored for assessment. 

The order of fetching pages does not matter. 

You do not need to fetch more than 100 distinct pages (but there may be more than 100 URLs). (Hint: How can you use this knowledge to simplify your code?) 

2.2 HTTP 

The servers hosting the URLs you are to crawl will conform to HTTP 1.1 and all content will be served over port 80. 

You must provide a User-Agent header consisting of your username (only), such as 

User-Agent: joeb This will be used to observe which pages you fetch. Without this, you risk getting no marks. 

Note that all requests must also contain the Host header1. 

The server may fail to serve a page that you request, and either indicate that with an error code or simply send a file shorter than the length in the header. You should keep reading data until the connection is closed by the server or Content-Length bytes have been read. For transient (non-permanent) errors, you can re-request such pages without penalty, but not that the transient error condition may last for quite a while. 

No server response will be longer than 100,000 bytes. 

2.3 Parsing HTML 

The HTML files will be valid HTML (unlike nearly all real web pages; web browsers silently fix many mistakes). 

The file will not contain the characters  <  or  >  except at the start and end of tags. No anchor tags will be split across lines. 

There will be no <base> tags or canonical link elements. 

Both inline and external javascript should be ignored. 

Submissions may (optionally) use the PCRE library for pattern manipulation, or the standard functions declared in regex.h. Please read the submission instructions carefully if you plan to use pcre.

This IT Assignment has been solved by our IT Experts at My Uni Paper. Our Assignment Writing Experts are efficient to provide a fresh solution to this question. We are serving more than 10000+ Students in Australia, UK & US by helping them to score HD in their academics. Our Experts are well trained to follow all marking rubrics & referencing style.

Be it a used or new solution, the quality of the work submitted by our assignment experts remains unhampered. You may continue to expect the same or even better quality with the used and new assignment solution files respectively. There’s one thing to be noticed that you could choose one between the two and acquire an HD either way. You could choose a new assignment solution file to get yourself an exclusive, plagiarism (with free Turnitin file), expert quality assignment or order an old solution file that was considered worthy of the highest distinction.