Lecture 5: Web Crawling

Question 1

What is a web crawler?

Answer 1

An autonomous data collection script

Question 2

Why are web crawlers used?

Answer 2

• Discover and index pages
• Perform market/sentiment analysis
• Intelligence gathering purposes

Question 3

What is web scrapping?

Answer 3

Typically performed over a set of
known pages

Used to extract data into a condensed format

Question 4

What is the difference between web crawlers and web scraping?

Answer 4

A crawler is interested in finding new pages

A scraper is interested in extracting data from existing, known-format pages

Question 5

What is the web crawling loop?

Answer 5

1. Fetch the contents of a web page at a URL
2. Comb through the contents and extract additional URLs
3. Add the extracted URLs to a list of URLs to be crawled (to-crawl)
4. Add the current URL to the list of completed URLs
5. Pick a new URL from the to crawl list, return to step 1

Question 6

What are the methodologies for picking the next URL to crawl?

Answer 6

• Breadth first: Crawl all links on that page first, before moving on
• Depth first: Crawl the linked URLs first always, rapidly getting further from your start point
• Alcohol first: Drunkenly stagger around from URL-to-URL with no discernible pattern