Web Scraping/Crawling

Question 1

What type of request is used to fetch the content of a web page from a web server?

Answer 1

GET

Question 2

What standard Python library contains functions for requesting data across the web, handling cookies, and even changing metadata?

Answer 2

urllib

Question 3

What import statement was used in our Lab and Group Project to import BeautifulSoup?

Answer 3

from bs4 import BeautifulSoup

Question 4

What Wikipedia page did you webscrape in Lab 8?

Answer 4

1999-2000 FA Premier League - Wikipedia

Question 5

What two fields(columns) required you to parse the text within the <a>…</a> tags?

Answer 5

Manager & Captain

Question 6

When dealing with HTML elements mapped out as a tree, which elements are exactly one tag below a parent tag?

Answer 6

children

Question 7

What store was the subject of the web scraping program reviewed during this lesson?

Answer 7

Family Dollar

Question 8

What is the latest version of HTML?

Answer 8

HTML5

Question 9

What is it called when the code accesses a URL, examines that page for another URL, retrieves that page, in a recursive process?

Answer 9

web crawling

Question 10

What is web scraping?

Answer 10

An automated process of gathering large amounts of data from the Internet

Question 11

Other names for web scraping:

Answer 11

• screen scraping
• data mining
• web harvesting

Question 12

Why use web scraping?

Answer 12

• useful data is available on the web, but isn’t available via downloads or APIs
• price comparison info
• social media scraping (what’s trending?)
• research (stats, weather data, etc)

Question 13

Difference between web scraping vs web crawling:

Answer 13

Web scraping: generally inspect a single web page or two to get the data they’re looking for.

Web crawling: “crawl across the web” following links from web page to web page recursively

Question 14

Applications for web crawling:

Answer 14

• generating a site map

- gathering data about a specific topic from a large number of websites