Session 2

Question 1

Is Webscraping legal?

Answer 1

Webscraping falls in a gray area both in the EU and USA.

So, it depends

Question 2

Almost any data source can be used in the context of a Data Mining project

Answer 2

• Data Mining is an exploratory process with uncertain outcomes
• Being able to collect data from different systems allows for fast prototyping and adjustments as needed

Question 3

Data Mining is…

Answer 3

… an exploratory process with uncertain outcomes

Question 4

A proper engineering solution should be deployed once…

Answer 4

the prototype demonstrates its merits

Question 5

Four major ways of collecting data from online sources

Answer 5

1 Manually browsing a web site (copy & paste)
2 Manually downloading a file
3 Pretending you are a human browsing a web site (web scraping)
4 Using an Application Programming Interface (API)

Question 6

Web scraping can be done using

Answer 6

• A modern programming language which offers complete
  flexibility but requires more effort to implement; or
• Specialized tools which allow faster implementation but provide less flexibility and make it harder to replicate data collection

Question 7

Web scraping using a programming language

Answer 7

Many languages provide functionality for reading and writing data from web sites, just like a regular web browser

Question 8

webscraper.io

Answer 8

is a more sophisticated tool that allows the user to select which elements of a web site are important and which links should be followed in order to gather more information

Question 9

Potential issues with web scraping

Answer 9

1. Many sites do not allow gathering information automatically
2. Not all data are publicly available online

Question 10

Ways to detect whether the site is being viewed by a human

Answer 10

• Detection of frequent requests
• Cookies
• Other trackers

Question 11

Robots Exclusion Protocol

Answer 11

The robots exclusion protocol or the robots.txt protocol is a way to communicate with crawlers or web bots with instructions on whether you can automatically scrape parts of a web site

Question 12

Potential issue web scraping: Not all data are publicly available online

Workaround

Answer 12

Workarounds include using authentication to access the protected information or using an API access

Question 13

Application Programming Interfaces (APIs)

Answer 13

are protocols to interact with specific web sites that can be used by any registered user

Question 14

Three steps for API access

Answer 14

1. Get an API key
2. Query an API endpoint using the API key
3. Process the response

Question 15

API key is…

Answer 15

like a valet key for the web

• Provides access to a limited set of functions
• Can be revoked by issuer at anytime

Question 16

An API usually provides multiple endpoints or functions. Examples:

Answer 16

e.g., most recent movies, most popular movies, search movies by keyword