Webscraping

Question 1

What are the 3 types of http requests? What do they mean?

Answer 1

get : receive data from server
post : send data to server
put : idempotent post (multiple put requests are treated as one)

Question 2

What are the 3 types of http codes? What do they mean?

Answer 2

200 : good
300 : redirection
400 : error

Question 3

How can you convert a json string to a dictionary?

Answer 3

import json
json_string = ‘….’
json_dict = json.loads(json_string)

Question 4

Why is Beautiful Soup useful? How can you create a soup object from html?

Answer 4

It allows you to parse html and index by tag and attributes. soup = BeautifulSoup(html_string, ‘html.parser’)

Question 5

How can you use BS to find the html with given tags and attributes?

Answer 5

soup.find_all(tag (str), attributes (dict))

Question 6

How can you customize your request headers?

Answer 6

Change user agent to be the same as browser:

requests.get(url, headers = {‘User-Agent’ : user_agent})

Question 7

When would you want to use selenium?

Answer 7

When webpage html is dynamic and requires input from the user (e.g. log in, accept cookies, wait for text to be displayed etc….)