Chapters 12&13: Networked Programmes and Using Web Services

Question 1

socket

Answer 1

like a file, but a 2-way connection between two programmes. Can edit and read from either side.

Question 2

http protocol

Answer 2

set of rules to determine which end of a socket goes first, what they do, the response to that message, who sends next message, etc.

Question 3

simple HTTP protocol programme which:

• connects to port 80 of www.py4e.com
• contains a loop which receives data in 512 character chunks from socket and prints until there is no data left

Answer 3

import socket

mysock=socket.socket(socket.AF_INET, socket.SOCK_STREAM)
mysock.connect((‘data.pr4e.org’, 80))
cmd=’GET http://data.pr4e.org/romeo.txt HTTP/1.0\r\n\r\n’.encode()
mysock.send(cmd)

while True:
   data=mysock.recv(512)
   if len(data)<1:
      break
   print(data.decode(),end='')

mysock.close()

Question 4

\r\n

\r\n\r\n

Answer 4

EOL (end of line)

signifies nothing between 2 EOL sequences, equiv to blank line

Question 5

output of above programme

Answer 5

sends headers on each line with info eg date, server, last modified, content-length, content-type (eg text/plain)

Then a blank line, then the actual data from romeo.txt.

Question 6

encode/decode meaning, and alternative

Answer 6

HTTP protocol requires data to be received and sent as bytes objects, not strings. hence these convert from string to bytes and back again.
‘x’.encode() is equiv to b’x’

Question 7

programme to retrieve an image:

body type
number of characters received per call

Answer 7

http://www.py4e.com/code3/urljpeg.py
-rather than copying data to screen, we accumulate the data in a string, trim off headers, and save image data to a file
image/jpeg
3200-5120 - depends how much data has been sent from server before the call, ie network speed. last call can be less than 3200 if less than that left.

Question 8

how to slow down successive recv() calls to more consistently get specified number of characters (5120 above):

Answer 8

time.sleep() (not sure where)
This gives a 0.25s delay, allows more consistent retrieval of 5120 chars unless speed v poor. may make it take longer obvs

Question 9

easier, less manual way of retrieving data using urllib (similar to getting a file):

Answer 9

import urllib.request

fhand=urllib.request.urlopen(‘http://data.pr4e.org/romeo.txt’)
for line in fhand:
print(line.decode().strip())

Question 10

find frequency of words in a web page

Answer 10

same as above
for line in fhand:
   words=line.decode().split()
   for word in words:
      counts[word]=counts.get(word, 0)+1
print(counts)

Question 11

retrieve binary file (eg image of vid) and save as a file locally

Answer 11

import=urllib.request, urllib.parse, urllib.error

img=urllib.request.urlopen(‘http://data.pr4e.org/cover3.jpg’).read()
fhand=open(‘cover3.jpg’, ‘wb’)
fhand.write(img)
fhand.close()
wb argument for open opens a binary file for writing only.

Question 12

if file too big (will take too long) - break into chunks and write bit by bit (100000 chars at a time):

Answer 12

.....
fhand=open('cover3.jpg', 'wb')
size=0
while True:
   info=img.read(100000)
   if len(info)<1: break
   size=size+len(info)
   fhand.write(info)

print(size, ‘charactcers copied.’)
fhand.close()

Question 13

web scraping

Answer 13

write a programme that pretends to be a web browser and retrieves pages, then looks for patterns in those pages’ data. this is what google does - detects word frequency and links to a page etc to determine importance of millions of pages, and suggests the best

Question 14

parse an HTML with regular expression to find links, where links take the form: href=”http://www.blah.. (and you know the rough structure on the link)

Answer 14

# Search for link values within URL input
import urllib.request, urllib.parse, urllib.error
import re
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input(‘Enter - ‘)
html = urllib.request.urlopen(url, context=ctx).read()
links = re.findall(b’href=”(http[s]?://.*?)”’, html)
for link in links:
print(link.decode())

Question 15

ssl library

how you import it

Answer 15

allows a programme to access websites which strictly enforce HTTPS.
import ssl

Question 16

read method use here:

Answer 16

returns HTML source code as bytes object rather than returning HTTPResponse object

Question 17

risk using REs:

solution (vague)

Answer 17

lots of broken HTML pages out there - only using REs may miss some or retrieve bad data.
solved by using a robust HTML parsing library

Question 18

HTML parsing libraries in python

Answer 18

many of them to help parse HTML, where XML parsers may reject and HTML as improperly formatted. each has its own strengths and weaknesses.

Question 19

BeautifulSoup (BS)

Answer 19

HTML parsing library which tolerates highly flawed HTML and lets you extract data you need easily.. requires prior downloading.

Question 20

pip

Answer 20

tool to install Python packages. once installed, can use:
pip install (package...) in cmd prompt.

Question 21

use urllib to read a page, then BeautifulSoup to extract href attributes from the anchor (a) tags

Answer 21

import urllib.request, urllib.parse, urllib.error
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
html = urllib.request.urlopen(url, context=ctx).read()
soup = BeautifulSoup(html, 'html.parser')

# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
    print(tag.get('href', None))

Question 22

above programme explained

Answer 22

prompts for a web address, opens page, reads data, passes data to BS parser, retrieves all anchor tags, prints out href attribute for each tag.
Output: includes HTML anhor tags which are relative paths (end in eg .html) or in-page references (eg #). Unlike out RE programme, this doesnts select for those starting with http:// etc

Question 23

use BS to pull out various parts of each tag (* denotes a line is different to programme above):

Answer 23

from urllib.request import urlopen
from bs4 import BeautifulSoup
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = input('Enter - ')
html = urlopen(url, context=ctx).read() #*
soup = BeautifulSoup(html, "html.parser")

# Retrieve all of the anchor tags
tags = soup('a')
for tag in tags:
    # Look at the parts of a tag
    print('TAG:', tag) #* and all the rest!
    print('URL:', tag.get('href', None))
    print('Contents:', tag.contents[0])
    print('Attrs:', tag.attrs)