Web Mining

Question 1

What is web mining?

Answer 1

Data mining on the web

Question 2

Is web mining the same as web searching?

Answer 2

No, web mining is about finding patterns.

Question 3

Various Web-Data Types

Answer 3

Web Pages
Intra-page structures
Inter-page structures
Usage data
Supplemental data
-  Profiles
-  Registration information

Question 4

What concepts and technologies for search engines?

Answer 4

Crawlers/Index

Profiles/Personalisation

Question 5

What’s a web crawler?

Answer 5

A spider that traverses the hyperlinks and to build out a popularity of web pages.

Question 6

What is the process of a web crawler?

Answer 6

1. Start with a seed

2. Send crawlers

Question 7

What is an issue with Web Crawling ?

Answer 7

Time wasted for waiting for responses to requests.
To reduce inefficiency, web crawlers use threads

Web crawlers use politeness policies to stop them from flooding sites with requests.

Question 8

What’s freshness?

Answer 8

Web crawlers need to revisit a page in order to maintain the freshness of a document because web pages are always being added, deleted and modified.

Question 9

What is focused crawling?

Answer 9

Attempt to download only those pages that about a particular topic. Popular pages tend to have links to other pages on the same topic.

Crawlers uses text classifiers to decide whether a page is on topic.

Example: Google Scholar (search by citations), Google patterns

Question 10

What is the challenge for focused crawling?

Answer 10

Finding relevant links.

Question 11

How do we do relevance prediction? What strategies do we use?

Answer 11

Define a score as cond. prob. that a page is relevant given its text content.

Parent-based: score a fetched page and extend score to all URLs in that page

Anchor-based: score each URL based on that anchor text to that URL: “semantic linkage”

Question 12

What is the purpose of personalization in web content mining?

Answer 12

Web access or contents to be tuned to fit the preferences of the user.

• Edititorial and hand curated - “editor’s pick”
• Simple aggregagates (top 10, most popular) - all users
• Tailored to individual users -> we recommend for you

Question 13

What is a utility matrix?

Answer 13

Assigning recommendations. Trying to assign a score to things you haven’t seen. 1

Question 14

What is a challenge of a recommendation system?

Answer 14

Sparcity. Hard to recommend when you don’t watch.

People not giving reviews.

Question 15

How do you overcome the challenge of sparcity in a recommendation system?

Answer 15

Be explicit:

• Ask people to rate items
• Doesn’t work well in practice

How? Give a reward

Be Implicit:

Learn ratings from user actions
- purchase implies high rating

Question 16

What is a cold start?

Answer 16

The utility matrix is sparse. Cold-start, new users have no ratings, nor history.

Question 17

What approaches do you have to recommender systems?

Answer 17

1) Content-based

2) Collaborative

Question 18

What is content-based recommendation?

Answer 18

Look at the preferences of the user and give a recommendation on this.

Question 19

What is an advantage of content-based approach?

Answer 19

No need for data on other users
Able to recommend to users with unique tastes
Able to recommend new & unpopular items
Able to provide explanations

Question 20

What is a con of content-based approach?

Answer 20

• Finding the appropriate feature is hard
• Recommendations for new users
• Overspecialization (too specific, you watch 1 documentary, you get many documentaries)

Question 21

What is collaborative filtering?

Answer 21

Look at similar users who have rated something highly, and recommend to another user.

Question 22

How do you find similar users?

Answer 22

A couple of methods,

Jaccard Similarity
Cosine Similarity

Question 23

What is a pro/con of collaborative filtering?

Answer 23

Pro
- Works for any kind of item

Con

• Need enough users in the system to find a match
• Hard to find users that have rated the same items
• First rate: hard to recommend an item that hasn’t been previously rated
• Popularity bias: Popular items tend to dominate the collaborative items

Question 24

How does PageRank work?

Answer 24

Rank based on Backlinks, the number of pages that point to a webpage.

Weighting - based on the importance of the pages that link to it.