System Design Week 2 - Search Engine

Question 1

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 1) List down the functional requirements

Answer 1

• Given static content data, data is available to search.
• User should be able to search content with keyword,
• user persistence
• Ignore security

Question 2

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 2) List down the non-functional Requirements

Answer 2

• size of corpus : hundreds of billions of documents
• throughput : 70,000 qps
• Availability

Question 3

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 3) What are the microservices?

Answer 3

• Search Service
• Indexing service

Question 4

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 4) What is the logical architecture?

Answer 4

Diagram: https://drive.google.com/file/d/1pyhZdUyBzeEhK5K6TipXgaoWnt8l9KLg/view?usp=sharing

Question 5

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 5) What is the schema design?

Answer 5

• K-V : keyword with list of documents (nosql)
• Document of large size (Object store)

Question 6

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 6) What is the API design?

Answer 6

• Search(word)
• Search(expression)
• Index(document)

Question 7

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 7) What is the Business Logic?

Answer 7

Core business logic includes the solution with the indexing of documents, and store the data into different shards.
Reverse indexer keep track of the keyword is present in which all databases(shards)
If a keyword is searched with the union of another word then the reverse indexer will give the shards on which to look for, the common shards are scanned to find the actual document.
The Vertical sharding is key to shard the data on the values.Sharded data to be stored as sorted.
[keyword] -> doc1, doc2, doc3 . . .

Question 8

Problem 1) Search Engine

Problem statement : Design a search engine system based on available dataset.
Clarifying questions :
questions 1 : Is data constantly changing or its static data ? For now assume static data.
questions 2 : example of the query and expected output.
→ Brutus and Caesar : should respond with the documents list in which these 2 keywords are present.
→ caesar and Brutus Not calpurnia.

Question 7) What are the Design Considerations?

Answer 8

Design Considerations: https://drive.google.com/file/d/1_4ZAENR6XOtRIctvrfixoD55g0nGWx2w/view?usp=sharing