SOLR

Question 1

What is it good for? What are common usecases too?

Answer 1

Text-centric (Can even do spell-check and synonym handling)
Read-dominant
Document-oriented (Flat structure, no sub-fields)
Flexible schema

Common usecase:

• Keyword search
• Ranked retrieval
• facets
• Geospacial ( indexing latitude and longitude values as well as sorting or ranking documents by geographical distance)

Dont use for deep analytics, document relationships, permission based storage

Question 2

What is inverted index in context of Lucene?

Answer 2

Every term maps to LIST of document IDs. Also for every document there is a term frequency in the index itself, as well as the term position to denote position in doc (This helps in finding EXACT phrase match).

Question 3

What are some of the query features?

Answer 3

Conditional logic using AND, OR, and NOT
    Wildcard matching
    Range queries for dates and numbers
    Phrase queries with slop to allow for some distance between terms
    Fuzzy string matching
    Regular expression matching
    Function queries

Question 4

How can trailing wildcards be supported? e.g. *ing

Answer 4

Enable reverse filtering too. Terms will be reversed and indexed as well, hence twice the index space.

Question 5

How is exact phrase match achieved? “homes for rent”

Answer 5

Solr still uses individual terms but after the results achieved it additionally looks for ‘term position’ in index to see if positions are continuous.

Question 6

What is inverse document frequency?

Answer 6

Inverse document frequency (idf), a measure of how “rare” a search term is, is calculated by finding the document frequency (how many total documents the search term appears within), and calculating its inverse

Question 7

Limitations of solr?

Answer 7

Relationships and joins. Denormalized schema makes it not suitable for some updates.
Field level updates not possible. Whole doc to be updated.

Question 8

What is ‘warming a new searcher’ mean?

Answer 8

Solr uses a searcher to run queries against. Searcher has caching built in as well. On any commit, a new searcher needs to be created. For perf, config can define warming queries to run on any new searcher. Until the new searcher is warmed, the old searcher will continue to serve current traffic.
Hence commit could be a costly operation.

Question 9

What is autowarming and warming query?

Answer 9

There is a newSearcher and a firstSearcher. Latter is created on startup and latter when new searcher is created after every commit.
Auto warming is ability to copy over cache to new searcher. Warming query is a query to be run on new searcher creation to warm the cache.

Question 10

What are kinds of caches in solr?

Answer 10

Filter cache: Cache on filters like fq=manu:Belkin are cached to improve results from next query.

Query result cache: Document IDs of all results of a query. There is a max setting to limit docs per query that go in cache.

Document cache: Cache the actual docs so that disk seek is avoided. Query result cache will use this cache internally.

Field value cache: Lucene concept to internally return results faster.

For every cache, autowarming count can be set.

Question 11

What are special field types in solr?

Answer 11

• dynamic fields allow you to apply the same definition to any fields in your documents whose names match either a prefix or suffix pattern, such as s_ or _s
• copy fields allow you to populate one field from one or more other fields (Using this can support catch-all search OR catch provide different analyzer to field like case insensitive)

Question 12

How does solr handle date?

Answer 12

Using the ‘tdate’ field. Range queries can be supported even e.g. timestamp:[NOW/DAY TO NOW/DAY+1DAY}
It uses a trie data structure under the covers.

Question 13

What tools allow data import?

Answer 13

SolrJ - uses javabin efficient protocol
Data Import Handler - can support JDBC connections
ExtractingRequestHander - can extract from rich docs like PDF
Nutch - integrates with web crawler by default

Question 14

What is soft commit?

Answer 14

Soft commit is a mechanism to make documents searchable in near real-time by skipping the costly aspects of hard commits.

Question 15

Does solr support field level updates?

Answer 15

Yes. PK needs to be provided. Under the covers it fetches the whole doc and makes changes. Optimistic concurrency control is in place using a version field.

Question 16

What are some types of text transformations solr does?

Answer 16

lowercase, removing apostrophes, stemming (drinking -> drink), stop words removal (a, an, with), remove hyphenated words (i-pad -> ipad and i pad), collapse repeated letters (yummmm -> yumm).

Text analyzers need to be added to fields that need to be transformed. In solr terms they are filters that act on (in sequence of definition) on text fields. They can be defined on indexing or on query times (many times both for e.g. lowercasing has to be done on both to match)

Question 17

What is omitNorms attribute on fields?

Answer 17

The document length norm is used to boost smaller documents. Lucene gives smaller documents a slight boost over longer documents to improve relevance scoring.
omitNorms can be enabled on fields so that this boost is disabled.

Question 18

How to determine what to send in fq and q parameters? What is ‘cost’ param for each fq?

Answer 18

fq = filter query. The only difference is that fq query results will be cached. So fq terms will be a cache hit first and then other query will be looked at. (fq can be multiple params while q can only be single)
You can also send in a number to denote ‘cost’ where lower cost fq will be run first (denote it on param which filters docs most). Cost specified of 100 and above kicks on ‘post filtering’ which is fq applied AFTER collecting results from both q and fq.

Question 19

How do ‘term proximity’ based queries work?

Answer 19

It’s also possible to search for terms close together, but not necessarily right beside each other, by adding a tilde and the number of positions the terms can be away from each other:
“apache solr”~3
(Default is 0 and hence quoted text is exact match)

Question 20

How do ‘character proximity’ based queries work?

Answer 20

Not only can you perform proximity searching between terms, you can also perform edit distances on the characters within a term you’re querying to find similarly spelled terms.
solr~1 (Matches sol, sor …)

Question 21

What is eDisMax parser?

Answer 21

Extended Disjunction Maximum parser is used for user facing applications generally where user can type queries to search on. Key differences with Lucene parser:

• Searches mutliple fields
• Liberal towards query syntax
• By default, uses max of score if two terms found rather than sum (Lucene sums up the score)
• Flexible relevancy calculation

But flexibility comes with a slight overhead.

Question 22

What is boost query parser?

Answer 22

Parser that allows relevancy boost only and does not filter results. e.g.
/select?q=query:”{!edismax qf=title content}data science” AND
query:”{!boost b=log(popularity)}:” AND
query:”{!boost b=recip(
ms(NOW,articledate),3.16e-11,1,1)}category:news”

Above is nested query where boost parser does not filter only edismax parser filters on data science.

Question 23

What fields can facets be on? What are some of the features for facets?

Answer 23

Only on indexed fields. You can specify sorting, num count to return for facets. Also you can specify how many threads to use to calculate facets if multiple fields facets requested for.

Question 24

What is query faceting and range faceting?

Answer 24

Query faceting is returning counts of potential queries if no fields are defined for that data. e.g. 
select?q=*:*&facet=true&
  facet.query=price:[* TO 5}&
  facet.query=price:[5 TO 10}&
  facet.query=price:[10 TO 20}

Range faceting is similar but more suited for evenly spread data:
select?q=*:*&facet=true&
  facet.range=price&
  facet.range.start=0&
  facet.range.end=50&
  facet.range.gap=5

Question 25

How is spell-checker and auto-suggest implemented?

Answer 25

Spell checker is on search handler and uses levesthein distance to suggest. Also gives number of hits for suggestions.
Auto suggest only has a prefix and needs to be fast. Its a separate handler and uses prefix-tree data structure to quickly match on index.

Question 26

What is results grouping?

Answer 26

Similar to facets, results are grouped under desired group by a field.
group=true&group.field=type&group.limit=3

Has perf impact