Indexing

Question 1

What is the system recommodation for the reference indexer?

Answer 1

12 cores, 2+ GHz, 12 GB RAM, 800 IOPS

Question 2

What is the system recommodation for the high end indexer?

Answer 2

48 cores, 2+ GHz, 128 GB RAM, 1200 IOPS

Question 3

Give an example of how 800 IOPS can be reached?

Answer 3

By using eight x-GB, 15,000 RPM, serial-attached SCSI (SAS) HDs in a Redundant Array of Independent Disks (RAID) 1+0 fault tolerance scheme as the disk subsystem.

Each hard drive is capable of about 200 average IOPS. The combined array produces a little over 800 average IOPS.

Question 4

What is a realiable methode to meassure IOPS on a disk subsystem?

Answer 4

bonnie++ or FIO

Question 5

In case you need to meassure the IOPS at customer site with shared storage, what do you need to consider?

Answer 5

To perform the test on all indexers at the same time to get a reliable results

Question 6

List the index artifacts and where they are located?

Answer 6

The indexing artifacts are stored under $SPLUNK_DB/etc/var/lib/

Data is stored in buckets. One index can contain several buckets.

There are different types of buckets (hot,warm,cold)

Frozen buckets per default will be deleted. You can specify to archive them.

Question 7

Does hot and warm buckets can be seperated on disk?

Answer 7

No, they do live under the same directory.

The path of hot/and warm buckets and be configured with homePath.maxDataSizeMB

Question 8

Can warm/hot buckets be seperated from cold? If so, what would be a common use case?

Answer 8

Yes, they can be seperated.

A common use case would be different underlying storage systems, eg hot/warms on high performance storage and cold on slower storage.

Question 9

What is the default time until data in an index gets frozen?

Answer 9

~6 years

Question 10

What is the rolling behaivor for maxDataSize?

Answer 10

Hot to warm

Question 11

What is the rolling behavior of maxWarmDBCount?

Answer 11

Warm to cold

Question 12

How do you configure maximum size for cold storage?

Answer 12

coldPath.maxDataSizeMB

Question 13

What is the default setting for maxTotalDataSizeMB?

Answer 13

500000 MB [~500GB]

Question 14

What is the rolling behavior maxTotalDataSizeMB?

Answer 14

Cold to frozen [based on size]

Question 15

How do you configure maximum size of an index?

Answer 15

maxTotalDataSizeMB

Question 16

What is the rolling behavior for frozenTimePeriodInSeconds?

Answer 16

Cold to frozen [based on time]

Question 17

What is the default setting for maxHotBuckets?

Answer 17

3

Question 18

What 3 bucket controls are settable in the GUI?

Answer 18

1) maxTotalDataSizeMB
2) maxDataSize
3) timePeriodInSecsBeforeTsidxReduction

Question 19

What is the default setting for maxDataSize?

Answer 19

auto (sets the size of hot buckets to 750MB)

You should use “auto_high_volume” for high-volume indexes (such as a firewall index); otherwise, use “auto”. A “high volume index” would typically be considered one that gets over 10GB of data per day

Question 20

How do you configure the maximum number of warm buckets?

Answer 20

maxWarmDBCount

Question 21

What is maxTotalDataSizeMB applied to?

Answer 21

to both, homepath and coldpath

Question 22

Why using volumes can be a good aproach?

Answer 22

Using volumes helps to prevent failures in index size calculation.

A volume offers the possibility to assing several index to one volume. The volume has a maximum limit. That does prevent indexes to growth until the maximum disk space reaches and keeps the index size under control.

hot/warm and cold can be on different volumes

Question 23

Which bucket type needs to have a volume defintion to work?

Answer 23

tstatsHomePath (Accelerated Data Models)

Question 24

What is the differene between a pipline and processor?

Answer 24

• Pipeline : A thread. Splunk creates a thread for each pipeline. Multiple pipelines run in parallel.
• Processor: Processes in pipeline

Question 25

How does the word ‘queue’ fit into the picture of a pipline and processors?

Answer 25

Each pipepline has a seperat queue where the data ‘waits’ to be processed, similar to a mail or printer queue. Its a memory space between pipelines to store data.

Question 26

In which pipepline is the UTF-8 processor located?

Answer 26

Parsing Pipeline

Question 27

In which pipepline is the annotator processor located?

Answer 27

Typing Pipeline

Question 28

In which piepline is the indexandforward processor located?

Answer 28

Indexing Pipeline

Question 29

What happens when an event enters the linebreaker processor?

Answer 29

Splits data stream into events based on the linebreaker configuration

Question 30

What processor anonymizing sensitive data?

Answer 30

regexreplacement processor in the typing pipeline

Question 31

Which pipelines use props.conf?

Answer 31

All of them using props.conf, except indexing pipeline

Question 32

Which pipeline uses outputs.conf?

Answer 32

Indexing pipeline

Question 33

Which pipeline and which processor does the timestamp extraction?

Answer 33

Typing pipeline, aggregator processor

Question 34

In case the ‘great eight’ have been configured, which pipeline will have a significant lower load?

Answer 34

Parsing pipeline and merging pipeline

Question 35

What is the average percentage of compressed data itself (journal.gz) residing in buckets?

Answer 35

15%

Question 36

What is the average size in percent of the lexicon (TSIDX) files which reside in a bucket?

Answer 36

35%

Question 37

What is the average compression for all indexed data?

Answer 37

50% - which containts 15% journal.gz and 35% TSIDX

Question 38

What kind of data does live under the tstatsHomePath?

Answer 38

Accelerated Data Models

Question 39

What kind of data does live under the summaryHomePath?

Answer 39

Accelerated Reports

Question 40

Which pipepline forwards data to the nullQueue?

Answer 40

Typing pipeline

Question 41

What are thawed buckets?

Answer 41

Data restored from an archive. If you archive frozen data, you can later return it to the index by thawing it.