Indexer Clustering

Question 1

Describe the Master Node:

Answer 1

Cluster Master - Coordinates replication activity, and distributes configuration informatio

Question 2

What are Indexer Clusters?

Answer 2

groups of SPlunk Indexers that provide data redundancy and disaster recovery capabilities, as well as feature automatic failover from one indexer to the next

Question 3

What does high replication or search factor require?

Answer 3

Higher RF or SF requires greater storage. Plan accordingly.

Question 4

What are the number of machines needed for a standard cluster?

Answer 4

Indexers (RF+)
Cluster Master (1)
Search Heads (1+)

Question 5

What is the best practice for number of indexers in a cluster?

Answer 5

Do not set replication_factor = # of indexers

Question 6

How should you use the org_cluster_master_base config?

Answer 6

set mode, RF, SF, pass4SymKey, cluster_label in the [clustering] stanza
- Example config in server.conf:

[clustering]
Mode = master
Replication_factor = 3
Search_factor = 2
pass4SymKey = 12345
Cluster_label = cluster1

Question 7

How do you configure peer nodes?

Answer 7

Use org_cluster_search_base to configure cluster SH

Example config in server.conf
[clustering]
Mode = searchhead
Master_uri = https://master:8089
pass4SymKey = 12345

Question 8

How do you enable the search head that searches multiple clusters?

Answer 8

[clustering]
Mode = searchhead
Master_uri = clusternaster:one, clustermaster:two

[clustermater:one]
Master_uri = https://master:8089
pass4SymKey = 12345

[clustermaster:two]
master_uri = https://master2:8089
pass4SymKey = 67890

Question 9

How do you gaurd against data loss on the forwarders to the indexers?

Answer 9

By enabling indexer discovery on forwarders.

• Must set unique pass4SymKey on server.conf in the [indexer_discovery] stanza on the CM
• Also set the pass4SymKey and master_uri in the outputs.conf of the forwarder.
• This allows the forwarder to know (from the indexer) that data has been received and processed.

Question 10

What must all peers and the Cluster Master share?

Answer 10

The same set of indexes

Question 11

What is the upgrade procedure?

Answer 11

• Clustermaster node
• Searchhead tier
  License Master, DMC, etc.)
• Indexer tier

Question 12

What is the Maximum latency between CM and Indexers?

Answer 12

20 ms

Question 13

What happens to an indexer in Detention?

Answer 13

An indexer in Detention (Manual or Automatic) will stop indexing data, but will still remain searchable

• Indexer in Manual detention will continue to index internal data
• ./splunk edit cluster-config -manual_detention on

Question 14

How should you tune the indexer heartbeat interval?

Answer 14

one second for every 50k buckets

Question 15

What is the Hot bucket lifecycle?

Answer 15

1. The indexers notify the CM when a new hot bucket is minted.
2. The CM replies with a list of streaming targets, for replication. This list is randomly chosen from the list of available indexers (generation), with enough to make sure we meet policy.
3. Original indexer begins replicating data to target indexers.

Question 16

What is the warm bucket lifecycle?

Answer 16

1. The indexers notify the CM when a bucket rolls from hot to warm.
2. The replication target is then notified that this bucket is complete, and the hot bucket will be rolled to warm on the target as well.

Question 17

What is the frozen bucket lifecycle?

Answer 17

1. In the case of an indexer cluster, when a peer freezes a copy of a bucket, it notifies the master. The master then stops doing fix-ups on that bucket. It operates under the assumption that the other peers will eventually freeze their copies of that bucket as well.

Question 18

What is the benefit of putting an indexer that is almost out of disk space in manual detention?

Answer 18

It does not disable the indexer’s ability to participate in search, although it does disable the hosts ability to index.

Question 19

How do you replace the Cluster Master?

Answer 19

No Failover capability

• Back up server.conf and /etc/master-apps
• Ensure communication is working
• Replace the master

Question 20

Can you recover the contents of master-app from the clients?

Answer 20

Yes, it lives in $SPLUNK_HOME/var/run/splunk/cluster/remote-bundle on the indexers.

Question 21

What are the steps, in order, to deploy the indexer cluster?

Answer 21

1. Identify your requirements
2. install the Splunk Enterprise instances
3. Enable Clustering
4. Complete Peer Node configuration
5. Forward master node data to peer nodes

Question 22

What is the preferred method to replace a Cluster Master Node?

Answer 22

Cloning is the preferred method as it allows for preparation/configuration of a new master before the failure event occurs.