1.0 Deploying Splunk Flashcards
What is an SVA
Proven reference architectures for stable, efficient, and repeatable deployments. Guidelines and certified architectures to ensure that their initial deployment is built on a solid foundation.
Why and how does Splunk grow from standalone to distributed?
- Ingests more data
- Distributed search across indexers
- Adding high availability
- Dedicating LM and CM
- Adding ES
- SH Cluster for searching
- Disaster Recovery
What does and doesn’t SVA provide?
- Implementation choices (OS, baremetal vs. virtual vs. Cloud etc.).
- Deployment sizing.
- A prescriptive approval of your architecture.
- A topology suggestion for every possible deployment scenario.
What does and doesn’t SVA provide?
Does:
- Implementation choices (OS, baremetal vs. virtual vs. Cloud etc.).
- Deployment sizing.
- A prescriptive approval of your architecture.
Doesn’t
- A topology suggestion for every possible deployment scenario.
What is HA? How can Splunk accomplish?
continuously operational system bounded by a set of tolerances
ex. IDX cluster. 1 node goes down - still send data to others
SHC - multiple SHs can look at the data.
What is DR? How can Splunk accomplish?
Process of backing-up and restoring service in case of disaster.
- Standby nodes - backed up copies of node managers
- Multisite
- SF and RF
What instances are suitable to become MC?
- Dedicated SH that is has connectivity to entire environment.
NEVER INSTALL ON:
- Prod (distributed) SH
- Member of SHC
- An IDX
- A DS OR LM with > 50 clients
- Deployer sharing with CM
https://docs.splunk.com/Documentation/Splunk/8.0.3/DMC/WheretohostDMC
How to configure MC for single or distributed environment?
Single:
1) In Splunk Web, navigate to Monitoring Console > Settings > General Setup.
2) Check that search head, license master, and indexer are listed under Server Roles, and nothing else. If not, click Edit to correct.
3) Click Apply Changes.
Distributed:
1) Log into the instance on which you want to configure the monitoring console. The instance by default is in standalone mode, unconfigured.
2) In Splunk Web, select Monitoring Console > Settings > General Setup.
3) Click Distributed mode.
4) Confirm the following:
The columns labeled instance and machine are populated correctly and show unique values within each column.
- The server roles are correct. For example, a search head that is also a license master must have both server roles listed. If not, click Edit > Edit Server Roles and select the correct server roles for the instance.
- If you are using indexer clustering, make sure the cluster master instance is set to the cluster master server role. If not, click Edit > Edit Server Roles and select the correct server role.
- If you are hosting the monitoring console on an instance other than the cluster master, you must add the cluster master instance as a search peer and configure the monitoring console instance as a search head in that cluster.
- Make sure anything marked as an indexer is actually an indexer.
5) (Optional) Set custom groups. Custom groups are tags that map directly to distributed search groups. You might find groups useful, for example, if you have multisite indexer clustering in which each group can consist of the indexers in one location, or if you have an indexer cluster plus standalone peers. Custom groups are allowed to overlap. For example, one indexer can belong to multiple groups. See Create distributed search groups in the Distributed Search manual.
6) Click Apply Changes.
If you add another node to your deployment later, click Settings > General Setup and check that these items are accurate.
Why do server roles matter MC?
Server roles are used to create searches, reports, and alerts based off what server roles are specified.
Why do groups matter MC?
Groups are used to in order to correlate among similar instances. Single clusters etc.
How are health checks performed on the MC?
Each health check item runs a separate search. The searches run sequentially. When one search finishes, the next one starts. After all searches have completed, the results are sorted by severity: Error, Warning, Info, Success, or N/A.
You are able to disable and enable certain health check items as needed as well as change their threshold.
The Health Check page lets you download new health check items provided by the Splunk Health Assistant Add-on on splunkbase.
Or you can create a new health check option.
What authentication methods are supported by Splunk?
LDAP - can’t use if SAML is enabled
SAML and SSO
Native Splunk accounts (created locally/internally)
Scripted authentication
Describe LDAP concepts.
Standard for accessing AD creds and services.
LDAP directories are arranged in a tree-like structure. The information model is based on entries:
- The distinguished name (DN) is based off attributes
cn=admin1,ou=people,dc=splunk,dc=com
Tree structure with cn at bottom and dc at top.
Describe LDAP configs.
https://docs.splunk.com/Documentation/Splunk/8.0.3/Security/ConfigureLDAPwithSplunkWeb
Authentication.conf
- host =
- port =
- groupBaseDN =
- groupMemberAttribute =
- groupNameAttribute =
- realNameAttribute =
- userBaseDN =
- userNameAttribute =
List SAML and SSO options
Review Slide Deck
Configure:
1) download the Splunk Service Provider Metadata file
2) Import the IdP metadata into Splunk
- SSO
- SLO (optional)
- IdP cert path
- IdP cert chains
- Replicate certs
- Issuer ID
- Entity ID
- Sign AuthnRequest
- Verify SAML Response
Roles in Splunk?
admin – this role has the most capabilities assigned to it.
power – this role can edit all shared objects (saved searches, etc) and alerts, tag events, and other similar tasks.
user – this role can create and edit its own saved searches, run searches, edit its own preferences, create and edit event types, and other similar tasks.
can_delete – This role allows the user to delete by keyword. This capability is necessary when using the delete search operator.
How can roles secure data?
Restrict index access capability by roles.
How can data be ingested by indexer?
Monitored Batch Script - opt/spl/etc/apps/bin/ Modular inputs Syslog Network inputs - http Splunk tcp REST
How does Splunk communicate with Splunk?
Ports: 8000 - web 8089 - mgmt 8088 - HEC 9997 - tcp listening 9887 - replication indexers - shc replication 8191 - kv store 514 - network input
Troubleshoot data inputs - monitor:
TailingProcessor for monitor inputs:
splunk _internal call /services/admin/inputstatus/TailingProcessor:FileStatus
Will show:
- what files it found
- whether they matched the wild card
- how far into the file it read
./splunk list monitor
list of currently monitored inputs.
Troubleshoot data inputs - conf files:
splunk btool conf-name list –debug
gives on-disk configs
What are examples of indexing artifacts
rawdata - compressed form (journal.gz)
Time Series Index (tsidx) - indexes that point to raw data
Buckets - directories of index files organized by age
- splunk_home/var/lib/splunk/myindex/db
- bucket locations defined in indexes.conf
Describe event processing
Splunk processes incoming data, stores result events in index
When Splunk indexes events:
- configures character set encoding
- configures line breaking for multi line events
- identifies timestamps
- extracts fields
- segments events
Name and describe the data pipelines
Parsing:
UTF-8 - Splunk will attempt to apply UTF-8 encoding to data
Line Breaker - Splunk will split data stream into events using default line breaker
Header - Splunk can multiplex different data streams into one “channel”
Merging/Aggregator:
- Splunk will merge lines separated by line breaker into events
Line breaking v line merging: LINE_BREAKER & SHOULD_LINEMERGE
Determining Time: TIME_PREFIX, TIME_FORMAT, MAX_TSL, DATETIME_CONFIG
Typing:
Regex replacement – performs any regular expression replacements called on in props.conf/tranforms.conf
Annotator – extracts the punct field
Indexing:
TCP/Syslog out – sends data to a remote server
Indexer – writes the data to disk
Describe the underlying text parsing process:
Splunk breaks events into segments at index and search time.
Index time - uses segments to create lexicons - which point to where on disk
Search time - uses segments to split terms
Describe Times Series Index
Optimized to execute arbitrary boolean keyword searches and return millions of events in revers time order
Inverted index
- allows fast full text searches
- maps keywords to locations in raw data
2 components:
lexicon
value arrays containing info about events
What is a lexicon TSIDX?
Each unique term from the raw event has its own row. Each row has a list of events containing the given term.
Looks like a table Term. Postings List (event #) bacon. 0 beets. 2, 5, 7 crab 0, 1, 9
What is a lexicon TSIDX?
Each unique term from the raw event has its own row. Each row has a list of events containing the given term.
Looks like a table Term. Postings List (event #) bacon. 0 beets. 2, 5, 7 crab 0, 1, 9
What is a TSIDX Value Array?
Each raw event has its own row
Each row contains metadata including the seek address
Looks like a table #. Seek address. _time. host. source. sourcetype.
What is data retention?
Management of the storage of indexed data. Allows Splunk to expire old data and make room for new data.
*Most restrictive rule wins - prompts a change in state
What are retention bucket controls?
maxDataSize - max size for hot bucket
maxWarmDBCount - max num of warm buckets
maxTotalDataSizeMB - max size of an index -> cold to frozen
frozenTimePeriodInSecs - max age of bucket -> cold to frozen
homePath.maxDataSizeMB - max size for hot/warm storage
coldPath.maxDataSizeMB - max size for cold storage
maxVolumeDataSizeMB - max size for volume
maxHotBuckets - max number of hot buckets
timePeriodInSecBeforeTsidxReduction - how long indexers retain tsidx files
Bucket Controls - Volumes
- Allows you to manage dis usage across multiple indexes
- Allows you to create max data size for them
- Typically separates hot/warm from cold storage
- Take precedence over other bucket controls
What is the bucket control precedence?
“Most restrictive rule wins”
- Oldest bucket will be frozen first
- Age determined by most recent event
- Hot buckets are measured by size but are exempt from age controls
What are typical storage multiplication factors?
.35 SF and .15 RF
Review formulas
What is the search head dispatch search sequence? (same with stats)
User Query -> SH -> Check search quota -> Check disk -> dispatch directory -> indexers
Sequence of events for searching for events in Splunk?
SH sends search request:
1) request is received by indexer
2) Indexer checks disk
3) Creates dispatch directory in var/run/spl/dispatch
4) configures subsystem - initializes configs (props, transforms etc) using bundle identified by SH
5) Implement time range - finds buckets in range
6) uses bloom filters to minimize resource usage
7) checks the lexicon- find events matching keywords within the lexicon (tsidx files)
8) Use results returned to find the event offsets within raw data from the values array
9) uncompresses raw data - uncompresses appropriate raw data to get the _raw event
10) Process field extractions
11) Send results to the search head
What metrics does the job inspector provide?
Time spent in search
Time spent searching the index
Time spent fetching data
Workload undertaken by search peers
What is the REST endpoint to find job properties?
REST /services/search/jobs
Search job inspector components?
Header
Execution costs
- categories listed as command.* and command.search.* reflect various phases of the search process
Search job properties
- Bundle
- Can summarize
- Create time
- Cursor time
- diskUsage
- dropCount
What are the type of search commands?
Generating Streaming Transforming Centralized (stateful) streaming Non-streaming
Characteristic of a generating search command?
Invoked at beginning of a search.
Does not expect or require an input
| search is implied
Characteristic of a transforming search command?
Generates a report data structure
Operate on the entire event set
ex. chart, timechart, stats
Characteristic of a streaming search command?
Operates on each event individually
Distributable streaming - run on indexers
ex. eval, fields, rename, regex
Characteristic of a centralized (stateful) streaming search command?
Runs on SH
ex. head, streamstats
