Troubleshooting 101

Question 1

What is SNOW?

Answer 1

a ticketing system

Question 2

What does ps -aux do?

Answer 2

lists all processes

Question 3

How to end a process?

Answer 3

kill -9 [PID]

Question 4

How to check the processes that are taking the most resources on a system?

Answer 4

with a top command

Question 5

What could we do if for some reason ./splunk restart command was not working?

Answer 5

kill -9 splunk process, and then try to run splunk

Question 6

what is /proc/meminfo?

Answer 6

stores info about how much memory is available

Question 7

what is /proc/cpuinfo?

Answer 7

stores information about cpu utilization

Question 8

What does fdisk -l do?

Answer 8

lists available drives

Question 9

What does rpm -qa do?

Answer 9

lists installed rpm packages

Question 10

What does netstat -tanpu do?

Answer 10

Finds which ports are open and listening for inbound data

Question 11

What is minimum hardware specification for Search Heads?

Answer 11

• 16 physical cpu cores or 32 vcpu at 2ghz or greater speed core
• 12 gb ram

Question 12

What is minimum hardware requirement for indexers?

Answer 12

• 12 physical cpu cores or 24 vcpu at 2ghz or greater speed core
• 12 gb ram

Question 13

What is MID-range hardware requirement for indexers?

Answer 13

• 24physical cpu cores or 48 vcpu at 2ghz or greater speed core
• 64 gb ram

Question 14

What is high performance hardware requirement for indexers?

Answer 14

• 48 physical CPU cores, or 96 vCPU or greater per core
• 128 gb RAM

Question 15

Where we can access Splunk internal logs?

Answer 15

We can find internal splunk logs in _internal index (so we woul run a search on SH GUI) or we can access the files directly through CLI : $SPLUNK_HOME/var/log/splunk

Question 16

What does tail -f do?

Answer 16

Prints last 10 lines of a file nad output appended data as the file grows

Question 17

Name some of splunk .log files

Answer 17

audit.log

license_usage.log

metrics. log
splunkd. log

slunk_ui_access.log

splunk_web_access.log

Question 18

What does audit.log store?

Answer 18

Audits what users do on a system, gives information about users activity such as failed login, running a search, modyfing a setting and more

Question 19

What does license_usage.log store?

Answer 19

It keeps track of how much license is being used

Question 20

What does metrics.log store?

Answer 20

Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processes and queue usage in Splunk’s data processing

Question 21

What is throughput?

Answer 21

It is about how much data is flowing through splunk

Question 22

What does splunkd.log store?

Answer 22

It stores information on what is going in splunk. It is a troubleshooting file - look for “errors” and “warns” in it.

Question 23

What does splunk_ui_access.log store?

Answer 23

It stores gui response time

Question 24

What does splunk_web_access.log store?

Answer 24

It stores web server logs

Question 25

What is btool?

Answer 25

It shows you the settings in the given .conf files. Shows you possibles mistakes in configurations.

Question 26

How to display a sum of a given configuration file in a given splunk component?

Answer 26

./splunk btool [.conf file name] list i.e: ./splunk btool inputs list

Question 27

How to display a sum of a given configuration file in a given splunk component and show filepaths to each line?

Answer 27

./splunk btool [file] list --debug

Question 28

How to check for typos in stanzas and settings names?

Answer 28

./splunk btool check

Question 29

How to prepere and send diag file?

Answer 29

1. ./splunk diag 2. Using SFTP transfer diag file (double check permissions and ownership) to your machine, and from it send it to Splunk support or, if you have access to web use this command which will create diag file and send it ./splunk diag --upload

Question 30

How to troubleshoot forwarder?

Answer 30

- Check if splunk is running (./splunk status) - Check deploymentclient.conf - check inputs.conf/outputs.conf - check splunkd.log to ensure that the forwarder is correctly connected to the indexers

Question 31

How to troubleshoot users not having accesds to their data?

Answer 31

a) check if the data is there b) look at client's role Settings\>Access Control \>User Access \> Users c) Ensure that the following user's role has access to the index

Question 32

lists all processes

Answer 32

What does ps -aux do?

Question 33

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. kill -9 [PID]

Answer 33

How to end a process?

Question 34

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. stores info about how much memory is available

Answer 34

what is /proc/meminfo?

Question 35

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. stores information about cpu utilization

Answer 35

what is /proc/cpuinfo?

Question 36

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. lists available drives

Answer 36

What does fdisk -l do?

Question 37

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. lists installed rpm packages

Answer 37

What does rpm -qa do?

Question 38

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. Finds which ports are open and listening for inbound data

Answer 38

What does netstat -tanpu do?

Question 39

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. - 16 physical cpu cores or 32 vcpu at 2ghz or greater speed core - 12 gb ram

Answer 39

What is minimum hardware specification for Search Heads?

Question 40

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. - 12 physical cpu cores or 24 vcpu at 2ghz or greater speed core - 12 gb ram

Answer 40

What is minimum hardware requirement for indexers?

Question 41

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. - 24physical cpu cores or 48 vcpu at 2ghz or greater speed core - 64 gb ram

Answer 41

What is MID-range hardware requirement for indexers?

Question 42

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. - 48 physical CPU cores, or 96 vCPU or greater per core - 128 gb RAM

Answer 42

What is high performance hardware requirement for indexers?

Question 43

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. Prints last 10 lines of a file nad output appended data as the file grows

Answer 43

What does tail -f do?

Question 44

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. Audits what users do on a system, gives information about users activity such as failed login, running a search, modyfing a setting and more

Answer 44

What does audit.log store?

Question 45

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It keeps track of how much license is being used

Answer 45

What does license\_usage.log store?

Question 46

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. Contains periodic snapshots of Splunk performance and system data, including information about CPU usage by internal processes and queue usage in Splunk's data processing

Answer 46

What does metrics.log store?

Question 47

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It is about how much data is flowing through splunk

Answer 47

What is throughput?

Question 48

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It stores information on what is going in splunk. It is a troubleshooting file - look for "errors" and "warns" in it.

Answer 48

What does splunkd.log store?

Question 49

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It stores gui response time

Answer 49

What does splunk\_ui\_access.log store?

Question 50

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It stores web server logs

Answer 50

What does splunk\_web\_access.log store?

Question 51

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. It shows you the settings in the given .conf files. Shows you possibles mistakes in configurations.

Answer 51

What is btool?

Question 52

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. ./splunk btool [.conf file name] list i.e: ./splunk btool inputs list

Answer 52

How to display a sum of a given configuration file in a given splunk component?

Question 53

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. ./splunk btool [file] list --debug

Answer 53

How to display a sum of a given configuration file in a given splunk component and show filepaths to each line?

Question 54

# This is the "reversed" card. It first displays you the answer and you have to guess the question part of it. ./splunk btool check

Answer 54

How to check for typos in stanzas and settings names?