Troubleshooting Splunk 101

Question 1

Linux Troubleshooting Commands

Answer 1

1. ps -aux = check processes
2. kill a process = kill -9 <pid></pid>
3. top =check which process is using the most resources on the server
4. cat /proc/meminfo = check how much memory is available
5. cat /proc/cpuinfo = check CPU utilization
6. df -h = check how much space is left on a volume
7. fdisk -1 = list available drives
8. rpm -qa =check installed rpm packages
9. netstat= find which ports are open and listening for inbound data

Question 2

What are the benefits of Ad Hoc Searches in Splunk?

Answer 2

Ad hoc searches can be run at any time, without the need to schedule them in advance.

This makes them ideal for troubleshooting problems, investigating security incidents, and exploring your data for new insights.

Question 3

Where are Splunk log files stored?

Answer 3

opt/splunk/var/log/splunk

Question 4

Key Splunk log files

Answer 4

1. audit.log = Audit events aka records of user and system activity
2. license_usage.log = info about how your Splunk license is being used i.e. the number of users, the amount of data being indexed, and the number of searches being performed.
3. metrics.log = contains metrics about the Splunk platform aka measure the performance or behavior of a system.
4. splunkd.log=troubleshooting ERROR or WARN messages
5. splunk_ui_access.log = response time for UI access
6. splunk_web_access.log= the web server logs of splunk=splunk uses apache as web server

Question 5

Benefits to using Btool to show .conf

Answer 5

1. Troubleshooting: Btool can be used to troubleshoot configuration problems. For example, if you are having a problem with a particular search, you can use Btool to identify the .conf settings that are affecting the search.
2. Understanding your configuration: Btool can be used to understand your Splunk configuration. For example, you can use Btool to see how different .conf files are merged and how precedence is applied.
3. Documenting your configuration: Btool can be used to document your Splunk configuration. For example, you can use Btool to generate a report of all of your .conf settings.
4. Validating your configuration: Btool can be used to validate your Splunk configuration. For example, you can use Btool to check for errors and conflicts in your .conf settings.

Question 6

Btool command to check stanza and setting names

Answer 6

./ splunk cmd btool check

Question 7

What is Splunk Support Diag

Answer 7

a diagnostic file that contains information about your Splunk environment.

This information can be used by Splunk support to troubleshoot problems and help you get the most out of your Splunk deployment.

Question 8

Steps to execute Splunk Diag

Answer 8

1.Find file output in opt/splunk/<filepath>
2.Move diag file to tmp directory or home directory
3.Change ownership of diag file to ID
4. Put in host of server in WinSCP
5. windows version of server will show on right side (go to root and then tmp directory)
-then move file to whereve</filepath>

Question 9

What is WinSCP

Answer 9

Open source file manager for file transfer between a local and a remote computer

Question 10

Forwarder Communication Issues

Answer 10

1. Phoning Home: On the forwarder run Splunk status command. Then check the deploymentclient.conf file to ensure it has the correct IP address of the deployment server.
2. Log forwarding: Check splunkd.log to ensure that the forwarder is correctly connected to the indexer
3. Problem: I can’t find my data!

Your data is being indexed, but it is being filtered out. Use the Splunk props.conf file to configure Splunk to index and display all of your data.

Your data is being indexed, but it is not being parsed correctly. Use the Splunk transforms.conf file to configure Splunk to parse your data correctly.

4. Problem: Too many search jobs!

You have too many scheduled searches running at the same time. Reduce the number of scheduled searches or schedule them to run at different times.

5. Problem: Dashboard in app is not showing the expected results.

Your dashboard is not refreshing properly. Make sure that your dashboard is configured to refresh properly.

Question 11

Users not having access to their data

Answer 11

Step 1: As an admin, run a search to find the data in order to ensure there are results and to determine what indexer the data is going into because your role as an admin is not restricted.

Step 2: Look at client’s role by logging into the SHC and going to Settings > Access Control > User Access > Users

Step 3: Ensure the following–user’s role has access to the index that stores the data.

***9 times out of 10 it is a permission issue of the user’s role not having access to the data.