Day2 log monitoring Flashcards
what is log files?
Log files are records of events, activities, incidents and transactions stored in a file
› Generated by systems, applications, network appliances, middleboxes, security devices, etc.
Provide critical visibility into system operations, user actions, and potential security incidents
EXAMPLE 1 – WEB SERVER LOG
SCENARIO: A web server receives one HTTP GET request and one HTTP POST
request from two clients on the Internet. For the first request, the resource is
available and returned (code 200 – OK). For the second request, the client
attempted an unauthorized action and hence, is denied (code 403 – Forbidden)
182.138.17.50 - 137.58.101.53 [21/Mar/2025:14:35:22] “GET /index.html HTTP/1.1” 200 1024
103.218.87.13 - 137.58.101.53 [21/Mar/2025:14:36:10] “POST /login.php HTTP/1.1” 403 2048
EXAMPLE 2 – WINDOWS SECURITY LOG
SCENARIO: A user attempted to log into a Windows machine but provided incorrect credentials. The authentication request failed, triggering a security event in the Windows Event Log under the Security category. This log entry
records details such as the username, source IP, timestamp, and failure reason
Source: Microsoft-Windows-Event-Log
Log Type: Security
Event ID: 4625
Task Category: Logon
Level: Information
User: admin
Computer: SERVER21
Date/Time: 2025-03-21 14:32:10
Description: An account failed to log on.
- Account Name: admin
- Workstation Name: DESKTOP-KU21
- Source IP: 192.168.1.100
- Failure Reason: Unknown username or bad password
TYPES OF LOG FILES
Understanding different types of logs and their sources is critical for
effective log monitoring and analysis
» System Logs (e.g., Windows Event Logs, Linux Syslog)
» Network Logs (e.g., Firewalls, IDS/IPS, Load Balancers, Routers)
» Application Logs (e.g., Web Servers, Databases, Cloud Services)
» Security Logs (e.g., SIEM, Antivirus, Honeypot, Endpoint Detection & Response)
» Operational Technology (OT) Logs (e.g., SCADA, Data Historian, HMI logs)
Each log type provides unique insights into system behavior, security
incidents, and operational performance
LOG CREATION – WINDOWS SYSTEMS
› In Windows systems, logs are created by the Windows Event Logging service
» Collects, stores and manages logs from various system components (OS, services, apps)
› Categorizes records into four different types:
» Security Logs (Records any security related events)
» System Logs (OS events like driver failures)
» Application Logs (Software and application events)
» Setup Logs (Installation and update-related logs)
› Logs are stored in two directories:
» C:\Windows\System32\winevt\Logs (new location)
» C:\Windows\System32\config (old location but still used)
› Logs can be viewed & analyzed in the Windows Event Viewer utility
› Users can also perform targeted security logging through Windows Security
Auditing feature
» Takes in a user-specified auditing policy to track certain types of events and activities
LOG CREATION – LINUX-BASED SYSTEMS
› In Linux systems, logging is generally performed through a Syslog-based
utility, such as rsyslog, syslog-ng or Graylog
» Syslog captures a wide range of system, application, and security events
» Well-defined and widely-used logging standard
» Syslog will be covered in more detail in the subsequent slides
› Logs are stored in /var/log/ directory (most apps/utilities share this directory
for storing logs of different kinds)
› For targeted logging of security events and incidents, Linux Audit Framework
(AuditD) is used
» Equivalent to the Windows Security Auditing feature
» Tracks security events across the system based on audit policies
» Logs are stored in /var/log/audit/audit.log
WHY ARE LOGS IMPORTANT
AND HOW DO THEY HELP
ROLE OF LOG FILES
› Logs play a critical role in both cybersecurity and digital forensics
» Provide a recorded history of system, network, and user activity
» Important source of evidence in investigating incidents
› Help answer key questions about attack timeline and attribution
» Who accessed the system and when?
» What commands or actions were performed on the system?
» Was any sensitive data stolen or exfiltrated?
» Were there any security policies violated?
» Did the infection spread to other machines in the network?
» And many others!!
GENERAL BENEFITS OF LOG MONITORING
› Log monitoring refers to the continuous collection, analysis, and real-time
tracking of log data generated by systems, networks, applications, and
security devices
» Supports troubleshooting performance-related problems, slow response times &
crashes
» Ensures system integrity by tracking changes to configuration files and registry settings
» Helps detect anomalies, security incidents, and operational issues
» Facilitates the process of addressing cyber threats before they escalate
» Essential for incident response and compliance requirements
» Heavily used to monitor infrastructure state via Security Information and Event
Management (SIEM) and Security Operations Center (SOC)
› Let’s see some more details of log monitoring and its applications
EXAMPLE USE CASES & APPLICATIONS
› Threat Detection and Incident Response:
» User Authentication logs help detect brute force attacks and unauthorized logins
» Firewall and IDS/IPS logs reveal suspicious network traffic (e.g., port scans, DDoS attacks)
» Endpoint Security logs detect malware infections, unauthorized software installations, and suspicious
command executions
› Security Monitoring and Anomaly Detection:
» By combining logs from various sources (e.g., firewalls, servers, endpoint devices), organizations can
detect anomalies that might indicate an attack
› Compliance and Regulatory Requirements:
» GDPR & HIPAA: Require logs to track access to personal or sensitive data
» PCI-DSS: Mandates logging of all access to cardholder data
LOG MANAGEMENT APPROACHES
CENTRALIZED:
› All logs are collected and stored in a
central repository (e.g., SIEM
solutions)
› Enables correlation across different
systems for better insights
› Allows for efficient long-term storage
and retrieval
DECENTRALIZED:
› Logs are stored locally on devices and
are analyzed independently
› Common in legacy or air-gapped
environments (e.g., ICS/OT networks)
› Devices retain control over log data but
makes correlation harder
WHAT IS SYSLOG
› Syslog is a comprehensive logging standard for centralized message logging
› Modular design allows for the separation of the software that generates messages, the
system that stores them, and the software that reports and analyzes them
» Frees programmers from managing log files
» Gives sysadmins control over log management
› Each message includes a:
» Facility Code (what is the source of a message or where did a certain event take place)
» Severity Level (what is the criticality of a message or how serious is an event)
› Admins and devs may use syslog for system management and security auditing as well as
general informational, analysis, and debugging messages
› A wide variety of devices, such as printers, routers, middleboxes, etc., across many
platforms use the Syslog standard
› Consolidates logging data from different types of systems into a central repository for
processing and analysis
SYSLOG – ARCHITECTURE
› Syslog Client
» Daemon that does the actual logging
» Can be configured to track and record events of different types at different granularity
» Shares the log data with the server
› Syslog Server
» Also known as the Syslog Collector/Receiver/Listener
» Collects all Syslog messages sent by the network devices in a database
» Responsible for filtering the data and generating alerts (or appropriate response)
› In a typical network, numerous Syslog clients are simultaneously sending log data to
the Syslog server
SYSLOG – FACILITY CODES
A facility value is used to specify the type of system that generated an event. Is also used to
compute the priority of the event (PRI).
0= kernel messages
1=user-level messages
2= mail system
3=system daemons
4= Security abd authorization-related messages
…=…
15= Clock daemon
16-23=Eight local levels for other programs
SYSLOG – SEVERITY LEVELS
A severity code is used to define the severity level (or criticality) of an event that is being logged
CODE = SEVERITY = DESCRIPTION
0 = Emergency = System is
unusable, panic situations (hardware failure, crash)
1 = Alert = Urgent situations, immediate action required
2 = Critical = Critical situations or conditions
3 = Error = Non-critical errors
4 = Warning = Warnings
5 = Notice = Might merit investigation
6 = Informational = Informational messages
7 = Debug = Debugging (typically enabled temporarily)
SYSLOG – PRIORITY VALUE (PRI)
› The two values (Facility value and Severity code) are combined to produce a
Priority Value (PRI) sent with the message
› The Priority Value is calculated by multiplying the Facility value by eight and
then adding the Severity code to the result
› PRI = (Facility Value x 8) + Severity Code
› The lower the PRI, the higher the priority
» Higher priority items require immediate attention
» Lower priority items can be deferred
SYSLOG – MESSAGE FORMAT
› The Syslog message consists of three parts:
» HEADER (with identifying information)
» STRUCTURED DATA (machine readable data in “key=value” format)
» MSG (the message itself or the payload)
› FORMAT (RFC5424): HEADER + STRUCTURED DATA + MSG
» OLD FORMAT (RFC3164): PRI + HEADER + MSG
› Some messages are simple, readable text, others may be quite long and contain
fine-grained details covering every aspect of an event
SYSLOG – HEADER COMPONENT
› HEADER
» Priority Value (PRI)
» Version
» Timestamp
» Hostname
» Application
» Process ID
» Message ID
SYSLOG– STRUCTURED DATA COMPONENT
› STRUCTURED DATA
» Provides a mechanism to express information in a well-defined, easily parseable and
interpretable data format in the form of key=value pairs.
› Can contain zero, one, or multiple structured data elements (SD-Elements)
› In case of zero SD-Elements, the STRUCTURED DATA field MUST contain the NILVALUE
› Example:
[exampleSDID@32473 iut=”3” eventSource=”Application” eventID=”1011”]
SYSLOG – MSG COMPONENT
› MSG
» The MSG part (also called the payload) contains a free-form message that provides information about the event.
› If a Syslog application encodes the message body in UTF-8 encoding, the
string MUST start with the Unicode Byte Order Mask or Mark (BOM)
» The hex representation of UTF-8 BOM is EF BB BF
» For other encodings, the BOM will be different
› The MSG component is often used to describe the event being recorded,
for example:
» Failed login attempt by remote user
» Configuration settings changed
» Patch C157 installed by admin user
SYSLOG – EXAMPLE
<165>1 2025-02-11T22:14:15.003Z kaust.server123.com evntslog 1187 ID47 [sampleSDID@786 interface=“eth1”
eventSource=“NginX” protocol=“TCP”] [SDID@KAUST471 severity=“warning”] An Application event log entry was
deleted unexpectedly
› In this example, we have the following information:
» HEADER is in red font, STRUCTURED DATA elements are in blue font and MSG is in green font
» The PRI value is 165
» The Syslog version is 1
» The message was created on 11 February 2025 at 10:14:15pm UTC, 3 milliseconds into the next second
» The message originated from the host “kaust.server123.com”
» The name of the application that generated the message is “NginX“
» The process ID is 1187
» The message ID is ID47
» There are two structured data elements in the STRUCTURED DATA component. The first has SD-ID
“sampleSDID@786” and three parameters and the second has SD-ID “SDID@KAUST471” with only one
parameter
» The message or payload is “An application event log entry was deleted unexpectedly”
WHAT IS A LOG MANAGEMENT PLATFORM?
Logs constitute large amounts of data
» Once aggregated, logs can be gigabytes or terabytes of data
» Makes management and analysis very challenging and time-consuming
› Log management platforms help deal with this challenge
› Provide several desirable functions to make dealing with log data manageable:
» Collection & Aggregation
» Log Storage
» Log Analysis & Reporting
» Log Disposal
› Multiple components work together to generate, transmit, store, analyze and
dispose of log data
LOG MANAGEMENT – FUNCTIONS
› Collection & Aggregation
» Log Parsing
» Event Filtering
» Event Aggregation
› Analysis
» Event Correlation
» Log Viewing
» Log Reporting
› Storage
» Log Rotation
» Log Archiving
» Log Compression
» Log Reduction
» Log Normalization / Conversion
» Log File Integrity Checking
› Disposal
» Log Clearing
› Collection & Aggregation
» Log Parsing
* Extracts specific data fields from raw log entries, transforming unstructured logs into structured data that can be easily analyzed or used in other logging processes.
» Event Filtering
* Not all log entries are valuable. Event filtering identifies and suppresses log entries that are deemed low-priority or irrelevant, reducing noise and optimizing storage.
» Event Aggregation
* When multiple log entries describe the same event, aggregation merges them into a single record while maintaining a count of occurrences. This minimizes redundancy and reduces size of data.
also LOG MANAGEMENT – FUNCTIONS
› Storage
» Log Rotation
* To prevent logs from growing indefinitely, log rotation closes an active log file and starts a new
one based on a predefined schedule (e.g., hourly, daily) or when a file reaches a set size.
» Log Archiving
* Security logs often need to be stored long-term to meet legal, regulatory, or forensic
requirements. Logs may be moved to external or secondary storage (e.g., SAN, cloud storage,
or dedicated log servers) for future reference.
» Log Compression
* To conserve storage, log compression reduces file size without altering content. This is
commonly applied during log rotation or archiving.
» Log Reduction
* Log reduction is removing unneeded entries from a log to create a new log that is smaller. A
similar process is event reduction, which removes unneeded data fields from all log entries.
› Storage
» Log Normalization / Conversion
* Logs often exist in different formats. Conversion translates logs from one format to another
(e.g., from a database format to a structured XML file) to ensure compatibility across tools and
systems.
» Log File Integrity Checking
* To detect tampering, integrity checks compute and store cryptographic hashes (message
digests) of log files. Any unauthorized modification is flagged as a security concern
also also LOG MANAGEMENT – FUNCTIONS
› Analysis
» Event Correlation
* This technique connects related log entries to detect patterns, anomalies, or security incidents.
Rule-based correlation is commonly used to link events based on timestamps, IPs, or user actions.
» Log Viewing
* Raw logs can be complex. Log viewers format and display logs in a human-readable way, often
with search, filtering, and aggregation capabilities.
» Log Reporting
* Reports summarize log data over a defined period, highlighting critical security events, trends, or
compliance insights. These reports are essential for audits and incident investigations.
› Disposal
» Log Clearing
* When logs are no longer needed, log clearing removes old entries while ensuring important data
has been archived. This prevents unnecessary log buildup and optimizes system performance.
UNDERSTANDING EVENT CORRELATION
› Event correlation is a technique that relates or links various events across logs to identify relationships and attack patterns and determine the cause and methodology of an attack
› Events can be linked or correlated based on several attributes:
» Similar IP addresses, usernames/accounts, hostnames, etc.
» Events triggered by the same process, application or executable
» Close physical proximity or geolocation
» Temporally sequential events (log entries occurring in quick succession having close timestamps)
» Events originating from the same device, service, or cloud provider
› Used for making sense of a large number of events and pinpointing the few events that are really
important in a mass of information
what is Root Cause Analysis (RCA
is a major component of event correlation
Method of problem solving used for identifying the root causes (or primary causes) of faults or problems
In log analysis, event correlation is usually a four-step process carried out on a Log Management Platform:
1-Event Filtering (Discarding & Prioritizing)
2-Event Aggregation (Summarizing & De-Duplication)
3-Event Masking (Ignoring & Excluding)
4-Root Cause Analysis (Dependency & Relationship Analysis)
EVENT CORRELATION – EXAMPLE
› Scenario Overview:
» A cybersecurity incident has occurred where an attacker gained access to an
enterprise network through a phishing attack. The attacker then escalated
privileges, moved laterally (pivoted), and exfiltrated sensitive data.
› Phishing Email → PowerShell Execution → C2 Communication →
Credential Theft → Lateral Movement → Data Exfiltration
› We have logs from different network devices and security systems
› We will analyze the logs and correlate the events
EVENT CORRELATION – TYPES
1- AI/ML-Based
Approach
2- Graph-Based
Approach
3-Rule-Based
Approach
what is AI/ML-Based
Approach
A neural network is
constructed and
trained to detect the
anomalies in the event
stream. It can also
highlight root causes
and various other
indicators of interest.
what is Graph-Based
Approach
A graph is constructed
with each node as a
system component
and each edge as a
dependency/relation
among two
components. The
graph is then searched
for peculiar patterns
and sub-graphs
indicative of a
problem.
what is Rule-Based
Approach
Events are correlated
according to a set of
rules and conditions.
The system can take
appropriate actions
based on which rules
and conditions are
triggered
WHAT ARE IOCs(indicators of compromise)
Artifact or sign that indicates a system or network may have been breached
Common types of IoCs
» File Hashes (MD5, SHA-1) of malware samples
» IP Addresses / Domains used for command-and-control (C2)
» File Paths / Registry Keys modified by malware
» Malicious Email Addresses or URLs in phishing campaigns
» A few others (unusual ports or services, suspicious cron jobs, malicious macros, etc.)
› Very important for monitoring an organization’s infrastructure for malicious
activity
» Enable early detection of threats
» Help in incident response and containment
» Support threat hunting and intelligence sharing
USING SIGMA & YARA TO FIND IOCs
› Logs contain a lot of information pertaining to different kinds of malicious
activities, which leaves behind IoCs in the records
› Sigma and YARA are YAML-based detection languages (or tools) that search
for malicious patterns or indicators in log files via user-defined rules
» Sigma was designed specifically to scan and search through log data
» YARA is mostly used for scanning files and executables/binaries but can also be used for
log files
› Provide rich searching capabilities to analyze log files, fish out relevant data
that matches the search criteria and raise alerts
› Technology agnostic standards with large open-source repositories
containing thousands of “ready to go” rules
HUTING FOR IOCs – YARA
› In YARA, each rule contains a textual or binary pattern to match a particular
malware family
» This is called a signature (a binary value that indicates the presence of the malware)
› Specifically, each rule has three sections:
» Meta Section
» General description and meta-level information about the rule
» Strings Definition Section
» Specific strings to be searched in file or memory
» Condition Section
» Logic of the rule goes here
» Usually refers to strings defined in the Strings section
HUNTING FOR IOCs – SIGMA
› Sigma rules contain information required to detect odd, bad or malicious
behavior when inspecting log files (usually within the context of a SIEM –
coming later)
› Rules are similar to YARA in appearance as both are YAML-based
› Each rule is separated into three main components:
» Detection
* What malicious behavior the rule should search for
* Most important component of any Sigma rule as it specifies exactly what the rule is looking for
across relevant logs
» Logsource
* What types of logs this detection should search over
» Metadata
* Other information about the detection
CYBER SECURITY PLATFORM – SIEM
› Security Information and Event Management (SIEM)
» Collects and aggregates data from various devices and performs correlation
» Examines and analyzes data for IoCs and signs of compromise using YARA/Sigma rules & user queries
what is ELK stack
is the most popular open-source log analysis and management platform used
to build custom SIEM solutions (OSSEC Wazuh, Azure Sentinel, Apache Metron, etc.)
» E – Elasticsearch
* A search and analytics engine
* Stores and indexes massive amounts of log data quickly
* Think of it as the brain that lets you query everything fast
» L – Logstash (often combined with Beats)
* A data processing pipeline
* Collects logs from various sources, processes them (e.g., filtering, conversion, etc.,), and ships to Elasticsearch
* Like a smart conveyor belt for logs
» K – Kibana
* A visualization tool
* Let’s you explore, plot (e.g., extrapolation, trend lines, etc.,), and dashboard your log data
* The UI of the stack used for user inputs/outputs and alerting
BEYOND SIEMs – SOCs
SIEM = TOOL
› Think log collection + detection +
correlation + dashboards
› Like a security camera system
Security op center = people + process + tools
› The operational team uses tools (like
SIEM) to defend the organization
through structured processes
› Like a security guard team monitoring
the infrastructure via cameras
What is Log Parsing
Extracts specific data fields from raw logs and converts them into structured key=value format for easy analysis.
What is Event Filtering?
Removes low-priority or irrelevant log entries to reduce noise and optimize storage.
What is Event Aggregation?
Merges multiple identical log entries into a single entry and keeps a count to reduce redundancy.
What is Event Correlation?
Links related log events to identify patterns or detect complex attacks.
What is Log Viewing?
Displays log entries in a user-friendly format, like tables or dashboards.
What is Log Reporting?
Generates summaries and reports based on logs for monitoring or compliance.
What is Log Rotation?
Replaces old log files with new ones once a size or time limit is reached to prevent uncontrolled growth.
What is Log Archiving?
Stores old log files for long-term retention and future reference.
What is Log Compression?
Reduces the size of log files using compression (e.g., ZIP or GZIP) to save disk space.
What is Log Reduction?
Removes repetitive or unnecessary data to shrink the size of logs.
What is Log Normalization / Conversion?
Converts logs from different formats into a unified, standardized structure.
What is Log File Integrity Checking?
Ensures logs haven’t been tampered with by verifying hashes or using integrity tools.
What is Log Clearing?
Secure deletion of log files when they are no longer needed, based on data retention policies.
……………..Mandates logging of all access to cardholder data
PCI-DSS
………….Require logs to track access to personal or sensitive data
GDPR & HIPAA
T/F FORMAT (RFC5424): HEADER + STRUCTURED DATA + MSG
» OLD FORMAT (RFC3164): PRI + HEADER + MSG
T
Provide several desirable functions to make dealing with log data manageable: 4 THINGS
» Collection & Aggregation
» Log Storage
» Log Analysis & Reporting
» Log Disposal
……………is a technique that relates or links various events across logs to identify
relationships and attack patterns and determine the cause and methodology of an attack
Event correlation
YARA RULE Specifically, each rule has three sections:
» Meta Section
General description and meta-level information about the rule
» Strings Definition Section
-Specific strings to be searched in file or memor
» Condition Section:
- Logic of the rule goes here
- Usually refers to strings defined in the Strings section
what is a wildcard mean in YARA ?
can ignore this value & only check the rest of the signature
……… contain information required to detect odd, bad or malicious behavior when inspecting log files
Sigma rules
SIGMA Each rule is separated into three main components
» Detection
* What malicious behavior the rule should search for
* Most important component of any Sigma rule as it specifies exactly what the rule is looking for
across relevant logs
» Logsource
* What types of logs this detection should search over
» Metadata
* Other information about the detection
……………. is used to declare
the exact log file on which this Sigma rule should be applied
logsource tag
………….. is used to declare
the search criteria and the condition that should trigger this rule
detection tag
T/F Search criteria is often defined under
the “selection” heading
T
