Different types of logs in SIEM and their log formats
In this page
- Types of log data
- Different log formats
- Common Event Format
A security information and event management (SIEM) solution ensures a healthy security posture for an organization's network by monitoring different types of data from the network. Log data records every activity happening on the device, and applications across the network. To assess the security posture of a network, SIEM solutions must collect and analyze different types of log data.
This article elaborates the different types of log data that you should collect and analyze using a SIEM solution to ensure network security.
Types of log data
There are six different types of logs monitored by SIEM solutions:
1. Perimeter device logs
Perimeter devices monitor and regulate traffic to and from the network. Firewalls, virtual private network (VPNs), intrusion detection systems (IDSs), and intrusion prevention systems (IPSs) are some of the perimeter devices. These devices generate logs containing a large amount of data, and perimeter device logs are vital for understanding the security events occurring in the network. Log data in the syslog format helps IT admins perform security audits, troubleshoot operational issues, and better understand the traffic passing through and from the corporate network.
Why do you need to monitor a perimeter device's log data?
- To detect malicious traffic to your network: These logs contain details about incoming traffic, IP addresses of the websites browsed by users, and unsuccessful logon attempts which helps you track down anomalous traffic behavior.
- To detect security misconfigurations: Security misconfigurations are the most important cause for firewall breaches. A few changes to the firewall configurations can open the doors to malicious network traffic. Monitoring firewall logs helps you detect unauthorized security configuration changes.
- To detect attacks: Analyzing firewall logs helps you detect patterns in network activity. For example, when the server receives, within a short time, a large number of SYN packets to connect the client to a server, this might indicate a distributed denial-of-service (DDoS) attack.
Dissecting a typical perimeter device (firewall) log data
2015-07-06 11:35:26 ALLOW TCP 10.40.4.182 10.40.1.11 63064 135 0 - 0 0 0 - - - SEND
The log entry above specifies the time stamp of the event, followed by the action. In this instance, it indicates the day and time the firewall allowed traffic. It also contains information about the protocol used, as well as the IP addresses and port numbers of the source and destination. From log data like this, you can detect attempts to connect to ports that you do not use, indicating that the traffic is malicious.
2. Windows event logs
Windows event logs are a record of everything that happens on a Windows system. This log data is further classified into:
- Windows application logs: These are events logged by the applications in the Windows operating system. For example, an error forcing the application to close is recorded in this application log.
- Security logs: These are any events that may affect the security of the system. It includes failed login attempts and file deletion instances.
- System logs: It contains events that are logged by the operating system. The logs indicate if processes and drivers were loaded successfully.
- Directory service logs: It contains events logged by the Active Directory (AD) service. It records AD operations such as authentication and modification of privileges. These logs are available only for domain controllers.
- DNS server logs: These are logs from domain name system (DNS) servers with information such as client IP addresses, the domain queried, and the record requested. It is available only for DNS servers.
- File replication service logs: It contains events of domain controller replication. It is available only for domain controllers.
Why do you need to monitor Windows event logs?
- To ensure server security: Most critical servers, such as file servers, and AD domain controllers, run on the Windows platform. It is essential to monitor this log data to understand what is happening to your critical resources.
- Windows workstation security: Event logs provide valuable insights into the functioning of a workstation. By monitoring Windows event logs generated from a device, user activities can be monitored for anomalous behavior which can help detect attacks in the early stages. In case of an attack, the logs can help reconstruct the user's activities for forensic purposes.
- To monitor hardware components: An analysis of Windows event logs helps diagnose problems with malfunctioning hardware components of a workstation by indicating the cause for malfunction.
Dissecting a typical Windows event log
Warning 4/28/2020 12:32:47 PM WLAN-AutoConfig 4003 None
Windows classifies every event based on its severity as Warning, Information, Critical, and Error. The security level in this case is Warning. The log entry above is from the WLAN AutoConfig service, which is a connection management utility enabling users to connect to a wireless local area network (WLAN) dynamically. The next segment indicates the date and time the event took place. The log specifies that WLAN AutoConfig detected limited network connectivity, and is attempting automatic recovery. Using this log, a SIEM solution can check for similar logs on other devices at the time stamp referenced in this log, to resolve the network connectivity issue.
3. Endpoint logs
Endpoints are devices that are connected across the network and communicate with other devices across servers. Some examples include desktops, laptops, smartphones, and printers. With organizations increasingly adopting remote work, endpoints create points of entry to the network that could be exploited by malicious actors.
Why do you need to monitor endpoint logs?
- To monitor activities on removable disk drives: Removable disk drives are often vulnerable to malware installations and data exfiltration attempts. By monitoring endpoint logs, these attempts can be detected.
- To monitor user activity: Users are required to abide by their organization's internal as well as external regulatory policies related to installation and use of software on their workstations. Endpoint logs can be used to monitor these policies and provide notifications if violations occur.
Dissecting a typical endpoint device log
Error 6/20/2019 5:00:45 PM Terminal Services- Printers 1111 None
The log above specifies that an error has occurred with the Terminal Services Easy Print driver. This is indicated by the error source, and the Event ID (1111). If a user faces issues while printing a file, the logs can be checked to understand the exact cause for the issue and resolve it.
4. Application logs
Businesses run on various applications such as databases, web server applications, and other in-house apps to perform specific functions. These applications are often vital for the effective functioning of the business. All of these applications generate log data that provide insights about what is happening within the applications.
Why do you need to monitor application logs?
- To troubleshoot issues: These logs help identify and correct issues relating to performance and security of the applications.
- To monitor activities: Logs generated from a database indicate requests and queries from users. This can be used to detect unauthorized file access, or data manipulation attempts by users. The logs are also helpful for troubleshooting problems in the database.
Dissecting a typical application log
02-AUG-2013 17:38:48 * (CONNECT_DATA=(SERVICE_NAME=dev12c)
(CID=(PROGRAM=sqlplus)(HOST=oralinux1)(USER=oracle))) *
(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.121)(PORT=21165))
* establish * dev12c * 0
The above log entry is from an Oracle database system. The log is for a connection attempt from a host computer. The log references the time and date when the request was received by the database server. It also indicates the user and the host computer from which the request originated, along with its IP address, and the port number.
5. Proxy logs
Proxy servers play an important role in an organization's network by providing privacy, regulating access, and saving bandwidth. Since all web requests and responses pass through the proxy server, proxy logs can reveal valuable information about usage statistics and the browsing behavior of endpoint users.
Why do you need to monitor proxy logs?
- To baseline user behavior: Analyzing users' browsing activities from the proxy logs collected can help form a baseline of their behavior. Any deviation from the baseline might reveal a data breach, and indicate that further inspection is required.
- To monitor the length of packets: Proxy logs can help monitor the length of packets exchanged through the proxy server. For example, a user repeatedly sending or receiving packets of the same length within a given interval of time might indicate a software update, or uncover malware exchanging signals with control servers.
Dissecting a typical proxy log
4/8/2020 2:20:55 PM User-001 192.168.10.10 GET https://wikipedia.com/
The log above specifies that User-001 requested pages from Wikipedia.com on the date and time indicated in the log. Analyzing the requests, URLs, and time stamps in the logs help detect patterns, and aids in evidence recovery in case of an event.
6. IoT logs
Internet of Things (IoT) refers to a network of physical devices that exchange data with other devices on the internet. These devices are embedded with sensors, processors, and software to enable data collection, processing, and transmission. Like endpoints, devices that make up an IoT system generate logs. Log data from IoT devices provides insights into the functioning of hardware components, such as microcontrollers, the firmware update requirements of the device, and the flow of data in and out of the device. A crucial part of logging data from IoT systems is the storage location of log data. These devices do not possess sufficient memory to store the logs. So, the logs must be forwarded to a centralized log management solution where they can be stored for extended periods of time. The SIEM solution then analyzes the logs to troubleshoot errors and detect security threats.
The logs from all of the above sources are usually forwarded to the centralized logging solution that correlates and analyzes the data to provide a security overview of your network. The logs are stored and transmitted in different formats, such as CSV, JSON, Key Value Pair, and Common Event Format.
Different log formats
CSV
CSV is a file format that stores values in a comma-separated format. It is a plain-text file format, which allows CSV files to be easily imported into a storage database, regardless of the software used. Because CSV files are not hierarchical or object-oriented, they are also easier to convert to other file types.
JSON
JavaScript Object Notation (JSON) is a text-based format for storing data. It is a structured format, which makes it easier to analyze the stored logs. It can also be queried for specific fields. These additional features make JSON a very reliable format for log management.
Key Value Pair
A key-value pair consists of two elements: a key and a value mapped to it. The key is a constant, and the value is variable across different entries. The formatting involves grouping similar sets of data under a common key. By running the query for a specific key, all the data under that key can be extracted.
Common Event Format
Common Event Format, commonly referenced as CEF, is a log management format that promotes interoperability by making it easier to collect and store log data from different devices and applications. It uses the syslog message format. The most widely used logging format, it is supported by a variety of vendors and software platforms, and consists of a CEF header and a CEF extension that contains log data in key-value pairs.
These are the different types of log data and their formats. Manually collecting these logs from all the different sources in a network and correlating them is a tedious and time-consuming process. A SIEM solution can help you with this. A SIEM solution analyzes the logs collected from different sources, correlates the log data, and provides insights to help organizations detect and recover from cyberattacks.
