Log and NetFlow collection and processing

Log data, generated by every device and application in the network, along with NetFlow data, which monitors network traffic, both provide insights into network activities, making them the main sources of input to security information and event management (SIEM) solutions.

A SIEM solution collects, stores, and analyzes this information to gain deeper insights into network behavior, detect threats, and proactively mitigate attacks. This article will discuss some techniques used for collecting and processing this information.

How are logs collected?

Logs are collected from all devices such as databases, routers, firewalls, servers, IDS/IPS devices, domain controllers, workstations, and applications.

Log collection can be done two ways:

• Agentless log collection

  Agentless log collection is the predominant method SIEM solutions use to collect logs. In this method, the log data generated by the devices is automatically sent to a SIEM server securely. There is no need for an additional agent to collect the logs, which reduces the load on the devices.

• Agent-based log collection

  Agent-based log collection demands an agent be deployed on every device that can generate logs. This method can help filter out logs while collecting them, based on defined parameters. Agents also take up less bandwidth and resources, and help provide filtered and structured log data. This method is employed when the devices are in a secure zone where communication is restricted, and it is difficult to send logs to a SIEM server.

How is NetFlow data collected?

The NetFlow collection corresponds to information about IP traffic, mainly:

• Source IP address
• Destination IP address
• Ports accessed
• Services carried out
  And more.

Gathering NetFlow data is done with a NetFlow collector, which also records timestamps, the packets requested, entry and exit interfaces of the IP traffic, and more. The process of NetFlow collection includes gathering the NetFlow data and analyzing the data for bandwidth speed, resource utilization, transmission, and reception in a network by NetFlow collectors. The main functions of NetFlow collectors include collecting flow data transmitted using the User Datagram Protocol (UDP) from NetFlow-enabled devices, and filtering the collected data to reduce its volume. The Simple Network Management Protocol (SNMP) collects information about traffic at every observation point in a network. This method of data collection is enabled on all devices that have an ethernet port, and the data is monitored and analyzed to detect anomalies and prevent threats.

Log processing

The different techniques used for log processing are listed below.

Log parsing

A parser can take unstructured raw log data and format it by grouping similar data under relevant attributes. Parsing makes the retrieval and searching of logs easier. Every SIEM solution includes multiple parsers to process the collected log data.

Log normalization

Normalization is the process of mapping only the necessary log data under relevant attributes, which can be configured by the IT security admin. To monitor the important activities in a network, logs have to be normalized. Log normalization can help distingush between regular and irregular activities in a network.

Log indexing

Normalized log data is separated and stored in files that contain indexed log information; admins can apply queries to the indexed data to accelerate the searching process. SIEM solutions can be configured by the network admin to record the data under a particular index for easier retrieval and interpretation.

Log analysis

Correlating log data helps in identifying if the different log sources correspond to one particular event that threatens network security. Forensic reports help verify where the network was compromised and how an attack was carried out. Log analysis plays a vital role in understanding user behavior and detecting threats, and also helps in preventing an attack before it occurs.