Log collection 101: Covering the basics

Logs are a record of everything that is happening inside the IT environment of your organization. They're typically a series of timestamped messages that give you firsthand information about all the activities in your network.

Every device and application in the network generates log data, along with NetFlow data, which is used to monitor network traffic. Logs are the main source of input to security information and event management (SIEM) solutions. A SIEM solution, at its root, is a log management platform that also performs security analytics and alerting, insider risk mitigation, response automation, threat hunting, and compliance management.

What is log management?

Log management involves the collection, storage, normalization, and analysis of logs to generate reports and alerts. Log management ensures that the network activity data hidden in logs is converted to meaningful, actionable security information. Log management is a prerequisite for network and security administrators to monitor and secure the network. SIEM logging combines event logs with contextual information about users, assets, threats, and vulnerabilities and processes them using algorithms, rules, and statistics.

Log management is a challenging task. To collect and process log data in real time, regardless of the volume of log data and the number of devices in the network, organizations need a robust log management mechanism. All in all, log management needs to be flexible enough to accommodate all network devices and applications.

Log collection

Log collection is the first step in log management. A SIEM solution collects logs and events from a diverse set of systems in the network and aggregates them in one place. Logs are typically collected from workstations, servers, domain controllers, network devices, IDSs, IPSs, endpoint security solutions, databases, web servers, public cloud infrastructure, and cloud platforms.

Every network has different systems and environments that generate various log formats, such as event logs, syslogs, and other application logs. Log collectors need to be flexible enough to accommodate all network devices and applications.

Logs can be collected via:

• Agent-based log collection.
• Agent-less log collection.

Agent-based log collection

Agent-based log collection requires the deployment of an agent on the devices that generate logs. The agent not only collects and filters the logs, but it also parses and converts them into other formats before forwarding them to the log collection server.

Windows, Unix, and most other systems create logs in areas of the file system that require high-level privileges to view, rotate, or relocate. Agents were developed to collect security-related information from the local system and then convert it to a format suitable for transmission over the network to a central collector. The agents are designed to run in the background with sufficient privileges to monitor and manage the logging subsystem, utilizing only those system resources necessary to collect, process, filter, and send the logs to the SIEM host with minimal overhead.

Agent-based log collection comes in handy for collection of logs across WANs and through firewalls. It also helps in log collection from devices residing in the restricted zones of your network such as DMZs. Using an agent for log collection reduces the CPU utilization of the server and thereby provides more control over the events per second rate. Windows Server, NXLog, and OSSEC are some of the popular agents used for log collection.

The agent can be deployed on any server in the network or sub-net and on all types of operating systems. It is installed as a service in that server. The agent collects the logs remotely, pre-processes the logs, and transfers them to the server in real time and without interruption.

How an agent works:

• Once an agent is installed on a device, it gains access to the internal activities of the device and obtains the log data from it.
• After the log data is collected, the agent pre-processes it, and carries out field extraction. It then zips the log data and sends it over to the SIEM server securely.
• The server will then index the logs and proceed further.

Advantages of agent-based log collection:

• Log transmission is secure and reliable, as agents can communicate with the centralized logging server using TLS and SSL.
• Log data is usually sent in compressed batches and is buffered, so no events are lost in transmission.
• Logs are processed and sent to the SIEM in real time, rapidly and efficiently.
• Log filtering is much better in agent-based log collection.
• Helps meet the various compliance requirements.
• Agents can collect logs from various platforms like Windows, Linux, and other systems, and log them in a usable format.
• Due to log filters, unnecessary log data is removed and the log data aggregated is compact. Therefore, agents take up less bandwidth and resources.

Agentless log collection

In SIEM solutions, agentless log collection is the predominant method used to collect logs. In dynamic cloud environments, agentless auditing is critical to reduce costs, unlock visibility, and to accelerate the speed of deployment.

There are embedded devices such as routers, printers, switches, and firewalls in which third-party software installation is not supported. In highly regulated systems, installation of additional software is not permitted. In these cases, an agentless log collection approach can be implemented instead, allowing devices to send logs to a remote data collector. One factor that forces the deployment of agents for log collection is the unavailability of an established network connection.

In agentless log collection, the log data generated by the devices is automatically sent to a SIEM server securely, eliminating the need for an additional agent to collect the logs, which reduces the load on the devices.

How agentless log collection works:

• A client, host, system, or device has previously installed software on it, or in most cases, already has the necessary programming needed to collect all the required data. This software or programming is used to collect the log data.
• The log data is forwarded using native protocols such as SNMP traps, WECS, WMI, and syslogs.
• The log-generating host may directly transmit its logs to the SIEM or there could be an intermediate logging server involved, such as a syslog server.

Advantages of agentless log collection:

• It's easier and fast to deploy, as it does not involve any software installation.
• Lower maintenance cost because there are no software or version updates required as there is no agent.
• Since there is no software installation, maintenance or operations required, agentless log collection can significantly reduce administration efforts.

Between agent-based or agentless log collection, neither is better than the other. The choice should be made considering the needs of the organization. So it is best to have a SIEM solution that offers both agent based and agentless log collection methods.

Log360 is your one-stop solution for all log management and network security challenges. It is an integrated solution that combines EventLog Analyzer, ADAudit Plus, and Cloud Security Plus into a single console to help you manage your network security, Active Directory auditing, and public cloud management. EventLog Analyzer is designed to support both agent based and agentless log collection mechanisms to cater to all devices and applications in the network.

The following table lists some of the important log sources and what methods can be used to collect these logs in Log360.

Learn more about Log360 or write to our support team at support@log360.com for any product questions.