CSC570G - OSSEC Open Source Security Assignment

Assignment Task

Introduction

For my research paper, I chose to study the OSSEC (Open Source Security) software. This paper consists of several sections, starting with a description of what OSSEC is, its architecture, and my rationale for using Linux. Additional details are then explored, including OSSEC software requirements, the installation process, file locations, and instructions on how to run it. Next, I delve into applicability for industry use, the current user base, related software, improving availability, backup strategies and disaster recovery.

I then introduce and describe in detail a case study I performed, where I installed OSSEC using the server-agent approach on 21 Internet-accessible servers. I discuss some of the attacks detected and my observations, both of the OSSEC software and of the nature of Internet-based threats for which OSSEC can be an effective tool.

The paper closes with conclusions and the bibliography. An appendix is attached which contains the interactive session log of the complete installation process, with highlighting to focus attention on key aspects of the installation process.

Description

OSSEC is a host-based instruction detection system (HIDS). Host-based IDSs function by monitoring events from inside the server rather than by inspecting network packets as they travel between machines. This exposes a wealth of information not available to network-based IDSs since encrypted traffic is available to the host-based system in plaintext form. OSSEC is open source, runs on nearly every operating system, and has active user and developer communities behind it.

OSSEC has a sophisticated analysis engine that performs the following functions:

• Rootkit Detection
• System Integrity Checking
• Log File Monitoring
• Alert Generation
• Active Response

A rootkit is unauthorized software installed into an operating system by an adversary with the dual goals of ensuring continued privileged access to the system and hiding its own existence from other processes and users on the system. OSSEC uses a number of techniques to detect the presence of a rootkit.

OSSEC validates the integrity of the system by monitoring the files within critical directories on the file system such as /etc, /bin, and /sbin. OSSEC calculates both MD5 and SHA1 hashcodes for the files in those directories and maintains them in a database. If those hashcodes change, an alert is generated.

When an event is detected for which an alert to a system or security administrator needs to be sent, OSSEC can use one of several methods, including emails, SMS messages, pagers, etc.

Further, OSSEC can be configured to take immediate action if necessary. For example, if a brute force attack is detected against the SSH daemon, OSSEC can dynamically insert a rule into the iptables firewall to block subsequent traffic from the offending IP address. This is termed “Active Response” in OSSEC terminology, and is a very powerful feature which qualifies OSSEC not only as an IDS but as an Intrusion Prevention System (IPS) as well. With that power comes.

the risk of unintended consequences, so its enablement and use should be carefully considered. However, my experience with this feature has been very positive. OSSEC is not limited to just modifying firewall rules. It can also update the hosts.deny file, lock a user account, etc. In fact, this facility merely executes a shell script, so quite literally any action can be taken by OSSEC in response to an alert.

The configuration of the alerts, email notifications, and active responses are defined by a set of rules written in a very expressive rules language, coded as XML files. Customizing these rules for specific site needs is straightforward.

The OSSEC development team also provides a web interface that displays agent status information, alert statistics, and an alert search facility that provides for the filtered display of alerts, both current and historical. This is a traditional Apache web application written in PHP, and is provided as an optional component via a separate download and installation process.

This configuration is preferred when monitoring multiple servers because events can be correlated across hosts. What this means is that OSSEC has the ability to detect types of attacks that would otherwise go unnoticed by a HIDS analyzing data from just a single host. An example of this is provided later in this report, where an attacker tries to compromise a single account on many different servers at the same time. The Server-Agent deployment is also easier to maintain.

The alert rules reside in only one location, email notifications originate from just one machine, etc. An additional advantage is that analysis processing is offloaded from the monitored servers. Depending on the existing workload on the monitored machines, this can be an important consideration. The network communication between agents and the server is secure. All traffic is encrypted. Further, public-key cryptography is used to ensure the integrity and privacy of the OSSEC communications.

The process for adding each new agent to the OSSEC environment involves creating a public/private key pair on the server, then copying and pasting the public key into the agent. OSSEC also supports the situation in which only log monitoring is desired (but not system integrity checking or rootkit detection). In this case, the syslog facility can simply send its log records across the network (unencrypted) to the OSSEC server, and OSSEC will include them in its analysis. This is represented in the lower left-hand box of Figure 2.

Rationale for using Linux

The OSSEC Server runs on Linux, Solaris, *BSD, AIX, HP-UX, and Mac. Therefore, I could have chosen any of those as the operating system on which to run OSSEC. The main reason I selected Linux (other than being a specific requirement for this research paper) is because Linux is stable, secure, performs well, and is free. Linux is a natural platform for this type of functionality.

Using Linux to improve Security, Availability, Load Balancing

There are several ways to use Linux to make OSSEC more secure, available, and load balanced. Page 11 of 28 First, use the Server-Agent approach, which offloads processing away from the servers being monitored. This improves performance of the monitored servers, and improves the security of the OSSEC environment, since a compromise of an agent machine will not impact the OSSEC Server or its processing.

Dedicate a separate Linux server to function as the OSSEC Server. Maintain strict security policies on that machine, and enforce all the common securityrelated best practices, such as shutting down unneeded services, default deny rules on the local firewall, applying all the relevant security patches in a timely manner, etc. Agent servers are not dependent on the OSSEC Server's availability, so ensuring high availability via high-end redundancy techniques such as dual servers and network-based load balancing of the OSSEC Server is not a major concern.

Using RAID drives for data protection may be enough in most cases. Network bandwidth does not appear to be an issue as scale (the number of agents) increases. The main concern with scaling up is with the storage of alerts on the OSSEC Server. By default, OSSEC stores alerts in regular flat files. As the number of agents increase, that approach becomes a bottleneck. Previous OSSEC users on the mailing lists have described their solutions, for which the common theme is to use MySQL to store the alerts and partition the database across devices.

This CSC570G – IT and Computer Science has been solved by our PHD Experts at My Uni Paper.