SEC - simple event correlator

Introduction

SEC is an open source and platform independent event correlation tool that was designed to fill the gap between commercial event correlation systems and homegrown solutions that usually comprise a few simple shell scripts. SEC accepts input from regular files, named pipes, and standard input, and can thus be employed as an event correlator for any application that is able to write its output events to a file stream. The SEC configuration is stored in text files as rules, each rule specifying an event matching condition, an action list, and optionally a Boolean expression whose truth value decides whether the rule can be applied at a given moment. Regular expressions, Perl subroutines, etc. are used for defining event matching conditions. SEC can produce output events by executing user-specified shell scripts or programs (e.g., snmptrap or mail), by writing messages to pipes or files, and by various other means.

SEC has been successfully applied in various domains like network management, system monitoring, data security, intrusion detection, log file monitoring and analysis, etc. The applications SEC has been used or integrated with include HP OpenView NNM and Operations, CiscoWorks, BMC Patrol, Nagios, SNMPTT, Snort IDS, Prelude IDS, etc.

Event correlation operations supported by SEC

Following event correlation rule types are currently implemented in SEC:

Single - match input event and execute an action list.
SingleWithScript - match input event and execute an action list, if an external script or program returns a certain exit value.
SingleWithSuppress - match input event and execute an action list, but ignore the following matching events for the next t seconds.
Pair - match input event, execute an action list, and ignore the following matching events until some other input event arrives. On the arrival of the second event execute another action list.
PairWithWindow - match input event and wait for t seconds for other input event to arrive. If that event is not observed within the given time window, execute an action list. If the event arrives on time, execute another action list.
SingleWithThreshold - count matching input events during t seconds and if a given threshold is exceeded, execute an action list and ignore the following matching events during the remaining time window. The window of t seconds is sliding.
SingleWith2Thresholds - count matching input events during t1 seconds and if a given threshold is exceeded, execute an action list. Then start the counting of matching events again and if their number per t2 seconds drops below the second threshold, execute another action list. Both event correlation windows are sliding.
Suppress - suppress matching input event (used to keep the event from being matched by later rules).
Calendar - execute an action list at specific times.
Jump - submit matching input events to specific ruleset(s) for further processing.
Options - set processing options for a ruleset.

Rules allow not only shell commands to be executed as actions, but they can also:

create and delete contexts that decide whether a particular rule can be applied at a given moment,
associate events with a context and report collected events at a later time (similar feature is supported by logsurfer),
generate new events that will be input for other rules,
reset correlation operations that have been started by other rules,
spawn external event, fault, or knowledge analysis modules,
etc.

This makes it possible to combine several rules and form more complex event correlation schemes.

Documentation

See SEC manpage and FAQ for a detailed information about SEC. You can also check the following sources for additional information:

"Working with SEC - the Simple Event Correlator" by Jim Brown - an excellent paper that not only provides a good introduction to SEC but also covers a number of advanced issues (here are the links for the part 1 and part 2 of the paper).
"Simple Event Correlator for real-time security log monitoring" - a paper about SEC that was published in Hakin9 Magazine 1/2006 (6).
"Hardening Linux" (Apress, 2005) by James Turnbull - Chapter 5 of the book contains a discussion and examples how to employ SEC for log monitoring.
"Real-time log file analysis using the Simple Event Correlator (SEC)" by John P. Rouillard - a paper with SEC ruleset examples that was presented at USENIX LISA'2004.
"SEC - a Lightweight Event Correlation Tool" - an early paper about SEC that was presented at IEEE IPOM'2002.
SEC rule repository - if you have developed a ruleset that might be interesting to others as well, please contribute.

Installation information and dependencies

SEC has been primarily tested on Linux and Solaris, but since it is written in Perl and does not use any platform-dependent subroutines, it should work on most operating systems. The author has received reports about SEC working on FreeBSD, OpenBSD, HP-UX, AIX, Tru64 UNIX, Mac OS X, and Win2000 (with CygWin Perl).

In order to install SEC check where your Perl executable is located, and change the first line in sec.pl file accordingly, if necessary. If your Perl executable is /usr/local/bin/perl, for instance, set the first line to #!/usr/local/bin/perl -w. Then copy sec.pl and sec.pl.man (SEC manpage) to appropriate directories, e.g.,
cp sec.pl /usr/local/bin
cp sec.pl.man /usr/local/man/man1/sec.pl.1

Since SEC is generally not tested against outdated Perl releases, it is recommended to run SEC with at least Perl 5.6 (see http://www.perl.org for the latest stable Perl release). SEC also uses Perl Getopt, POSIX, Fcntl, IO::Handle, and Sys::Syslog modules, but those modules are included in the standard installation of Perl.

Mailing list

There is a mailing list for SEC users. The purpose of this list is to facilitate discussion between SEC users, so that you can ask questions from more experienced users and share your experience with others.

Download

sec-2.5.2.tar.gz (July 16 2009)

... or you can visit SEC download page at Sourceforge (has also older versions available).

SEC has also been packaged for:
- Debian Linux
- Gentoo Linux
- Ubuntu Linux
- Fedora Linux
- RHEL
- FreeBSD
- OpenBSD
(if any of the links are broken or you spot packages for other platforms, please contact the author).

Related tools

Logpp is a tool that can be employed for reducing the load of SEC by filtering out irrelevant input data, for converting multi-line log messages into syslog format, and for other log preprocessing tasks. You might also be interested in SLCT and LogHound that were designed for mining patterns from log files.

Author

Risto Vaarandi (ristov at users d0t s0urcef0rge d0t net)

Please don't contact the author with SEC usage questions - you should post such questions to the SEC mailing list.

Acknowledgments

This work is supported by SEB.

The author wishes to thank the following people for supplying software patches and documentation updates:

Al Sorrell
James Brown
John P. Rouillard
Jon Frazier
Mark D. Nagel
Rick Casey
William Gertz