Network session data analysis with Snort and Argus

This edition of Snort Report departs from the standard format to introduce Argus, a session data collecting tool that can work alongside Snort. Learn how session data can complement Snort's alert data for network session analysis

Service provider takeaway: Service providers will learn how flow/session data can complement the alert data supplied by the Snort intrusion detection system for network session data analysis.

This edition of the Snort Report departs from the standard format by introducing a data format and data collecting tool that can work alongside Snort. The data format is session data, and the tool is Argus 3.0.

Why session data?

The Snort intrusion detection system can identify suspicious and malicious activity by inspecting network traffic. Snort makes a judgment based on its analytical capabilities and notifies the operator of its decision by generating an alert. I call the output of this collect-inspect-report process "alert data."

While this is a good and necessary methodology, it has one important flaw. In most configurations, Snort is not told to report on what it sees if the traffic in question is deemed to be "normal." One might consider this aspect of Snort to be a benefit. Why generate an alert if the traffic is "normal" and not suspicious or malicious?

No alerting system can perfectly identify all suspicious or malicious activity. In many cases it's simply not possible -- especially on a packet-by-packet basis -- to identify a packet or stream as being worthy of an operator's attention. In those cases it makes sense to keep a log of the traffic. Recording traffic or characteristics of traffic for later analysis has recently been labeled retrospective network analysis (RNA), not to be confused with Sourcefire's Real-time Network Awareness. Others call recording traffic in this manner "network forensics," but that implies a degree of care and evidence handling that exceeds the methodology I present here.

Previous Snort Reports
  • How to use shared object rules in Snort
  • Snort limitations
  • Output options for Snort data
  • Detect events without Snort IDS rules
  • When you collect data about traffic that Snort didn't consider to be suspicious or malicious, you have the opportunity to look back (hence the term "retrospective") to see what happened during an incident. How do you know to look back? Perhaps you receive a tip from law enforcement. Maybe a client reports odd activity. Or you perform a manual investigation and realize you'd like to know as much as possible about the network traffic of a certain host. In all of these situations, Snort might not have provided any clue that something was amiss.

    Despite my attention to Snort in this series, I never deploy Snort as a stand-alone tool. I always supplement Snort with additional data sources. One of the most important supplementary data sources I collect is session data.

    What is session data?

    As I wrote in my first book, The Tao of Network Security Monitoring (Addison-Wesley, 2005): "Session data represents a summary of a conversation between two parties. [...] A session, also known as a flow, a stream or a conversation, is a summary of a packet exchange between two systems." I also defined the seven minimum elements of a session, namely:

  • Timestamp (generally when the session began, preferably including the end of the session, too).
  • Protocol.
  • Source IP address.
  • Source port (for protocols that offer ports, like TCP, SCTP and UDP).
  • Destination IP address.
  • Destination port.
  • Measure of the amount of information exchanged during the session (to include bytes sent by the source, bytes sent by the destination, packets sent by the source and packets sent by the destination).

    I don't consider records that lack these seven basic elements to be session records. If you have a record showing only a source IP talking to a destination IP, it isn't completely worthless. However, a real session record includes the other five elements listed in addition to source IP and destination IP.

    What does a session look like?

    The following is a session record. I'll explain how I generated this record later.

             StartTime  Proto            SrcAddr  Sport            DstAddr  Dport  
    SrcPkts SAppBytes  DstPkts    DAppBytes
    1204514703.744796    tcp          
    10         1953        9         4049

    The columns contain the following:

    1204514703.744796: date in Unix time format. Using the date command we can convert to something more obvious:

    p># date -r 1204514703
    Sun Mar 2 22:25:03 EST 2008

    tcp: the protocol of the session the source IP and source port the destination IP and destination port

    10: number of packets sent by the source
    1953: number of bytes of application (above layer 4) data sent by the source
    9: number of packets sent by the destination
    4049: number of bytes of application (above layer 4) data sent by the destination

    This is a bidirectional session record. In other words, it displays data sent by the source and destination in a single record. Other types of session records, such as NetFlow (not shown here) are unidirectional. That means two records are needed to describe a conversation. The first lists data from source to destination, while the second lists data from destination to source.

    Getting started with Argus

    Let's look at how you can collect session data using Argus 3.0, developed by Carter Bullard. Argus dates back to 1993, the days before Snort and Wireshark/Ethereal, when Tcpdump was one of the few tools a network security analyst had. For the last two years, Carter has been coding Argus 3.0, and this article uses code posted to ftp.qosient.com/dev for introductory purposes.

    Argus 3.0 is a complicated and powerful tool. Those who frequent the Argus Development Mailing List are not going to learn anything new here! This article is for those who are unfamiliar with Argus but would like to try it for network session data analysis.

    The following examples are run on FreeBSD 6.2:

    Argus is currently packaged as a server component (argus) and client programs (argus-clients). I retrieve both using "fetch" and then extract them.

    hacom:/usr/local/src# fetch ftp://ftp.qosient.com/dev/argus-3.0/argus-3.0.0.tar.gz
    argus-3.0.0.tar.gz       100% of 391 kB 833 kBps
    hacom:/usr/local/src# fetch ftp://ftp.qosient.com/dev/argus-3.0/argus-clients-3.0.0.rc.69.tar.gz
    argus-clients-3.0.0.rc.69.tar.gz      100% of 1422 kB 1058 kBps

    hacom:/usr/local/src# tar -xzf argus-3.0.0.tar.gz
    hacom:/usr/local/src# tar -xzf argus-clients-3.0.0.rc.69.tar.gz

    Because I like to experiment and keep multiple versions of programs like Argus, I install each program into its own directory in /usr/local. (Yes, I could use Stow.)

    hacom:/usr/local/src# mkdir /usr/local/argus-3.0.0
    hacom:/usr/local/src# mkdir /usr/local/argus-clients-3.0.0.rc.69

    Next, I compile the Argus server package. Only an "argus" binary is created in the "bin" directory:

    hacom:/usr/local/src/argus-3.0.0# ./configure --prefix=/usr/local/argus-3.0.0
    hacom:/usr/local/src/argus-3.0.0# make
    hacom:/usr/local/src/argus-3.0.0# make install
    hacom:/usr/local/src/argus-3.0.0# ls /usr/local/argus-3.0.0/sbin/

    Finally, I compile the Argus clients package. A variety of tools are created in the "bin" directory:

    hacom:/usr/local/src/argus-3.0.0# cd ../argus-clients-3.0.0.rc.69
    hacom:/usr/local/src/argus-clients-3.0.0.rc.69# ./configure --prefix=/usr/local/argus-clients-3.0.0.rc.69
    hacom:/usr/local/src/argus-clients-3.0.0.rc.69# make
    hacom:/usr/local/src/argus-clients-3.0.0.rc.69# make install
    hacom:/usr/local/src/argus-clients-3.0.0.rc.69# ls /usr/local/argus-clients-3.0.0.rc.69/bin/
    argusbug        radark          rahisto         rapolicy        rastrip
    ra              radump          rahosts         raports         ratemplate
    rabins          rafilteraddr    ralabel         rasort          ratimerange
    racluster       ragraph         ranonymize      rasplit         ratop
    racount         ragrep          rapath          rastream        ratree

    Now I'm ready to collect session data with Argus.

    Starting the Argus server

    Let's look at Argus 3.0's more easily demonstrated features. The help screen summarizes command-line options; I don't create a config file here:

    hacom:/root# /usr/local/argus-3.0.0/sbin/argus -h
    Argus Version 3.0.0
    usage: argus [options] [-i interface] [filter-expression]
    usage: argus [options] -r packetfile [filter-expression]

    options: -A                   Generate application byte metrics.
             -b                   dump filter compiler output.
             -B              specify bind interface address.
             -c  <dir>            daemon chroot directory.
             -d                   run Argus in daemon mode.
             -e             specify Argus Identifier  .
             -h                   print help.
             -F          read configuration from  .
             -J                   generate packet performance data.
             -M              set MAR Status Report Time Interval (300s).
             -m                   turn on MAC Layer Reporting.
             -O                   turn off filter optimizer.
             -p                   don't go into promiscuous mode.
             -P           enable remote access on   (561).
             -r     use packet file as data source.
             -R                   generate response time data.
             -S              set FAR Status Report Time Interval (60s).
             -t                   indicate that packetfile is MOAT Tsh format. 
             -u            specify user id for daemon.
             -g           specify group id for daemon.
             -U             specify the number of user bytes to capture.
             -w   write output to  , or '-', for stdout,
                                  against optional filter expression.
             -X                   reset argus configuration.
             -Z                   generate packet size data.

    Note that the help page is missing the -i option to specify a sniffing interface.

    Now I start the Argus server.

    hacom:/root# /usr/local/argus-3.0.0/sbin/argus -A -B -P 651 -R -u sguil -g sguil -U 256 -w /nsm/argus/argus3.test.arg -Z -i bridge0 - ip
    ArgusWarning: argus[42020]: 02 Mar 08 22:10:18.647475 started
    ArgusWarning: argus[42020]: 02 Mar 08 22:10:18.648282 ArgusGetInterfaceStatus: interface bridge0 is up

    I tell Argus to generate application byte metrics, bind its server to IP and port 651 TCP, to generate response time data, to run as user sguil group sguil, to capture 256 bytes of user data, where to write its output, to generate packet size data, to sniff on interface bridge0 and to watch only IP traffic.

    The sockstat command shows Argus has bound a server to port 651 TCP. We'll see what that offers soon.

    hacom:/root# sockstat -4 | grep argus
    sguil    argus      41996 4  tcp4         *:*
    sguil    argus      41996 6  udp4   *:*                   *:*

    As Argus collects data it appears in the specified directory:

    hacom:/root# ls -lh /nsm/argus/
    total 3586
    drwxr-xr-x  2 root   sguil   512B Mar  2 22:04 .old
    -rw-r--r--  1 sguil  sguil   3.5M Mar  2 22:04 argus3.test.arg

    With Argus saving data, I need to use one of its client programs to read it.

    Using the Argus ra client

    One of the fundamental client programs packaged with Argus is ra. The help options in the 3.0 ra client are extensive:

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./ra -h
    Ra Version 3.0.0.rc.69
    usage: ra
    usage: ra [options] -S remoteServer [- filter-expression]
    usage: ra [options] -r argusDataFile [- filter-expression]

    options: -A                    print record summaries on termination.
             -b                    dump packet-matching code.
             -c               specify a delimiter   for output columns.
             -C <[host]:port>      specify Cisco Netflow source.
             -e             convert user data using   method.
                                   Supported types are   and  .
             -E               write records that are rejected by the filter
             -F           read configuration from  .
             -h                    print help.
             -M rmon               convert bi-directional flow data to RMON in/out stats
                poll               attach to remote server to get MAR and then disconnect
             -n                    don't convert numbers to names.
             -p             print fractional time with   precision.
             -q                    quiet mode. don't print record outputs.
             -r               read argus data  . '-' denotes stdin.
             -R &ltdir>              recursively process files in directory
             -s [-][+[#]]field[:w] specify fields to print.
                       fields:     srcid, stime, ltime, sstime, dstime, sltime, dltime,
                                   trans, seq, flgs, dur, avgdur, stddev, mindur, maxdur,
                                   saddr, daddr, proto, sport, dport, stos, dtos, sdsb, ddsb
                                   sco, dco, sttl, dttl, sipid, dipid, smpls, dmpls, svlan, dvlan
                                   svid, dvid, svpri, dvpri, [s|d]pkts, [s|d]bytes,
                                   [s||d]appbytes, [s|d]load, [s|d]loss, [s|d]ploss, [s|d]rate,
                                   smac, dmac, dir, [s|d]intpkt, [s|d]jit, state, suser, duser,
                                   swin, dwin, trans, srng, erng, stcpb, dtcpb, tcprtt, inode,
                                   offset, smaxsz, dmaxsz, sminsz, dminsz
             -S        specify remote argus and optional port number
             -t          specify   for reading records.
                       format:     timeSpecification[-timeSpecification]
                                   timeSpecification: [[[yyyy/]mm/]dd.]hh[:mm[:ss]]
             -T               attach to remote server for T seconds.
             -u                    print time in Unix time format.
             -w               write output to  . '-' denotes stdout.
             -X                    don't read default rarc file.
             -z                    print Argus TCP state changes.
             -Z              print actual TCP flag values.
                                   <'s'rc | 'd'st | 'b'oth>

    The following is one way to invoke ra. I tell ra to show time in Unix format, with TCP flags, column headers, no resolution of IP addresses, while reading the specified file. The format of the remainder of the output is the same as demonstrated earlier. I conclude with a filter "- tcp" to only display TCP records, and I use the head command to limit the output to 10 lines.

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./ra -u -Z b -L0 -n -r /nsm/argus/argus3.test.arg -s stime proto saddr sport daddr dport spkts sappbytes dpkts dappbytes state - tcp | head
        StartTime  Proto     SrcAddr  Sport     DstAddr  Dport  SrcPkts    SAppBytes  DstPkts    DAppBytes State 
     1204513818.770060    tcp     1  0   1            0   A_A
     1204513828.429709    tcp    1   0   1   0  FA_A
     1204513828.858757    tcp     1   0   1            0   A_A
     1204513830.041338    tcp     15   421  17    3974 FSPA_

    This sort of raw output is helpful if you want to inspect specific records. For a more top-down approach I turn to racluster. Note the first and third records describe the same session. This must be a persistent connection. We can use the racluster client to aggregate these, assuming Argus has seen most or all of the session from the beginning.

    Using the Argus racluster client

    The Argus client racluster is useful for getting a higher-level view of session data. The client merges records from the same flows. First I show the help options for racluster:

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -h
    Racluster Version 3.0.0.rc.69
    usage:  racluster [-f racluster.conf]
    usage:  racluster [-f racluster.conf] [ra-options] [- filter-expression]

    options:  -f        read aggregation rules from  .
              -m flow key fields       specify fields to be used as flow keys.
              -M modes                 modify mode of operation.
                 Available modes:      
                    ind                aggregate multiple files independently
                    norep              do not report aggregation statistics
                    rmon               convert bi-directional data into rmon in/out data
                    replace            replace input files with aggregation output
              -V                       verbose mode.

    In the following example I tell racluster to collect all records involving host and present the first four results:

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -L0 -M norep -u -n -r /nsm/argus/argus3.test.arg - host and tcp | head -n 5
     StartTime   Flgs  Proto  SrcAddr  Sport   Dir  DstAddr  Dport  TotPkts   TotBytes State 
     1204513828.429709  e    tcp  6   421   RST
     1204513862.958283  e    tcp  ->  25  7741   RST
     1204513878.329589  e    tcp  ->  23  5684   RST
     1204514058.412265  e    tcp  ->  23  5684   RST

    The following shows aggregation based on unique source and destination IP address pairs:

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -L0 -M norep -m saddr daddr -u -n -r /nsm/argus/argus3.test.arg - tcp | head -n 5
             StartTime    Flgs  Proto       SrcAddr  Sport   Dir       DstAddr  Dport  TotPkts   TotBytes State 
     1204516393.988996  e     ip     <->          4        276   CON
     1204513828.429709  e     ip     <->       1513     377809   CON
     1204513831.418396  e     ip     <->       1679     446483   CON
     1204516648.005118  e     ip     <->          8        504   CON

    To change the order of the output, call rasort. Although I won't show the output here, you could use the following command to sort the racluster output by bytes.

    hacom:/usr/local/argus-clients-3.0.0.rc.69/bin# ./racluster -M norep -m saddr daddr –u
    -n -r /nsm/argus/argus3.test.arg -w - - tcp | ./rasort -m bytes -L0 | head -5

    This article has only scratched the surface of Argus 3.0. If you're interested in using Argus for the collection and analysis of network session data, I recommend starting to record some data using the Argus server. Once you have data to inspect you can begin trying the client programs. Expect to hear more about this in future Snort Reports.

    About the author
    Richard Bejtlich is the founder of TaoSecurity, author of several books on network security monitoring, including
    Extrusion Detection: Security Monitoring for Internal Intrusions , and operator of the TaoSecurity blog.

    Dig Deeper on MSP technology services

    Cloud Computing
    Data Management
    Business Analytics