Host-based IDS with Snort, Barnyard2 and Snorby in AWS

13 minute read , Oct 30, 2014

Snort is open source network-based intrusion detection system (NIDS) that has the ability to perform real-time traffic analysis and packet logging on Internet Protocol (IP) networks. Snort performs protocol analysis, content searching, and content matching.

Snort can be configured in three main modes: sniffer, packet logger, and network intrusion detection. In sniffer mode, the program will read network packets and display them on the console. In packet logger mode, the program will log packets to the disk. In intrusion detection mode, the program will monitor network traffic and analyze it against a rule set defined by the user. The program will then perform a specific action based on what has been identified.

On each host we are going to use Snort in network intrusion detection mode. To speed up its traffic processing it will log in binary mode. That’s where Barnyard2 comes into play.

Barnyard2 is an open source interpreter for Snort unified2 binary output files. Its primary use is allowing Snort to write to disk in an efficient manner and leaving the task of parsing binary data into various formats to a separate process that will not cause Snort to miss network traffic. It can also operate in three modes: Batch, Continual and Continual with bookmarking.

In batch (or one-shot) mode, barnyard2 will process the explicitly specified file(s) and exit. In continual mode, barnyard2 will start with a location to look and a specified file pattern and continue to process new data (and new spool files) as they appear. Continual mode with bookmarking will also use a checkpoint file (or waldo file in the snort world) to track where it is. In the event the barnyard2 process ends while a waldo file is in use, barnyard2 will resume processing at the last entry as listed in the waldo file.

Barnyard2 processing is controlled by two main types of directives: input processors and output plugins. The input processors read information in from a specific format (currently the spo_unified2 output module of Snort) and output them in one of several ways.

Snorby is an open source Ruby on Rails web application that interacts with the data populated by Snort and Barnyard in a database and presents them in a really beautiful and easy to manage user interface.

Installation and Setup

Server side

This is going to be the host where the central snort database will reside. It will be a Ubuntu host launched from AWS Ubuntu-14.04 LTS x86-64 AMI. Being the latest stable Ubuntu LTS release it should have most recent packages so we can try to reduce compiling from sources to minimum.


We start with installing some dependencies:

root@server:~$ tasksel install lamp-server
root@server:~$ aptitude install libwww-perl libnet1 libnet1-dev libpcre3 libpcre3-dev autoconf libcrypt-ssleay-perl libtool libssl-dev build-essential automake gcc make flex bison
root@server:~$ aptitude install libdnet libdaq-dev libpcap-dev

After that we can install snort which is on version in the 14.04 repositories and not much behind Snort’s master which is on atm:

root@server:~# aptitude install snort

Then we can easily configure the package Debian way /etc/snort/snort.debian.conf:


To install the Snort Rules I used the OinkMaster script. But first we need to register on the Snort site and get so called oink code which I can use to download the latest rules for my snort major version (mine is, the source is at

root@server:/tmp# wget -O snortrules-snapshot-2962.tar.gz<MY-OINK-CODE>
root@server:/tmp# cd oinkmaster-2.0/
root@server:/tmp/oinkmaster-2.0# cp /usr/local/bin/
root@server:/tmp/oinkmaster-2.0# cp oinkmaster.conf /usr/local/etc/
root@server:/tmp/oinkmaster-2.0# -b /var/tmp -o /etc/snort/rules -C /usr/local/etc/oinkmaster.conf -u file:///tmp/snortrules-snapshot-2962.tar.gz

We will also use the OinkMaster source scripts to create the file for Snort:

root@server:/tmp/oinkmaster-2.0# ./contrib/ /etc/snort/rules > /etc/snort/


First we need to create a MySQL database we gonna use for Barnyard:

root@server:~# mysql -u root -p -e 'create database snorby'

Next we move to installing and configuring Barnyard from source:

root@server:~# aptitude install libmysqld-dev libpcap-dev libprelude-dev git
root@server:~# git clone barnyard
root@server:~/barnyard# cd barnyard/
root@server:~/barnyard# autoreconf -fvi -I ./m4
root@server:~/barnyard# ./configure --with-mysql --with-mysql-libraries=/usr/lib/x86_64-linux-gnu
root@server:~/barnyard# make
root@server:~/barnyard# make install
root@server:~/barnyard# cp etc/barnyard2.conf /etc/snort/
root@server:~/barnyard# mkdir /var/log/barnyard2
root@server:~/barnyard# chmod 666 /var/log/barnyard2
root@server:~/barnyard# touch /var/log/snort/barnyard2.waldo
root@server:~/barnyard# chown snort:snort /var/log/snort/barnyard2.waldo
root@server:~/barnyard# mysql -u root -p -D snorby < ./schemas/create_mysql

Now Barnyard2 should be installed and configured and the database tables populated. We can now create a user we gonna use to access and write the snort records in the database:

root@server:~# mysql -u root -p
mysql> GRANT ALL ON snorby.* TO 'snorby'@'localhost' IDENTIFIED BY '<snort-password>';
Query OK, 0 rows affected (0.00 sec)
mysq> quit

Next we will update the Barnyard config file as follows /etc/snort/barnyard2.conf:

config reference_file:      /etc/snort/reference.config
config classification_file: /etc/snort/classification.config
config gen_file:            /etc/snort/
config sid_file:            /etc/snort/
config logdir: /var/log/snort
config archivedir: /var/log/snort/archive
config hostname:   localhost
config interface:  eth0
config daemon
config waldo_file: /var/log/snort/barnyard2.waldo
input unified2
output alert_fast: stdout
output database: log, mysql, user=snorby password=<snort-password> dbname=snorby host=localhost

This tells Barnyard how to connect to the database and were to find the needed files. At the end we create the archive directory where Barnyard will store the processed files:

root@server:~/barnyard# mkdir /var/log/snort/archive
root@server:~/barnyard# chown snort /var/log/snort/archive/


We install some dependencies and build the package from source:

root@server:~# aptitude install imagemagick wkhtmltopdf ruby1.9.3 libyaml-dev libxml2-dev libxslt1-dev zlib1g-dev build-essential openssl libssl-dev libmysqlclient-dev libreadline6-dev
root@server:~# cd /usr/local/src/
root@server:~/usr/local/src# git clone
root@server:~/usr/local/src# cd snorby
root@server:~/usr/local/src/snorby# aptitude install bundler
root@server:~/usr/local/src/snorby# bundle install

We have already setup the database in the previous steps so we can move to setting up the configuration files:

root@server:~/usr/local/src/snorby# cp config/database.yml.example config/database.yml
root@server:~/usr/local/src/snorby# vi config/database.yml
# Snorby Database Configuration
# Please set your database password/user below
# NOTE: Indentation is important.
snorby: &snorby
  adapter: mysql
  username: snorby
  password: "<snort-password>"
  host: localhost
  database: snorby
  <<: *snorby
  database: snorby
  <<: *snorby
  database: snorby
  <<: *snorby
root@server:~/usr/local/src/snorby# cp config/snorby_config.yml.example config/snorby_config.yml
root@server:~/usr/local/src/snorby# vi config/snorby_config.yml
  # in case you want to run snorby under a suburi/suburl under eg. passenger:
  baseuri: ''
  # baseuri: '/snorby'
  domain: ''
  wkhtmltopdf: /usr/bin/wkhtmltopdf
  ssl: false
  mailer_sender: ''
  geoip_uri: ""
    - ""
  authentication_mode: database
  # If timezone_search is undefined or false, searching based on time will
  # use UTC times (historical behavior). If timezone_search is true
  # searching will use local time.
  timezone_search: true
  # uncomment to set time zone to time zone of box from /usr/share/zoneinfo, e.g. "America/Cancun"
  # time_zone: 'UTC'
  time_zone: Australia/Sydney

To finish off the installation we run:

root@server:~/usr/local/src/snorby# bundle exec rake snorby:setup

And finally we can start the service:

root@server:~/usr/local/src/snorby# bundle exec rails server -e production -d
=> Booting WEBrick
=> Rails 3.1.12 application starting in production on
=> Call with -d to detach
=> Ctrl-C to shutdown server
[2014-07-22 15:34:41] INFO  WEBrick 1.3.1
[2014-07-22 15:34:41] INFO  ruby 1.9.3 (2013-11-22) [x86_64-linux]
[2014-07-22 15:34:41] INFO  WEBrick::HTTPServer#start: pid=21888 port=3000

Just a quick note here on a error I faced. In case we get error message in the console “The Snorby worker is not currently running!” we need to restart it:

root@server:~# rails c production
Loading production environment (Rails 3.1.0)
irb(main):001:0> Snorby::Worker.stop
=> ""
irb(main):002:0> Snorby::Jobs.clear_cache
=> nil
irb(main):003:0> Snorby::Worker.start
=> ""
irb(main):004:0> exit


We want to setup Apache SSL proxy as front-end for the Snorby dashboard. We install the needed packages and enable the modules needed by apache proxy:

root@server:~# aptitude install libapache2-mod-evasive libapache2-mod-spamhaus libapache2-mod-proxy-html apache2-mpm-worker libapache2-mod-security2 libapache2-mod-auth-mysql libapache2-mod-auth-radius libapache2-mod-authn-sasl
root@server:~# a2enmod rewrite
root@server:~# a2enmod ssl
root@server:~# a2enmod auth_digest
root@server:~# a2enmod proxy_html
root@server:~# a2enmod xml2enc
root@server:~# a2enmod proxy_connect
root@server:~# a2enmod proxy_http
root@server:~# a2enmod authnz_ldap

Now to configure the proxy in /etc/apache2/sites-available/snorby.conf file:

<VirtualHost *:80>
  DocumentRoot /var/www/html/
  ErrorLog /var/log/apache2/snorby_error.log
  CustomLog /var/log/apache2/snorby_access.log combined
  RewriteEngine On
  RewriteCond %{HTTPS} !=on
  RewriteRule ^(.*)$$1
  <Directory /var/www/html/>
    Options None
    Order allow,deny
    allow from all
<VirtualHost *:443>
  DocumentRoot /var/www/html/
  ErrorLog /var/log/apache2/snorby_error.log
  CustomLog /var/log/apache2/snorby_access.log combined
  LogLevel info
  SSLEngine on
  <FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
  <Directory /var/www/html/>
      Options None
      Order allow,deny
      allow from all
      AuthType Digest
      AuthName "Secure"
      AuthUserFile "/etc/apache2/user.passwd"
      AuthDigestProvider file
      Require valid-user
  # Proxy
  #ProxyHTMLLogVerbose on
  SSLProxyEngine On
  ProxyRequests Off
  ProxyPreserveHost On
  ProxyTimeout 60
  #ProxyHTMLExtended On
  ProxyPass /
  ProxyPassReverse /
  ProxyHTMLURLMap / /
  SetEnvIf User-Agent ".*MSIE [2-6].*" \
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0
  SetEnvIf User-Agent ".*MSIE [7-9].*" \

Enable the configuration and restart apache:

root@server:~# a2enconf encompass-security.conf
root@server:~# a2enconf security
root@server:~# a2ensite snorby
root@server:~# a2dissite 000-default
root@server:~# service apache2 restart

Now when we navigate to we will be met with the Snorby Dashboard.


Install some dependencies first:

# aptitude install libpcre3 libpcre3-dev autoconf libcrypt-ssleay-perl libtool libssl-dev build-essential automake gcc make flex bison libdnet-dev libpcap-dev nbtscan g++ libpcap-ruby zlib1g-dev libmysqld-dev libdnet libdnet-dev libpcre3 libpcre3-dev gcc byacc bison linux-headers-generic libxml2-dev libdumbnet-dev zlib1g zlib1g-dev


For Ubuntu-12.04:

# cd /usr/local/src
# wget
# tar -xzvf daq-2.0.2.tar.gz
# cd daq-2.0.2
# ./configure && make && make install
# snort --daq-list
Available DAQ modules:
pcap(v3): readback live multi unpriv
ipfw(v3): live inline multi unpriv
dump(v2): readback live inline multi unpriv
afpacket(v5): live inline multi unpriv

For CentOS/Redhat:

# yum remove libpcap libpcap-devel
# wget
# tar -xzvf libpcap-1.1.1.tar.gz
# cd libpcap-1.1.1
# ./configure --prefix=/usr
# make && make install
# ldconfig -v
# ldconfig -p | grep pcap (libc6,x86-64) => /usr/lib/ (libc6,x86-64) => /usr/lib/
# cd /usr/local/src
# wget
# tar -xzvf daq-2.0.2.tar.gz
# cd daq-2.0.2
# ./configure --with-libpcap-libraries=/usr/lib/
# make && make install
# ldconfig -v
# cd /usr/local/src
# wget
# tar -xzvf pcre-8.35.tar.gz
# cd pcre-8.35
# ./configure --enable-utf8
# make && make install
# ldconfig -v


We start with creating the snort system user:

# mkdir /var/log/snort
# groupadd -r --gid 513 snort
# useradd -r -c " Snort IDS" -s /bin/false -d /var/log/snort -G adm -g snort -u 513 snort
# chown snort:adm /var/log/snort
# chmod 750 /var/log/snort
# chmod g+s /var/log/snort

Then we fetch the snort source:

# cd /usr/local/src
# wget --no-check-certificate -O snort-
# tar -xzvf snort-
# cd snort-

For Ubuntu-12.04:

# ./configure --prefix /usr/local/snort --enable-sourcefire
# make && make install

For CentOS/RedHat:

# yum install libdnet-devel
# ./configure --prefix /usr/local/snort --enable-sourcefire --with-libpcap-libraries=/usr/lib --with-daq-libraries=/usr/local/lib/daq --enable-zlib --enable-gre --enable-mpls --enable-targetbased --enable-ppm --enable-perfprofiling
# make && make install

Next steps are distribution independent. First we create the needed system links to the path were we installed and couple of rules directories:

# ln -s /usr/local/snort/bin/snort /usr/sbin/snort
# ln -s /usr/local/snort/bin/u2spewfoo /usr/sbin/u2spewfoo
# ln -s /usr/local/snort/bin/u2boat /usr/sbin/u2boat
# mkdir /usr/local/snort/lib/snort_dynamicrules
# ln -s /usr/local/snort/lib/snort_dynamicpreprocessor /usr/local/lib/snort_dynamicpreprocessor
# ln -s /usr/local/snort/lib/snort_dynamicengine /usr/local/lib/snort_dynamicengine
# ln -s /usr/local/snort/lib/snort_dynamicrules /usr/local/lib/snort_dynamicrules
# ldconfig -v

To finish the setup we need to download the Snort rules for our particular version and deploy them:

# cd /usr/local/src/snort-
# wget --no-check-certificate -O snortrules-snapshot-2962.tar.gz<MY-OINK-CODE>
# tar -xzvf snortrules-snapshot-2962.tar.gz
# cp -R etc/ /usr/local/snort/
# rm -f /usr/local/snort/etc/Makefile*
# ln -s /usr/local/snort/etc /etc/snort
# cp -R rules/ /usr/local/snort/
# cp -R preproc_rules/ /usr/local/snort/
# rm -f /usr/local/snort/preproc_rules/Makefile*
# cp so_rules/precompiled/Ubuntu-12-04/x86-64/*.so /usr/local/snort/lib/snort_dynamicrules/
# mkdir /usr/local/snort/so_rules
# ldconfig -v
# snort -c /usr/local/snort/etc/snort.conf --dump-dynamic-rules=/usr/local/snort/so_rules
# touch /usr/local/snort/rules/white_list.rules /usr/local/snort/rules/black_list.rules

On CentOS/RedHat the only difference would be the source directory of the pre-compiled dynamic rules. So we replace Ubuntu-12-04 in the above line:

cp so_rules/precompiled/Ubuntu-12-04/x86-64/*.so /usr/local/snort/lib/snort_dynamicrules/

with Centos-5-4 or RHEL-5-5. The above section of commands crates the snort so_rules as well and we also create couple of list files.

At last we configure the local network Snort should sniff the traffic for, the format of its the output file (unified2 so it is ready for Barnyard processing) and enable the dynamic rule sets we just installed /etc/snort/snort.conf:

ipvar HOME_NET <my-subnet>
# unified2
output unified2: filename snort.u2, limit 128
output log_unified2: filename snort.u2, limit 128
# dynamic library rules
 include $SO_RULE_PATH/bad-traffic.rules
 include $SO_RULE_PATH/browser-ie.rules
 include $SO_RULE_PATH/chat.rules
 include $SO_RULE_PATH/dos.rules
 include $SO_RULE_PATH/exploit.rules
 include $SO_RULE_PATH/file-flash.rules
 include $SO_RULE_PATH/icmp.rules
 include $SO_RULE_PATH/imap.rules
 include $SO_RULE_PATH/misc.rules
 include $SO_RULE_PATH/multimedia.rules
 include $SO_RULE_PATH/netbios.rules
 include $SO_RULE_PATH/nntp.rules
 include $SO_RULE_PATH/p2p.rules
 include $SO_RULE_PATH/smtp.rules
 include $SO_RULE_PATH/snmp.rules
 include $SO_RULE_PATH/specific-threats.rules
 include $SO_RULE_PATH/web-activex.rules
 include $SO_RULE_PATH/web-client.rules
 include $SO_RULE_PATH/web-iis.rules
 include $SO_RULE_PATH/web-misc.rules


We need to constantly update our Snort rules from the public repository on the clients and the server in order to stay up-to-date with the recent threats. We can use the Pulledpork script for this purpose.

root@server:~# mkdir -p /usr/local/pulledpork
root@server:~# cd /usr/local/pulledpork/
root@server:/usr/local/pulledpork# wget
root@server:/usr/local/pulledpork# chmod u+x

Then we need to configure it /usr/local/pulledpork/pulledpork.conf:


And to update the rules we just need to run it:

root@server:/usr/local/pulledpork# ./ -k -K /usr/local/snort/rules -c pulledpork.conf -o /usr/local/snort/rules -s /usr/local/snort/so_rules

After that we need to restart the snort service so the new rules get loaded. To make the updates run automatically we add this to the root crontab:

# Pulledpork - Update Snort rules
00 01 1 * * /usr/local/pulledpork/ -k -K /usr/local/snort/rules -c /usr/local/pulledpork/pulledpork.conf -o /usr/local/snort/rules -s /usr/local/snort/so_rules

so it runs every first day of the week at 1am. The same setup goes into each client too.


Same as for the server configuration we clone the current master from Git and build from source:

# cd /usr/local/src
# aptitude install git
# git clone barnyard
# cd barnyard/

For Ubuntu-12.04:

# autoreconf -fvi -I ./m4
# ./configure --with-mysql --with-mysql-libraries=/usr/lib/x86_64-linux-gnu
# make && make install

For CentOS/RedHat:

# ./
# ./configure --with-mysql --with-mysql-libraries=/usr/lib64/mysql/
# make && make install

Now we need to configure Branyard so it can find the Snort output files and connect to the MySQL database we setup on the server /usr/local/etc/barnyard2.conf:

config reference_file:      /etc/snort/reference.config
config classification_file: /etc/snort/classification.config
config gen_file:            /etc/snort/
config sid_file:            /etc/snort/
config logdir: /var/log/snort
config waldo_file: /var/log/snort/barnyard2.waldo
config archivedir: /var/log/snort/archive
input unified2
config hostname:   client
config interface:  eth0
config process_new_records_only
#config event_cache_size: 32768
#config dump_payload
output alert_fast: /var/log/snort/barnyard_alert
output database: log, mysql, user=snorby password=<snort-password> dbname=snorby

This file will be practically same on all of the clients apart from the different host name in the config hostname: line.

Before we start Barnyard we need to open TCP port 3306 in the server’s Security Group for the client’s Security Group and grant access to the main database from this host(s):

mysql> GRANT ALL ON snorby.* TO 'snorby'@'' IDENTIFIED BY '<snort-password>';

and test the connection:

# mysql -h -u snorby -p<snort-password>

If all good we move to the next step to start the daemons. We need to repeat this for each client we want to monitor.

After all is in place we can start the processes:

# /usr/sbin/snort -u snort -g snort -dev -l /var/log/snort -c /etc/snort/snort.conf -D
# /usr/local/bin/barnyard2 -c /usr/local/etc/barnyard2.conf -d /var/log/snort -f snort.u2 -w /var/log/snort/barnyard2.waldo -l /var/log/snort -a /var/log/snort/archive -D

Converting the compiled packages into services

Of course manually starting and killing the services is not really good especially with high number of servers. Hence I have set service scripts and config files on each host.


For Ubuntu-12-04 hosts download the file and place it as /etc/init.d/snort. Make it executable:

root@server:~# chmod +x /etc/init.d/snort

and copy the example config file from snort source on each host:

root@server:~# cp rpm/snort.sysconfig /etc/default/snort

then set the values as needed, in our case /etc/default/snort:


Then set the default run levels:

root@server:~# update-rc.d snort defaults

For rpm distros like CentOS/RedHat download the file and place it as /etc/init.d/snort. Make it executable:

root@content~# chmod +x /etc/init.d/snort

and copy the example config file from snort source on each host:

root@content:~# cp rpm/snort.sysconfig /etc/sysconfig/snort

then set the values as needed, in our case /etc/sysconfig/snort:

# /etc/sysconfig/snort
# $Id: snort.sysconfig,v 1.8 2003/09/19 05:18:12 dwittenb Exp $
#### General Configuration
#### Logging & Alerting

Then set the default run levels:

root@content~# chkconfig --add snort


Ubuntu-12.04 files: /etc/default/barnyard2

# Config file for /etc/init.d/barnyard2

CentOS/RedHat files: /etc/sysconfig/barnyard2

# Config file for /etc/init.d/barnyard2

Leave a Comment