Monday, September 19, 2011

Sysadmin guide to using git and etckeeper for tracking configuration changes

When administering any Linux or Unix system over a period time, the problem of tracking what changes occurred to which files quickly arises. This becomes increasingly difficult when there are multiple system administrators configuring a single server. How do you easily track who made what changes, and when? This is where git comes in handy.

Git is a distributed revision version control system, which enables you to track changes to a file(s) by using a repository, also known as an index. Changes are tracked each commit. A commit can be thought of as a snapshot of a file's current state, similar to virtual machine snapshots in a virtual environment.

The index, which is basically the main database, can be initiated in any directory. However, wherever the Git index is initiated, it will only track files that are a child of that directory. For example, if we were to track /home/localadmin, it would track all files and sub-directories within localadmin's home directory, but nothing above it in the directory tree. A system can have many indexes initiated in different directories.

On Linux and Unix servers, changes primarily occur within the /etc directory. This is where most of the important .conf files are found. When a new package is installed, existing files may change and several new ones are added to /etc. All of these files and changes will eventually need to be automatically committed and added to the git index. This is where etckeeper comes in.

Etckeeper works with git, allowing automatic commits to the /etc directory after each package installation (shown and explained later.) It also tracks file metadeta that revision control systems do not normally support, but are important for /etc, such as permissions of /etc/shadow.

You may have heard of git heavily in the software development community, but I try to show how it can also be leveraged as a powerful tool for system administrators.

NOTE: This tutorial demonstrates instructions based on an Ubuntu 10.04 LTS 32-bit server installation. However, I have successfully configured the same setup on Ubuntu 11.04, Debian 5, Debian 6, CentOS 5.6, CentOS 6.0, and Red Hat Enterprise Linux 6.0.

Installing git and etckeeper

Install the git and etckeeper packages:

$ sudo aptitude -yvVR install git-core etckeeper

  • -y assume yes when prompted.
  • -v verbose output.
  • -V show which versions of packages will be installed.
  • -R do not treat recommended packages as dependencies.

Etckeeper supports other software version control systems (VCS) besides git, double-check the configuration file to make sure it is using git.

$ sudo vi /etc/etckeeper/etckeeper.conf
# The VCS to use.
# VCS="hg"
VCS="git"
# VCS="bzr"
# VCS="darcs"

Now initialize the /etc directory:

$ cd /etc
$ sudo etckeeper init

Now make your first commit:

$ sudo git commit

You will be brought to a screen where you can create a commit message along with the commit itself. Enter a brief description above the commented lines.

Initial commit for server build

* Initializing git repository in /etc

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Committer: localhost root
#
# On branch master
#
# Initial commit
#
# Changes to be committed:
# (use "git rm --cached …" to unstage)
#
# new file: .etckeeper
# new file: .gitignore
# new file: adduser.conf
# new file: aliases
# new file: alternatives/README
# new file: alternatives/awk
# new file: alternatives/awk.1.gz
# new file: alternatives/builtins.7.gz

In each commit message, git will display which files will be commited to the repository.

sudo is required for running the git commands because the repository was initialized in /etc by sudo; it is readable and writable only by root.

Any git command should be run after changing your working directory to /etc.

Making commits

After the first initialization, if a file is changed, git will notice the change.

$ sudo git status

  • Show status of current repository, changes, etc.

rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: issue
#
no changes added to commit (use "git add" and/or "git commit -a")

Above I added the line "Add new message of the day." to the /etc/issue file.

Using the following command, we can see what changes happened to /etc/issue since the previous commit:

$ sudo git diff

rsabalburo@localhost:/etc$ sudo git diff

diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

  • The diff output is identical to the regular diff command (using -u unified output switch).
  • The + shows the changes that were added since the previous commit.


Now that we are satisfied with the change we can git commit it:

$ sudo git commit -a

  • -a commit all changes.

Again, you will be prompted to create a description along with your commit.

Add a new message of the day for users logging in

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# Committer: localhost root
#
# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# modified: issue
#

If you made an error in your commit message or would simply like to change it, you can go back and edit it with git commit --amend:

$ sudo git commit --amend

  • This will only allow you to change the commit message of your most recent commit.

NOTE: You generally cannot amend any commit messages past the previous commit, doing so requires you to rewrite the entire git repository which is a very complicated process.

Important: If you have more than one modified file showing up in the "Changes to be committed:" staging area, you can commit each single file rather than all at once (-a), which is highly recommended.

$ sudo git commit

How do I display commits I've made?

To view a log of all the commits that you have made:

$ sudo git log

rsabalburo@localhost:/etc$ sudo git log

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

commit 30636b4bdfb84cdd4db756df60dcc2c49b53842b
Author: localhost root
Date: Thu Jun 17 11:43:47 2010 -1000

Initial commit for server build

* Initializing git repository in /etc

Each commit message has a unique SHA-1 hash, commit author, date of commit, and description of commit.

In the above git log, the author is currently localhost root. It is recommended that you change author and email address, especially if there are multiple administrators configuring the same server.

$ git config --global user.name "Rodolf Sabalburo"
$ git config --global user.email "rsabalburo@localhost"

Notice sudo is not needed for these commands. These commands will automatically create the .gitconfig file in the current users home directory:

hccitc@localhost:~$ cd ~/
hccitc@localhost:~$ cat .gitconfig

[core]
editor = vim
[user]
name = Rodolf Sabalburo
email = rsabalburo@localhost

From this point on, Rodolf Sabalburo will display in the commit log for any commit made by the user rsabalburo:

rsabalburo@localhost:/etc$ sudo git log

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

commit 30636b4bdfb84cdd4db756df60dcc2c49b53842b
Author: localhost root
Date: Thu Jun 17 11:43:47 2010 -1000

Initial commit for server build

* Initializing git repository in /etc

Important: Ensure that other privileged users set their git user.name and user.email, otherwise the commit author will show up as localhost root .

The git log command supports several additional switches, one of the most important being the -u switch which displays a diff unified output for each commit:

rsabalburo@localhost:/etc$ sudo git log

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

commit d23affc7b89da8966826ab0fef6891d87dbd3ee1
Author: localhost root
Date: Thu Jun 17 12:37:55 2010 -1000

Add a new message of the day for users logging in

diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

It also supports the --color switch, sudo git log -u --color which displays commit messages in yellow, removed text/lines in red, added text/lines in green.

To make the color behavior default use git config:

$ git config --global color.diff auto

Additional git log switches:

sudo git log --pretty=oneline
sudo git log --pretty=email
sudo git log --decorate

See 'man git-log' for more and the Additional git commands near the end of the tutorial for examples.

How do I compare different commits?

When viewing the different commits, you can compare changes with the git diff command:

$ sudo git diff -u

  • You do not have to type the entire hash, the first 4-6 hash charactes will suffice.

rsabalburo@localhost:/etc$ sudo git diff -u 30636b a208e373

diff --git a/hosts b/hosts
index 4eaa7c5..b5fe3df 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,7 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway
+192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
diff --git a/issue b/issue
index 9029f91..2100bfd 100644
--- a/issue
+++ b/issue
@@ -1,2 +1,4 @@
+A new message of the day.
+
Ubuntu 10.04 LTS n l

You can compare commits using git log as well.

By default when issuing git log under /etc, it will display ALL commits within the /etc git repository.

You can run git log on a single file (or directory) to see all changes made to it:

$ sudo git log

  • -u unified diff output

rsabalburo@localhost:/etc$ sudo git log -u hosts

commit a208e373fc8c5ddff86e752128bada592ccd2f60
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:26:24 2010 -1000

Add new host entry for 192.168.1.2 localproxy

* Make more changes to hosts to demonstrate git log on single file.

diff --git a/hosts b/hosts
index 12a111b..b5fe3df 100644
--- a/hosts
+++ b/hosts
@@ -1,6 +1,7 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
+192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback

Tracking new files

Git will track new files added to the repository (any file under /etc because we initiated the repository here) and display a notification in it's status output.

rsabalburo@localhost:/etc/shorewall$ sudo cp /usr/share/doc/shorewall/default-config/masq .

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq
nothing added to nothing added to commit but untracked files present (use "git add" to track)

Add the file or directory using git add:

$ sudo git add

rsabalburo@localhost:/etc/shorewall$ sudo git add masq

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# new file: masq
#

Git will now notify you that the masq file has changes to be commited. After committing the changes, git status will display that there are none.

rsabalburo@localhost:/etc/shorewall$ sudo git commit -a
[master 23d7db8] Copy masq file to shorewall/
1 files changed, 11 insertions(+), 0 deletions(-)
create mode 100644 shorewall/masq

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
nothing to commit (working directory clean)

How do I untrack files?

If there is a file that you do not want committed or tracked by git use the git rm --cached command along with the .gitignore file.

rsabalburo@localhost:/etc/shorewall$ sudo git rm --cached masq
rm 'shorewall/masq'

rsabalburo@localhost:/etc/shorewall$ sudo git status
# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# deleted: masq
#
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq

Notice that the masq file shows "deleted", this does not mean it has been deleted from the system only deleted from the git repository. The masq file still shows up under the Untracked files staging area.

rsabalburo@localhost:/etc/shorewall$ sudo git commit -a

[master bf06da0] Remove shorewall/masq from git repository
1 files changed, 0 insertions(+), 11 deletions(-)
delete mode 100644 shorewall/masq


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit bf06da0f52c16002dbf1a134ffc97b8f30d22673
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:56:04 2010 -1000

Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Untracked files:
# (use "git add …" to include in what will be committed)
#
# masq

To untrack masq completely, add the file to /etc/.gitignore. This file is responsible for ignoring any files or directories to be tracked from git.

hccitc@localhost:/etc$ sudo vi .gitignore
# Ignore masq file to demonstrate .gitignore
shorewall/masq


rsabalburo@localhost:/etc$ sudo git status
# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: .gitignore
#

Git also tracks the .gitignore file, commit the change describing which file was ignored in the commit message.

rsabalburo@localhost:/etc$ sudo git commit -a
[master 13957e9] Add shorewall/masq to .gitignore
1 files changed, 6 insertions(+), 0 deletions(-)

rsabalburo@localhost:/etc$ sudo git log
commit 13957e9dc87a5207a11081298f48ff37c1011b6b
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:01:12 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

commit bf06da0f52c16002dbf1a134ffc97b8f30d22673
Author: Rodolf Sabalburo
Date: Thu Jun 17 14:56:04 2010 -1000

Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.

How do I reset to an older commit?

In the instance you need to rollback to an earlier or snapshot, use git reset --hard.

$ sudo git reset --hard

WARNING!: Be very careful with this command! Not only does it abandon any uncommitted changes when rolling back to a previous commit i.e. any changes you have not committed will not be saved and wiped out, it also wipes out any changes in commits found after the one you are resetting to! ONLY perform a hard reset if you are absolutely sure you want to reset to an earlier commit.

For example if I had made changes to /etc/shorewall/params and /etc/shorewall/rules and wanted to roll back prior those changes:

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

rsabalburo@localhost:/etc/shorewall$ sudo git log --pretty=oneline

9f3a7708d600ed44913d1f7a6cacc0919825740f Add rule to allow ssh from net zone to $FW
d8c2337ed3127a9d091a3075857ae46b6d7284a2 Add iv-rover to trusted ssh jumpserver variable
8512a7dac4565bf6ec8fb459ca4faf0f08471429 Add shorewall/masq to .gitignore
bf06da0f52c16002dbf1a134ffc97b8f30d22673 Remove shorewall/masq from git repository

The commit I would want to reset to would be "8512a… Add shorewall/masq to .gitignore" using:

$ sudo git reset --hard 8512a7d

rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard 8512a7d

HEAD is now at 8512a7d Add shorewall/masq to .gitignore


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit 8512a7dac4565bf6ec8fb459ca4faf0f08471429
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:11:23 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

Undoing a reset

In the event you do a hard reset, it is still possible to recover by finding the commit using git reflog. reflog is a history log of all actions regarding commits in git.

$ sudo git reflog

rsabalburo@localhost:/etc/shorewall$ sudo git reflog

8512a7d HEAD@{0}: 8512a7d: updating HEAD
9f3a770 HEAD@{1}: commit: Add rule to allow ssh from net zone to $FW
d8c2337 HEAD@{2}: commit: Add iv-rover to trusted ssh jumpserver variable
8512a7d HEAD@{3}: commit: Add shorewall/masq to .gitignore

  • Notice "updating HEAD" in reference to commit "8512a7d Add shorewall/masq to .gitignore", which we previously reset to.
  • We want to roll back to 9f3a770 the most recent commit before our reset.

rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard 9f3a770

HEAD is now at 9f3a770 Add rule to allow ssh from net zone to $FW


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc/shorewall$ sudo git log

commit 9f3a7708d600ed44913d1f7a6cacc0919825740f
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:48 2010 -1000

Add rule to allow ssh from net zone to $FW

commit d8c2337ed3127a9d091a3075857ae46b6d7284a2
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:17 2010 -1000

Add iv-rover to trusted ssh jumpserver variable

commit 8512a7dac4565bf6ec8fb459ca4faf0f08471429
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:11:23 2010 -1000

Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

The git repository head is now at the commit before our initial reset.

How do I reset to the previous commit?

If you just want to reset or revert a just a SINGLE file to the previous commit use:

$ sudo git checkout

For example, lets say I add a PING rule to /etc/shorewall/rules and test it out. I did not commit the changes yet because they didn't work -- I can simply reset the single file to the most previous commit.

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: rules
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc/shorewall$ sudo git diff

diff --git a/shorewall/rules b/shorewall/rules
index 09dfd26..b47ea62 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -20,7 +20,8 @@ SSH(ACCEPT) net $FW

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

-Ping(DROP) net $FW
+#Ping(DROP) net $FW
+Ping(ACCEPT) net $FW

# Permit all ICMP traffic FROM the firewall TO the net zone


rsabalburo@localhost:/etc/shorewall$ sudo git checkout rules

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

If you just want to reset or revert ALL files (which were changed, but uncommited) to the previous commit use:

$ sudo git reset --hard

In this example, lets say I add a PING rule to /etc/shorewall/rules and change a policy in /etc/shorewall/policy. If I test out these changes and I'm not satisfied, I can reset everything to the most previous commit rather than having to use git checkouton each individual file.

rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: policy
# modified: rules
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc/shorewall$ sudo git diff

diff --git a/shorewall/policy b/shorewall/policy
index dd1d7a6..f43371a 100644
--- a/shorewall/policy
+++ b/shorewall/policy
@@ -15,8 +15,8 @@

# RS: If you want open access to the Internet from the firewall, change the
# $FW to net policy to 'ACCEPT' and remove 'info' under LOG LEVEL.
-#$FW net ACCEPT
-$FW net REJECT info
+$FW net ACCEPT
+#$FW net REJECT info
net all DROP info
# The FOLLOWING POLICY MUST BE LAST
all all REJECT info
diff --git a/shorewall/rules b/shorewall/rules
index 09dfd26..b47ea62 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -20,7 +20,8 @@ SSH(ACCEPT) net $FW

# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

-Ping(DROP) net $FW
+#Ping(DROP) net $FW
+Ping(ACCEPT) net $FW

# Permit all ICMP traffic FROM the firewall TO the net zone



rsabalburo@localhost:/etc/shorewall$ sudo git reset --hard

HEAD is now at 9f3a770 Add rule to allow ssh from net zone to $FW


rsabalburo@localhost:/etc/shorewall$ sudo git status

# On branch master
nothing to commit (working directory clean)

How do I reset a SINGLE file to a much older commit?

If you have a single file you want to revert to a much older commit e.g. maybe you want a revert a configuration file to it's original form in one of your first commits, use the git checkout command and specify the hash and filename as arguments.

sudo git checkout

  • Below I use sudo git checkout cae7 hosts, cae7 is the hash, hosts is the filename.

  • I found the correct hash I wanted to revert to by using sudo git log -u hosts.

rsabalburo@localhost:/etc$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc$ head hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
192.168.1.2 localproxy

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes


rsabalburo@localhost:/etc$ sudo git show cae7

commit cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b
Author: Rodolf Sabalburo
Date: Thu Jun 17 13:57:07 2010 -1000

Add hosts entry for 192.168.1.1

* Showing different author and email in commit message.

diff --git a/hosts b/hosts
index 4eaa7c5..12a111b 100644
--- a/hosts
+++ b/hosts
@@ -1,5 +1,6 @@
127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
+192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback


rsabalburo@localhost:/etc$ sudo git checkout cae7 hosts


rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changes to be committed:
# (use "git reset HEAD …" to unstage)
#
# modified: hosts
#


rsabalburo@localhost:/etc$ head hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Git shows in the staging area that hosts has been modified (in this case, to our previous commit cae7…). The hosts file would still need to be commited if we were satisfied with the changes. If not satisfied, we can reset to the previous commit using git reset --hard.

How do I modify my previous commit and include additional changes without using a reset?

For example if I added the '192.168.1.3 homecomputer' host entry to /etc/hosts and commit the changes:

rsabalburo@localhost:/etc$ head /etc/hosts

127.0.0.1 localhost
127.0.1.1 localhost.localdomain localhost
192.168.1.1 localgateway
192.168.1.2 localproxy
192.168.1.3 homecomputer

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix


rsabalburo@localhost:/etc$ sudo git log | head

commit c853bfcc0fd65e37be4959f208af8122e1390806
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

If I felt that I could have included another host entry to the same commit I could use git commit --amend on the file after I made the changes:

$ sudo git commit --amend

rsabalburo@localhost:/etc$ sudo vi hosts

(Here I added the homeserver entry)

rsabalburo@localhost:/etc$ sudo git diff

diff --git a/hosts b/hosts
index 77fdaee..0f0ac1f 100644
--- a/hosts
+++ b/hosts
@@ -3,6 +3,7 @@
192.168.1.1 localgateway
192.168.1.2 localproxy
192.168.1.3 homecomputer
+192.168.1.4 homeserver

# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback


rsabalburo@localhost:/etc$ sudo git status

# On branch master
# Changed but not updated:
# (use "git add …" to update what will be committed)
# (use "git checkout -- …" to discard changes in working directory)
#
# modified: hosts
#
no changes added to commit (use "git add" and/or "git commit -a")


rsabalburo@localhost:/etc$ sudo git log

commit c853bfcc0fd65e37be4959f208af8122e1390806
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4


rsabalburo@localhost:/etc$ sudo git commit --amend /etc/hosts

(update our new change, create a commit message)

[master bfdb9ff] Add new host entry for 192.168.1.3 homecomputer and 192.168.1.4 homeserver
2 files changed, 2 insertions(+), 2 deletions(-)


rsabalburo@localhost:/etc$ sudo git log

commit bfdb9ffc7956a6123d447f0996b9f23b6858abce
Author: Rodolf Sabalburo
Date: Thu Jun 24 12:08:04 2010 -1000

Add new host entry for 192.168.1.3 homecomputer and 192.168.1.4 homeserver

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

Notice the previous commit was overwritten after we used the git commit --amend command.

You can also include changes made to several other files if needed:

$ sudo git commit --amend

  • Redo previous commit, and include changes made to , , etc.

etckeeper

Etckeeper performs the simple role of automatically committing changes within /etc after package installations, shown below:

rsabalburo@localhost:/etc$ sudo git status

# On branch master
nothing to commit (working directory clean)


rsabalburo@localhost:/etc$ sudo aptitude -yvVR install openvpn

Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done
The following NEW packages will be installed:
liblzo2-2{a} [2.03-2] libpkcs11-helper1{a} [1.07-1build1] openssl-blacklist{a} [0.5-2] openvpn [2.1.0-1ubuntu1] openvpn-blacklist{a} [0.4]
The following packages are SUGGESTED but will NOT be installed:
resolvconf
0 packages upgraded, 5 newly installed, 0 to remove and 25 not upgraded.
Need to get 0B/7,937kB of archives. After unpacking 16.3MB will be used.
Writing extended state information… Done
Preconfiguring packages …
(Reading database … 44756 files and directories currently installed.)
Unpacking openssl-blacklist (from …/openssl-blacklist_0.5-2_all.deb) …
Unpacking liblzo2-2 (from …/liblzo2-2_2.03-2_i386.deb) …
Unpacking libpkcs11-helper1 (from …/libpkcs11-helper1_1.07-1build1_i386.deb) …
Unpacking openvpn-blacklist (from …/openvpn-blacklist_0.4_all.deb) …
Unpacking openvpn (from …/openvpn_2.1.0-1ubuntu1_i386.deb) …
Processing triggers for man-db …
Processing triggers for ureadahead …
Setting up openssl-blacklist (0.5-2) …
Setting up liblzo2-2 (2.03-2) …

Setting up libpkcs11-helper1 (1.07-1build1) …

Setting up openvpn-blacklist (0.4) …
Setting up openvpn (2.1.0-1ubuntu1) …
* Restarting virtual private network daemon(s)… * No VPN is running.

Processing triggers for libc-bin …
ldconfig deferred processing now taking place
[master ac1d59b] committing changes in /etc after apt run
Author: Rodolf Sabalburo
13 files changed, 412 insertions(+), 0 deletions(-)
create mode 100644 bash_completion.d/openvpn
create mode 100644 default/openvpn
create mode 100755 init.d/openvpn
create mode 100755 network/if-down.d/openvpn
create mode 100755 network/if-up.d/openvpn
create mode 100755 openvpn/update-resolv-conf
create mode 120000 rc0.d/K80openvpn
create mode 120000 rc1.d/K80openvpn
create mode 120000 rc2.d/S16openvpn
create mode 120000 rc3.d/S16openvpn
create mode 120000 rc4.d/S16openvpn
create mode 120000 rc5.d/S16openvpn
create mode 120000 rc6.d/K80openvpn
Reading package lists… Done
Building dependency tree
Reading state information… Done
Reading extended state information
Initializing package states… Done
Writing extended state information… Done

Current status: 0 broken [+0], 25 updates [+0], 29045 new [+0].


rsabalburo@localhost:/etc$ sudo git show

commit ac1d59be8222ff8f0d956434c2c2ba57e938047e
Author: Rodolf Sabalburo
Date: Wed Jun 23 13:55:15 2010 -1000

committing changes in /etc after apt run

Package changes:
+liblzo2-2 2.03-2
+libpkcs11-helper1 1.07-1build1
+openssl-blacklist 0.5-2
+openvpn 2.1.0-1ubuntu1
+openvpn-blacklist 0.4

  • Notice the "create mode"s on files that were added to the system.
  • Also an automatic commit message is generated show in the git log.
  • If you want to change this message and be more verbose on the packages installed you can use sudo git commit --amend.

Stanardized git commit message format

Here I have standardized my git commit messages to keep git logs uniform, tidy, and easy to read.

  • The first line should be a summary or short phrase of what change was made on which file.
  • Use present tense only, and omit period or other punctuation at end.
  • Reference the file from relative path, (everything is a child of /etc) e.g. shorewall/rules.
  • Second line should be blank.
  • Third line indented with three spaces, use an asterisk * as a bullet to describe in more detail about change.
  • Feel free to be as verbose as possible, and make notes for self and other administrators.
  • Use proper punctuation and grammar.

Example of a good commit message:

Add SNMP rule from net zone to firewall zone in shorewall/rules

* Allow snmp traffic to reach firewall.
* This will remain commented out for now as a hook, enable when needed.
* Note that Shorewall 4.0 to 4.4 have different "macro" formats.

Example of my git log from an OpenVPN server I manage:

commit 94591e98136adcbe10212e69b30964326044dad9
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:49:41 2010 -1000

Update shorewall/params add OpenVPN variables

* Add variable for NET_OPENVPN_TRUST and ROAD_OPENVPN_TRUST.
* See shorewall/params for description of variables.

commit b2bb776167537cb6cfcead81e1cca90766e5ae80
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:39:23 2010 -1000

Fix shorewall/policy

* Add explicit policy for net -> $FW drop, with logging.

commit b28940adfdfa4c2b5370831c990d84d1364c52c1
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:37:08 2010 -1000

Fix shorewall/interfaces

* Change eth0 interface to eth+.
- + denotes wildcard, or "all eth interfaces".

commit 974cd9f34c4f06ecabea414006453d610e2af0e3
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:27:16 2010 -1000

Deprecate shorewall/tunnels file

* tunnels to be deprecated, use rules-hcc.inc instead to allow related
inbound and outbound OpenVPN traffic.
* tunnels creates open-type policies in iptables ($FW -> net udp openvpn,
net -> $FW udp openvpn) that take higher priority in the net2fw chain.
* rules-hcc.inc will be used to accomplish close-type policies, only
allowing trusted hosts or networks from the net zone.
* See "Eliminating the /etc/shorewall/tunnels file"
http://www.shorewall.net/VPNBasics.html

commit 6cbab1296f351e069857a035df077fa55911a745
Author: Rodolf Sabalburo
Date: Tue Jun 22 05:19:07 2010 -1000

Build is now a release candidate: ubuntu1004-x86-hccvpn-rc1

* Hostname change using /usr/local/sbin/hcc-ubuntu-hostname-change.

Additional git commands

Change git commit editor

For some distributions the default editor for commit messages is nano or emacs, this can be changed with the following command to vim (or vi):

$ git config --global core.editor "vim"

  • On a per user basis like user.name and user.email options; sudo not needed.

Commit verbosely

You can commit verbosely i.e. include the diff of the contents being committed in the commit message screen with:

$ sudo git commit -a -v
$ sudo git commit -av

  • You can commit a single file if instead, replace -a with filename.

git blame

git blame will show what revision and author last modified each line of a file. Example:

$ sudo git blame

rsabalburo@localhost:/etc$ sudo git blame /etc/hosts

^30636b4 (localhost root 2010-06-17 11:43:47 -1000 1) 127.0.0.1 localhost
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 2) 127.0.1.1 localhost.localdomain localhost
cae791d4 (Rodolf Sabalburo 2010-06-17 13:57:07 -1000 3) 192.168.1.1 localgateway
a208e373 (Rodolf Sabalburo 2010-06-17 14:26:24 -1000 4) 192.168.1.2 localproxy
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 5)
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 6) # The following lines are desirable for IPv6 capable hosts
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 7) ::1 localhost ip6-localhost ip6-loopback
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 8) fe00::0 ip6-localnet
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 9) ff00::0 ip6-mcastprefix
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 10) ff02::1 ip6-allnodes
^30636b4 (localhost root 2010-06-17 11:43:47 -1000 11) ff02::2 ip6-allrouters

  • Notice lines 3 and 4 of /etc/hosts was edited by Rodolf Sabalburo, both on June 17.
  • The far left hand column shows the truncated commit hash.

More git log switches

$ sudo git log --pretty=oneline

ac1d59be8222ff8f0d956434c2c2ba57e938047e committing changes in /etc after apt run
9f3a7708d600ed44913d1f7a6cacc0919825740f Add rule to allow ssh from net zone to $FW
d8c2337ed3127a9d091a3075857ae46b6d7284a2 Add iv-rover to trusted ssh jumpserver variable
8512a7dac4565bf6ec8fb459ca4faf0f08471429 Add shorewall/masq to .gitignore
bf06da0f52c16002dbf1a134ffc97b8f30d22673 Remove shorewall/masq from git repository
23d7db8e818ff27c789b229f47d8614a99f6ce11 Copy masq file to shorewall/
a208e373fc8c5ddff86e752128bada592ccd2f60 Add new host entry for 192.168.1.2 localproxy
cae791d4ac3aeb206f108815f7a5d4fc0a3bd49b Add hosts entry for 192.168.1.1
d23affc7b89da8966826ab0fef6891d87dbd3ee1 Add a new message of the day for users logging in
30636b4bdfb84cdd4db756df60dcc2c49b53842b Initial commit for server build.

$ sudo git log --pretty=oneline --abbrev-commit

ac1d59b committing changes in /etc after apt run
9f3a770 Add rule to allow ssh from net zone to $FW
d8c2337 Add iv-rover to trusted ssh jumpserver variable
8512a7d Add shorewall/masq to .gitignore
bf06da0 Remove shorewall/masq from git repository
23d7db8 Copy masq file to shorewall/
a208e37 Add new host entry for 192.168.1.2 localproxy
cae791d Add hosts entry for 192.168.1.1
d23affc Add a new message of the day for users logging in
30636b4 Initial commit for server build.

$ sudo git log --pretty=email

From 9f3a7708d600ed44913d1f7a6cacc0919825740f Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:15:48 -1000
Subject: [PATCH] Add rule to allow ssh from net zone to $FW


From d8c2337ed3127a9d091a3075857ae46b6d7284a2 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:15:17 -1000
Subject: [PATCH] Add iv-rover to trusted ssh jumpserver variable


From 8512a7dac4565bf6ec8fb459ca4faf0f08471429 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 15:11:23 -1000
Subject: [PATCH] Add shorewall/masq to .gitignore

* This file will no longer be tracked or show up in untracked files
staging area.

From bf06da0f52c16002dbf1a134ffc97b8f30d22673 Mon Sep 17 00:00:00 2001
From: Rodolf Sabalburo
Date: Thu, 17 Jun 2010 14:56:04 -1000
Subject: [PATCH] Remove shorewall/masq from git repository

* masq itself is not deleted from the file system, only from the
repository.
* To remove from staging area, use .gitignore file.

  • See man git-log for additional switches.

Best practices and tips

  • Git commits should be atomic i.e. make a commit anytime you make a single change to a file or similar changes across a few files.
  • For example, if I had the same IP I had to change in multiple configuration files I can commit it all as one with a short description describing why the IP was changed.
  • If I changed /etc/fstab and /etc/hosts; they should be commited separately as they are entirely unrelated.
  • Always set the git author immediately using git config user.name and user.email.
  • Use git status and git diff often to see the progress of your work, and if there were any changes you did not commit yet.
  • Anytime you make a change, whether a commit, revert, reset; git will always update the timestamp on the related file!

Reading diff output

The git log command supports the diff -u (unified format) output, which is much easier to read, and allows the log to show differences made to each commit:

rsabalburo@localhost:/etc/shorewall$ sudo git log -u

commit 9f3a7708d600ed44913d1f7a6cacc0919825740f
Author: Rodolf Sabalburo
Date: Thu Jun 17 15:15:48 2010 -1000

Add rule to allow ssh from net zone to $FW

diff --git a/shorewall/rules b/shorewall/rules
index 0e33b66..09dfd26 100644
--- a/shorewall/rules
+++ b/shorewall/rules
@@ -14,6 +14,10 @@
#ACTION SOURCE DEST PROTO DEST SOURCE ORIGINAL RATE USER/ MARK
# PORT PORT(S) DEST LIMIT GROUP

+# Allow ssh connections to $FW
+
+SSH(ACCEPT) net $FW
+
# Drop Ping from the "bad" net zone.. and prevent your log from being flooded..

Ping(DROP) net $FW

Notice the --- a/shorewall/rules and the +++ b/shorewall/rules. These are the files that are being compared. In this case, the older rules file and the newer rules file.

  • Spaces are where the two files are the same. (no - or + )
  • - denotes changes in the previous file, but were usually removed in the newer file.
  • + denotes changes in newer file, usually additions.

Also notice the format @@ -14,6 +14,10 @@. These are called change hunks or chunks. The -14 denotes the first/older file line number, and the 6 is the line range. This means that at line 14 in the older file there were changes within 6 lines. The second part +14 denotes the newer file, and 10 is again the line range. In other words, line 14 onward for 10 lines there is a change.

By default the -u switch gives 3 lines of context, or 3 lines BEFORE and AFTER there was a change. This can be changed with the -UN format, where N is the number of lines before or after the changes that show. If you only want to see changes without surrounding context use:

$ sudo git log -U0

Resources

Very helpful Git command cheat sheet: http://cheat.errtheblog.com/s/git
Git on Wikipedia: http://en.wikipedia.org/wiki/Git_(software)
Git for the lazy: http://www.spheredev.org/wiki/Git_for_the_lazy
Starting git using just 10 commands: http://blog.xkoder.com/2008/08/13/git-tutorial-starting-with-git-using-just-10-commands/

If this is your first time reading this tutorial, I recommend reading it again and practicing it hands-on to fully understand it.

Thursday, September 15, 2011

Snort 2.9.1 on CentOS 6.0

This tutorial demonstrates a Snort 2.9.1 installation on CentOS 6.0 32-bit using Emerging Threats community rules. These rules will be automatically configured and updated by Oinkmaster.

It is important to note that Snort will be installed from RPMs, and not from source. There are also many tutorials on the web that cover a Snort installation from source in addition to MySQL, Apache2, and Barnyard to support the BASE front-end. This can be a very daunting and discouraging task to first-time Snort users. This tutorial, however, is not another one of those.

I try to reduce the overall complexity of learning Snort by showing a very simple setup to help those first-time Snort users get "their feet wet".

Bridging interfaces

In this setup Snort will be placed inline between a cable modem and home router.

To use Snort inline, two separate interfaces (eth0 and eth1) will need to be bridged as a single interface (br0). Using a bridge interface will allow Snort to transparently inspect all traffic passing in and out of the network.

Configure /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
BRIDGE=br0
NM_CONTROLLED=no
USERCTL=no
ONBOOT=yes
BOOTPROTO=none

Configure /etc/sysconfig/network-scripts/ifcfg-eth1:

DEVICE=eth1
BRIDGE=br0
NM_CONTROLLED=no
USERCTL=no
ONBOOT=yes
BOOTPROTO=none

Now create /etc/sysconfig/network-scripts/ifcfg-br0:

DEVICE=br0
TYPE=Bridge
NM_CONTROLLED=no
USERCTL=no
ONBOOT=yes
BOOTPROTO=none
DELAY=0

Restart the networking service:

# service networking restart

Installing Snort

Download the snort-2.9.1-1.F13.i386.rpm RPM from the Vincent Cojot unofficial Snort repository:

# wget http://vscojot.free.fr/dist/snort/snort-2.9.1/RHEL6/i386/snort-2.9.1-14.el6.i686.rpm


Download and install the following dependencies needed by Snort 2.9.1 from the Vincent Cojot repository:
  • libdnet
  • libpcap library version 1.1.1
  • Data Acquisiion Library

# wget http://vscojot.free.fr/dist/snort/snort-2.9.1/RHEL6/i386/libdnet-1.12-7.el6.i686.rpm
# wget http://vscojot.free.fr/dist/snort/snort-2.9.1/RHEL6/i386/libpcap1-1.1.1-10.el6.i686.rpm
# wget http://vscojot.free.fr/dist/snort/snort-2.9.1/RHEL6/i386/daq-0.6.1-10.el6.i686.rpm
# rpm -Uvh libdnet-1.12-7.el6.i686.rpm libpcap1-1.1.1-10.el6.i686.rpm daq-0.6.1-10.el6.i686.rpm

After all dependency packages have been installed, install the Snort RPM:

# rpm -Uvh snort-2.9.1-14.el6.i686.rpm

  • This will automatically create the Snort user and group, startup scripts, and setup the correct directories and paths.

Edit /etc/snort/snort.conf:

  • Configure the HOME_NET variable with your public IP address. If using Snort behind a router, configure HOME_NET with your internal network address, e.g. 192.168.1.0/24.
  • Define the RULE_PATH. The SO_RULE_PATH and PREPROC_RULE_PATH can be safely commented out as they are not used by the Emerging Threat rulesets.
  • Configure the unified2 output, change the filename from merged.log to snort.log and remove the additional options.

# Setup the network addresses you are protecting
#ipvar HOME_NET any
ipvar HOME_NET 1.2.3.4/32

# Path to your rules files (this can be a relative path)
# Note for Windows users: You are advised to make this an absolute path,
# such as: c:\snort\rules
#var RULE_PATH ../rules
#var SO_RULE_PATH ../so_rules
#var PREPROC_RULE_PATH ../preproc_rules
var RULE_PATH /etc/snort/rules

# output unified2: filename merged.log, limit 128, nostamp, mpls_event_types, vlan_event_types
output unified2: filename snort.log, limit 128

Verify that Snort will start with the following command:

# snort -c /etc/snort/snort.conf -i br0

  • -c denotes the configuration file.
  • -i denotes the interface to listen on.

Although no rules have been loaded yet, Snort should display a similar message if everything is configured correctly so far:

        --== Initialization Complete ==--

,,_ -*> Snort! <*-
o" )~ Version 2.9.1 IPv6 GRE (Build 71)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team
Copyright (C) 1998-2011 Sourcefire, Inc., et al.
Using libpcap version 1.1.1
Using PCRE version: 7.8 2008-09-05
Using ZLIB version: 1.2.3

Rules Engine: SF_SNORT_DETECTION_ENGINE Version 1.15
Preprocessor Object: SF_SIP (IPV6) Version 1.1
Preprocessor Object: SF_FTPTELNET (IPV6) Version 1.2
Preprocessor Object: SF_DCERPC2 (IPV6) Version 1.0
Preprocessor Object: SF_DNS (IPV6) Version 1.1
Preprocessor Object: SF_SDF (IPV6) Version 1.1
Preprocessor Object: SF_SMTP (IPV6) Version 1.1
Preprocessor Object: SF_REPUTATION (IPV6) Version 1.1
Preprocessor Object: SF_IMAP (IPV6) Version 1.0
Preprocessor Object: SF_SSH (IPV6) Version 1.1
Preprocessor Object: SF_POP (IPV6) Version 1.0
Preprocessor Object: SF_SSLPP (IPV6) Version 1.1
Commencing packet processing (pid=10957)

Oinkmaster and Emerging Threats rules

Emerging Threats is a fantastic community-driven project that provides free Snort rules for both personal and commercial use. They update their rulesets on a daily basis and even provide a commercial subscription with support, known as Emerging Threats Pro. See also ruleset comparison.

Oinkmaster is a simple perl script with accompanying .conf file that allows for that automatic configuration and management of Snort rules.

In a real-production network, not all Snort rules will be used, and certain rules may be disabled. When manually downloading and updating the newest rules, rules that have been disabled may be re-enabled again during the manual update process. Oinkmaster simplifies management of rules by allowing an administrator to keep track of which rules are disabled or enabled by default during an update as well as any other additional files to ignore.

Download the Oinkmaster tar archive from http://oinkmaster.sourceforge.net/download.shtml and extract:

# gunzip oinkmaster-2.0.tar.gz
# tar xvf oinkmaster-2.0.tar

The following files will be extracted to the current working directory:

oinkmaster-2.0
oinkmaster-2.0/README.templates
oinkmaster-2.0/oinkmaster.pl
oinkmaster-2.0/FAQ
oinkmaster-2.0/UPGRADING
oinkmaster-2.0/README
oinkmaster-2.0/README.win32
oinkmaster-2.0/README.gui
oinkmaster-2.0/LICENSE
oinkmaster-2.0/INSTALL
oinkmaster-2.0/ChangeLog
oinkmaster-2.0/oinkmaster.1
oinkmaster-2.0/template-examples.conf
oinkmaster-2.0/oinkmaster.conf
oinkmaster-2.0/contrib
oinkmaster-2.0/contrib/create-sidmap.pl
oinkmaster-2.0/contrib/addsid.pl
oinkmaster-2.0/contrib/makesidex.pl
oinkmaster-2.0/contrib/addmsg.pl
oinkmaster-2.0/contrib/README.contrib
oinkmaster-2.0/contrib/oinkgui.pl

Change into the oinkmaster-2.0/ directory and edit the oinkmaster.conf file.

Edit oinkmaster-2.0/oinkmaster.conf:

  • Add the url to the Emerging Threat rules Snort 2.9.0 rules. (2.9.0 rules are compatible with 2.9.1)
  • Set the tmpdir to /tmp.

# Example for Community rules
# url = http://www.snort.org/pub-bin/downloads.cgi/Download/comm_rules/Community-Rules.tar.gz

# Example for rules from the Bleeding Snort project
# url = http://www.bleedingsnort.com/bleeding.rules.tar.gz

# Emerging Threats
url = http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz

# Example for UNIX.
# tmpdir = /home/oinkmaster/tmp/
tmpdir = /tmp

Create a directory to backup old rules when Oinkmaster updates rules:

# mkdir /tmp/rules.old

Run the Oinkmaster perl script for the first time:

# oinkmaster.pl -o /etc/snort/rules/ -b /tmp/rules.old -C oinkmaster.conf

  • -o outputs the downloaded rules to the specified directory
  • -b defines where to backup old rules
  • -C defines which configuration file to use.
  • NOTE: oinkmaster.pl uses the wget utility, install it if it is not present on the system.

Output from above command:

# ./oinkmaster.pl -o /etc/snort/rules/ -b /tmp/rules.old -C oinkmaster.conf
Loading /root/oinkmaster-2.0/oinkmaster.conf
Downloading file from http://rules.emergingthreats.net/open/snort-2.9.0/emerging.rules.tar.gz... done.
Archive successfully downloaded, unpacking... done.
Setting up rules structures... done.
Processing downloaded rules... disabled 0, enabled 0, modified 0, total=15999
Setting up rules structures... done.
Comparing new files to the old ones... done.
Updating local rules files... done.

[***] Results from Oinkmaster started 20110915 20:06:35 [***]

[*] Rules modifications: [*]
None.

[*] Non-rule line modifications: [*]
None.

[+] Added files (consider updating your snort.conf to include them if needed): [+]

-> BSD-License.txt
-> classification.config
-> compromised-ips.txt
-> emerging.conf
-> emerging-activex.rules
-> emerging-attack_response.rules
-> emerging-botcc.rules
-> emerging-chat.rules
-> emerging-ciarmy.rules
-> emerging-compromised.rules
-> emerging-current_events.rules
-> emerging-deleted.rules
-> emerging-dns.rules
-> emerging-dos.rules
-> emerging-drop.rules
-> emerging-dshield.rules
-> emerging-exploit.rules
-> emerging-ftp.rules
-> emerging-games.rules
-> emerging-icmp.rules
-> emerging-icmp_info.rules
-> emerging-imap.rules
-> emerging-inappropriate.rules
-> emerging-malware.rules
-> emerging-misc.rules
-> emerging-mobile_malware.rules
-> emerging-netbios.rules
-> emerging-p2p.rules
-> emerging-policy.rules
-> emerging-pop3.rules
-> emerging-rbn-malvertisers.rules
-> emerging-rbn.rules
-> emerging-rpc.rules
-> emerging-scada.rules
-> emerging-scan.rules
-> emerging-shellcode.rules
-> emerging-smtp.rules
-> emerging-snmp.rules
-> emerging-sql.rules
-> emerging-telnet.rules
-> emerging-tftp.rules
-> emerging-tor.rules
-> emerging-trojan.rules
-> emerging-user_agents.rules
-> emerging-virus.rules
-> emerging-voip.rules
-> emerging-web_client.rules
-> emerging-web_server.rules
-> emerging-web_specific_apps.rules
-> emerging-worm.rules
-> gen-msg.map
-> gpl-2.0.txt
-> rbn-ips.txt
-> rbn-malvertisers-ips.txt
-> reference.config
-> sid-msg.map
-> snort-2.9.0-open.txt
-> unicode.map

Edit /etc/snort/snort.conf:

  • Now add the emerging.conf configuration file as an include.

###################################################
# Step #7: Customize your rule set
# For more information, see Snort Manual, Writing Snort Rules
#
# NOTE: All categories are enabled in this conf file
###################################################

# Emerging Threats rules
include $RULE_PATH/emerging.conf

By default /etc/snort/rules/emerging.conf has all rule groups commented out, and they must be enabled manually.

Edit /etc/snort/rules/emerging.conf to enable specific Emerging Threats rules:

  • Ensure classification.config and reference.config are uncommented!
  • Note do not enable all rules, you should enable rules relevant to your environment.
  • As an example I enabled the rules shown below.

include $RULE_PATH/classification.config
include $RULE_PATH/reference.config

include $RULE_PATH/emerging-policy.rules
include $RULE_PATH/emerging-games.rules
include $RULE_PATH/emerging-virus.rules
include $RULE_PATH/emerging-attack_response.rules
include $RULE_PATH/emerging-icmp.rules
include $RULE_PATH/emerging-scan.rules
include $RULE_PATH/emerging-chat.rulesles
include $RULE_PATH/emerging-shellcode.rules
include $RULE_PATH/emerging-current_events.rules
include $RULE_PATH/emerging-malware.rules
include $RULE_PATH/emerging-worm.rules
include $RULE_PATH/emerging-misc.rules
include $RULE_PATH/emerging-exploit.rules
include $RULE_PATH/emerging-p2p.rules
include $RULE_PATH/emerging-mobile_malware.rules

include $RULE_PATH/emerging-botcc.rules
include $RULE_PATH/emerging-compromised.ruless
include $RULE_PATH/emerging-drop.rulesK.rules
include $RULE_PATH/emerging-dshield.rules
include $RULE_PATH/emerging-rbn.rules
include $RULE_PATH/emerging-rbn-malvertisers.rules

Take time to go through each file to determine whether or not to use the rulesets provided by Emerging Threats. Again, not all will be relevant to your environment.

Because we edited /etc/snort/rules/emerging.conf, we need to ensure that it is not accidentally "updated" or overwritten by Oinkmaster. Remember, Oinkmaster outputs to and updates the /etc/snort/rules directory.

Edit oinkmaster-2.0/oinkmaster.conf again:

#######################################################################
# Files to totally skip (i.e. never update or check for changes) #
# #
# Syntax: skipfile filename #
# or: skipfile filename1, filename2, filename3, ... #
#######################################################################

# Do not modify or update emerging.conf
skipfile emerging.conf

NOTE: You can optionally ignore the rulesets that are commented out as well, i.e. skipfile emerging-tor.rules,emerging-sql etc.

Now anytime you need to update your Snort rules, simply run the ./oinkmaster.pl -o /etc/snort/rules/ -b /tmp/rules.old -C oinkmaster.conf command. It is recommended to put this command in a daily cron job, as Emerging Threats updates their rules on a daily basis.

Running Snort

Finally, run Snort in the foreground with the following command:

# snort -c /etc/snort/snort.conf -A fast -b -i br0

  • -A specifies the alert mode.
  • -b will log packets that were alerted on in tcpdump format.
  • -i specifies what interface to listen on.

Alerts will be logged to /var/log/snort/alert, if using the fast alert mode, alerts will appear as a single line.

Example output of /var/log/snort/alert using Snort alert-mode fast:

09/15-20:55:08.642022  [**] [1:2012141:2] ET POLICY Protocol 41 IPv6 encapsulation potential 6in4 IPv6 tunnel active [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {IPV6-ICMP} fe80:0000:0000:0000:0000:5efe:c0a8:0bf1 -> fe80:0000:0000:0000:0000:5efe:181c:c109
09/15-21:01:55.234549 [**] [1:2003089:4] ET GAMES STEAM Connection (v2) [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 1.2.3.4:49158 -> 68.142.72.250:27031
09/15-21:02:22.628590 [**] [1:2012170:2] ET GAMES Blizzard Web Downloader Install Detected [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 1.2.3.4:49196 -> 12.129.206.133:1119

I recommend running Snort for at least a day or two, you are guaranteed to see several alerts especially if your sensor is facing an external public network.

Tuning alerts

After running Snort for a period of time, even for a day or two as previously mentioned, you can quickly get a baseline of the alerts on your network. Some of these alerts can occur quite frequently and may unnecessarily flood your log file. Ideally you want to reduce the amount of "noise" in the alert log file. Tuning your alerts either by disabling the rule signature entirely, or suppressing them using the threshold.conf file is one of the many best practices in maintaining and optimizing Snort.

Disabling an alert

For example, on my development sensor I saw several of the following alerts:

09/16-17:39:08.567824  [**] [1:2012170:2] ET GAMES Blizzard Web Downloader Install Detected [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 1.2.3.4:54741 -> 173.223.52.219:80
09/16-17:39:08.677677 [**] [1:2012170:2] ET GAMES Blizzard Web Downloader Install Detected [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 1.2.3.4:54741 -> 173.223.52.219:80
09/16-17:39:13.197098 [**] [1:2012170:2] ET GAMES Blizzard Web Downloader Install Detected [**] [Classification: Potential Corporate Privacy Violation] [Priority: 1] {TCP} 1.2.3.4:54742 -> 12.129.242.21:80

As seen above, Snort detected a Blizzard Web Downloader which was triggered from a desktop running World of Warcraft within my network. If this were a corporate network, company policy would likely dictate that employees should not be playing games during work. However, because this development Snort sensor is on my home network, this alert is completely harmless and therefore can simply be disabled.

The signature or rule ID that triggered the above alert is 2012170. The 2 denotes that this is the second revision of the rule.

Using grep to search for 2012170 within the /etc/snort/rules directory, the file that contains the rule can be found:

# cd /etc/snort/rules
# grep -n 2012170 *

emerging-games.rules:326:alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET GAMES Blizzard Web Downloader Install Detected"; flow: established,to_server; content: "User-Agent|3a| Blizzard Web Client"; nocase; classtype:policy-violation; sid:2012170; rev:2;)
sid-msg.map:11036:2012170 || ET GAMES Blizzard Web Downloader Install Detected

The command found the rule in emerging-games.rules on line 326. To disable the rule, simply comment it out with #, like so:

#emerging-games.rules:326:alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"ET GAMES Blizzard Web Downloader Install Detected"; flow: established,to_server; content: "User-Agent|3a| Blizzard Web Client"; nocase; classtype:policy-violation; sid:2012170; rev:2;)

Restart Snort and the rule ID will no longer trigger an alert to the /var/log/snort/alert log file.

IMPORTANT: The oinkmaster.conf still needs to be configured to ensure that the above rule we disabled stays disabled during an update.

Edit your oinkmaster.conf:

  • Use a disablesid statement to disable the rule.

########################################################################
# SIDs to comment out, i.e. disable, after each update by placing a #
# '#' in front of the rule (if it's a multi-line rule, it will be put #
# in front of all lines). #
# #
# Syntax: disablesid SID #
# or: disablesid SID1, SID2, SID3, ... #
########################################################################

disablesid 2012170 # ET GAMES Blizzard Web Downloader Install Detected

The configuration file allows for comments to be appended to the disablesid parameter. For reference, putting the description of the rule, or why the rule is being disabled is strongly recommended.

Now, during an Oinkmaster update the above signature will stay disabled.

Suppressing an alert

There may be scenarios, where you do not want to disable a rule entirely, but rather configure it so that it does not alert on specific factors, such as source or destination IP addresses.

Take for instance the following alerts that were lightly flooding my alert file:

09/16-12:20:43.317651  [**] [1:1390:6] GPL SHELLCODE x86 inc ebx NOOP [**] [Classification: Executable Code was Detected] [Priority: 1] {TCP} 74.125.239.6:80 -> 1.2.3.4:54066
09/16-12:20:43.329227 [**] [1:1390:6] GPL SHELLCODE x86 inc ebx NOOP [**] [Classification: Executable Code was Detected] [Priority: 1] {TCP} 74.125.239.9:80 -> 1.2.3.4:54229
09/16-12:20:43.365007 [**] [1:1390:6] GPL SHELLCODE x86 inc ebx NOOP [**] [Classification: Executable Code was Detected] [Priority: 1] {TCP} 74.125.224.170:80 -> 1.2.3.4:54222

Resolving the above IP addresses shows that they actually belong to Google:

# for i in 74.125.239.6 74.125.239.9 74.125.224.170; do whois -h asn.shadowserver.org origin $i; done
[Querying asn.shadowserver.org]
[asn.shadowserver.org]
15169 | 74.125.239.0/24 | GOOGLE | US | GOOGLE.COM | GOOGLE INC
[Querying asn.shadowserver.org]
[asn.shadowserver.org]
15169 | 74.125.239.0/24 | GOOGLE | US | GOOGLE.COM | GOOGLE INC
[Querying asn.shadowserver.org]
[asn.shadowserver.org]
15169 | 74.125.224.0/24 | GOOGLE | US | GOOGLE.COM | GOOGLE INC

After further investigation it is safe to assume that the alerts were false positives. Optionally the triggering rule can be disabled, as shown in the previous example above, however, doing so would allow any IP address meeting the requirements to alert signature ID 1390 to pass through the network undetected. The correct way to tune the alert is to ignore only if it originates from Google, which can be done using the /etc/snort/threshold.conf file.

Configure /etc/snort/threshold.conf to ignore alerts from signature ID 1390 if it originates from Google's networks:

# Suppression:
#
# Suppression commands are standalone commands that reference generators and
# sids and IP addresses via a CIDR block (or IP list). This allows a rule to be
# completely suppressed, or suppressed when the causitive traffic is going to
# or comming from a specific IP or group of IP addresses.
#
# Suppress this event completely:
#
# suppress gen_id 1, sig_id 1852
#
# Suppress this event from this IP:
#
# suppress gen_id 1, sig_id 1852, track by_src, ip 10.1.1.54
#
# Suppress this event to this CIDR block:
#
# suppress gen_id 1, sig_id 1852, track by_dst, ip 10.1.1.0/24
#

# Suppress [1:1390:6] GPL SHELLCODE x86 inc ebx NOOP [**] if originating from Google's networks.
suppress gen_id 1, sig_id 1390, track by_src, ip 74.125.0.0/16

The Snort developers were kind enough to provide different examples in thethreshold.conf file.

Lastly, restart Snort and the the signature ID will only alert on non-Google networks.

Nothing needs to be changed on the Oinkmaster side, as no rule was explicitly being disabled or enabled. Suppression is handled by threshold.conf which is outside of the /etc/snort/rules directory that Oinkmaster "tracks".

Resources

Snort Official Documention: http://www.snort.org/docs
Emerging Threats Rule Documentation Wiki: http://doc.emergingthreats.net/

Thursday, September 1, 2011

Network analysis of malware infected PC

So I took a short break from work early this morning to run a small errand. I decided to check my email on my desktop computer before I headed back out. Man was I in for a surprise. I moved the mouse cursor to wake the monitor and I was greeted by a nice little program that looked like this:



"Security Protection" my ass. My desktop was infected with malware and I'm a network and systems engineer wanting to move into the realm of information security and information assurance. I feel like a dentist getting a cavity or a police officer getting pulled over for speeding.

I was caught with my pants down...at my ankles...while bending over. See http://nooooooooooooooo.com/ for my initial reaction upon seeing the above image. I don't think I've been infected by malware in close to 4-5 years.

My "check my email for before I head out" quickly became "figure out what the hell happened and block your desktop from connecting back to out to the Internet, and fix it when you get home."

I remembered that no more than a month ago I had installed Snort 2.9.0.5 inline (between my cable modem and embedded x86 ALIX router running Linux) on an old Pentium III desktop to vet my CentOS 6.0 32-bit hardened build for work. Cool, I thought, maybe I can analyze the packet captures, log files, and alerts and see what else I can find.

Alas, my poor desktop PC, became a case study for incident response and network analysis of a malware infected PC.

It was almost 9:30 AM by this time, and I had to head back to work so I came up with a game plan:

1. Check Snort logs to see if anything was alerted.
2. Block all traffic from the entire local subnet reaching out to the Internet, log it, and see
what shows up in the logs.
3. Try to run a quick diagnostics and scans on infected desktop before I head back to work.
4. Analyze everything further and decide how I will re-mediate my desktop.

Game Plan Item #1: check Snort logs

I had been running Snort in the background with snort -c /etc/snort/snort.conf -A fast -b -K pcap -i br0 on and off for the past two weeks to get an idea of what kind of alerts I would see over time.

A quick look at /var/log/snort/alerts, the default log file for Snort alerts, and I saw a couple suspicious entries:

09/01-02:33:13.716611  [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54027 -> 76.74.152.98:8080
09/01-02:33:13.716718 [**] [1:2011582:8] ET POLICY Vulnerable Java Version 1.6.x Detected [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} X.X.X.X:54028 -> 76.74.152.98:8080

The good news was Snort did detect something malicious going on; I didn't have time to check out it further until later.

Game Plan Item #2: block local egress traffic

I had to cage this beast quickly before it downloaded even more malware and did more damage. I went with a quick rule in Shorewall on my Linux router to block (and log) the entire local subnet from reaching the Internet until I got home.

/etc/shorewall/rules:

# BLOCK!
DROP:info loc net

Shorewall is a front-end for iptables; the actual iptables command would have been iptables -I FORWARD 0 -s 192.168.11.0/24 -j DROP.

I restarted Shorewall, went to grab a drink, and actually started to do my errands. You know, what I originally had intended to do when I made a quick stop home.

These were a couple entries in /var/log/messages:

Sep  1 08:57:42 voyage localadmin: Shorewall restarted
Sep 2 09:04:14 voyage kernel: [2084722.015845] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.94.220 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=429 DF PROTO=TCP SPT=49192 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2
Sep 2 09:04:17 voyage kernel: [2084725.097856] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.94.220 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=430 DF PROTO=TCP SPT=49192 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2
Sep 2 09:04:23 voyage kernel: [2084731.247840] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.94.220 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=431 DF PROTO=TCP SPT=49192 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2

Sep 2 09:06:13 voyage kernel: [2084841.657851] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.53.156 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=440 DF PROTO=TCP SPT=49193 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2
Sep 2 09:06:16 voyage kernel: [2084844.741827] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.53.156 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=441 DF PROTO=TCP SPT=49193 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2
Sep 2 09:06:22 voyage kernel: [2084850.890854] Shorewall:loc2net:DROP:IN=eth0 OUT=eth2 SRC=192.168.11.10 DST=65.55.53.156 LEN=48 TOS=0x00 PREC=0x00 TTL=127 ID=442 DF PROTO=TCP SPT=49193 DPT=443 WINDOW=8192 RES=0x00 SYN URGP=0 MARK=0x2

As you can see I restarted Shorewall at 8:57 AM, and at 9:04 AM my infected desktop (192.168.11.10) was trying to tunnel out via SSL (TCP 443) to destination address 65.55.94.220! It obviously failed 3 times to reach that address, so it tried again but a different IP at 65.55.53.156. I had an idea of what was to come, so I went on to my next plan of action.

Game Plan Item #3: attempt diagnostics

Since I cut it off from the Internet, I tried to characterize the behavior of the malware and its effect it had on my desktop. I didn't click anything on the Security Protection Center window, I wanted to kill the process directly.

Open up task manager to try to kill process directly? The window automatically closed.
Open up command prompt from Start > Run? The window automatically closed.
Open up anti-virus software? The window automatically closed.
Open up anti-spyware software? The window automatically closed.

Well that was characterized quite quickly.

I headed back to work and had an excited yet anxious feeling during my shift. On one hand my desktop had malware on it, on the other hand I had Snort packet captures and log files of real malware I could analyze; something I've always wanted to do.

Game Plan Item #4: additional analysis

When I got home, the first thing I did was check out what new log entries had appeared since I started blocking traffic from loc -> net. Since I normally don't log traffic from loc -> net, a simple grep command on my Linux router would filter my exact results: grep Shorewall:loc2net /var/log/messages. Let's just say there were A LOT of log entries.

I did a word count on my grep command to tell me exactly how many log entries there were:

# grep loc2net messages | wc -l
3869

So between approximately 9:00 AM this morning and 6:00 PM, there were close to 4000 denied attempts from my desktop to reach other malicious hosts on the Internet. That's some persistent malware, to say the least.

I ran the following command to get an idea of how many of those destination IPs were unique:

# grep loc2net messages | awk '{print $10}' | sort -rn | uniq -c

3 DST=91.209.196.169
4 DST=83.145.197.2
14 DST=74.54.61.194
95 DST=74.125.53.188
6 DST=74.125.53.141
9 DST=74.125.53.132
9 DST=74.125.53.125
34 DST=74.125.47.108
15 DST=74.125.239.9
39 DST=74.125.239.8
48 DST=74.125.239.7
42 DST=74.125.239.6
48 DST=74.125.239.5
48 DST=74.125.239.4
39 DST=74.125.239.3
6 DST=74.125.239.2
6 DST=74.125.239.15
6 DST=74.125.239.14
6 DST=74.125.239.13
12 DST=74.125.239.12
8 DST=74.125.239.11
6 DST=74.125.239.10
12 DST=74.125.239.1
12 DST=74.125.239.0
6 DST=74.125.224.82
51 DST=74.125.224.239
36 DST=74.125.224.238
42 DST=74.125.224.237
39 DST=74.125.224.236
24 DST=74.125.224.235
30 DST=74.125.224.234
48 DST=74.125.224.233
66 DST=74.125.224.232
84 DST=74.125.224.231
72 DST=74.125.224.230
90 DST=74.125.224.229
99 DST=74.125.224.228
81 DST=74.125.224.227
84 DST=74.125.224.226
75 DST=74.125.224.225
69 DST=74.125.224.224
9 DST=74.125.224.191

...additional output omitted...

3 DST=69.171.228.14
12 DST=69.171.224.67
5 DST=66.235.120.98
105 DST=66.220.146.22
3 DST=65.55.94.220
3 DST=65.55.94.216
21 DST=65.55.53.190
3 DST=65.55.53.156
18 DST=65.55.27.219
24 DST=65.55.25.59
9 DST=65.55.200.156
3 DST=65.55.200.155
9 DST=65.55.200.139
12 DST=65.55.184.16
9 DST=65.55.184.152
4 DST=65.55.17.39
9 DST=65.55.119.90
6 DST=65.54.75.98
9 DST=65.54.75.95
9 DST=65.54.75.93
15 DST=65.54.75.92
15 DST=65.54.75.8
3 DST=65.54.75.71
18 DST=65.54.75.6
3 DST=65.54.75.51
3 DST=65.54.75.40
3 DST=65.54.75.25

...additional output omitted...

49 DST=24.28.193.9
12 DST=24.25.230.9
45 DST=24.25.230.8
42 DST=24.25.230.18
30 DST=24.25.230.16
9 DST=24.25.230.10
4 DST=217.149.52.196
84 DST=216.228.124.39
63 DST=216.18.194.133
3 DST=216.156.213.179
3 DST=216.156.213.152
3 DST=213.35.100.25
4 DST=209.62.68.168
15 DST=209.18.46.99
15 DST=209.18.46.91
6 DST=209.18.46.66
3 DST=209.18.46.50
6 DST=209.18.46.42
3 DST=209.18.46.123
3 DST=209.151.233.98
28 DST=208.49.52.91
26 DST=208.49.52.106
1 DST=208.43.217.90
12 DST=207.46.21.123
10 DST=207.171.162.56
2 DST=207.171.162.142
36 DST=204.160.114.254
3 DST=199.66.201.169
3 DST=199.47.217.174
35 DST=199.47.217.173
30 DST=199.47.217.144
3 DST=178.255.83.0
9 DST=173.223.52.217
9 DST=173.223.52.186
12 DST=107.20.249.120

I piped the above grep command through word count to give me an exact number:

# grep loc2net messages | awk '{print $10}' | sort -rn | uniq | wc -l
163

That makes 163 unique IP addresses my malware infected desktop was trying to contact, and on many of those IPs, they were trying to be contacted several times.

I decided to check out those Snort alerts in further detail.

I accumulated roughly 642 Snort alerts in /var/log/snort/alerts in the past week. There were a bunch of false positives that were triggered by my PC games, IMs, etc. A quick and dirty regex filtered out all that and gave me what I really needed to see:

# cd /var/log/snort
# egrep "X.X.X.X:.* -> .*" alert | egrep -v "GAME|P2P|Skype|Google" | less -S

Output from the above command:

08/30-02:27:54.042284  [**] [1:2012607:2] ET USER_AGENTS Lowercase User-Agent header purporting to be MSIE [**] Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:33852 -> 70.37.68.225:80
08/30-02:29:11.933137 [**] [1:2012607:2] ET USER_AGENTS Lowercase User-Agent header purporting to be MSIE [**][Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54918 -> 70.37.68.225:80
09/01-02:32:38.984298 [**] [1:2010937:2] ET POLICY Suspicious inbound to mySQL port 3306 [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 116.255.188.165:6000 -> X.X.X.X:3306
09/01-02:33:13.716611 [**] [1:2011582:8] ET POLICY Vulnerable Java Version 1.6.x Detected [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} X.X.X.X:54027 -> 76.74.152.98:8080
09/01-02:33:13.716611 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54027 -> 76.74.152.98:8080
09/01-02:33:13.716718 [**] [1:2011582:8] ET POLICY Vulnerable Java Version 1.6.x Detected [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} X.X.X.X:54028 -> 76.74.152.98:8080
09/01-02:33:13.716718 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54028 -> 76.74.152.98:8080
09/01-02:33:13.716987 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54026 -> 76.74.152.98:8080
09/01-02:33:13.717114 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54029 -> 76.74.152.98:8080
09/01-02:33:13.717287 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54024 -> 76.74.152.98:8080
09/01-02:33:13.717449 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54025 -> 76.74.152.98:8080
09/01-02:33:13.900829 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54033 -> 76.74.152.98:8080
09/01-02:33:13.901049 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54032 -> 76.74.152.98:8080
09/01-02:33:13.901278 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54031 -> 76.74.152.98:8080
09/01-02:33:13.901574 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54030 -> 76.74.152.98:8080
09/01-02:33:13.907570 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54034 -> 76.74.152.98:8080
09/01-02:33:13.907705 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54035 -> 76.74.152.98:8080
09/01-02:36:19.794010 [**] [1:2011894:8] ET TROJAN TDSS/TDL/Alureon MBR rootkit Checkin [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54071 -> 178.238.233.154:80
09/01-02:36:55.080334 [**] [1:2003579:4] ET MALWARE Findwhat.com Spyware (clickthrough) [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54215 -> 206.123.102.103:80
09/01-02:37:12.889356 [**] [1:2003579:4] ET MALWARE Findwhat.com Spyware (clickthrough) [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54323 -> 206.123.102.103:80
09/01-02:37:13.026983 [**] [1:2003579:4] ET MALWARE Findwhat.com Spyware (clickthrough) [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54323 -> 206.123.102.103:80
09/01-02:37:35.670005 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54480 -> 109.236.82.46:80
09/01-02:37:35.670716 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54482 -> 109.236.82.46:80
09/01-02:37:36.505659 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54484 -> 109.236.82.46:80
09/01-02:37:36.508068 [**] [1:2012609:2] ET CURRENT_EVENTS Java Exploit Attempt Request for .class from octal host [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54485 -> 109.236.82.46:80
09/01-02:37:41.788934 [**] [1:2012612:4] ET TROJAN Hiloti Style GET to PHP with invalid terse MSIE headers [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54489 -> 205.234.129.135:80
09/01-02:37:41.997022 [**] [1:2012612:4] ET TROJAN Hiloti Style GET to PHP with invalid terse MSIE headers [**] [Classification: A Network Trojan was Detected] [Priority: 1] {TCP} X.X.X.X:54490 -> 205.134.252.251:80

I'm unsure as to whether the initial alerts on August 30 from 2:27-2:29 AM are related or have something to do with the alerts that started firing off on September 1st. Regardless, the bulk of it started at 2:32 AM on September 1st.