Friday, July 25, 2014

fail2ban with Shorewall

When it came to blacklisting attackers trying to brute-force my services, like SSH, my go-to package has always been DenyHosts. However, issues such as recent vulnerabilities and most notably, its removal from the default repositories for Ubuntu 14.04 LTS caused me to finally switch to fail2ban. The biggest advantage fail2ban provides over DenyHosts is that it is more flexible in its actions and types of services it can monitor (DenyHosts only supports services using TCP wrappers).

Installation

Install the fail2ban package:

root@localhost:~# aptitude -yvV install fail2ban

After installation, the fail2ban service will automatically be started:


root@localhost:/etc/fail2ban# ps aux | grep [f]ail2ban
root     11232  0.0  1.8 268164  9360 ?        Sl   02:47   0:00 /usr/bin/python /usr/bin/fail2ban-server -b -s /var/run/fail2ban/fail2ban.sock -p /var/run/fail2ban/fail2ban.pid

root@localhost:/etc/fail2ban# service fail2ban status
 * Status of authentication failure monitor
 *  fail2ban is running

Configuration

First, make a copy of the /etc/fail2ban/jail.conf file as /etc/fail2ban/jail.local, as per http://www.fail2ban.org/wiki/index.php/MANUAL_0_8:

Every .conf file can be overridden with a file named .local. The .conf file is read first, then .local, with later settings overriding earlier ones. Thus, a .local file doesn't have to include everything in the corresponding .conf file, only those settings that you wish to override. Modifications should take place in the .local and not in the .conf. This avoids merging problem when upgrading.

root@localhost:/etc/fail2ban# cp -v jail.conf jail.local
‘jail.conf’ -> ‘jail.local’

By default fail2ban uses iptables to control its default banning actions. Luckily using Shorewall is as simple as changing a single line in the /etc/fail2ban/jail.local configuration file.

/etc/fail2ban/jail.local

#
# ACTIONS
#

# Default banning action (e.g. iptables, iptables-new,
# iptables-multiport, shorewall, etc) It is used to define
# action_* variables. Can be overridden globally or per
# section within jail.local file
#banaction = iptables-multiport
banaction = shorewall

The /etc/fail2ban/action.d/shorewall.conf configuration file comments explains this in more detail.

All files in the /etc/fail2ban/action.d/ drop configuration directory control actions to an accompanying service. For example, action.d/ipfw.conf controls how fail2ban tells ipfw, used in FreeBSD systems, what command to run to ban an offending host. Likewise, action.d/shorewall.conf controls what commands are run by shorewall in order to ban an offender.

In /etc/fail2ban/action.d/shorewall.conf the important directives to pay attention to for our setup are actionban, actionunban, and blocktype:

# Option:  actionban
# Notes.:  command executed when banning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionban = shorewall <blocktype> <ip>

# Option:  actionunban
# Notes.:  command executed when unbanning an IP. Take care that the
#          command is executed with Fail2Ban user rights.
# Tags:    See jail.conf(5) man page
# Values:  CMD
#
actionunban = shorewall allow <ip>

[Init]

# Option:  blocktype
# Note:    This is what the action does with rules.
#          See man page of shorewall for options that include drop, logdrop, reject, or logreject
# Values:  STRING
#blocktype = reject
blocktype = drop
  • To reiterate, when fail2ban bans a host, the command shorewall drop <offending IP address> will be run.

In the above configuration I changed the blocktype to drop rather than reject. Rather than getting an ICMP destination unreachable response from my server when attackers are brute-forcing my services like SSH after they have been banned, drop will cause their connections to timeout instead.

Also, shorewall needs to be updated to handle the banning action correctly. By default, it will only blacklist hosts part of a new TCP connection, but not already existing or established connections.

/etc/shorewall/shorewall.conf

#BLACKLIST="NEW,INVALID,UNTRACKED"
BLACKLIST="ALL"

Restart the fail2ban and shorewall services:

root@localhost:/etc/fail2ban# service fail2ban restart
 * Restarting authentication failure monitor fail2ban                         [ OK ]

root@localhost:/etc/shorewall# service shorewall restart
Restarting "Shorewall firewall": done.

Seeing it in action

After leaving fail2ban running for a couple hours, you are inevitably going to get some banned hosts. You can see that fail2ban is working correctly by checking the /var/log/fail2ban.log log file. Here's an example of some IPs that were banned:

2014-07-23 19:58:16,068 fail2ban.actions: WARNING [ssh] Ban 111.74.238.167
2014-07-23 20:08:16,818 fail2ban.actions: WARNING [ssh] Unban 111.74.238.167
2014-07-23 20:46:02,506 fail2ban.actions: WARNING [ssh] Ban 117.21.225.116
2014-07-23 20:56:03,249 fail2ban.actions: WARNING [ssh] Unban 117.21.225.116
  • Note the hosts were unbanned after 10 minutes, which is the default timer.
  • To increase the ban time, change the bantime value in the /etc/fail2ban/jail.local configuration file.

To verify that fail2ban did indeed pass the IPs to shorewall use the shorewall show dynamic command:

root@localhost:/etc/fail2ban# shorewall show dynamic
Shorewall 4.5.21.6 Chain dynamic at localhost - Fri Jul 25 07:21:26 EDT 2014

Counters reset Wed Jul 23 03:07:05 EDT 2014

Chain dynamic (4 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      *       183.56.129.146       0.0.0.0/0           
    0     0 DROP       all  --  *      *       54.210.100.16        0.0.0.0/0           
    0     0 DROP       all  --  *      *       115.239.248.61       0.0.0.0/0           
    0     0 DROP       all  --  *      *       61.147.103.71        0.0.0.0/0           
    0     0 DROP       all  --  *      *       115.239.248.50       0.0.0.0/0           
    0     0 DROP       all  --  *      *       117.21.191.209       0.0.0.0/0           
    0     0 DROP       all  --  *      *       117.21.191.210       0.0.0.0/0           
    0     0 DROP       all  --  *      *       110.12.207.5         0.0.0.0/0  

Resources

https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-12-04
http://www.ehow.com/how_12187342_fail2ban-shorewall.html

Sunday, July 13, 2014

Redirecting Shorewall log messages to custom file

By default Shorewall will log firewall messages to /var/log/messages if you are running CentOS, or /var/log/syslog if you are running Ubuntu.

Sample log message from Shorewall using default format:

Jul 13 01:30:42 localhost kernel: [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC=128.72.226.39 DST=192.168.0.16 LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 

Despite the LOGFILE=/var/log/messages parameter in the shorewall.conf file, this directive isn't used to redirect logs to a custom file. As described from man shorewall.conf:

LOGFILE=[pathname]
    This parameter tells the /sbin/shorewall program where to look for Shorewall messages
    when processing the dump, logwatch, show log, and hits commands. If not assigned or if
    assigned an empty value, /var/log/messages is assumed. For further information, see
    http://www.shorewall.net/shorewall_logging.html.

Using rsyslog to redirect messages

Configurations done through rsyslog will allow Shorewall log messages to be redirected to a custom file.

Create the /etc/rsyslog.d/40-shorewall.conf file:

# touch /etc/rsyslog.d/40-shorewall.conf
# ls -l /etc/rsyslog.d
total 8
-rw-r--r-- 1 root root  311 Mar 17  2012 20-ufw.conf
-rw-r--r-- 1 root root    0 Jul 13 02:04 40-shorewall.conf
-rw-r--r-- 1 root root 1655 May 19  2013 50-default.conf
  • The numerical prefix determines the order at which the configuration files are read by rsyslog, i.e. settings in 40-shorewall.conf will be configured before 50-default.conf.

Edit /etc/rsyslog.d/40-shorewall.conf:

:msg, contains, "Shorewall:" -/var/log/shorewall.log
& ~
  • This is an rsyslog property-based filter which matches any log messages which contain the string Shorewall: and redirects them to the /var/log/shorewall.log log file.
  • The "& ~" tells rsyslog to stop processing the message after it was written to the log. Without the "& ~", messages would continue to be written to local files such as /var/log/syslog.
    • If you wanted messages to still show up in /var/log/{syslog,messages} then the "& ~" can be omitted.

Anytime custom log files are made, an accompanying /etc/logrotate.d configuration file should be created or updated. Shorewall by default creates /etc/logrotate.d/shorewall which controls rotation of the /var/log/shorewall-init.log file. Add a directive to control /var/log/shorewall.log:

# cat /etc/logrotate.d/shorewall
/var/log/shorewall-init.log {
    weekly
    rotate 4
    compress
    missingok
    create 0640 root adm
}

/var/log/shorewall.log {
    weekly
    rotate 52
    compress
    compressext .gz
    missingok
    create 0640 root adm
}
  • This will rotate the shorewall.log log file every week and store archived log files for 52 weeks (1 year).

Restart the rsyslog service:

# service rsyslog restart

The restart should have triggered the /var/log/shorewall.log file to be created, if not, try to port-scan or trigger an event and see if the file was created:

# ls -l /var/log/shorewall.log
-rw-r----- 1 syslog adm 0 Jul 13 02:27 /var/log/shorewall.log

rsyslog template

I created a template to shorten the log messages, as Shorewall log messages can be quite long and verbose, by adding another line to /etc/rsyslog.d/40-shorewall.conf:

$template shorewall-template,"%timegenerated% %msg%\n"
:msg, contains, "Shorewall:" -/var/log/shorewall.log;shorewall-template
& ~
  • rsyslog filtering does not follow a simple syntax, so I recommend reading their official documentation to learn exactly how it works, as rsyslog can be leveraged to do very complex and powerful configurations.
  • Though not much, the template removes the localhost kernel: portion in the message:
  • From this:
    
    Jul 13 01:30:42 localhost kernel: [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC=128.72.226.39 DST=192.168.0.16 LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 
    
    To this:
    
    Jul 13 01:30:42 [37078.720986] Shorewall:net2fw:DROP:IN=eth0 OUT= MAC=00:0c:29:85:73:f2:00:1d:d0:cd:3b:81:08:00 SRC=128.72.226.39 DST=192.168.0.16 LEN=129 TOS=0x08 PREC=0x20 TTL=104 ID=19418 PROTO=UDP SPT=11888 DPT=51413 LEN=109 
    
  • Full list of properties available here.

Wednesday, July 9, 2014

Initial network analysis of stray BitTorrent traffic

In my previous blog post I detailed a basic installation of ntopng. Interestingly, I was able to detect odd traffic destined to one of my desktops in my network ‒ no less than a day after I had setup ntopng on a development server. Network forensics and analysis is a domain I've been meaning to level up, so this was a perfect opportunity quest to explore.

In the ntopng web interface under the Flows section, ntopng had identified an active flow with a duration of over 4h, 14 min originating from the IP address 82.207.24.165, which resides in Kiev, Ukraine. There was a nice little Ukraine flag by the IP in the web interface, so it quickly caught my eye. In addition, the flow was identified as the BitTorrent protocol.

Example screenshot of Active Flows section inside web interface

Foreign IP addresses are to be expected, especially in regards to the BitTorrent protocol. The problem, however, was that my desktop wasn't running a BitTorrent client at the time and BitTorrent almost always connects to several peers at once for both (sometimes simultaneously) uploads and downloads. So my immediate questions were how was this traffic getting through the router firewall and why was it only this single IP address? Seeing a bunch of foreign flags in the web interface at once would have actually worried me less.

Since the flow was still active, I decided to see what it looked like on the wire with tcpdump by using a BPF to match any traffic to or from that Ukraine host:

root@trustytahr:/var/tmp# tcpdump -v -n -i eth0 not tcp port 22 and ip host 82.207.24.165
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
13:14:49.233184 IP (tos 0x0, ttl 107, id 5434, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.356179 IP (tos 0x0, ttl 107, id 5449, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.438760 IP (tos 0x0, ttl 107, id 5460, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.544598 IP (tos 0x0, ttl 107, id 5470, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.583587 IP (tos 0x0, ttl 107, id 5475, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.735788 IP (tos 0x0, ttl 107, id 5489, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.804994 IP (tos 0x0, ttl 107, id 5498, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.839829 IP (tos 0x0, ttl 107, id 5502, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.926611 IP (tos 0x0, ttl 107, id 5512, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
13:14:49.965995 IP (tos 0x0, ttl 107, id 5516, offset 0, flags [none], proto UDP (17), length 129)
    82.207.24.165.6881 > 192.168.0.7.31369: UDP, length 101 
  • UDP packets were being sent several times a second by 82.207.24.165 with random source ports.
  • All of these packets were being sent to the same UDP port, 31368, on 192.168.0.7.
  • -v specifies verbose mode, and -n prevents tcpdump from doing a reverse DNS lookup on the IP addresses.

Seeing that the traffic was identified as BitTorrent and destined to 192.168.0.7:31369, I checked three things on 192.168.0.7 (a desktop running Windows 7):

  1. Find any connections matching port 31369, and also see if there were any interesting or malicious processes using the port.
  2. C:\Windows\system32>netstat -nabo | findstr "31369"
    C:\Windows\system32>
    
    • No connections found.
  3. Double-check if µTorrent was running. A quick check in the system tray and Windows Task Manager verified it wasn't.
  4. See what random port µTorrent was using; µTorrent assigns itself a random UDP port for incoming connections.

  5. 31389 is indeed the randomly assigned port for incoming connections
    Also take note how Enable UPnP port mapping and Enable NAT-PMP port mapping were enabled

For further exploration, I re-ran tcpdump and included the -w switch which outputs the captured network traffic to a named .pcap file. I then opened it in Wireshark and clicked Follow UDP Stream.

Screenshot of Follow UDP Stream in Wireshark

Searching Google for strings in the output lead me to For Developers section on the BitTorrent.org website (not to be used with BitTorrent.com). On the page it describes the find_node packets:

Find node is used to find the contact information for a node given its ID. "q" == "find_node" A find_node query has two arguments, "id" containing the node ID of the querying node, and "target" containing the ID of the node sought by the queryer. When a node receives a find_node query, it should respond with a key "nodes" and value of a string containing the compact node info for the target node or the K (8) closest good nodes in its own routing table.

Also, it shows an example packet for a find_node Query, which is the type of packet that I was receiving from the Ukraine host:

find_node Query = {"t":"aa", "y":"q", "q":"find_node", "a": {"id":"abcdefghij0123456789", "target":"mnopqrstuvwxyz123456"}}
bencoded = d1:ad2:id20:abcdefghij01234567896:target20:mnopqrstuvwxyz123456e1:q9:find_node1:t2:aa1:y1:qe

Alternatively, the find_node Response packet looks like so (though my desktop never replied to a query in the time it was an active flow listed in ntopng):

Response = {"t":"aa", "y":"r", "r": {"id":"0123456789abcdefghij", "nodes": "def456..."}}
bencoded = d1:rd2:id20:0123456789abcdefghij5:nodes9:def456...e1:t2:aa1:y1:re

UPnP, NAT-PMP and BitTorrent

UPnP stands for “Universal Plug and Play.” Using UPnP, an application can automatically forward a port on your router, saving you the hassle of forwarding ports manually. We’ll be looking at the reasons people recommend disabling UPnP, so we can get a clear picture of the security risks. — http://www.howtogeek.com/122487/htg-explains-is-upnp-a-security-risk/
NAT-PMP allows a computer in a private network (behind a NAT router) to automatically configure the router to allow parties outside the private network to contact it. NAT-PMP runs over UDP port 5351. It essentially automates the process of port forwarding. — Wikipedia

UPnP and NAT-PMP, though different protocols, essentially do the same thing. UPnP or NAT-PMP run as a service on many home-based routers, and is usually enabled by default. BitTorrent often relies on these protocols to allow traffic through networks behind a NAT, which is nearly every home network. All of the major BitTorrent applications support support UPnP and NAT-PMP, which allows these clients to map ports on the router.

Turning off UPnP or NAT-PMP is as simple as going into the router web interface and looking for the checkbox to disable it.

Monitoring

Though the flow eventually stopped after a few hours, before I could test to see if turning off UPnP and NAT-PMP made a difference, I've been using ntopng to continually monitor and analyze other anomalous traffic.

Sunday, July 6, 2014

ntopng on Ubuntu 14.04 LTS Server

Following directions are deprecated, updated instructions are available at ntopng 1.2.2 on Ubuntu 14.04, revisited.


ntopng is the next generation version of the original ntop, a network traffic probe that shows the network usage, similar to what the popular top Unix command does. ntopng is based on libpcap and it has been written in a portable way in order to virtually run on every Unix platform, MacOSX and on Win32 as well. — http://www.ntop.org/products/ntop/

ntopng is different from the original ntop package found in the default Ubuntu repositories. Perhaps the biggest different between the two is that ntopng has an updated and modern HTML5 web interface which has data visualizations that update in real-time, whereas ntop provides a static and very generic looking web interface.

Installing ntopng

Note that packages from the repository below are x64 only. If you are using i386, then ntopng must be built from source.

Following the download pages on ntop.org and looking for stable Ubuntu packages will eventually lead you to this website repository. The directions below follow those in the listed website.

Add the http://www.nmon.net/apt-stable/ repository and its accompanying public key:

ubuntu@trustytahr:~$ sudo -i
[sudo] password for ubuntu: 

root@trustytahr:~# /bin/echo -e "deb http://www.nmon.net/apt-stable/ x64/\ndeb http://www.nmon.net/apt-stable/ all/" > /etc/apt/sources.list.d/ntop.list

root@trustytahr:~# wget -qO - http://www.nmon.net/apt-stable/ntop.key | sudo apt-key add -
OK
  • Switch completely to the root user, rather than using sudo to run the above commands.

Update apt repositories with apt-get update and ignore the conflicting messages at the end:

root@trustytahr:~# apt-get update

Reading package lists... Done
W: Conflicting distribution: http://www.nmon.net x64/ Release (expected x64 but got )
W: Conflicting distribution: http://www.nmon.net all/ Release (expected all but got )

Install the ntopng and ntopng-data packages:

root@trustytahr:~# apt-get -y install ntopng ntopng-data

The installation creates startup script /etc/init.d/ntopng, and the ntopng binary itself as /usr/local/bin/ntopng. The rest of the relevant files can be all found under the /usr/local/share/ntopng directory. See output of dpkg -L ntopng for a full listing.

Running ntopng

Starting ntopng is as simple as executing the /usr/local/bin/ntopng binary:

root@trustytahr:/usr/local/bin# ./ntopng 
06/Jul/2014 01:15:34 [Ntop.cpp:555] Setting local networks to 192.168.1.0/24,0.0.0.0/32,224.0.0.0/8,239.0.0.0/8,255.255.255.255/32,127.0.0.0/8
06/Jul/2014 01:15:34 [Redis.cpp:50] Successfully connected to Redis 127.0.0.1:6379
06/Jul/2014 01:15:34 [PcapInterface.cpp:81] Reading packets from interface eth0...
06/Jul/2014 01:15:34 [Ntop.cpp:662] Registered interface eth0 [id: 0]
06/Jul/2014 01:15:34 [PcapInterface.cpp:81] Reading packets from interface lo...
06/Jul/2014 01:15:34 [Ntop.cpp:662] Registered interface lo [id: 1]
06/Jul/2014 01:15:34 [Utils.cpp:251] User changed to nobody
06/Jul/2014 01:15:34 [main.cpp:152] PID stored in file /var/tmp/ntopng.pid
06/Jul/2014 01:15:34 [HTTPserver.cpp:351] HTTPS Disabled: missing SSL certificate /usr/local/share/ntopng/httpdocs/ssl/ntopng-cert.pem
06/Jul/2014 01:15:34 [HTTPserver.cpp:352] Please read README.SSL if you want to enable SSL
06/Jul/2014 01:15:34 [HTTPserver.cpp:389] Web server dirs [/usr/local/share/ntopng/httpdocs][/usr/local/share/ntopng/scripts]
06/Jul/2014 01:15:34 [HTTPserver.cpp:392] HTTP server listening on port 3000
06/Jul/2014 01:15:34 [main.cpp:186] Using RRD version 1.4.7
06/Jul/2014 01:15:34 [main.cpp:202] Working directory: /var/tmp/ntopng
06/Jul/2014 01:15:34 [main.cpp:204] Scripts/HTML pages directory: /usr/local/share/ntopng
06/Jul/2014 01:15:34 [Ntop.cpp:181] Welcome to ntopng x86_64 v.1.1.4 (r7806) - (C) 1998-14 ntop.org
06/Jul/2014 01:15:34 [PeriodicActivities.cpp:53] Started periodic activities loop...
06/Jul/2014 01:15:34 [NetworkInterface.cpp:770] Started packet polling on interface eth0...
06/Jul/2014 01:15:34 [NetworkInterface.cpp:770] Started packet polling on interface lo...
  • Above output shows ./ntopng run as a foreground process, optionally it could have been ran as a background process with ./ntopng &.
  • By default the working directory is /var/tmp and the listening port for the web interface is at port 3000.

While the process is running, the web interface should now be accessible via browser at http://<SERVER IP>:3000 — http://localhost:3000 if viewing the web page locally.

The default username is admin and password is admin.

Running ntopng as a service

Running ntopng as a service, rather than a foreground or background script requires additional steps.

Note that ntopng has several options that can be passed at either the command-line or via configuration file. All the options are listed in the man page.

A configuration file should be created at /etc/ntopng/ntopng.conf, below is a configuration file I used (the commented descriptions are excerpts directly taken from man ntopng):

# /etc/ntopng/ntopng.conf
#
#        The  configuration  file is similar to the command line, with the exception that an equal
#        sign '=' must be used between key and value. Example:  -i=p1p2  or  --interface=p1p2  For
#        options with no value (e.g. -v) the equal is also necessary. Example: "-v=" must be used.
#
#
#       -G|--pid-path
#        Specifies the path where the PID (process ID) is saved.
#
-G=/var/tmp/ntopng.pid
#
#       -e|--daemon
#        This  parameter  causes ntop to become a daemon, i.e. a task which runs in the background
#        without connection to a specific terminal. To use ntop other than as a casual  monitoring
#        tool, you probably will want to use this option.
#
-e=
#
#       -i|--interface
#        Specifies  the  network  interface or collector endpoint to be used by ntopng for network
#        monitoring. On Unix you can specify both the interface name  (e.g.  lo)  or  the  numeric
#        interface id as shown by ntopng -h. On Windows you must use the interface number instead.
#        Note that you can specify -i multiple times in order to instruct ntopng to create  multi‐
#        ple interfaces.
#
-i=eth0
#
#       -w|--http-port
#        Sets the HTTP port of the embedded web server.
#
-w=3000
#
#       -m|--local-networks
#        ntopng determines the ip addresses and netmasks for each active interface. Any traffic on
#        those  networks  is considered local. This parameter allows the user to define additional
#        networks and subnetworks whose traffic is also considered local in  ntopng  reports.  All
#        other hosts are considered remote. If not specified the default is set to 192.168.1.0/24.
#
#        Commas  separate  multiple  network  values.  Both netmask and CIDR notation may be used,
#        even mixed together, for instance "131.114.21.0/24,10.0.0.0/255.0.0.0".
#
-m=192.168.1.0/24
#
#       -n|--dns-mode
#        Sets the DNS address resolution mode: 0 - Decode DNS responses  and  resolve  only  local
#        (-m)  numeric  IPs  1  -  Decode DNS responses and resolve all numeric IPs 2 - Decode DNS
#        responses and don't resolve numeric IPs 3 - Don't decode DNS responses and don't  resolve
#
-n=1
#
#       -S|--sticky-hosts
#        ntopng  periodically purges idle hosts. With this option you can modify this behaviour by
#        telling ntopng not to purge the hosts specified by -S. This parameter requires  an  argu‐
#        ment  that  can  be  "all"  (Keep  all hosts in memory), "local" (Keep only local hosts),
#        "remote" (Keep only remote hosts), "none" (Flush hosts when idle).
#
-S=
#
#       -d|--data-dir
#        Specifies the data directory (it must be writable). Default directory is ./data
#
-d=/var/tmp/ntopng
#
#       -q|--disable-autologout
#        Disable web interface logout for inactivity.
#
-q=

Create the /etc/ntopng/ntopng.start as an empty file:

root@trustytahr:~# touch /etc/ntopng/ntopng.start

root@trustytahr:~# ls -l /etc/ntopng/ntopng.start
-rw-r--r-- 1 root root 0 Jul  6 01:41 /etc/ntopng/ntopng.start
  • For some reason the start_ntopng() function in the /etc/init.d/ntopng startup script requires this file in order for the service to start.

Start the ntopng service:

root@trustytahr:~# service ntopng start
Starting ntopng

The ntopng process can be verified with the service ntopng status or in ps aux output:

root@trustytahr:~# ps aux | grep [n]topng
nobody    2343  0.6  1.3 764432 27900 ?        Ssl  01:45   0:00 /usr/local/bin/ntopng /etc/ntopng/ntopng.conf

root@trustytahr:~# service ntopng status
ntopng running as 2343

Changing default password for web interface

Command-line Method

Verify connectivity to redis-server with redis-cli ping command:

root@trustytahr:~# redis-cli ping
PONG

Generate an md5 hash of our new password with md5sum:

root@trustytahr:~# echo -n "NewPassword_4ntopng" | md5sum
a13a83f9a23740e9ba453b2df3c41f9a  -

Use this hash in the redis-cli SET user.admin.password command:

root@trustytahr:~# redis-cli SET user.admin.password a13a83f9a23740e9ba453b2df3c41f9a
OK

Web Interface Method

After logging into the web interface:

Settings (Gear Icon) > Manage Users > Set Password

Troubleshooting

  • If the service fails to start, run /usr/local/bin/ntopng /etc/ntopng/ntopng.conf directly to see all messages on stdout.
  • The log file can be found at: /var/tmp/ntopng/ntopng.log
  • ntopng requires the redis-server service to be running:
  • root@trustytahr:~# service redis-server status
    redis-server is running
    
    root@trustytahr:~# netstat -tunlp | grep -i redis-server
    tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      2139/redis-server 1
    

Bug Fixes

The /etc/init.d/ntopng startup script has a few minor bugs that could use a couple fixes:

  • /usr/local/bin/ntopng can actually be run without the root user; however, the ntopng process will not be able to enter the listening interface into promiscuous mode and will still return a successful exit status.
  • The startup script parses for PID_FILE using grep to find the -G switch, but doesn't alternatively search for its synonymous long switch --pid-file.
  • The service ntopng status command will return nothing if ntopng hasn't started.

diff -u output for the above fixes:

root@trustytahr:~# diff -u /etc/init.d/ntopng.orig /etc/init.d/ntopng
--- /etc/init.d/ntopng.orig 2014-07-06 02:30:21.133552653 -1000
+++ /etc/init.d/ntopng 2014-07-06 02:44:55.057578340 -1000
@@ -13,6 +13,11 @@
 # Short-Description: Start/stop ntopng web
 ### END INIT INFO
 
+if [ $(id -u) -ne 0 ]; then
+    echo "ntopng requires root."
+    exit 1
+fi
+
 start_ntopng() {
     FORCE=$1
 
@@ -34,7 +39,7 @@
     if [ -f /etc/ntopng/ntopng.start ] || [ $FORCE -eq 1 ]; then
  echo "Starting ntopng"
 
- PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+ PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
         if [ -f $PID_FILE ]; then
      PID=$(cat $PID_FILE)
             if [ $PID -gt 0 ]; then
@@ -60,7 +65,7 @@
     fi
 
     if [ -f /etc/ntopng/ntopng.conf ]; then
- PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+ PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
  if [ -f "$PID_FILE" ]; then
      PID=$(cat $PID_FILE)
      if [ $PID -gt 0 ]; then
@@ -86,7 +91,7 @@
  return 0
     fi
 
-    PID_FILE=$(cat /etc/ntopng/ntopng.conf | grep '\-G='|cut -d '=' -f 2)
+    PID_FILE=$(grep '^-G\|^--pid-file' /etc/ntopng/ntopng.conf | cut -d '=' -f 2)
     if [ -f $PID_FILE ]; then
  PID=$(cat $PID_FILE)
  if [ $PID -gt 0 ]; then
@@ -94,6 +99,8 @@
  else
      echo "No running ntopng pid [$PID] in [$PID_FILE]"
  fi
+    else
+        echo "ntopng not running: process id file $PID_FILE not found."
     fi
     
     return 0

References

http://xmodulo.com/2013/10/set-web-based-network-traffic-monitoring-linux.html
http://blog.redbranch.net/2013/12/12/reset-ntopng-admin-password/

Friday, July 4, 2014

SSD tweaks under Ubuntu 14.04 LTS Server

Change IO Scheduler

"The scheduler helps organise reads and writes in the I/O queue to maximise performance. The default scheduler in the Linux kernel is CFQ (Completely Fair Queuing) which is designed with the rotational latencies of spinning platter drives in mind. So while it works well for standard hard drives it doesn’t work so well when it comes to SSDs." — http://apcmag.com/how-to-maximise-ssd-performance-with-linux.htm

Edit /etc/default/grub and add elevator=deadline to GRUB_CMDLINE_LINUX_DEFAULT:

#GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX_DEFAULT="elevator=deadline"

Update GRUB settings with update-grub2:

root@localhost:~# update-grub2
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.13.0-30-generic
Found initrd image: /boot/initrd.img-3.13.0-30-generic
Found linux image: /boot/vmlinuz-3.13.0-24-generic
Found initrd image: /boot/initrd.img-3.13.0-24-generic
Found memtest86+ image: /memtest86+.elf
Found memtest86+ image: /memtest86+.bin
done

Verify deadline scheduler is in use:

root@localhost:~# cat /sys/block/sda/queue/scheduler 
noop [deadline] cfq 

/etc/fstab options

Add discard,noatime, and nodiratime options to all partitions on the SSD in the /etc/fstab file.

Descriptions of the options found in the mount man page:

discard/nodiscard
              Controls  whether ext4 should issue discard/TRIM commands to the underlying block device
              when blocks are freed.  This is useful for  SSD  devices  and  sparse/thinly-provisioned
              LUNs, but it is off by default until sufficient testing has been done.

noatime
              Do not update inode access times on this filesystem (e.g., for faster access on the news
              spool to speed up news servers).

nodiratime
              Do not update directory inode access times on this filesystem.

Backup /etc/fstab first:

root@localhost:~# cp -v /etc/fstab{,.bak}
‘/etc/fstab’ -> ‘/etc/fstab.bak’

Edit and add options in /etc/fstab:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
/dev/mapper/vg0-lv_root /       ext4    errors=remount-ro,discard,noatime,nodiratime    0       1

References

http://www.howtogeek.com/62761/how-to-tweak-your-ssd-in-ubuntu-for-better-performance/
http://apcmag.com/how-to-maximise-ssd-performance-with-linux.htm

Wednesday, July 2, 2014

Configuring OpenVPN and SELinux

up and down user scripts

OpenVPN supports custom user scripts to be run as the service is started or stopped — known as the --up and --down parameters in the man page. On CentOS or other Red Hat-based distributions, these scripts may not work correctly because SELinux is preventing them from executing, which results in the OpenVPN service failing to start.

[root@localhost ~]# service openvpn start
Starting openvpn:                                          [FAILED]

[root@localhost ~]# tail -15 /var/log/openvpn.log
Tue Jul  1 10:48:44 2014 TUN/TAP TX queue length set to 100
Tue Jul  1 10:48:44 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Tue Jul  1 10:48:44 2014 /sbin/ip link set dev tun0 up mtu 1500
Tue Jul  1 10:48:44 2014 /sbin/ip addr add dev tun0 local 172.16.1.1 peer 172.16.1.2
Tue Jul  1 10:48:44 2014 PLUGIN_CALL: POST /usr/lib64/openvpn/plugins/openvpn-plugin-down-root.so/PLUGIN_UP status=0
Tue Jul  1 10:48:44 2014 /etc/openvpn/scripts/up.sh tun0 1500 1542 172.16.1.1 172.16.1.2 init
/etc/openvpn/scripts/up.sh: line 14: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 15: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 16: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 17: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 18: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 19: /sbin/iptables: Permission denied
/etc/openvpn/scripts/up.sh: line 20: /proc/sys/net/ipv4/ip_forward: Permission denied
Tue Jul  1 10:48:44 2014 WARNING: Failed running command (--up/--down): external program exited with error status: 1
Tue Jul  1 10:48:44 2014 Exiting due to fatal error
  • OpenVPN fails to start with the service command.
  • Error messages in /var/log/openvpn.log show failed execution of commands regarding the up.sh script.

Also to further verify SELinux is causing this problem, error messages can be found in /var/log/audit/audit.log:

type=AVC msg=audit(1404247863.966:166): avc:  denied  { getattr } for  pid=2946 comm="up.sh" path="/sbin/iptables-multi-1.4.7" dev=sda1 ino=705 scontext=unconfined_u:system_r:openvpn_t:s0 tcontext=system_u:object_r:iptables_exec_t:s0 tclass=file

type=SYSCALL msg=audit(1404247863.966:166): arch=c000003e syscall=4 success=no exit=-13 a0=26c6890 a1=7fff1691d490 a2=7fff1691d490 a3=50 items=0 ppid=2939 pid=2946 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=2 comm="up.sh" exe="/bin/bash" subj=unconfined_u:system_r:openvpn_t:s0 key=(null)
  • Notice up.sh was trying to run shell commands and /bin/bash.

To allow custom scripts to be executed two things must be done:

  1. The up or down script must have the correct file contexts set.
  2. The correct SELinux boolean must be set to allow for unconfined scripts to be run by OpenVPN.

1. OpenVPN SELinux file contexts

Anytime the restorecon command is run, it reads the /etc/selinux/targeted/contexts/files/file_contexts file to know what file context to set on a regular file or directory. Therefore we can search through this file to find all the default file contexts for OpenVPN using grep -i openvpn /etc/selinux/targeted/contexts/files/file_contexts, see below.

[root@localhost ~]# grep -i openvpn /etc/selinux/targeted/contexts/files/file_contexts
/etc/openvpn(/.*)? system_u:object_r:openvpn_etc_t:s0
/var/log/openvpn.* system_u:object_r:openvpn_var_log_t:s0
/etc/openvpn/ipp.txt -- system_u:object_r:openvpn_etc_rw_t:s0
/var/lib/openvpn(/.*)? system_u:object_r:openvpn_var_lib_t:s0
/var/run/openvpn(/.*)? system_u:object_r:openvpn_var_run_t:s0
/etc/openvpn/scripts(/.*)? system_u:object_r:openvpn_unconfined_script_exec_t:s0
/usr/sbin/openvpn -- system_u:object_r:openvpn_exec_t:s0
/etc/rc\.d/init\.d/openvpn -- system_u:object_r:openvpn_initrc_exec_t:s0
/usr/share/munin/plugins/openvpn -- system_u:object_r:munin_services_plugin_exec_t:s0

Note all the different context types, though, the type we are most interested in is system_u:object_r:openvpn_unconfined_script_exec_t:s0 which applies to all files and directories under /etc/openvpn/scripts/, and the directory itself.

Create the /etc/openvpn/scripts directory, move your custom script to it, and run a recursive restorecon:

[root@localhost ~]# mkdir /etc/openvpn/scripts

[root@localhost ~]# ls -Zd /etc/openvpn/scripts/
drwx------. root root unconfined_u:object_r:openvpn_etc_t:s0 /etc/openvpn/scripts/

[root@localhost ~]# mv -v up.sh /etc/openvpn/scripts/
`up.sh' -> `/etc/openvpn/scripts/up.sh'

[root@localhost ~]# restorecon -Rv /etc/openvpn/scripts/
restorecon reset /etc/openvpn/scripts context unconfined_u:object_r:openvpn_etc_t:s0->unconfined_u:object_r:openvpn_unconfined_script_exec_t:s0
restorecon reset /etc/openvpn/scripts/roadwarrior context unconfined_u:object_r:openvpn_etc_t:s0->unconfined_u:object_r:openvpn_unconfined_script_exec_t:s0
restorecon reset /etc/openvpn/scripts/up.sh context unconfined_u:object_r:admin_home_t:s0->unconfined_u:object_r:openvpn_unconfined_script_exec_t:s0
  • Notice the file context type for the scripts/ directory and the scripts/up.sh switch from openvpn_etc_t to openvpn_unconfined_script_exec_t.

Verify the directory and files have the correct file contexts with ls -Z:

[root@localhost ~]# ls -Zd /etc/openvpn/scripts/
drwx------. root root unconfined_u:object_r:openvpn_unconfined_script_exec_t:s0 /etc/openvpn/scripts/

[root@localhost ~]# ls -Zl /etc/openvpn/scripts/
total 3
-rwxr-xr-x. 1 unconfined_u:object_r:openvpn_unconfined_script_exec_t:s0 root root    0 Jul  1 12:21 up.sh

2. OpenVPN SELinux booleans

Find the relevant SELinux booleans by listening them using getsebool -a:

[root@localhost ~]# getsebool -a | grep -i openvpn
openvpn_enable_homedirs --> on
openvpn_run_unconfined --> off

Change the openvpn_run_unconfined boolean to on with the setsebool command and verify:

[root@localhost ~]# setsebool openvpn_run_unconfined on
[root@localhost ~]# getsebool -a | grep -i openvpn
openvpn_enable_homedirs --> on
openvpn_run_unconfined --> on

Now the OpenVPN service should be able to start without any errors relating to the execution of --up or --down scripts:

[root@localhost ~]# service openvpn start
Starting openvpn:                                          [  OK  ]

[root@localhost ~]# tail -15 /var/log/openvpn.log 
Tue Jul  1 12:33:45 2014 TUN/TAP TX queue length set to 100
Tue Jul  1 12:33:45 2014 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Tue Jul  1 12:33:45 2014 /sbin/ip link set dev tun0 up mtu 1500
Tue Jul  1 12:33:45 2014 /sbin/ip addr add dev tun0 local 172.16.1.1 peer 172.16.1.2
Tue Jul  1 12:33:45 2014 PLUGIN_CALL: POST /usr/lib64/openvpn/plugins/openvpn-plugin-down-root.so/PLUGIN_UP status=0
Tue Jul  1 12:33:45 2014 /etc/openvpn/scripts/up.sh tun0 1500 1542 172.16.1.1 172.16.1.2 init
Tue Jul  1 12:33:46 2014 /sbin/ip route add 172.16.1.0/24 via 172.16.1.2
Tue Jul  1 12:33:46 2014 GID set to nobody
Tue Jul  1 12:33:46 2014 UID set to nobody
Tue Jul  1 12:33:46 2014 UDPv4 link local (bound): [undef]
Tue Jul  1 12:33:46 2014 UDPv4 link remote: [undef]
Tue Jul  1 12:33:46 2014 MULTI: multi_init called, r=256 v=256
Tue Jul  1 12:33:46 2014 IFCONFIG POOL: base=172.16.1.4 size=62, ipv6=0
Tue Jul  1 12:33:46 2014 IFCONFIG POOL LIST
Tue Jul  1 12:33:46 2014 Initialization Sequence Completed

Custom OpenVPN listening port

The default listening port for OpenVPN is UDP port 1194. If you decide to use another port, SELinux settings must be updated using semanage to reflect the newly assigned port.

List SELinux port information and find port contexts matching OpenVPN with semanage port -l | grep -i openvpn:

[root@localhost ~]# semanage port -l | grep -i openvpn
openvpn_port_t                 tcp      1194
openvpn_port_t                 udp      1194

If we decide to use UDP port 2194 instead, use the following semanage command:

[root@localhost ~]# semanage port -a -t openvpn_port_t -p udp 2194

Optionally you can swap udp with tcp if you are using OpenVPN in TCP mode.

Verify SELinux port contexts are updated:

[root@localhost ~]# semanage port -l | grep -i openvpn
openvpn_port_t                 tcp      1194
openvpn_port_t                 udp      2194, 1194

References

http://sysadmin-notepad.blogspot.com/2013/05/custom-openvpn-port-with-selinux-enabled.html
http://www.mankier.com/8/openvpn_unconfined_script_selinux