Sunday, February 15, 2015

ntopng 1.2.2 on Ubuntu 14.04, revisited

I was doing some random research on ntopng a few days ago and I happened to stumble upon a page that was linking to my own blog. I didn't ever think I'd see the day where someone would link to one of my tutorials, especially on a sub-Reddit. This blog was mainly created for myself, little things like this motivate me to post more often.

My ntopng on Ubuntu 14.04 LTS Server post was created only 7 months ago. Unfortunately my tutorial didn't work out for that individual user, and it goes to show how quickly documentation can become inconsistent, especially in the open source world. So I've decided to revisit the topic and redocument it again from scratch; below are directions for installing ntopng 1.2.2 on Ubuntu 14.04.

Installing ntopng

Directions for installing ntopng seem liked they are far more streamlined compared to when I first did this last July. I'll be simply following the ntop.org official directions for their stable build packages.

Pull down the apt-ntop-stable.deb package using wget, and install with dpkg:

ubuntu@ubuntu-14-04:~$ sudo -i
[sudo] password for ubuntu: 

root@ubuntu-14-04:~# wget http://www.nmon.net/apt-stable/14.04/all/apt-ntop-stable.deb
root@ubuntu-14-04:~# ls
apt-ntop-stable.deb

root@ubuntu-14-04:~# dpkg -i apt-ntop-stable.deb 
Selecting previously unselected package apt-ntop-stable.
(Reading database ... 55712 files and directories currently installed.)
Preparing to unpack apt-ntop-stable.deb ...
Unpacking apt-ntop-stable (2.1-288) ...
Setting up apt-ntop-stable (2.1-288) ...
Adding ntop key to apt keyring
OK

The apt-ntop-stable.deb package doesn't install ntopng itself, it's simply files to add the repository. See below:

root@ubuntu-14-04:~# dpkg -l | grep ntop
ii  apt-ntop-stable       2.1-288        all          ntop apt package repository

root@ubuntu-14-04:~# dpkg -L apt-ntop-stable
/.
/etc
/etc/nbox
/etc/nbox/ntop-apt.key
/etc/apt
/etc/apt/sources.list.d
/etc/apt/sources.list.d/ntop-stable.list

Run apt-get update to update your system repositories and install the packages as per directions:

root@ubuntu-14-04:~# apt-get update
root@ubuntu-14-04:~# apt-get -y install pfring nprobe ntopng ntopng-data n2disk nbox

Here's a short description of what each package in the family does:

root@ubuntu-14-04:~# dpkg -s pfring nprobe ntopng ntopng-data n2disk nbox | egrep "^Package|^Description"

Package: pfring
Description: PF_RING (http://www.ntop.org/pf_ring/)

Package: nprobe
Description: A network probe.

Package: ntopng
Description: Web-based traffic monitoring.

Package: ntopng-data
Description: Data files (geoip) for ntopng.

Package: n2disk
Description: A packet-to-disk application.

Package: nbox
Description: Web management interface for ntop apps.
  • pfring is a module that allows for high-speed package captures, it's recommended to enable this if you plan on capturing on high-traffic interfaces.
  • nProbe is simply the NetFlow probe, for example you can setup multiple probes throughout your network and send all the NetFlow data to a central ntopng instance to visualize all the traffic.
  • n2disk allows you to efficiently write huge volumes of packet captures to disk without packet loss.

A lot of packages will be installed, and at the very end you should see the following message:

IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT

You can now point your browser to https://localhost/

The default user is nbox with password nbox

IMPORTANT IMPORTANT IMPORTANT IMPORTANT IMPORTANT
  • Don't forget that the address is https and not regular https. If you use http it will direct you to the Apache2 Ubuntu Default Page.

Before you visit the https://localhost page, however, restart the apache2 service:

root@ubuntu-14-04:~# service apache2 restart
 * Restarting web server apache2                                                                                              AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

I was getting a Service Unavailable error, and the restart fixed this.

Starting ntopng with the nBox web GUI

Unlike before where we had to configure things manually, the new ntop UI or nBox web GUI makes many of the configurations trivial.

After logging into https://localhost with the default credentials (nbox:nbox), at the top of the nBox dashboard:

  • Applications > ntopng
  • Under the Configuration > General
  • Select which interfaces you want to monitor. If you want to select multiple, hold <Ctrl> and click.
  • Enable the service to startup automatically, if needed.

You can edit other settings under Hosts, Flows, Directory, and Advanced.

When finished, click Save Changes. Then click back to the Status tab, and click On for the interface you selected.

The interface will tell you that you can now access ntopng at the http://<server IP>:3000 address.

Note that additional changes under the Configuration tab first require you to stop the ntopng service by clicking the Off button for your interface under the Status tab.

Change those default credentials

nBox GUI

On the of the nBox web GUI, System > Users. Web Users > for the already selected nbox user click Change Pwd.

ntopng GUI

The default username and password for the ntopng web interface is admin. To change the defaults, after logging into the web interface, click the Gear Icon > Manage Users, for the admin user, click Manage and change the password.

Conclusion

The nBox web UI greatly simplified configuration of all the components of the ntop family, in addition, it's refreshing to see that the ntopng UI is becoming more and more refined with each release.

Monday, February 9, 2015

PPPoE on Ubuntu 14.04

It took me over the course of a week and probably close to 20 hours of troubleshooting to finally get my Linux router to establish a PPPoE connection to my provider. There were two major hurdles I overcame:

  1. The first was figuring out that the VLAN going to the DSL modem needed to be manually tagged. I found out this out by connecting my Windows laptop and tagging it's VLAN on the network adapter after seeing that it was also tagged in the router they used to setup my initial connection. I suspect they do this to differentiate between the IP phone and cable box traffic.
  2. The second hurdle was properly configuring PPPoE properly in Linux. I emphasize properly because PPPoE in Linux is one of those topics that are barely documented, or if it is documented its done 50 different ways, all which don't work correctly for you — kind of like LDAP.

Adding VLAN support

Note this part may not be needed, double check your settings on a working router to see if the VLAN is configured for the WAN interface. If you are able to receive a DHCP lease, but can't establish a PPPoE connection or see any response in the PPPoE logs, you may need to tag the VLAN on your network interface.

This is the part which probably took up the majority of my time because when I ran the pppoeconf utility (like many tutorials and StackOverflow responses tell you to do), it would simply hang. I would suggest trying to use pppoeconf first to configure PPPoE (there are several tutorials out there), if that doesn't work for you, try the procedures outlined in this tutorial.

Tagging VLANs isn't supported by default in Ubuntu 14.04, luckily, the vlan package can do that for us. Install the the vlan package:

root@ubuntu:~# aptitude -yvV install vlan

Load the 8021q module and verify it is loaded:

root@ubuntu:~# modprobe 8021q
root@ubuntu:~# lsmod | grep ^8021q
8021q                  24712  0 

Ensure this module is loaded each time at boot by appending it to /etc/modules:

root@ubuntu:~# echo "8021q" >> /etc/modules

Use the vconfig utility to add a VLAN to the specified interface:

root@ubuntu:/etc/network# vconfig add eth0 20
Added VLAN with VID == 20 to IF -:eth0:-
  • In the above I tagged my eth0 interface with VLAN 20.

As additional verification you can cat the contents of /proc/net/vlan/config which would give you output similar to the following:

root@ubuntu:/etc/network# cat /proc/net/vlan/config 
VLAN Dev name  | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
eth0.20        | 20  | eth0

This will allow you to reference the VLAN-tagged interface as eth0.20. As another example, if you were to tag your eth1 interface with VLAN 50 the interface name would be eth1.50.

Network interfaces

After issuing vconfig, you should be able to create and reference the interface in the /etc/network/interfaces file.

#
# WAN interface
#
auto eth0
iface eth0 inet dhcp

#
# WAN interface tagged with VLAN 20
#
auto eth0.20
iface eth0.20 inet manual
 vlan-raw-device eth0

Now if you have a working physical connection from your ISP facing interface to the DSL modem, you should be able to get a DHCP lease with an private IP address on the original interface, in this case eth0.

The trick was understanding that I had to tell my PPPoE client to use the eth0.20 interface, not eth0, to send the initial request for the PPPoE connection. This is where the pppoeconf setup would hang for me.

RP-PPPoE

Unfortunately it's very easy to get confused between all the PPP/PPPoE packages, e.g. ppp, pppconfig, pppoe, pppoeconf, rp-pppoe, etc. Even worse, is they all dump their configuration files and scripts in the same place, the /etc/ppp directory.

The client that I ended up using was rp-pppoe by Roaring Penguin Software.

Download the tar archive from their website:

root@ubuntu:~# wget https://www.roaringpenguin.com/files/download/rp-pppoe-3.11.tar.gz
2015-02-07 17:15:55 (167 KB/s) - ‘rp-pppoe-3.11.tar.gz’ saved [223234/223234]

Install the build-essential package which contains additional utilities needed to build packages from source:

root@ubuntu:~# aptitude -yvV install build-essential

Unarchive, change into the unarchived directory, and run the ./go script:

root@ubuntu:~# tar xvf rp-pppoe-3.11.tar.gz

root@ubuntu:~# cd rp-pppoe-3.11/

root@ubuntu:~/rp-pppoe-3.11# ls
configs  doc  go  go-gui  gui  man  README  rp-pppoe.spec  scripts  SERVPOET  src

root@ubuntu:~/rp-pppoe-3.11# ./go

Now if rp-pppoe compiled correctly it should kick off a script immediately afterwards that prompts you for input:

Welcome to the Roaring Penguin PPPoE client setup.  First, I will run
some checks on your system to make sure the PPPoE client is installed
properly...

Looks good!  Now, please enter some information:

USER NAME

>>> Enter your PPPoE user name (default bxxxnxnx@sympatico.ca): dsluser

INTERFACE

>>> Enter the Ethernet interface connected to the DSL modem
For Solaris, this is likely to be something like /dev/hme0.
For Linux, it will be ethn, where 'n' is a number.
(default eth0): eth0.20

Do you want the link to come up on demand, or stay up continuously?
If you want it to come up on demand, enter the idle time in seconds
after which the link should be dropped.  If you want the link to
stay up permanently, enter 'no' (two letters, lower-case.)
NOTE: Demand-activated links do not interact well with dynamic IP
addresses.  You may have some problems with demand-activated links.
>>> Enter the demand value (default no): 

DNS

Please enter the IP address of your ISP's primary DNS server.
If your ISP claims that 'the server will provide DNS addresses',
enter 'server' (all lower-case) here.
If you just press enter, I will assume you know what you are
doing and not modify your DNS setup.
>>> Enter the DNS information here: server

PASSWORD

>>> Please enter your PPPoE password:    
>>> Please re-enter your PPPoE password: 

FIREWALLING

Please choose the firewall rules to use.  Note that these rules are
very basic.  You are strongly encouraged to use a more sophisticated
firewall setup; however, these will provide basic security.  If you
are running any servers on your machine, you must choose 'NONE' and
set up firewalling yourself.  Otherwise, the firewall rules will deny
access to all standard servers like Web, e-mail, ftp, etc.  If you
are using SSH, the rules will block outgoing SSH connections which
allocate a privileged source port.

The firewall choices are:
0 - NONE: This script will not set any firewall rules.  You are responsible
          for ensuring the security of your machine.  You are STRONGLY
          recommended to use some kind of firewall rules.
1 - STANDALONE: Appropriate for a basic stand-alone web-surfing workstation
2 - MASQUERADE: Appropriate for a machine acting as an Internet gateway
                for a LAN
>>> Choose a type of firewall (0-2): 0

** Summary of what you entered **

Ethernet Interface: eth0.20
User name:          dsluser
Activate-on-demand: No
DNS addresses:      Supplied by ISP's server
Firewalling:        NONE

>>> Accept these settings and adjust configuration files (y/n)? y
Adjusting /etc/ppp/pppoe.conf
Adjusting /etc/ppp/pap-secrets and /etc/ppp/chap-secrets
  (But first backing it up to /etc/ppp/pap-secrets-bak)
  (But first backing it up to /etc/ppp/chap-secrets-bak)



Congratulations, it should be all set up!

Type 'pppoe-start' to bring up your PPPoE link and 'pppoe-stop' to bring
it down.  Type 'pppoe-status' to see the link status.
  • This script will create the /etc/ppp/pppoe.conf populated with configuration parameters that you provided as input.
  • It will also put the username and password into the /etc/ppp/pap-secrets and /etc/ppp/chap-secrets file for you.

Again, note that I specified my VLAN-tagged interface eth0.20 as my interface connected to the DSL modem, and not eth0.

Now you should be able to run pppoe-start:

root@ubuntu:~/rp-pppoe-3.11# pppoe-start
. Connected!

The plog command will show you logging information from the initiated connection:

root@d54250wyk:~# plog
Feb  7 15:32:08 ubuntu pppd[2134]: Remote message: Login ok
Feb  7 15:32:08 ubuntu pppd[2134]: PAP authentication succeeded
Feb  7 15:32:08 ubuntu pppd[2134]: not replacing existing default route via 10.150.32.1
Feb  7 15:32:08 ubuntu pppd[2134]: local  IP address 176.205.250.149
Feb  7 15:32:08 ubuntu pppd[2134]: remote IP address 31.215.80.1
Feb  7 15:32:08 ubuntu pppd[2134]: primary   DNS address 213.42.20.20
Feb  7 15:32:08 ubuntu pppd[2134]: secondary DNS address 195.229.241.222

You should now have a ppp0 interface, use ifconfig or ip addr ls to verify it is there.

In the above output you can see the message not replacing existing default route via 10.150.32.1. This was the default route that was obtained from the original DHCP lease on the eth0 interface. There was a bug in the rp-pppoe utility that regardless of settings in the /etc/ppp/pppoe.conf file, it would not obtain and correctly replace the default route via the ppp0 interface.

To resolve this issue, and correctly obtain the default route upon connect, delete all the files in the /etc/ppp/peers/ directory:

root@ubuntu:~# rm -v /etc/ppp/peers/*
removed ‘/etc/ppp/peers/dsl-provider’
removed ‘/etc/ppp/peers/dsl-provider.dpkg-old’
removed ‘/etc/ppp/peers/provider’

Run pppoe-stop and pppoe-start to connect again:

root@ubuntu:/etc/openvpn# pppoe-stop
Killing pppd (2769)
Killing pppoe-connect (2749)

root@ubuntu:/etc/openvpn# pppoe-start
. Connected!

After your connection is established and verified, your configurations for all your network applications, such as iptables should reference the ppp0 interface as your primary interface.

Friday, February 6, 2015

Configuring GeoIP support for Shorewall on Ubuntu 14.04

If you've run fail2ban on any of your servers, see fail2ban with Shorewall, you'll quickly find out that a majority of the banned IPs will originate from many of the same countries, usually China. One of the techniques I previously used to block out an entire country's network range was to use ipsets. I used custom bash scripts to pull down zone information from http://www.ipdeny.com/ipblocks/ and import them into separate ipsets for countries I wanted. There were certain limitations with this solution, however, such as maintaining up to date ipsets across multiple servers, some issues with Shorewall losing the ipsets across reboots or restarts, and the US netblock space being to large to fit into a single ipset.

Shorewall version 4.5.4 introduced the ability to support GeoIP 2-character ISO 3166 country codes. This method is far more efficient and easier to maintain as the GeoIP database holds all the netblocks for all the countries in the world in an offline hashed database.

xt_geoip module installation

The first step is to install the xt_geoip kernel module which allows you to reference two letter country codes (e.g. US, CN, UK, CA, FR, etc.) in iptables or Shorewall.

This module can be found in the xtables-addons-common 2.3-1 package, but do not install this default package to use the xt_geoip module. I probably spent over 4 hours trying to figure out why my virtual machine was crashing every time a rule triggered the use of the xt_geoip module only to find out that there's a verified bug that causes a kernel panic.

Patrick Domack PPA to the rescue

Fortunately, if you read the bug comments, a user by the name of Patrick Domack provided his own PPA with a packaged version of xtables-addons-common version 2.6-1~ppa1 which fixes this bug.

Add the PPA with apt-add-repository ppa:patrickdk/general-lucid:

root@ubuntu:~# add-apt-repository ppa:patrickdk/general-lucid

 Packages used for my personal productions systems where newer versions or special patches are needed.
 More info: https://launchpad.net/~patrickdk/+archive/ubuntu/general-lucid
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmp24yt3e8g/secring.gpg' created
gpg: keyring `/tmp/tmp24yt3e8g/pubring.gpg' created
gpg: requesting key 4D79B5B5 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmp24yt3e8g/trustdb.gpg: trustdb created
gpg: key 4D79B5B5: public key "Launchpad PPA for Patrick Domack" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

apt-get update to update the repository and verify the new version of xtables-addons-common is available:

root@ubuntu:~# apt-get update

root@ubuntu:~# apt-cache show xtables-addons-common
Package: xtables-addons-common
Source: xtables-addons
Priority: extra
Section: admin
Installed-Size: 326
Maintainer: Pierre Chifflier 
Architecture: amd64
Version: 2.6-1~ppa1

Install xtables-addons-common and verify the correct version is installed:

root@ubuntu:~# aptitude -yvV install xtables-addons-common

root@ubuntu:~# dpkg -l | grep xtables-addons-common
ii  xtables-addons-common               2.6-1~ppa1                    amd64        Extensions targets and matches for iptables [tools, libs]

Building the GeoIP database for xt_geoip

The package will install two scripts in the /usr/lib/xtables-addons directory:

root@ubuntu:~# ls /usr/lib/xtables-addons/
xt_geoip_build  xt_geoip_dl
  • According to http://xtables-addons.sourceforge.net/geoip.php the "xt_geoip_dl simply calls wget on the hardcoded URLs and unpacks the retrieved files into the current directory. Then use xt_geoip_build to transform the CSV into the packed format"

The xt_geoip_dl script uses the unzip utility to unpack the a .csv file, install unzip if it isn't already:

root@ubuntu:~# aptitude -yvV install unzip

Change into /tmp and run /usr/lib/xtables-addons/xt_geoip_dl:

root@ubuntu:/tmp# /usr/lib/xtables-addons/xt_geoip_dl

Note that both iptables and shorewall will look for the GeoIP database in the /usr/share/xt_geoip directory, by default it is not created. Also, the build script requires the libtext-csv-xs-perl module to parse the .csv file.

Create the /usr/share/xt_geoip directory and install the required perl module:

root@ubuntu:/tmp# mkdir /usr/share/xt_geoip
root@ubuntu:/tmp# aptitude -yvV install libtext-csv-xs-perl

Run the build script to create the GeoIP database from the .csv files and place the them in the /usr/share/xt_geoip directory:

root@ubuntu:/tmp# /usr/lib/xtables-addons/xt_geoip_build -D /usr/share/xt_geoip *.csv

Loading the module and configuring Shorewall

Load the kernel module and verify it was loaded:

root@ubuntu:~# modprobe xt_geoip
root@ubuntu:~# lsmod | grep ^xt_geoip
xt_geoip               12775  0 

For verification I'm going to add a rule to reject and log any ping from my virtual machine to any address in the United States. Note, I wouldn't recommend blocking all of U.S. in production, because the IP addresses to Google for other countries are still considered U.S. based. Though whitelisting the U.S. address space may be useful in some environments.

Edit the /etc/shorewall/rules file:

Ping(REJECT):info $FW  net:^[US]
  • The format for referencing countries is ^[<Country Code>].
  • Multiple countries can be specified in the brackets, e.g. ^[US,FR,CA,UK].
  • See http://shorewall.net/ISO-3661.html for more information.

Check and restart Shorewall:

root@ubuntu:/etc/shorewall# shorewall check
root@ubuntu:/etc/shorewall# shorewall restart

Now we'll test ping to www.google.com:

root@ubuntu:/etc/shorewall# ping -n -c 2 www.google.com
PING www.google.com (64.233.185.105) 56(84) bytes of data.
From 192.168.1.100 icmp_seq=1 Destination Host Unreachable
From 192.168.1.100 icmp_seq=1 Destination Host Unreachable

--- www.google.com ping statistics ---
0 packets transmitted, 0 received, +2 errors

Working as intended. Let's see what Shorewall logged:

Feb  6 23:34:27 ubuntu kernel: [  942.806263] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=192.168.1.100 DST=64.233.185.104 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9477 DF PROTO=ICMP TYPE=8 CODE=0 ID=5713 SEQ=1 
Feb  6 23:34:27 ubuntu kernel: [  942.807537] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=192.168.1.100 DST=64.233.185.104 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=9478 DF PROTO=ICMP TYPE=8 CODE=0 ID=5713 SEQ=1 
Feb  6 23:35:26 ubuntu kernel: [ 1002.101682] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=192.168.1.100 DST=64.233.185.105 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63477 DF PROTO=ICMP TYPE=8 CODE=0 ID=5730 SEQ=1 
Feb  6 23:35:26 ubuntu kernel: [ 1002.101924] Shorewall:fw2net:REJECT:IN= OUT=eth0 SRC=192.168.1.100 DST=64.233.185.105 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=63478 DF PROTO=ICMP TYPE=8 CODE=0 ID=5730 SEQ=1 

And for more verification, to see it isn't blocking IP addresses that are non-US owned, we'll ping www.pcengines.ch:

root@ubuntu:/etc/shorewall# ping -n -c 2 www.pcengines.ch
PING www.pcengines.ch (213.133.104.38) 56(84) bytes of data.
64 bytes from 213.133.104.38: icmp_seq=1 ttl=51 time=297 ms
64 bytes from 213.133.104.38: icmp_seq=2 ttl=51 time=296 ms

--- www.pcengines.ch ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 296.200/296.846/297.493/0.845 ms

Conclusion

Using GeoIP support in Shorewall is a much quicker way to blacklist or whitelist large country IP address ranges, when compared to using just ipsets. Though this does not diminish the use cases of ipsets, as they are very useful in many different solutions which I will eventually cover in the future.

References