Mirror, mirror… January 15, 2013

Recently I wanted to mirror traffic from one interface to another. There is a cool utility from Martin Roesch (of Snort fame) called daemonlogger. It’s use is pretty simple (note I created the ‘daemonlogger user and group on my Ubuntu system since I wanted it to drop privileges after starting):
$ sudo /usr/bin/daemonlogger -u daemonlogger \
  -g daemonlogger -i eth0 -o eth1


Sure enough, examing tcpcump output, anything coming in on eth0 was mirrored to eth1. I then used a crossover cable and used tcpdump on the other system and it all worked as advertised. This could be very useful for an IDS.

Then I thought how cool it would be to do this for multiple interfaces, but then I quickly realized that would be confusing for whatever was trying to interpret that traffic. I read that someone used a combination of daemonlogger and vtun for a remote snort monitor to work in EC2, which got me playing….

In my test environment, I again used a crossover cable to connect two systems (let’s call them the ‘server’ and the ‘monitor’). The server system runs vtund. The monitor system then connects as a vtun client. When this happens, tap devices are created on the server and the monitor. The vtund server is setup to start daemonlogger on connect to mirror the traffic from eth0 to the tap0 device. The monitor system can then listen on the tap device and get all the traffic. I then created a second vtun tunnel and mirrored from eth1 to tap1. The monitor could monitor on both tap devices. Neat!

To do accomplish this, I used this /etc/vtund.conf on the server:options {
 type stand;
 port 5000;
 syslog daemon;
 ifconfig /sbin/ifconfig;
}
default {
  type ether;
  proto udp;
  compress no;
  encrypt no;
  multi yes;
  keepalive yes;
  srcaddr {
   iface eth2;
   addr 10.0.3.1;
  };
}
internal {
  password pass0;
  device tap0;
  up {
   ifconfig "%d up";
   program /usr/bin/daemonlogger "-u daemonlogger -g daemonlogger -i eth0 -o %d";
  };
  down {
   ifconfig "%d down";
   program /usr/bin/pkill "-f -x '/usr/bin/daemonlogger -u daemonlogger -g daemonlogger -i eth0 -o %d'";
  };
}
external {
  password pass1;
  device tap1;
  up {
   ifconfig "%d up";
   program /usr/bin/daemonlogger "-u daemonlogger -g daemonlogger -i eth1 -o %d";
  };
  down {
   ifconfig "%d down";
   program /usr/bin/pkill "-f -x '/usr/bin/daemonlogger -u daemonlogger -g daemonlogger -i eth1 -o %d'";
  };
}


Ther server’s /etc/default/vtun has:RUN_SERVER=yes
SERVER_ARGS="-P 5000"


And here is the client /etc/vtund.conf:options {
  port 5000;
  ifconfig /sbin/ifconfig;
}
default {
  type ether;
  proto udp;
  compress no;
  encrypt no;
  persist yes;
}
internal {
  device tap0;
  password pass0;
  up {
   ifconfig "%d up";
  };
  down {
   ifconfig "%d down";
  };
}
external {
  device tap1;
  password pass1;
  up {
   ifconfig "%d up";
  };
  down {
   ifconfig "%d down";
  };
}


The monitor system’s /etc/default/vtun looks like this:CLIENT0_NAME=internal
CLIENT0_HOST=10.0.2.1
CLIENT1_NAME=external
CLIENT1_HOST=10.0.2.1


With the above, I was able to start suricata (a snort-compatible rewrite that is multithreaded) to listen to both tap0 and tap1 on the monitoring system and see all the traffic. Very cool. :)

Now, this isn’t optimized for anything– sending 2 interfaces’ worth of traffic through one interface is likely going to result in droppoed packets to the monitor if the links are busy. Could maybe use compression to help there. Also, the server really needs to have an extra interface for vtun traffic that isn’t mirrored, otherwise you end up mirroring the vtun traffic itself (in my test environment, eth2 on the server used 10.0.2.1 and the montior used 10.0.2.2). Finally, vtun doesn’t offer any meaningful line encryption so if you are doing this for anything resembling production, you’re going to want to use a dedicated network segment between the server and the monitor (a crossover cable worked fine in my test environment).

Enjoy!

OpenSSH and multiple identities November 29, 2012

While this may be old news to some, I only just now figured out how to conveniently use multiple identities in OpenSSH. I have several ssh keys, but only two that I want to use with the agent: one for personal use and one for work. I’d like to be able to not specify which identity to use on the command line most of the time and just use ssh like so:

$ ssh <personal>
$ ssh <work>
$ ssh -i ~/.ssh/other.id_rsa <other>


where the first two use the agent with ~/.ssh/id_rsa and ~/.ssh/work.id_rsa respectively, and the last does not use the agent. ‘man ssh_config’ tells me that the agent looks at all the different IdentityFile configuration directives (in order) and also the IdentitiesOnly option. Therefore, I can set up my ~/.ssh/config to have something like:
# This makes it so that only my default identity
# and my work identity are used by the agent
IdentitiesOnly yes
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/work.id_rsa

# Default to using my work key with work domains
Host *.work1.com *.work2.com
    IdentityFile ~/.ssh/work.id_rsa


With the above, it all works the way I want. Cool! :)

A fun hack: triggered reverse ssh connections for remote access to anywhere November 8, 2010

So, like a lot people, I get asked to install Ubuntu on friend’s and family’s computers. I talk to them about what their use cases are and more often than not (by far) I install it on their systems. That’s cool. What is less cool is tech support for said installation. Not that I mind doing it or that there is a lot to do, but what becomes problematic is when they go home and are sitting behind their NAT router from their ISP, and I can’t just connect to them to fix something I forgot or to help them out of a jam. Before I go any farther let me say that what I am going to describe is never done on machines without the owner’s permission and that I am always upfront in that my account on his/her machine is an administrative account that has access to everything on the system (barring any encryption they use, of course). I should also mention that what I am describing is more in the ‘fun hack that other people might like’ category, and not in the ‘serious systems administration’ category. In other words, this is total crack, but it is fun crack. :)

Now, there are a lot of ways to do this, and I have tried some and surely missed many others. Here are a few I’ve tried:

• Giving realtime tech support over the phone
• Giving realtime tech support over some secure/encrypted chat mechanism
• Email support
• OpenSSH access from my machine to theirs, including adjusting their router for port forwarding
• VPN access from their machine to my network, at which point I can OpenSSH to their machine

Rather than going through a comparison of all the different techniques, let’s just say they all have issues: realtime support almost always entails describing some obscure incantation to fix the problem, which is neither confidence inspiring for them and is extremely time consuming. Email is too slow. Routers get reset and straight OpenSSH doesn’t work when they are away from home. VPN access is not bad, especially with the use of OpenVPN client and server certificates. It has the added benefit of being opt-in by the user, and is easy to use with the network-manager-openvpn-gnome package. It is probably the most legitimate form of access, and should be highly considered, especially for corporate environments. The only real issues are that some draconian networks will block this VPN traffic and that my IP happens to change so they have to fiddle with their connection setting. So I started doing something different.

Remotely triggered reverse OpenSSH connections
The basic idea is this: on the client install OpenSSH, harden it a bit, install a firewall so that no one can connect to it, then create a cron job to poll an HTTP server you have access to for an IP address to connect to, then create a reverse SSH connection to that IP address. If this sounds a little shady and a bit like a botnet, well, you’d be right, but again, I did ask for permission first. :)

So, more specifically, on the remote machine (ie, the one you want to administer):

• Install openssh-server and set /etc/ssh/sshd_config with the following:
  # Force key authentication (ie, no passwords)
PasswordAuthentication no
# Only allow logins to my account
AllowUsers me
Obviously, you will need to copy your ssh key over to this machine (man ssh-copy-id) before restarting OpenSSH and putting the above into effect.
• Setup a firewall. Eg:$ sudo ufw enable
• Create some passwordless ssh keys. Eg:
  $ ssh-keygen -f revssh.id_rsa
• Create a script to poll some HTTP server, then create the reverse connection (this is an abridged script. I’ve omitted error checking and locking for brevity):#!/bin/sh
hname=`hostname | cut -f 1 -d '.'`
ip=`elinks -no-home -dump "http://your.web.server/$hname" | head -1 | awk '{print $1}' | egrep '^[1-9][0-9\.]*$'` || {
#echo "Could not obtain ip" >&2
exit 0
}
echo "Connecting to $ip"
# we use StrictHostKeyChecking=no so that keys gets added without prompting
ssh -oStrictHostKeyChecking=no -i $HOME/.ssh/revssh.id_rsa -p 8080 -NR 3333:localhost:22 revssh@"$ip" sleep 30 || true


To remotely administer a machine, create a file on the server with the name of the hostname of the client to have only the external IP address of your network in it, and then the remote client will pick it up and try to setup a reverse ssh connection on port 8080 (usually open even on the most draconian firewalls, but you could also use 53/tcp, 80/tcp or 443/tcp), after which you can connect to the remote machine with something like:$ ssh -t -p 3333 localhost

Caveats
This is hardly perfect. For one thing, the client is polling an HTTP server so it can easily be man-in-the-middled, but that isn’t a big deal because even if the attacker had full knowledge of this technique, all it gives is a connection to OpenSSH on the client, which is configured to only allow connections from ‘me’ and with my ssh key. This could of course be fixed by connecting with HTTPS and using connection.ssl.cert_verify=1 with elinks. Similarly, the HTTP server could be subverted and under attacker control. For the client, this is no different than the MITM attack in that the attacker really doesn’t have much to work with due to the OpenSSH configuration. Also, the client completely ignores the fingerprint of the server it is connecting to, but again, not huge deal because of our OpenSSH configuration on the client, but you will need to be extra careful in checking fingerprints when connecting to the reverse connection.

You need to remember to update your webserver (ie, just remove the file) so the client isn’t always trying to connect to you. Also, it is somewhat inconvenient that when you logout the reverse connection is still there, but instead of using ‘exit’ to logout of the remote machine, use something like: $ kill -9 `ps auxww | grep [s]sh | grep 3333 | awk '{print $2}'`

While is it relatively easy to setup the client, it is somewhat harder to setup the initiating end. First off, you need to have a webserver that can be accessed by the client. Then you need to have the ssh server on your local machine listen on port 8080 (done either via a port redirect or a separate ssh server). You also need to setup a non-privileged ‘revssh’ user (eg, /bin/false for the shell, a disabled password, etc) on your machine. If you are behind a firewall/router you need to allow connections through your firewall to your machine so that the connection to port 8080 from the client is not blocked. Finally, if you remotely administer multiple clients you will want to keep track of their ssh fingerprints, because when you connect to ‘-p 3333 localhost’ they will conflict with each other (most annoying, but workable). I have written a ‘revssh_allow’ script to automate the above for me (not included, as it is highly site-specific). It will: fire up an sshd server on port 8080 that is specially configured for this purpose, adjust my local firewall to open port 8080/tcp from the client, connect to my router to set up the port redirection to my machine, then poll (via ‘netstat -atn | grep “:3333.*LISTEN”‘) for the connection from the client, then remind me how to connect to the client and how to properly kill the connection.

Summary
So yeah, this is a fun hack. Is it something to put in production? Probably not. Does it work for administering friend’s and family’s computers? Absolutely, but I’ll have to see how well it works over the long haul.

Have fun!