jump to navigation

Mirror, mirror… January 15, 2013

Posted by jdstrand in security, ubuntu.

Recently I wanted to mirror traffic from one interface to another. There is a cool utility from Martin Roesch (of Snort fame) called daemonlogger. It’s use is pretty simple (note I created the ‘daemonlogger user and group on my Ubuntu system since I wanted it to drop privileges after starting):
$ sudo /usr/bin/daemonlogger -u daemonlogger \
  -g daemonlogger -i eth0 -o eth1

Sure enough, examing tcpcump output, anything coming in on eth0 was mirrored to eth1. I then used a crossover cable and used tcpdump on the other system and it all worked as advertised. This could be very useful for an IDS.

Then I thought how cool it would be to do this for multiple interfaces, but then I quickly realized that would be confusing for whatever was trying to interpret that traffic. I read that someone used a combination of daemonlogger and vtun for a remote snort monitor to work in EC2, which got me playing….

In my test environment, I again used a crossover cable to connect two systems (let’s call them the ‘server’ and the ‘monitor’). The server system runs vtund. The monitor system then connects as a vtun client. When this happens, tap devices are created on the server and the monitor. The vtund server is setup to start daemonlogger on connect to mirror the traffic from eth0 to the tap0 device. The monitor system can then listen on the tap device and get all the traffic. I then created a second vtun tunnel and mirrored from eth1 to tap1. The monitor could monitor on both tap devices. Neat!

To do accomplish this, I used this /etc/vtund.conf on the server:options {
 type stand;
 port 5000;
 syslog daemon;
 ifconfig /sbin/ifconfig;
default {
  type ether;
  proto udp;
  compress no;
  encrypt no;
  multi yes;
  keepalive yes;
  srcaddr {
   iface eth2;
internal {
  password pass0;
  device tap0;
  up {
   ifconfig "%d up";
   program /usr/bin/daemonlogger "-u daemonlogger -g daemonlogger -i eth0 -o %d";
  down {
   ifconfig "%d down";
   program /usr/bin/pkill "-f -x '/usr/bin/daemonlogger -u daemonlogger -g daemonlogger -i eth0 -o %d'";
external {
  password pass1;
  device tap1;
  up {
   ifconfig "%d up";
   program /usr/bin/daemonlogger "-u daemonlogger -g daemonlogger -i eth1 -o %d";
  down {
   ifconfig "%d down";
   program /usr/bin/pkill "-f -x '/usr/bin/daemonlogger -u daemonlogger -g daemonlogger -i eth1 -o %d'";

Ther server’s /etc/default/vtun has:RUN_SERVER=yes

And here is the client /etc/vtund.conf:options {
  port 5000;
  ifconfig /sbin/ifconfig;
default {
  type ether;
  proto udp;
  compress no;
  encrypt no;
  persist yes;
internal {
  device tap0;
  password pass0;
  up {
   ifconfig "%d up";
  down {
   ifconfig "%d down";
external {
  device tap1;
  password pass1;
  up {
   ifconfig "%d up";
  down {
   ifconfig "%d down";

The monitor system’s /etc/default/vtun looks like this:CLIENT0_NAME=internal

With the above, I was able to start suricata (a snort-compatible rewrite that is multithreaded) to listen to both tap0 and tap1 on the monitoring system and see all the traffic. Very cool. :)

Now, this isn’t optimized for anything– sending 2 interfaces’ worth of traffic through one interface is likely going to result in droppoed packets to the monitor if the links are busy. Could maybe use compression to help there. Also, the server really needs to have an extra interface for vtun traffic that isn’t mirrored, otherwise you end up mirroring the vtun traffic itself (in my test environment, eth2 on the server used and the montior used Finally, vtun doesn’t offer any meaningful line encryption so if you are doing this for anything resembling production, you’re going to want to use a dedicated network segment between the server and the monitor (a crossover cable worked fine in my test environment).


Application isolation with AppArmor – part I January 11, 2013

Posted by jdstrand in canonical, security, ubuntu.
Tags: , ,
add a comment

A lot of exciting work has been going on with AppArmor and this multipart series will discuss where AppArmor is now, how it is currently used in Ubuntu and how it fits into the larger application isolation story moving forward.

Brief History and Background

AppArmor is a Mandatory Access Control (MAC) system which is a Linux Security Module (LSM) to confine programs to a limited set of resources. AppArmor’s security model is to bind access control attributes to programs rather than to users. AppArmor confinement is provided via profiles loaded into the kernel. AppArmor profiles can be in one of two modes: enforcement and complain. Profiles loaded in enforcement mode will result in enforcement of the policy defined in the profile as well as reporting policy violation attempts (either via syslog or auditd) such that what is not allowed in policy is denied. Profiles in complain mode will not enforce policy but instead report policy violation attempts. AppArmor is typically deployed on systems as a targeted policy where only some (eg high risk) applications have an AppArmor profile defined, but it also supports system wide policy.

Some defining characteristics of AppArmor are that it:

  • is root strong
  • is path-based
  • allows for mixing of enforcement and complain mode profiles
  • uses include files to ease development
  • is very lightweight in terms of resources
  • is easy to learn
  • is relatively easy to audit

AppArmor is an established technology first seen in Immunix and later integrated into SUSE and Ubuntu and their derivatives. AppArmor is also available in Debian, Mandriva, Arch, PLD, Pardus and others. Core AppArmor functionality is in the mainline Linux kernel starting with 2.6.36. AppArmor maintenance and development is ongoing.

An example AppArmor profile

Probably the easiest way to describe what AppArmor does and how it works is to look at an example, in this case the profile for tcpdump on Ubuntu 12.04 LTS:

#include <tunables/global>
/usr/sbin/tcpdump {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/user-tmp>
  capability net_raw,
  capability setuid,
  capability setgid,
  capability dac_override,

  network raw,
  network packet,

  # for -D
  capability sys_module,
  @{PROC}/bus/usb/ r,
  @{PROC}/bus/usb/** r,

  # for finding an interface
  @{PROC}/[0-9]*/net/dev r,
  /sys/bus/usb/devices/ r,
  /sys/class/net/ r,
  /sys/devices/**/net/* r,

  # for tracing USB bus, which libpcap supports
  /dev/usbmon* r,
  /dev/bus/usb/ r,
  /dev/bus/usb/** r,

  # for init_etherarray(), with -e
  /etc/ethers r,

  # for USB probing (see libpcap-1.1.x/
  # pcap-usb-linux.c:probe_devices())
  /dev/bus/usb/**/[0-9]* w,

  # for -z
  /bin/gzip ixr,
  /bin/bzip2 ixr,

  # for -F and -w
  audit deny @{HOME}/.* mrwkl,
  audit deny @{HOME}/.*/ rw,
  audit deny @{HOME}/.*/** mrwkl,
  audit deny @{HOME}/bin/ rw,
  audit deny @{HOME}/bin/** mrwkl,
  owner @{HOME}/ r,
  owner @{HOME}/** rw,

  # for -r, -F and -w
  /**.[pP][cC][aA][pP] rw,

  # for convenience with -r (ie, read 
  # pcap files from other sources)
  /var/log/snort/*log* r,

  /usr/sbin/tcpdump r,

  # Site-specific additions and overrides. See 
  # local/README for details.
  #include <local/usr.sbin.tcpdump>

This profile is representative of traditional AppArmor profiling for a program that processes untrusted input over the network. As can be seen:

  • profiles are simple text files
  • comments are supported in the profile
  • absolute paths as well as file globbing can be used when specifying file access
  • various access controls for files are present. From the profile we see ‘r’ (read), ‘w’ (write), ‘m’ (memory map as executable), ‘k’ (file locking), ‘l’ (creation hard links), and ‘ix’ to execute another program with the new program inheriting policy. Other access rules also exists such as ‘Px’ (execute under another profile, after cleaning the environment), ‘Cx’ (execute under a child profile, after cleaning the environment), and ‘Ux’ (execute unconfined, after cleaning the environment)
  • access controls for capabilities are present
  • access controls for networking are present
  • explicit deny rules are supported, to override other allow rules (eg access to @{HOME}/bin/bad.sh is denied with auditing due to ‘audit deny @{HOME}/bin/** mrwkl,’ even though general access to @{HOME} is permitted with ‘@{HOME}/** rw,’)
  • include files are supported to ease development and simplify profiles (ie #include <abstractions/base>, #include <abstractions/nameservice>, #include <abstractions/user-tmp>, #include <local/usr.sbin.tcpdump>)
  • variables can be defined and manipulated outside the profile (#include <tunables/global> for @{PROC} and @{HOME})
  • AppArmor profiles are fairly easy to read and audit

Complete information on the profile language can be found in ‘man 5 apparmor.d’ as well as the AppArmor wiki.

Updating and creating profiles

AppArmor uses the directory heirarchy as described in policy layout, but most of the time, you are either updating an existing profile or creating a new one and so the files you most care about are in /etc/apparmor.d.

The AppArmor wiki has a lot of information on debugging and updating existing profiles. AppArmor denials are logged to /var/log/kern.log (or /var/log/audit/audit.log if auditd is installed). If an application is misbehaving and you think it is because of AppArmor, check the logs first. If there is an AppArmor denial, adjust the policy in /etc/apparmor.d, then reload the policy and restart the program like so:

$ sudo apparmor_parser -R /etc/apparmor.d/usr.bin.foo
$ sudo apparmor_parser -a /etc/apparmor.d/usr.bin.foo
$ <restart application>

Oftentimes it is enough to just reload the policy without unloading/loading the profile or restarting the application:

$ sudo apparmor_parser -r /etc/apparmor.d/usr.bin.foo

Creating a profile can be done either with tools or by hand. Due to the current pace and focus of development, the tools are somewhat behind and lack some features. It is generally recommended that you profile by hand instead.

When profiling, keep in mind that:

  • AppArmor provides an additional permission check to traditional Discretionary Access Controls (DAC). DAC is always checked in addition to the AppArmor permission checks. As such, AppArmor cannot override DAC to provide more access than what would be normally allowed.
  • AppArmor normalizes path names. It resolves symlinks and considers each hard link as a different access path.
  • AppArmor evaluates file access by pathname rather than using on disk labeling. This eases profiling since AppArmor handles all the labelling behind the scenes.
  • Deny rules are evaluated after allow rules and cannot be overridden by an allow rule.
  • Creation of files requires the create permission (implied by w) on the path to be created. Separate rules for writing to the directory of where the file resides are not required. Deletion works like creation but requires the delete permission (implied by w). Copy requires ‘r’ of the source with create and write at the destination (implied by w). Move is like copy, but also requires delete at source.
  • The profile must be loaded before an application starts for the confinement to take effect. You will want to make sure that you load policy during boot before any confined daemons or applications.
  • The kernel will rate limit AppArmor denials which can cause problems while profiling. You can avoid this be installing auditd or by adjusting rate limiting in the kernel:
    $ sudo sysctl -w kernel.printk_ratelimit=0


There is a lot of documentation on AppArmor (though some is still in progress):

Next time I’ll discuss the specifics of how Ubuntu uses AppArmor in the distribution.

Thanks to Seth Arnold and John Johansen for their review.

OpenSSH and multiple identities November 29, 2012

Posted by jdstrand in security, ubuntu.
add a comment

While this may be old news to some, I only just now figured out how to conveniently use multiple identities in OpenSSH. I have several ssh keys, but only two that I want to use with the agent: one for personal use and one for work. I’d like to be able to not specify which identity to use on the command line most of the time and just use ssh like so:

$ ssh <personal>
$ ssh <work>
$ ssh -i ~/.ssh/other.id_rsa <other>

where the first two use the agent with ~/.ssh/id_rsa and ~/.ssh/work.id_rsa respectively, and the last does not use the agent. ‘man ssh_config’ tells me that the agent looks at all the different IdentityFile configuration directives (in order) and also the IdentitiesOnly option. Therefore, I can set up my ~/.ssh/config to have something like:
# This makes it so that only my default identity
# and my work identity are used by the agent
IdentitiesOnly yes
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/work.id_rsa

# Default to using my work key with work domains
Host *.work1.com *.work2.com
    IdentityFile ~/.ssh/work.id_rsa

With the above, it all works the way I want. Cool! :)

Overfeeding the Planet April 15, 2011

Posted by jdstrand in canonical, security, ubuntu.

I just learned to my chagrin that wordpress.com would update my atom feed every time I clicked the ‘Update’ button in my blog. As a result, I have recently been spamming Planet Ubuntu pretty heavily lately, for which I apologize. Thanks to jcastro and especially nigelb for figuring out that the fix was simply to have the planet use the rss feed instead of the atom feed. While this post/apology could be argued to be yet more spam, I thought I’d pass this information along so others might check their feeds and adjust accordingly. The wiki has been updated for this change, and you can reference the change made to my planet-ubuntu settings in Launchpad if you want to check your settings.

Unity and Me April 8, 2011

Posted by jdstrand in ubuntu, uncategorized.

I have been wanting to write this post for a while, and thought I might do it now before Ubuntu releases Natty. :)

I’ll admit when Mark announced that Unity would be the new desktop for Ubuntu, I was skeptical. I always liked the indicators work, but had used an otherwise pretty standard Gnome desktop for years, and liked it. I upgraded to Natty very early in the cycle and have been using Unity for months. Things were quite rocky at first, with instability issues and features either gone or partially implemented. Of course that is to be expected since massive amounts of development work was being done on it. Today, the features are there and Unity is quite stable for me. It is getting very close to release, but AIUI the Unity developers are working very hard to squish the remaining stability bugs. You can read more about the decision to stick with Unity, but that is not what I am writing about today. I’m writing about why as an Ubuntu user I like Unity. Keep in mind, I am not a Unity developer and have just picked things up along the way and this isn’t meant to be an exhaustive list of features or bugs. Just what I like and and dislike.

The Cool
I will certainly miss some great stuff in Unity here, but I’ll mention what I know about:

  • Unity gets out of my way. It reminds me of minimalist desktops from ages ago, but is modern and beautiful
  • It is keyboard friendly
  • The datetime indicator. Not only does it show me the calendar when I click on it, it shows me my appointments for today
  • The global menu. I was pretty sure I didn’t like this at first, but I am actually at a point now where I like it. You always know where to go and it cuts down on wasted space. On my 1440×900 resolution latop screen, this is appreciated (as is the lack of bottom panel)
  • AppIndicators are cool. I’ve always liked them and they are better than ever in Natty. NetworkManager, sound, power, Me, social: all functional, easy to navigate and out of the way.
  • Launcher quicklists. It is easy to add them and they provide cool extra functionality for applications. Right click on evolution or the Applications list to see what I mean.
  • Software search via Alt+F2 or the BFB (aka Big Friendly Button in the upper left– you know, the Ubuntu logo :). While the search results aren’t always perfect, they often are and this functionality is way cool. Eg, try typing in ‘network’ and see what pops up.
  • The applications menu is really neat. Right click on the Launcher icon to see the familiar Applications menu folder entries. Clicking any of them brings you to a place with most recently used applications, installed applications for the category you selected, or suggestions for download.
  • Files and Folders is similarly implemented, with familiar places/categories available via right click. Selecting a place shows you things of that type you’ve edited today, yesterday or within the last week or more.
  • The trash is conveniently located. It doesn’t waste space on the Desktop or clutter the panel and you can drag and drop things onto it.
  • Update (2011/04/15): Superkey shortcuts (just learned about them :). Eg:
    • hold the ‘Super’ (aka ‘meta’) key and the first 10 Launcher icons will show a number. You can then do ‘Super+#’ to launch or bring to focus that application. ‘Super+shift+#’ will launch a second instance of the application.
    • ‘Super+s’ will bring up your workspaces, just as if you click the workspaces launcher. You can then navigate via the arrow keys to choose the workspace to move to.
    • ‘Super+w’ will give you an exploded view of all of your open applications, and again you can use the arrow keys to select the window to bring to focus. It is sorta like Alt+Tab, but with all your open applications.

Tweaks and other things you should know
There are probably also a lot more of these than what I will list here, but I’m just a simple man with simple tastes. ;)

  • To use Unity at all you need a 3D capable card that it supports. Apparently Unity/Compiz uses all kinds of OpenGL and if Unity doesn’t detect that it will run well on your computer, you can use Ubuntu Classic instead. unity-2d might be interesting as an alternative as well for those needing a strictly 2D experience.
  • Middle-clicking on a Launcher will launch another instance of the application
  • Use the power indicator/System Settings to get to the the old ‘System’ menu and preferences in the classic desktop
  • The Files & Folders menu provides similar functionality to the Places menu in the classic desktop
  • If you launch an application via Alt+F2 or the Applications place, you can keep it in the panel for later use by right clicking on the the icon and selecting ‘Keep in Launcher’
  • There is an open bug where when launching an application it shows up under the Launcher, so the Launcher autohides. This can be worked around via CompizConfig Settings Manager (Power Button/System Settings/Personal/CompizConfig Settings Manager, or ‘ccsm’ from a terminal). In ccsm, select Desktop and click on Ubuntu Unity Plugin and select ‘Hide Launcher: Never’. I’m told this bug is being fixed soon, but it might be good to know you can adjust the Launcher behavior anyway. Update (2011/04/09): This is fixed as of today. I confirmed it by setting autohide back to ‘Dodge Windows’ and opening a gagillion terminals. Yes, that’s right, a gagillion. :)
  • You can also adjust the Launcher icon size in ccsm. In the Ubuntu Unity Plugin settings, select the Experimental tab and adjust ‘Launcher icon size’. On my laptop the standard size is good, but on a big monitor, they are too large (might be neat if Unity could detect that somehow….)
  • I sorely missed my weather applet. Well, people stepped up and wrote ‘indicator-weather’
  • ‘Ctrl+Alt+t’ opens a terminal. Apparently this was around for awhile in Gnome but I didn’t know about it. Very handy regardless.
  • Use Alt+F2 to search for and run applications
  • The window selector:
    • right click: zooms in
    • left single click: selects window
    • left double click: selects window and zooms
  • Can drag and drop onto the trash
  • The Files & Folders menu provides similar functionality to the Places menu in the classic desktop
  • Custom launchers can be used, but are not readily supported via a GUI afaict. I use profiles in firefox and wanted to middle click on the firefox Launcher and launch the profile manager. This is how I did that:
    1. Create a new desktop file with:$ mkdir ~/bin/unity
      $ cp /usr/share/applications/firefox.desktop ~/bin/unity
      $ chmod 755 ~/bin/unity/firefox.desktop
    2. Update the ‘Exec=…’ entry in ~/bin/unity/firefox.desktop to have:Exec=firefox -Profile-Manager -no-remote %u
    3. Finally, open the file manager (Nautilus) and drag and drop ~/bin/unity/firefox.desktop onto the Launcher
  • Update (2011/04/10): Custom launchers for terminal applications can also be used, which can be very useful for applications such as irssi and mutt. The thing to remember is that you’ll want to specify a different window manager class for the terminal, otherwise after you start your application via the Launcher, it will show up with all your other terminals and you can’t use a superkey keyboard shortcut with it. For example, to create a launcher to login to another server, you can use something like the following for a .desktop file (see above for how to get this into the Launcher):[Desktop Entry]
    Name=My Server
    Comment=Login to my server
    Exec=gnome-terminal --sm-client-disable --disable-factory --class=MyServer -x ssh -t myserver.example.com
  • Update (2015/04/27): gnome-terminal removed ‘–disable-factory’ (https://bugzilla.gnome.org/show_bug.cgi?id=707899) so if you were relying on it to separate different window manager classes, this will no longer work. ‘mate-terminal’ seems to be a suitable replacement
  • Update (2011/04/09): There are a bunch of other tweaks and questions and answers at askubuntu.com:

The Uncool
Alas, there are a few things I miss and/or people might need to know about:

  • hamster-applet (a time tracker) integration in the panel. Someone is actually working on this, so we may have an implementation soon. Otherwise, simply adding the Time Tracker to the Launcher works ok enough
  • The workspace switcher isn’t as functional as the old Gnome one. It is certainly pretty, but if you use workspaces extensively, you may need to adjust your workflow (eg, by using superkey shortcuts). I actually use workspaces in the way that they are implemented in Unity, so this wasn’t a big deal for me. Work is ongoing in this area, but improved functionality won’t be in Natty release.
  • Unless an application has indicator support, it won’t be integrated into the panel. Things I miss here are the system monitor and sensors-applet. Someone has started work in this area as well. One could go really old school in the meantime (like I did) and use gkrellm to monitor things. There is a certain old-World charm there for such a modern desktop, but it would be nice to have these indicators already available.
  • Stability issues are much improved these days (they better be, we are almost at release! ;) and I haven’t had to use these in several weeks, but am listing them here for completeness:
    1. ‘compiz’ will restart Compiz, the 3D window manager (in the past I would use ‘compiz –replace’, but I’m told that is no longer the correct way)
    2. ‘unity –reset’ will completely reset Unity to shipped defaults and restart it (Note: any ccsm changes will have to be reapplied after running this command)
  • Update (2011/04/09): Focus follows mouse, aka point to focus, aka sloppy focus is not currently supported and using it will likely lead to frustration. Personally, I’ve been click to focus for many years now, so this didn’t get in my way….

I like Unity and to me Unity is clean, easy to use, out of the way, functional and fits my workflows. I didn’t know how much I liked it until I had to stop using it for a week and a half. The release schedule for Unity was very aggressive, but I am happy to say that if the stability bugs are addressed in time, Unity should be a very nice desktop for Natty users. If when trying it out you find a bug, please file it with:$ ubuntu-bug unity no matter where it is in the stack (eg, unity, compiz, global menu, etc). The developers are very responsive and I bet they will be adding more and more bug fixes before release as well as queuing up others for SRU after.


A bite out of the Apple December 27, 2010

Posted by jdstrand in ubuntu.

Being the “computer guy” for friends and family is both demanding and rewarding. On the one hand, I get to install Ubuntu on lots of computers. On the other hand, I end up having to deal with various hardware and devices. For many years I have successfully avoided Apple’s proprietary products, but recently I’ve had to deal with them. Let me first say that the hardware I’ve gotten my hands on is excellent, and let me then say that having worked with open source for so long, it is surprising at how hard Apple works at making sure you are locked into using only their stuff. Also, am I the only one who finds their faux-ubiquity in TV and movies annoying? I mean, how is it that virtually every laptop, monitor and computer has the Apple logo emblazoned on it? Clearly, their marketing is at least as excellent as their hardware, but I digress….

After an annoying foray into the world of the MacBook Pro for a friend (btw, with all the Ubuntu developers I see at UDS with Apple hardware, why isn’t it better supported in the default install? I mean, I felt like I stepped back 5 years with all the twiddling I had to do with the silly touchpad to get a halfway sane configuration — I’d fix it in Ubuntu myself, but the device is out of my hands now), I finally was confronted with trying to get a new, off-the-shelf iPod Touch to work with Ubuntu 10.10. Pretty and shiny as it was, let’s just say it did not work well with Ubuntu out of the box. Looking at https://help.ubuntu.com/community/PortableDevices/iPod and the video at http://www.libimobiledevice.org/ I thought it was going to be relatively easy, but there were several roadblocks:

  1. Currently, you can’t just open a new iPod and use it with a non-iTunes system (ie, Windows and OS X — last I checked, the latest iTunes doesn’t work too well in Wine). As soon as you plug it into a USB port, there is a helpful graphic telling you that you need to connect to iTunes. This step seems to be required to download the latest firmware for the device and register it.
  2. In addition to the above, you must do an initial music sync from iTunes to initialize the music databases on the device.
  3. This new iPod Touch only does USB 2.0 and won’t fallback to 1.1 — a frustrating tactic that works quite well at driving revenue since people all of a sudden feel that their perfectly good computer yesterday is old and outdated after getting their shiny new iPod today. Ranting aside, qemu-kvm (my preferred virtualization technology) does not provide a USB 2.0 host controller.
  4. The version of usbmuxd in Maverick (Ubuntu 10.10) is too old. Among other things, it installs a udev rule to start usbmuxd which is used to coordinate the communications between the device and Ubuntu (see the README file in /usr/share/doc/usbmuxd for a detailed explanation of an IMHO too complex protocol). The udev rule for Maverick did not catch the idProduct of my device (0x129e) and would not start usbmuxd.
  5. While hand-editing /lib/udev/rules.d/85-usbmuxd.rules and running ‘sudo udevadm control –reload-rules’ did get udev to start usbmuxd, usbmuxd didn’t know how to deal with my device. I guess there was a reason the udev rule was so specific to begin with. ;)
  6. Apple’s latest version of iOS (4.2.1) broke stuff for Linux users. AIUI, libimobiledevice is the (excellent) library that pulls all the disparate parts of dealing with iPods together and makes it so that music players like Rhythmbox work seamlessly with with the device. Upstream fixed the 4.2.1 problem quickly, but the version of libimobiledevice in Maverick is too old.
  7. As of the time of this writing, DbVersion 5 is not supported by libgpod, so after drag and dropping files in Rhythmbox to the device, the iTunes database on the iPod is not updated and the files don’t show up. This is also being tracked in Ubuntu as LP: #601251.

See what I mean about Apple trying very hard to make sure that their devices work with only their stuff? Merry Christmas Jamie, you get to fiddle with several highly proprietary systems for hours on end! *Sigh*. Of course this isn’t Ubuntu’s fault; Apple doesn’t provide iTunes for Linux and they keep changing things in incompatible ways while the open source developers furiously try to keep up (admirably).

So what did I do to get the thing to work? Well, I didn’t because of the libgpod DbVersion 5 support. That said, I am to the point now where I resolved everything except this, so when the new libgpod comes out, I’ll be ready. Here is what I did to address each of the above:

  • I didn’t have a Windows install handy, but do have a legal installation disk for Windows XP. I installed it via virt-manager, but ran into a bug where the domain xml used ‘kvm’ as the emulator instead of qemu-system-i386, so when I rebooted the machine after stage 1 of the install, I got a black screen (I still need to file a bug on this). Not sure if it is still required in Maverick, but I also disabled all the <features/> except <pae/> (ie, ACPI and APIC). I installed iTunes, plugged the device in and… nothing. qemu-kvm doesn’t provide a USB 2.0 controller. D’oh! Now, I could conceivably do PCI passthrough for the USB 2.0 controller on the host, but that required fiddling with the pci-stub kernel module so that the controller isn’t being used by the ehci_hcd module on the host. I’d like to play with this more, but haven’t found good documentation on it yet.
  • At this point, I still didn’t have Windows handy for what I needed. The non-OSE version of VirtualBox does provide a USB 2.0 controller if you also install the ‘Oracle_VM_VirtualBox_Extension_Pack-4.0.0-69151.vbox-extpack’. So I removed the kvm* modules from the kernel, installed VirtualBox, installed the extension pack, installed Windows XP in the guest, installed iTunes, plugged the device in, then told VirtualBox to make it available in the guest. This got me to some sort of wizard to install the firmware and register the device. Using iTunes on Windows XP in VirtualBox worked flawlessly.
  • From within iTunes, I did a music sync, ejected the device, then shutdown the VM
  • In Ubuntu, I added Paul McEnery’s ppa and performed an upgrade. This got me several library updates, but not everything (Update: it now has everything but a new libgpod). In particular, you need (all from the ppa, except where noted):
    • usbmuxd 1.0.6
    • libimobiledevice 1.0.4 (and be sure to install libimobiledevice-utils)
    • ipheth (for tethering, not strictly needed)
    • A version of libgpod4 that supports DbVersion 5 (in progress)

Now when I plug in the device in Ubuntu, I am prompted to open F-Spot and Rhythmbox, the device is mounted via gvfs, the device shows up on the Desktop and I can read anything on the device via nautilus. ideviceinfo works, and presumably so do all the other idevice* utilities. I can also copy files to the device via nautilus (though make sure you see the ‘Sync in Progress’ message on the iPod before unplugging!), but they don’t show up in the iPod (because I am not using libgpod). Now I just need a new libgpod to support the DbVersion 5 database so it all works in Rhythmbox (I can use iTunes in the VirtualBox VM using a shared folder to add files to the device in the meantime). Beyond getting Rhythmbox to work, I’d like to be able to copy videos and pictures to the device. Maybe once libgpod is in order I can use gtkpod (there are also notes in the aforementioned wiki documentation). If I can get this to all work right, I may document all this step by step.

While the person I am setting this up for is generally happy (and patiently waiting for proper Rhythmbox support), this whole experience has felt more like Apple took a bite out of me. Here’s to hoping this blog helps others with new iPods!

Update (2010/12/28): While I haven’t personally tried any of the following, I’m told:

Update (2010/12/29): As pointed out by Christophe in the comments, ideviceactivate does not work with the iTouch 4G, but patches have been submitted. Also, he says you only need to do the ihash/sync music step with DbVersion 4 devices, so other devices will be ok.

Show the date in Natty’s Unity December 13, 2010

Posted by jdstrand in ubuntu.

By default in Unity on Natty the clock only shows the time but doesn’t provide a GUI method to display the date. Since I never seem to know the date, and only sometimes the day, I like to have my clock display these to me. This is bug #601974. The quick fix:

$ gsettings set org.ayatana.indicator.datetime show-date true
$ gsettings set org.ayatana.indicator.datetime show-day true

Since this was my first foray into gsettings, I figured I’d also post how I figured this out. First I looked at the output of:

$ gsettings help

Then I drilled down until I found what I needed:

$ gsettings list-schemas|grep datetime
$ gsettings list-keys org.ayatana.indicator.datetime

$ gsettings get org.ayatana.indicator.datetime show-date
$ gsettings set org.ayatana.indicator.datetime show-date true
$ gsettings get org.ayatana.indicator.datetime show-date

I filed bug #689722 to request exposing this via the GUI.

Update: dconf-editor from dconf-tools can be used to edit gsettings in a graphical way (like gconf-editor for gconf). Thanks to mdeslaur for pointing this out! :)

Update: The gsettings path moved to /com/canonical/indicator/datetime in recent indicator-datetime versions. See bug #717318 for details.

Compiz vs Ubuntu Classic Desktop December 1, 2010

Posted by jdstrand in ubuntu.

I am running the development version of Ubuntu (the Natty Narwhal). I’ve tried the Unity desktop (and will continue to do so) but for reasons I won’t go into here, I need to use the Ubuntu Classic Desktop for now. After today’s update I could no longer login to a functional Ubuntu Classic Desktop because of bug #683686. There were a number of things that went wrong and I wasted an hour on trying to work it out (thank you to didrocks and seb128 for helping me). Here is what I’ve learned:

  • Do not disable the Unity plugin in CompizConfig Settings Manager while in Unity (or enable it when not in Unity)
  • To use Unity, login to GDM with ‘Ubuntu Desktop’
  • To use the traditional desktop, login to GDM with ‘Ubuntu Classic Desktop’
  • If after logging in to the Ubuntu Classic Desktop your window manager does not start, this might be bug #683686. To work around it, logout, move your ~/.config/compiz-1 aside (logging into a console first), then log back in with GDM like normal. This bug is actively being worked on.
  • There is a known bug with compiz and the gnome-panel that may cause applets to not load. Logging out and back in again usually solves this. This bug is actively being worked on.

Hope this helps anyone suffering from the same problems I did. Please file bugs if you are having other problems with Compiz, Unity or the Ubuntu Classic Desktop.

A fun hack: triggered reverse ssh connections for remote access to anywhere November 8, 2010

Posted by jdstrand in security, ubuntu.

So, like a lot people, I get asked to install Ubuntu on friend’s and family’s computers. I talk to them about what their use cases are and more often than not (by far) I install it on their systems. That’s cool. What is less cool is tech support for said installation. Not that I mind doing it or that there is a lot to do, but what becomes problematic is when they go home and are sitting behind their NAT router from their ISP, and I can’t just connect to them to fix something I forgot or to help them out of a jam. Before I go any farther let me say that what I am going to describe is never done on machines without the owner’s permission and that I am always upfront in that my account on his/her machine is an administrative account that has access to everything on the system (barring any encryption they use, of course). I should also mention that what I am describing is more in the ‘fun hack that other people might like’ category, and not in the ‘serious systems administration’ category. In other words, this is total crack, but it is fun crack. :)

Now, there are a lot of ways to do this, and I have tried some and surely missed many others. Here are a few I’ve tried:

  • Giving realtime tech support over the phone
  • Giving realtime tech support over some secure/encrypted chat mechanism
  • Email support
  • OpenSSH access from my machine to theirs, including adjusting their router for port forwarding
  • VPN access from their machine to my network, at which point I can OpenSSH to their machine

Rather than going through a comparison of all the different techniques, let’s just say they all have issues: realtime support almost always entails describing some obscure incantation to fix the problem, which is neither confidence inspiring for them and is extremely time consuming. Email is too slow. Routers get reset and straight OpenSSH doesn’t work when they are away from home. VPN access is not bad, especially with the use of OpenVPN client and server certificates. It has the added benefit of being opt-in by the user, and is easy to use with the network-manager-openvpn-gnome package. It is probably the most legitimate form of access, and should be highly considered, especially for corporate environments. The only real issues are that some draconian networks will block this VPN traffic and that my IP happens to change so they have to fiddle with their connection setting. So I started doing something different.

Remotely triggered reverse OpenSSH connections
The basic idea is this: on the client install OpenSSH, harden it a bit, install a firewall so that no one can connect to it, then create a cron job to poll an HTTP server you have access to for an IP address to connect to, then create a reverse SSH connection to that IP address. If this sounds a little shady and a bit like a botnet, well, you’d be right, but again, I did ask for permission first. :)

So, more specifically, on the remote machine (ie, the one you want to administer):

  • Install openssh-server and set /etc/ssh/sshd_config with the following:
    # Force key authentication (ie, no passwords)
    PasswordAuthentication no
    # Only allow logins to my account
    AllowUsers me
    Obviously, you will need to copy your ssh key over to this machine (man ssh-copy-id) before restarting OpenSSH and putting the above into effect.
  • Setup a firewall. Eg:$ sudo ufw enable
  • Create some passwordless ssh keys. Eg:
    $ ssh-keygen -f revssh.id_rsa
  • Create a script to poll some HTTP server, then create the reverse connection (this is an abridged script. I’ve omitted error checking and locking for brevity):#!/bin/sh
    hname=`hostname | cut -f 1 -d '.'`
    ip=`elinks -no-home -dump "http://your.web.server/$hname" | head -1 | awk '{print $1}' | egrep '^[1-9][0-9\.]*$'` || {
    #echo "Could not obtain ip" >&2
    exit 0
    echo "Connecting to $ip"
    # we use StrictHostKeyChecking=no so that keys gets added without prompting
    ssh -oStrictHostKeyChecking=no -i $HOME/.ssh/revssh.id_rsa -p 8080 -NR 3333:localhost:22 revssh@"$ip" sleep 30 || true

To remotely administer a machine, create a file on the server with the name of the hostname of the client to have only the external IP address of your network in it, and then the remote client will pick it up and try to setup a reverse ssh connection on port 8080 (usually open even on the most draconian firewalls, but you could also use 53/tcp, 80/tcp or 443/tcp), after which you can connect to the remote machine with something like:$ ssh -t -p 3333 localhost

This is hardly perfect. For one thing, the client is polling an HTTP server so it can easily be man-in-the-middled, but that isn’t a big deal because even if the attacker had full knowledge of this technique, all it gives is a connection to OpenSSH on the client, which is configured to only allow connections from ‘me’ and with my ssh key. This could of course be fixed by connecting with HTTPS and using connection.ssl.cert_verify=1 with elinks. Similarly, the HTTP server could be subverted and under attacker control. For the client, this is no different than the MITM attack in that the attacker really doesn’t have much to work with due to the OpenSSH configuration. Also, the client completely ignores the fingerprint of the server it is connecting to, but again, not huge deal because of our OpenSSH configuration on the client, but you will need to be extra careful in checking fingerprints when connecting to the reverse connection.

You need to remember to update your webserver (ie, just remove the file) so the client isn’t always trying to connect to you. Also, it is somewhat inconvenient that when you logout the reverse connection is still there, but instead of using ‘exit’ to logout of the remote machine, use something like: $ kill -9 `ps auxww | grep [s]sh | grep 3333 | awk '{print $2}'`

While is it relatively easy to setup the client, it is somewhat harder to setup the initiating end. First off, you need to have a webserver that can be accessed by the client. Then you need to have the ssh server on your local machine listen on port 8080 (done either via a port redirect or a separate ssh server). You also need to setup a non-privileged ‘revssh’ user (eg, /bin/false for the shell, a disabled password, etc) on your machine. If you are behind a firewall/router you need to allow connections through your firewall to your machine so that the connection to port 8080 from the client is not blocked. Finally, if you remotely administer multiple clients you will want to keep track of their ssh fingerprints, because when you connect to ‘-p 3333 localhost’ they will conflict with each other (most annoying, but workable). I have written a ‘revssh_allow’ script to automate the above for me (not included, as it is highly site-specific). It will: fire up an sshd server on port 8080 that is specially configured for this purpose, adjust my local firewall to open port 8080/tcp from the client, connect to my router to set up the port redirection to my machine, then poll (via ‘netstat -atn | grep “:3333.*LISTEN”‘) for the connection from the client, then remind me how to connect to the client and how to properly kill the connection.

So yeah, this is a fun hack. Is it something to put in production? Probably not. Does it work for administering friend’s and family’s computers? Absolutely, but I’ll have to see how well it works over the long haul.

Have fun!

What I Do September 15, 2010

Posted by jdstrand in ubuntu.

I’m often asked, “So Jamie, what do you do?” I find my answer is usually quite different depending on whom I am talking to. Normally I say something fairly bland like, “I’m a security engineer for Ubuntu, which is a Free alternative to Windows and Mac.” I try to say something about freedom and beer, but really by the time I get to the word ‘engineer,’ many people’s eyes go glassy (maybe they’re tearing up at the thought of working on free software for a living and I am just not empathetic enough to notice). There might be a follow-up question or two and I may even offer a free CD, but usually the response is a simple, “Oh, you work with computers?”

Yes, I work with computers.

The truth is I would love to talk in depth about what I do with people who ask, so when my employer asked people to blog about what they do, I was pretty stoked. So where to start? How about where I got started.

I started using Free Software in 1996, when I went back to school to expand my education. Not long after that, my wife was pregnant and I found myself needing a way to work on my school assignments from home. My computer graphics professor gave me the new RedHat 5.0. I went straight home, installed it and was hooked. A little while later I installed a pre-release version of Debian Slink. Like many others, I loved Debian’s package management, its policy and how it is community-based. These gifts of Free Software and the community around them were, and still are, very meaningful for me.

Fast forward a few years and you’ll find me setting up a business with Debian Woody. Back then Debian stable still had Gnome 1.4, so I was keen on finding a newer desktop on top of the reliable, stable and secure foundation that I admired in Debian. I found Gustavo Noronha Silva’s unofficial Gnome 2 packages, but I really wanted Gnome 2.2. He didn’t plan on providing 2.2 packages, so I took up that work by providing a full, modern desktop including Xfree86, evolution, Mozilla, and a whole lot more. I realized that I had a pretty good thing going and thought others could benefit, so I released this as the Gnome 2.2 Backport for Debian Woody. I provided security support and an upgrade path for the backport for more than 3 years until Woody’s end of life. During this same time I developed an intense interest in secure servers which led me to consulting and a strong advocacy of Free Software. These experiences helped me understand how much good you can bring to people by working on Free Software.

In 2007 I was interviewed for a position at Canonical and I’ll never forget Matt Zimmerman’s question in my interview: “What will stop you from quitting a year from now and going back to consulting?” Though I did not expect this question, the answer was immediate: “Because I know how much of an impact Free Software can have and I won’t have the opportunity to help more people than with Ubuntu.”

These days, I get paid by Canonical to work with computers.

As an Ubuntu Security engineer, I am on a team of people who are responsible for tending to known threats against Ubuntu. We track security vulnerabilities, triage bugs, interact with upstreams, coordinate with other vendors, sponsor patches from the community, liaise with upstreams and vendors on behalf of researchers, analyze vulnerabilities, add to the Debian CVE tracker and of course fix security bugs in Ubuntu. Quality assurance is an integral part of fixing a bug that can land on millions of users’ desktops, so I helped start and regularly participate in the QA Regression Testing (QRT) project. In addition to helping our team prevent regressions in our updates, it is regularly used by the Ubuntu QA team to test the development release and in stable bug fix updates. The scripts in QRT have on several occasions found bugs in software in our development release that led to upstream and/or Debian bug reports and fixes. I also regularly update the Ubuntu CVE Tracker and security team tools for tracking, building and publishing security updates.

Another part of what I do is help develop security features, tools and documentation for Ubuntu. I am the principal author of the Uncomplicated Firewall (ufw) which aims to help people unfamiliar with firewall concepts be safer while helping seasoned administrators get their job done faster. It’s the default firewall for Ubuntu and included in other distributions such as Debian and Arch Linux. Several projects have popped up around ufw and provide graphical frontends, and I coordinate features and bug fixes in ufw with those projects.

I have joined the AppArmor project. AppArmor is the default Mandatory Access Control (MAC) system in Ubuntu and OpenSUSE, and thanks to to the tireless work of John Johansen and many others, is now included in the mainline Linux kernel. My upstream focus is on AppArmor testing, profiling, documentation, userspace tools and ease of distribution integration. In addition, I regularly participate in upstream planning discussions and meetings. For Ubuntu, I have authored many of the profiles in Ubuntu and regularly provide testing, bug fixes and new versions of AppArmor in Ubuntu.

I’ve also authored a few smaller applications like auth-client-config, openssl-blacklist, and openvpn-blacklist. auth-client-config is a program for modifying nsswitch.conf and pam configuration, but has largely been superseded in Ubuntu by pam-auth-update. The openssl-blacklist and openvpn-blacklist tools and lists were developed by me to detect known-bad RSA keys, and are included in Debian. I’ve had patches accepted by upstream for random software such as Gnucash and Gourmet, and have submitted many patches to Debian over the years.

I use virtualization for much of my development work and testing, so I regularly triage and fix bugs in libvirt and other parts of the virtualization stack with the Ubuntu Server team. I wrote and regularly maintain the AppArmor security driver in upstream libvirt. In Ubuntu I tend to focus on libvirt’s bug triage, AppArmor integration, merges with Debian, and testing. Writing much of the vm-tools in the Ubuntu QA Tools, I hope these scripts help anyone be more efficient when trying to manipulate several machines at one time, such as when performing ISO testing or testing a patch on many different operating systems.

When not working at home, I might be at a conference such as the Ubuntu Developer Summit (UDS), where I collaborate with people from all over the world in the Ubuntu community and upstreams to help plan and implement new features with security in mind. I’ve also attended security conferences such as DefCon and BlackHat.

Yes, I work with computers and am happy for it! On any given day I might publish an update, audit a piece of software, discuss a vulnerability with upstream, submit a profile to AppArmor, forward a patch to Debian, plan a ufw feature, test and refine a security fix with other vendors, triage and comment on a security vulnerability, write up some documentation, develop a test script, and/or fire up a bunch of virtual machines. What I do is fun, challenging… satisfying. It is hugely rewarding working on Free Software with so many talented and intelligent individuals in Canonical, the Ubuntu community, and the upstreams I interact with every day. I am blessed to work with these fantastic people who continually inspire me to stretch to learn and do more. I believe by working together on Free Software all of us are in our own way changing the world for the better. That’s why I do what I do.