jump to navigation

Cloud images, qemu, cloud-init and snapd spread tests April 16, 2019

Posted by jdstrand in canonical, ubuntu, ubuntu-server, uncategorized.
add a comment

It is useful for testing to want to work with official cloud images as local VMs. Eg, when I work on snapd, I like to have different images available to work with its spread tests.

The autopkgtest package makes working with Ubuntu images quite easy:

$ sudo apt-get install qemu-kvm autopkgtest
$ autopkgtest-buildvm-ubuntu-cloud -r bionic # -a i386
Downloading https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img...
# and to integrate into spread
$ mkdir -p ~/.spread/qemu
$ mv ./autopkgtest-bionic-amd64.img ~/.spread/qemu/ubuntu-18.04-64.img
# now can run any test from 'spread -list' starting with
# 'qemu:ubuntu-18.04-64:'

This post isn’t really about autopkgtest, snapd or spread specifically though….

I found myself wanting an official Debian unstable cloud image so I could use it in spread while testing snapd. I learned it is easy enough to create the images yourself but then I found that Debian started providing raw and qcow2 cloud images for use in OpenStack and so I started exploring how to use them and generalize how to use arbitrary cloud images.

General procedure

The basic steps are:

  1. obtain a cloud image
  2. make copy of the cloud image for safekeeping
  3. resize the copy
  4. create a seed.img with cloud-init to set the username/password
  5. boot with networking and the seed file
  6. login, update, etc
  7. cleanly shutdown
  8. use normally (ie, without seed file)

In this case, I grabbed the ‘debian-testing-openstack-amd64.qcow2’ image from http://cdimage.debian.org/cdimage/openstack/testing/ and verified it. Since this is based on Debian ‘testing’ (current stable images are also available), when I copied it I named it accordingly. Eg, I knew for spread it needed to be ‘debian-sid-64.img’ so I did:

$ cp ./debian-testing-openstack-amd64.qcow2 ./debian-sid-64.img

I then resized it. I picked 20G since I recalled that is what autopkgtest uses:

$ qemu-img resize debian-sid-64.img 20G

These are already setup for cloud-init, so I created a cloud-init data file (note, the ‘#cloud-config’ comment at the top is important):

$ cat ./debian-data
password: debian
chpasswd: { expire: false }
ssh_pwauth: true

and a cloud-init meta-data file:

$ cat ./debian-meta-data
instance-id: i-debian-sid-64
local-hostname: debian-sid-64

and fed that into cloud-localds to create a seed file:

$ cloud-localds -v ./debian-seed.img ./debian-data ./debian-meta-data

Then start the image with:

$ kvm -M pc -m 1024 -smp 1 -monitor pty -nographic -hda ./debian-sid-64.img -drive "file=./debian-seed.img,if=virtio,format=raw" -net nic -net user,hostfwd=tcp:

(I’m using the invocation that is reminiscent of how spread invokes it; feel free to use a virtio invocation as described by Scott Moser if that better suits your environment.)

Here, the “59355” can be any unused high port. The idea is after the image boots, you can login with ssh using:

$ ssh -p 59355 debian@

Once logged in, perform any updates, etc that you want in place when tests are run, then disable cloud-init for the next boot and cleanly shutdown with:

$ sudo touch /etc/cloud/cloud-init.disabled
$ sudo shutdown -h now

The above is the generalized procedure which can hopefully be adapted for other distros that provide cloud images, etc.

For integrating into spread, just copy the image to ‘~/.spread/qemu’, naming it how spread expects. spread will use ‘-snapshot’ with the VM as part of its tests, so if you want to update the images later since they might be out of date, omit the seed file (and optionally ‘-net nic -net user,hostfwd=tcp:’ if you don’t need port forwarding), and use:

$ kvm -M pc -m 1024 -smp 1 -monitor pty -nographic -hda ./debian-sid-64.img

UPDATE 2019-04-23: the above is confirmed to work with Fedora 28 and 29 (though, if using the resulting image to test snapd, be sure to configure the password as ‘fedora’ and then be sure to ‘yum update ; yum install kernel-modules nc strace’ in the image).

UPDATE 2019-04-22: the above is confirmed to work with CentOS 7 (though, if using the resulting image to test snapd, be sure to configure the password as ‘centos’ and then be sure to ‘yum update ; yum install epel-release ; yum install golang nc strace’ in the image).

UPDATE 2020-02-16: https://wiki.debian.org/ContinuousIntegration/autopkgtest documents how to create autopkgtest images for Debian as ‘$ autopkgtest-build-qemu unstable autopkgtest-unstable.img’. This is confirmed to work on an Ubuntu 20.04 host but broken on an Ubuntu 18.04 host (seems to be https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=922501).

UPDATE 2020-06-27: the above is confirmed to work with Amazon Linux 2. Spread doesn’t yet have this setup for qemu, but the user is ‘ec2-user’. Booting without the seed image has password auth disabled, and the boot is slow due to nfs being enabled and it requires gssproxy to start before it and gssproxy.service doesn’t start due to lack of randomness. To workaround, can login and do (need to investigate and fix correctly via cloud-init (or just remove nfs-utils)):
$ sudo -i sh -c 'echo PasswordAuthentication yes >> /etc/ssh/sshd_config'
$ sudo mkdir /etc/systemd/system/gssproxy.service.d
$ sudo sh -c 'sed "s/Before=.*/Before=/" /usr/lib/systemd/system/gssproxy.service > /etc/systemd/system/gssproxy.service.d/local.conf'
$ sudo systemctl daemon-reload

Extra steps for Debian cloud images without default e1000 networking

Unfortunately, for the Debian cloud images, there were additional steps because spread doesn’t use virtio, but instead the default the e1000 driver, and the Debian cloud kernel doesn’t include this:

$ grep E1000 /boot/config-4.19.0-4-cloud-amd64
# CONFIG_E1000 is not set
# CONFIG_E1000E is not set

So… when the machine booted, there was no networking. To adjust for this, I blew away the image, copied from the safely kept downloaded image, resized then started it with:

$ kvm -M pc -m 1024 -smp 1 -monitor pty -nographic -hda $HOME/.spread/qemu/debian-sid-64.img -drive "file=$HOME/.spread/qemu/debian-seed.img,if=virtio,format=raw" -device virtio-net-pci,netdev=eth0 -netdev type=user,id=eth0

This allowed the VM to start with networking, at which point I adjusted /etc/apt/sources.list to refer to ‘sid’ instead of ‘buster’ then ran apt-get update then apt-get dist-upgrade to upgrade to sid. I then installed the Debian distro kernel with:

$ sudo apt-get install linux-image-amd64

Then uninstalled the currently running kernel with:

$ sudo apt-get remove --purge linux-image-cloud-amd64 linux-image-4.19.0-4-cloud-amd64

(I used ‘dpkg -l | grep linux-image’ to see the cloud kernels I wanted to remove). Removing the package that provides the currently running kernel is a dangerous operation for most systems, so there is a scary message to abort the operation. In our case, it isn’t so scary (we can just try again ;) and this is exactly what we want to do.

Next I cleanly shutdown the VM with:

$ sudo shutdown -h now

and try to start it again like with the ‘general procedures’, above (I’m keeping the seed file here because I want cloud-init to be re-run with the e1000 driver):

$ kvm -M pc -m 1024 -smp 1 -monitor pty -nographic -hda ./debian-sid-64.img -drive "file=./debian-seed.img,if=virtio,format=raw" -net nic -net user,hostfwd=tcp:

Now I try to login via ssh:
$ ssh -p 59355 debian@
debian@'s password:
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Apr 16 16:13:15 2019
debian@debian:~$ sudo touch /etc/cloud/cloud-init.disabled
debian@debian:~$ sudo shutdown -h now
Connection to closed.

While this VM is no longer the official cloud image, it is still using the Debian distro kernel and Debian archive, which is good enough for my purposes and at this point I’m ready to use this VM in my testing (eg, for me, copy ‘debian-sid-64.img’ to ‘~/.spread/qemu’).

Snappy app trust model January 30, 2015

Posted by jdstrand in canonical, security, ubuntu, ubuntu-server, uncategorized.
add a comment

Most of this has been discussed on mailing lists, blog entries, etc, while developing Ubuntu Touch, but I wanted to write up something that ties together these conversations for Snappy. This will provide background for the conversations surrounding hardware access for snaps that will be happening soon on the snappy-devel mailing list.


Ubuntu Touch has several goals that all apply to Snappy:

  • we want system-image upgrades
  • we want to replace the distro archive model with an app store model for Snappy systems
  • we want developers to be able to get their apps to users quickly
  • we want a dependable application lifecycle
  • we want the system to be easy to understand and to develop on
  • we want the system to be secure
  • we want an app trust model where users are in control and express that control in tasteful, easy to understand ways

Snappy adds a few things to the above (that pertain to this conversation):

  • we want the system to be bulletproof (transactional updates with rollbacks)
  • we want the system to be easy to use for system builders
  • we want the system to be easy to use and understand for admins

Let’s look at what all these mean more closely.

system-image upgrades

  • we want system-image upgrades
  • we want the system to be bulletproof (transactional updates with rollbacks)

We want system-image upgrades so updates are fast, reliable and so people (users, admins, snappy developers, system builders, etc) always know what they have and can depend on it being there. In addition, if an upgrade goes bad, we want a mechanism to be able to rollback the system to a known good state. In order to achieve this, apps need to work within the system and live in their own area and not modify the system in unpredictable ways. The Snappy FHS is designed for this and the security policy enforces that apps follow it. This protects us from malware, sure, but at least as importantly, it protects us from programming errors and well-intentioned clever people who might accidentally break the Snappy promise.

app store

  • we want to replace the distro archive model with an app store model
  • we want developers to be able to get their apps to users quickly

Ubuntu is a fantastic distribution and we have a wonderfully rich archive of software that is refreshed on a cadence. However, the traditional distro model has a number of drawbacks and arguably the most important one is that software developers have an extremely high barrier to overcome to get their software into users hands on their own time-frame. The app store model greatly helps developers and users desiring new software because it gives developers the freedom and ability to get their software out there quickly and easily, which is why Ubuntu Touch is doing this now.

In order to enable developers in the Ubuntu app store, we’ve developed a system where a developer can upload software and have it available to users in seconds with no human review, intervention or snags. We also want users to be able to trust what’s in Ubuntu’s store, so we’ve created store policies that understand the Ubuntu snappy system such that apps do not require any manual review so long as the developer follows the rules. However, the Ubuntu Core system itself is completely flexible– people can install apps that are tightly confined, loosely confined, unconfined, whatever (more on this, below). In this manner, people can develop snaps for their own needs and distribute them however they want.

It is the Ubuntu store policy that dictates what is in the store. The existing store policy is in place to improve the situation and is based on our experiences with the traditional distro model and attempts to build something app store-like experiences on top of it (eg, MyApps).

application lifecycle

  • dependable application lifecycle

This has not been discussed as much with Snappy for Ubuntu Core, but Touch needs to have a good application lifecycle model such that apps cannot run unconstrained and unpredictably in the background. In other words, we want to avoid problems with battery drain and slow systems on Touch. I think we’ve done a good job so far on Touch, and this story is continuing to evolve.

(I mention application lifecycle in this conversation for completeness and because application lifecycle and security work together via the app’s application id)


  • we want the system to be secure
  • we want an app trust model where users are in control and express that control in tasteful, easy to understand ways

Everyone wants a system that they trust and that is secure, and security is one of the core tenants of Snappy systems. For Ubuntu Touch, we’ve created a
system that is secure, that is easy to use and understand by users, and that still honors relevant, meaningful Linux traditions. For Snappy, we’ll be adding several additional security features (eg, seccomp, controlled abstract socket communication, firewalling, etc).

Our security story and app store policies give us something that is between Apple and Google. We have a strong security story that has a number of similarities to Apple, but a lightweight store policy akin to Google Play. In addition to that, our trust model is that apps not needing manual review are untrusted by the OS and have limited access to the system. On Touch we use tasteful, contextual prompting so the user may trust the apps to do things beyond what the OS allows on its own (simple example, app needs access to location, user is prompted at the time of use if the app can access it, user answers and the decision is remembered next time).

Snappy for Ubuntu Core is different not only because the UI supports a CLI, but also because we’ve defined a Snappy for Ubuntu Core user that is able to run the ‘snappy’ command as someone who is an admin, a system builder, a developer and/or someone otherwise knowledgeable enough to make a more informed trust decision. (This will come up again later, below)

easy to use

  • we want the system to be easy to understand and to develop on
  • we want the system to be easy to use for system builders
  • we want the system to be easy to use and understand for admins

We want a system that is easy to use and understand. It is key that developers are able to develop on it, system builders able to get their work done and admins can install and use the apps from the store.

For Ubuntu Touch, we’ve made a system that is easy to understand and to develop on with a simple declarative permissions model. We’ll refine that for Snappy and make it easy to develop on too. Remember, the security policy is there not just so we can be ‘super secure’ but because it is what gives us the assurances needed for system upgrades, a safe app store and an altogether bulletproof system.

As mentioned, the system we have designed is super flexible. Specifically, the underlying system supports:

  1. apps working wholly within the security policy (aka, ‘common’ security policy groups and templates)
  2. apps declaring specific exceptions to the security policy
  3. apps declaring to use restricted security policy
  4. apps declaring to run (effectively) unconfined
  5. apps shipping hand-crafted policy (that can be strict or lenient)

(Keep in mind the Ubuntu App Store policy will auto-accept apps falling under ‘1’ and trigger manual review for the others)

The above all works today (though it isn’t always friendly– we’re working on that) and the developer is in control. As such, Snappy developers have a plethora of options and can create snaps with security policy for their needs. When the developer wants to ship the app and make it available to all Snappy users via the Ubuntu App Store, then the developer may choose to work within the system to have automated reviews or choose not to and manage the process via manual reviews/commercial relationship with Canonical.

Moving forward

The above works really well for Ubuntu Touch, but today there is too much friction with regard to hardware access. We will make this experience better without compromising on any of our goals. How do we put this all together, today, so people can get stuff done with snappy without sacrificing on our goals, making it harder on ourselves in the future or otherwise opening Pandora’s box? We don’t want to relax our security policy, because we can’t make the bulletproof assurances we are striving for and it would be hard to tighten the security. We could also add some temporary security policy that adds only certain accesses (eg, serial devices) but, while useful, this is too inflexible. We also don’t want to have apps declare the accesses themselves to automatically adds the necessary security policy, because this (potentially) privileged access is then hidden from the Snappy for Ubuntu Core user.

The answer is simple when we remember that the Snappy for Ubuntu Core user (ie, the one who is able to run the snappy command) is knowledgeable enough to make the trust decision for giving an app access to hardware. In other words, let the admin/developer/system builder be in control.

immediate term

The first thing we are going to do is unblock people and adjust snappy to give the snappy core user the ability to add specific device access to snap-specific security policy. In essence you’ll install a snap, then run a command to give the snap access to a particular device, then you’re done. This simple feature will unblock developers and snappy users immediately while still supporting our trust-model and goals fully. Plus it will be worth implementing since we will likely always want to support this for maximum flexibility and portability (since people can use traditional Linux APIs).

The user experience for this will be discussed and refined on the mailing list in the coming days.

short term

After that, we’ll build on this and explore ways to make the developer and user experience better through integration with the OEM part and ways of interacting with the underlying system so that the user doesn’t have to necessarily know the device name to add, but can instead be given smart choices (this can have tie-ins to the web interface for snappy too). We’ll want to be thinking about hotpluggable devices as well.

Since this all builds on the concept of the immediate term solution, it also supports our trust-model and goals fully and is relatively easy to implement.


Once we have the above in place, we should have a reasonable experience for snaps needing traditional device access. This will give us time to evaluate how people are accessing hardware and see if we can make things even better by using frameworks and/or a hardware abstraction layer. In this manner, snaps can program to an easy to use API and the system can mediate access to the underlying hardware via that API.

Unity and Me April 8, 2011

Posted by jdstrand in ubuntu, uncategorized.

I have been wanting to write this post for a while, and thought I might do it now before Ubuntu releases Natty. :)

I’ll admit when Mark announced that Unity would be the new desktop for Ubuntu, I was skeptical. I always liked the indicators work, but had used an otherwise pretty standard Gnome desktop for years, and liked it. I upgraded to Natty very early in the cycle and have been using Unity for months. Things were quite rocky at first, with instability issues and features either gone or partially implemented. Of course that is to be expected since massive amounts of development work was being done on it. Today, the features are there and Unity is quite stable for me. It is getting very close to release, but AIUI the Unity developers are working very hard to squish the remaining stability bugs. You can read more about the decision to stick with Unity, but that is not what I am writing about today. I’m writing about why as an Ubuntu user I like Unity. Keep in mind, I am not a Unity developer and have just picked things up along the way and this isn’t meant to be an exhaustive list of features or bugs. Just what I like and and dislike.

The Cool
I will certainly miss some great stuff in Unity here, but I’ll mention what I know about:

  • Unity gets out of my way. It reminds me of minimalist desktops from ages ago, but is modern and beautiful
  • It is keyboard friendly
  • The datetime indicator. Not only does it show me the calendar when I click on it, it shows me my appointments for today
  • The global menu. I was pretty sure I didn’t like this at first, but I am actually at a point now where I like it. You always know where to go and it cuts down on wasted space. On my 1440×900 resolution latop screen, this is appreciated (as is the lack of bottom panel)
  • AppIndicators are cool. I’ve always liked them and they are better than ever in Natty. NetworkManager, sound, power, Me, social: all functional, easy to navigate and out of the way.
  • Launcher quicklists. It is easy to add them and they provide cool extra functionality for applications. Right click on evolution or the Applications list to see what I mean.
  • Software search via Alt+F2 or the BFB (aka Big Friendly Button in the upper left– you know, the Ubuntu logo :). While the search results aren’t always perfect, they often are and this functionality is way cool. Eg, try typing in ‘network’ and see what pops up.
  • The applications menu is really neat. Right click on the Launcher icon to see the familiar Applications menu folder entries. Clicking any of them brings you to a place with most recently used applications, installed applications for the category you selected, or suggestions for download.
  • Files and Folders is similarly implemented, with familiar places/categories available via right click. Selecting a place shows you things of that type you’ve edited today, yesterday or within the last week or more.
  • The trash is conveniently located. It doesn’t waste space on the Desktop or clutter the panel and you can drag and drop things onto it.
  • Update (2011/04/15): Superkey shortcuts (just learned about them :). Eg:
    • hold the ‘Super’ (aka ‘meta’) key and the first 10 Launcher icons will show a number. You can then do ‘Super+#’ to launch or bring to focus that application. ‘Super+shift+#’ will launch a second instance of the application.
    • ‘Super+s’ will bring up your workspaces, just as if you click the workspaces launcher. You can then navigate via the arrow keys to choose the workspace to move to.
    • ‘Super+w’ will give you an exploded view of all of your open applications, and again you can use the arrow keys to select the window to bring to focus. It is sorta like Alt+Tab, but with all your open applications.

Tweaks and other things you should know
There are probably also a lot more of these than what I will list here, but I’m just a simple man with simple tastes. ;)

  • To use Unity at all you need a 3D capable card that it supports. Apparently Unity/Compiz uses all kinds of OpenGL and if Unity doesn’t detect that it will run well on your computer, you can use Ubuntu Classic instead. unity-2d might be interesting as an alternative as well for those needing a strictly 2D experience.
  • Middle-clicking on a Launcher will launch another instance of the application
  • Use the power indicator/System Settings to get to the the old ‘System’ menu and preferences in the classic desktop
  • The Files & Folders menu provides similar functionality to the Places menu in the classic desktop
  • If you launch an application via Alt+F2 or the Applications place, you can keep it in the panel for later use by right clicking on the the icon and selecting ‘Keep in Launcher’
  • There is an open bug where when launching an application it shows up under the Launcher, so the Launcher autohides. This can be worked around via CompizConfig Settings Manager (Power Button/System Settings/Personal/CompizConfig Settings Manager, or ‘ccsm’ from a terminal). In ccsm, select Desktop and click on Ubuntu Unity Plugin and select ‘Hide Launcher: Never’. I’m told this bug is being fixed soon, but it might be good to know you can adjust the Launcher behavior anyway. Update (2011/04/09): This is fixed as of today. I confirmed it by setting autohide back to ‘Dodge Windows’ and opening a gagillion terminals. Yes, that’s right, a gagillion. :)
  • You can also adjust the Launcher icon size in ccsm. In the Ubuntu Unity Plugin settings, select the Experimental tab and adjust ‘Launcher icon size’. On my laptop the standard size is good, but on a big monitor, they are too large (might be neat if Unity could detect that somehow….)
  • I sorely missed my weather applet. Well, people stepped up and wrote ‘indicator-weather’
  • ‘Ctrl+Alt+t’ opens a terminal. Apparently this was around for awhile in Gnome but I didn’t know about it. Very handy regardless.
  • Use Alt+F2 to search for and run applications
  • The window selector:
    • right click: zooms in
    • left single click: selects window
    • left double click: selects window and zooms
  • Can drag and drop onto the trash
  • The Files & Folders menu provides similar functionality to the Places menu in the classic desktop
  • Custom launchers can be used, but are not readily supported via a GUI afaict. I use profiles in firefox and wanted to middle click on the firefox Launcher and launch the profile manager. This is how I did that:
    1. Create a new desktop file with:$ mkdir ~/bin/unity
      $ cp /usr/share/applications/firefox.desktop ~/bin/unity
      $ chmod 755 ~/bin/unity/firefox.desktop
    2. Update the ‘Exec=…’ entry in ~/bin/unity/firefox.desktop to have:Exec=firefox -Profile-Manager -no-remote %u
    3. Finally, open the file manager (Nautilus) and drag and drop ~/bin/unity/firefox.desktop onto the Launcher
  • Update (2011/04/10): Custom launchers for terminal applications can also be used, which can be very useful for applications such as irssi and mutt. The thing to remember is that you’ll want to specify a different window manager class for the terminal, otherwise after you start your application via the Launcher, it will show up with all your other terminals and you can’t use a superkey keyboard shortcut with it. For example, to create a launcher to login to another server, you can use something like the following for a .desktop file (see above for how to get this into the Launcher):[Desktop Entry]
    Name=My Server
    Comment=Login to my server
    Exec=gnome-terminal --sm-client-disable --disable-factory --class=MyServer -x ssh -t myserver.example.com
  • Update (2015/04/27): gnome-terminal removed ‘–disable-factory’ (https://bugzilla.gnome.org/show_bug.cgi?id=707899) so if you were relying on it to separate different window manager classes, this will no longer work. ‘mate-terminal’ seems to be a suitable replacement
  • Update (2011/04/09): There are a bunch of other tweaks and questions and answers at askubuntu.com:

The Uncool
Alas, there are a few things I miss and/or people might need to know about:

  • hamster-applet (a time tracker) integration in the panel. Someone is actually working on this, so we may have an implementation soon. Otherwise, simply adding the Time Tracker to the Launcher works ok enough
  • The workspace switcher isn’t as functional as the old Gnome one. It is certainly pretty, but if you use workspaces extensively, you may need to adjust your workflow (eg, by using superkey shortcuts). I actually use workspaces in the way that they are implemented in Unity, so this wasn’t a big deal for me. Work is ongoing in this area, but improved functionality won’t be in Natty release.
  • Unless an application has indicator support, it won’t be integrated into the panel. Things I miss here are the system monitor and sensors-applet. Someone has started work in this area as well. One could go really old school in the meantime (like I did) and use gkrellm to monitor things. There is a certain old-World charm there for such a modern desktop, but it would be nice to have these indicators already available.
  • Stability issues are much improved these days (they better be, we are almost at release! ;) and I haven’t had to use these in several weeks, but am listing them here for completeness:
    1. ‘compiz’ will restart Compiz, the 3D window manager (in the past I would use ‘compiz –replace’, but I’m told that is no longer the correct way)
    2. ‘unity –reset’ will completely reset Unity to shipped defaults and restart it (Note: any ccsm changes will have to be reapplied after running this command)
  • Update (2011/04/09): Focus follows mouse, aka point to focus, aka sloppy focus is not currently supported and using it will likely lead to frustration. Personally, I’ve been click to focus for many years now, so this didn’t get in my way….

I like Unity and to me Unity is clean, easy to use, out of the way, functional and fits my workflows. I didn’t know how much I liked it until I had to stop using it for a week and a half. The release schedule for Unity was very aggressive, but I am happy to say that if the stability bugs are addressed in time, Unity should be a very nice desktop for Natty users. If when trying it out you find a bug, please file it with:$ ubuntu-bug unity no matter where it is in the stack (eg, unity, compiz, global menu, etc). The developers are very responsive and I bet they will be adding more and more bug fixes before release as well as queuing up others for SRU after.


New blog June 26, 2009

Posted by jdstrand in uncategorized.
add a comment

My first post on wordpress.com. Thanks to my family for the snazzy name ‘penguindroppings’. Not sure what all will be here, but hopefully I can put up something useful.

If I can reach just one person…