Loop-AES blues on Debian testing

Fri, 22 Apr 2011 23:33:44 -0500

Tags: debian, sysadmin, encryption


Update: The situation seems to have improved, see this comment.

I use both in my desktop and laptop Debian testing (currently codenamed 'wheezy'). For backups, I have been using an AES256 encrypted ZFS external USB hard drive, with date-based snapshots and deduplication. I have two hard drives, one of which I keep off-site, doing weekly (or so) backups. I have all the same set-up in my laptop, and I have done some drill back-up recoveries from it (from the latest backup, not from the snapshots, as zfs-fuse doesn't support .snapshot folder, yet).

This solution has been working quite well until I went to Argentina for teaching. There, while I upgraded my Debian testing on the laptop, FUSE stop working properly in strange ways. When I got back to Canada and tried to backup my desktop, zfs-fuse was also non-working. The problem took me quite a bit of work to fix, I'm documenting here the problem and its solution (in a nutshell: loop-AES, in the package loop-aes-utils, diverts /bin/mount to a version that is incompatible with FUSE, solution: don't use loop-AES and move to dm-crypt).


In Debian there are several "flavors" for the distribution: stable, testing and unstable. Stable is well-tested, with timely security updates and all around rock solid (or that is the expectation at least). Nothing new happens in Debian stable for about year and a half to two years. In the mid-time, all the new packages, testing of new versions, etc., happens in Debian unstable (Debian 'sid'). When new versions of packages are uploaded to unstable, they automatically migrate into testing if there are no serious bugs filled against the upload in those first two weeks. I have been using testing for years and it is overall a much "happier" experience that testing as it usually Just Works (TM) and has very recent software.

Interestingly, when I start getting more involved in Debian, I realized testing is a little bit the worst of both worlds: you get no security updates (as in stable) and you get no interest from the developers (as in unstable). If you try to submit a bug against a package in testing, chances are you'll be asked to upgrade to unstable and see what happens. In general, submitting a bug against a package in testing when there is a newer version in unstable is a waste of everybody's time (and counter-intuitive, as the distro is called 'testing'!). Still, once in a blue moon a horrible update comes in unstable and the system breaks in a horrible way. I will stick to testing until I am comfortable with checking #debian-devel in OFTC before doing an upgrade. But over time using testing is becoming more and more of a nuisance. And this issue of testing vs. unstable is very related to my loop-AES blues, keep reading.

As usual in the FLOSS world, there is a variety of competing technologies to do anything, in this case disk encryption. I have had encrypted containers in my machine for a while and when I started using them, loop-AES was a recommended solution. The way loop-AES works is by "butchering" the loop functionality in the kernel (the functionality that allows you to mount a file as if it were a device) with extra encryption-fu. Therefore, you make sure your kernel has the loop device as a module, install the Debian package loop-aes-source, then use the module assistant to create a custom Debian package with the new loop.ko, install it and then you are ready to rumba. Of course, that does it at the kernel level, so you need modified user-space tools (mount, losetup, swapon, etc) to access this new functionality (selecting encryption algorithm, setting up keys, encryption parameters, etc). These modified versions of existing tools comes in the package loop-aes-utils, which diverts /bin/mount (and others) to their modified versions.

Loop-AES is half the story, the other half is FUSE, File System in User-space, a kernel module that allows you to have the file system driver in, well, user-space. Even though it doesn't have particularly good press, two file systems I rely on are FUSE-based: NTFS-3G and ZFS-FUSE. The NT file system moved out of the kernel due to a number of issues (the driver in Linux 2.4 caused data corruption and NTFS is structured as a database, ensuring data consistency is easier using recursion, which is usually unacceptable in kernel code). On the other hand, ZFS is Free Software, developed by then SUN Microsystems but its license under the CDDL, which is GPL-incompatible. There, using FUSE allows for a technical solution to a legal problem. The original ZFS code is kernel-quality, so the Debian version based on a FreeBSD kernel, Debian/kFreeBSD incorporates ZFS directly in the kernel (and this relates to a last-ditch solution I considered but didn't have to try).

The Problem

My loop-AES blues started when I upgraded my laptop back in Argentina and couldn't access the NTFS partition on the dual boot system. That got made even worse due to the fact that GrUB won't be able to recognize the NTFS partitions without being able to mount them (that might be a bug, I should look into reporting it) and then deleted the bootloader entry to boot the Windows partition (so I had to recreate the chainloading by hand, not very difficult, but not pretty either). If that happens to you, just add to /etc/grub.d/40_custom the following lines:

menuentry "windows" {
   set root=(hd0,2)
   chainloader +1

Replace hd0,2 for your partition: (hd0,2) is /dev/sda2. More worryingly, I couldn't access from Debian anymore an encrypted TrueCrypt partition which I share between the two OSes. Annoying, but none of them was a show-stopper. Unbeknown to me, these were all loop-AES blues. I could see it was a FUSE problem, but I couldn't find anything in the Debian BTS.

When I got back home, it took me some days to merge back my files from the laptop (yes, I should be using git more comprehensively next time) and then I was ready to backup the desktop. Then the loop-AES blues hit hard: I would mount the encrypted partition fine with losetup -p0 -e aes256 /dev/loop7 /dev/sdc1 but then zpool import zfs_backup will croak:

cannot mount 'zfs_backup': Input/output error.
Make sure the FUSE module is loaded.

A very unhappy puppy, indeed. Good news were, zfs list will see my data, everything seemed to be there. I tried with my laptop, but the running kernel had no loop-AES module, and trying to re-compile it failed miserably (there I missed a very good clue), so I tried with an old kernel which had it and it wouldn't work. Exactly the same error message as above. Searching for it on-line wouldn't help either. In syslog I found zfs-fuse: kstat: fuse_mount error - trying to umount but those went away when I started running zfs-fuse with --no-kstat-mount. Adding debug to fuse-mount-options in /etc/zfs/zfsrc didn't make a difference (and there was a good clue I missed completely as the diverted /bin/mount wasn't using /etc/zfs/zfsrc).

The Solution

By then I was fairly down on my loop-AES blues. No bugs filled, no idea where the bug was, no clear culprit. Besides, both FUSE and loop-AES didn't seem to be very popular en the BTS. One had a recent request to be orphaned, the other had a FTBS bug saying it was unfit for Wheezy. It seemed I had based my backups in technology that was going away. Fast. But AES256 and ZFS are available outside GNU/Linux. So just when I was resigned to set-up a kFreeBSD box to re-enable my backups, I read in some random bug post a request to upgrade /bin/mount.

"Here goes nothing", I said to myself. If I were to submit a bug report, I needed to be running unstable. So I upgraded my laptop to unstable. Now, I couldn't use encryption with the old kernel anymore: loop-aes-utils conflicted with fuse. Then it all made sense, the diverted version of /bin/mount was breaking FUSE and taking it out made NTFS-3G and TrueCrypt work again! Sadly, I needed the encrypted loop to mount the ZFS.

Luckily, AES256 is not anything particularly strange cryptographically speaking (and according to Manning it doesn't take the government more than a few weeks to break) so I followed a very good migration guide and found a great script that provides loop-AES using the dm-crypt capabilities of modern kernels (local copy).

That was the end of my loop-AES blues. Moral of the story: if things aren't working... switch to unstable.


Your name:

URL (optional):

Your e-mail (optional, won't be displayed):

Something funny using the word 'elephant' (spam filter):

Your comment: