Tags: debian, sysadmin, encryption
Update: The situation seems to have improved, see this comment.
I use both in my desktop and laptop Debian testing (currently codenamed 'wheezy'). For backups, I have been using an AES256 encrypted ZFS external USB hard drive, with date-based snapshots and deduplication. I have two hard drives, one of which I keep off-site, doing weekly (or so) backups. I have all the same set-up in my laptop, and I have done some drill back-up recoveries from it (from the latest backup, not from the snapshots, as zfs-fuse doesn't support .snapshot folder, yet).
This solution has been working quite well until I went to
Argentina for teaching. There,
while I upgraded my Debian testing on the laptop, FUSE stop working
properly in strange ways. When I got back to Canada and tried to
backup my desktop, zfs-fuse was also non-working. The problem took me
quite a bit of work to fix, I'm documenting here the problem and its
solution (in a nutshell: loop-AES, in the package loop-aes-utils,
diverts /bin/mount to a version that is incompatible with
FUSE, solution: don't use loop-AES and move to dm-crypt).
In Debian there are several "flavors" for the distribution: stable, testing and unstable. Stable is well-tested, with timely security updates and all around rock solid (or that is the expectation at least). Nothing new happens in Debian stable for about year and a half to two years. In the mid-time, all the new packages, testing of new versions, etc., happens in Debian unstable (Debian 'sid'). When new versions of packages are uploaded to unstable, they automatically migrate into testing if there are no serious bugs filled against the upload in those first two weeks. I have been using testing for years and it is overall a much "happier" experience that testing as it usually Just Works (TM) and has very recent software.
Interestingly, when I start getting more involved in Debian, I realized testing is a little bit the worst of both worlds: you get no security updates (as in stable) and you get no interest from the developers (as in unstable). If you try to submit a bug against a package in testing, chances are you'll be asked to upgrade to unstable and see what happens. In general, submitting a bug against a package in testing when there is a newer version in unstable is a waste of everybody's time (and counter-intuitive, as the distro is called 'testing'!). Still, once in a blue moon a horrible update comes in unstable and the system breaks in a horrible way. I will stick to testing until I am comfortable with checking #debian-devel in OFTC before doing an upgrade. But over time using testing is becoming more and more of a nuisance. And this issue of testing vs. unstable is very related to my loop-AES blues, keep reading.
As usual in the FLOSS world, there is a variety of competing
technologies to do anything, in this case disk encryption. I have had
encrypted containers in my machine for a while and when I started
using them, loop-AES was a recommended solution. The way loop-AES works is
by "butchering" the loop functionality in the kernel (the
functionality that allows you to mount a file as if it were a device)
with extra encryption-fu. Therefore, you make sure your kernel has the
loop device as a module, install the Debian package loop-aes-source,
then use the module assistant to create a custom Debian package with
the new loop.ko, install it and then you are ready to rumba. Of
course, that does it at the kernel level, so you need modified
user-space tools (mount, losetup, swapon, etc) to access this new
functionality (selecting encryption algorithm, setting up keys,
encryption parameters, etc). These modified versions of existing tools
comes in the package loop-aes-utils, which diverts
/bin/mount (and others) to their modified
versions.
Loop-AES is half the story, the other half is FUSE, File System in User-space, a kernel module that allows you to have the file system driver in, well, user-space. Even though it doesn't have particularly good press, two file systems I rely on are FUSE-based: NTFS-3G and ZFS-FUSE. The NT file system moved out of the kernel due to a number of issues (the driver in Linux 2.4 caused data corruption and NTFS is structured as a database, ensuring data consistency is easier using recursion, which is usually unacceptable in kernel code). On the other hand, ZFS is Free Software, developed by then SUN Microsystems but its license under the CDDL, which is GPL-incompatible. There, using FUSE allows for a technical solution to a legal problem. The original ZFS code is kernel-quality, so the Debian version based on a FreeBSD kernel, Debian/kFreeBSD incorporates ZFS directly in the kernel (and this relates to a last-ditch solution I considered but didn't have to try).
My loop-AES blues started when I upgraded my laptop back in Argentina and couldn't access the NTFS partition on the dual boot system. That got made even worse due to the fact that GrUB won't be able to recognize the NTFS partitions without being able to mount them (that might be a bug, I should look into reporting it) and then deleted the bootloader entry to boot the Windows partition (so I had to recreate the chainloading by hand, not very difficult, but not pretty either). If that happens to you, just add to /etc/grub.d/40_custom the following lines:
menuentry "windows" {
set root=(hd0,2)
chainloader +1
}
Replace hd0,2 for your partition: (hd0,2) is
/dev/sda2. More worryingly, I couldn't access from Debian anymore an
encrypted TrueCrypt
partition which I share between the two OSes. Annoying, but none of
them was a show-stopper. Unbeknown to me, these were all loop-AES
blues. I could see it was a FUSE problem, but I couldn't find anything
in the Debian
BTS.
When I got back home, it took me some days to merge back my
files from the laptop (yes, I should be using git more comprehensively
next time) and then I was ready to backup the desktop. Then the
loop-AES blues hit hard: I would mount the encrypted partition fine
with losetup -p0 -e aes256 /dev/loop7 /dev/sdc1 but then
zpool import zfs_backup will croak:
cannot mount 'zfs_backup': Input/output error. Make sure the FUSE module is loaded.
A very unhappy puppy, indeed. Good news were, zfs list
will see my data, everything seemed to be there. I
tried with my laptop, but the running kernel had no loop-AES module,
and trying to re-compile it failed miserably (there I missed a very
good clue), so I tried with an old kernel which had it and it wouldn't
work. Exactly the same error message as above. Searching for it on-line
wouldn't help either. In syslog I found zfs-fuse: kstat:
fuse_mount error - trying to umount but those went away when I
started running zfs-fuse with
--no-kstat-mount. Adding debug to
fuse-mount-options in /etc/zfs/zfsrc didn't
make a difference (and there was a good clue I missed
completely as the diverted /bin/mount wasn't using /etc/zfs/zfsrc).
By then I was fairly down on my loop-AES blues. No bugs filled, no
idea where the bug was, no clear culprit. Besides, both FUSE and
loop-AES didn't seem to be very popular en the BTS. One had a recent
request to be orphaned,
the other had a FTBS bug
saying it was unfit for Wheezy. It seemed I had based my
backups in technology that was going away. Fast. But AES256 and ZFS
are available outside GNU/Linux. So just when I was resigned to set-up
a kFreeBSD box to re-enable my backups, I read in some random bug post
a request to upgrade /bin/mount.
"Here goes nothing", I said to myself. If I were to submit a bug
report, I needed to be running unstable. So I upgraded my laptop to
unstable. Now, I couldn't use encryption with the old kernel anymore:
loop-aes-utils conflicted with
fuse. Then it all made sense, the diverted version of
/bin/mount was breaking FUSE and taking it out made
NTFS-3G and TrueCrypt work again! Sadly, I needed the encrypted loop to mount the
ZFS.
Luckily, AES256 is not anything particularly strange cryptographically speaking (and according to Manning it doesn't take the government more than a few weeks to break) so I followed a very good migration guide and found a great script that provides loop-AES using the dm-crypt capabilities of modern kernels (local copy).
That was the end of my loop-AES blues. Moral of the story: if things aren't working... switch to unstable.