Recently I had need to re-purpose a server and for convenience, I decided to do a complete wipe and reinstall since it had previously been used for all sorts of package testing, experiments, dak debugging, the list goes on. I took a careful backup and then cooked up some USB installation media, but it took so long to boot (USB1.1, yay) I ran out of time before the building was locked.

Since this box has two hard disks, and not being one to back down from a challenge, I eventually reinstalled it over the weekend with nothing – no install media, no reinstall robot or intelligent hands – just a reliable internet connection and a healthy dose of courage. Here’s how.

Target: reinstalled machine with the same network settings, ssh host keys, and other minor configuration ported. The disk layout is to be RAID-1 containing LVM, with separate /var volume and separate /boot partition, also RAID-1.

1. One disk in the box contained old data, so I cleared that out and wiped it (including the MBR for good measure) and partitioned it.
2. I set up a degraded RAID-1 array for a small /boot partition, a large RAID-1 array for the LVM and a swap partition.
3. I mounted the new partitions in the correct layout in /mnt and used debootstrap(8) to get a very basic root set up. I also bind-mounted /sys, /proc, /dev and /dev/pts for now, they can be done properly when the root is a bit more mature.
4. Next, I copied into the new root /etc/apt/sources.list and chroot(8)ed into it. Now I could apt-get update and tasksel install standard to get an almost fully-functional base system. At this point it is also sensible to install locales, tzdata and console-data and dpkg-reconfigure them, followed by mdadm and lvm2 if required and openssh-server so you can get back in after rebooting. Some or all of these may already be installed by tasksel.
5. Time to install a kernel before leaving the chroot: apt-get install linux-image-2.6, followed by grub-pc which should detect both installations and set up menu entries for them.
6. Back in the old system, I copied in the network, hosts, resolv and hostname configuration files, and set up /etc/fstab to my liking.
7. Install grub to both hard disks if it isn’t already so (dpkg-reconfigure grub-pc) and again check that it detects both installations and creates the right menu entries. At this stage, booting from either hard disk will allow the loading of either the new or old installations, which is exactly what we want. It’s now time to umount the new installation.
8. Now I followed the excellent guide for remote kernel upgrades at http://ariekanarie.nl, except in this case we are using the same method to try booting the new system and fall back to the old one if it’s a disaster.
9. Reboot and hope!

At this point I rebooted to find myself back in the old kernel, which was disappointing – this means the new kernel has panicked and rebooted, and grub has fallen back to the old system (exactly as planned). It turned out there was nothing in /dev at boot time, and udev doesn’t start early enough to populate it before panic. That’s easily solved by mounting the installation again and using MAKEDEV as a seed.

10. With a bit of luck, you’re now in the new installation and can dpkg-reconfigure grub-pc again to install grub to both hard disks again. This isn’t strictly necessary, but it records this choice in debconf so future upgrades will automatically upgrade the bootloader everywhere it’s needed.
11. Now I could do some tidying up, mount the old installation and copy over all the data I wanted, and after careful checking wipe the first disk clean ready to be added into the RAID arrays.
12. Finally, add the old disk to the RAID arrays so they are fully redundant.

Sources:
http://www.michael-hammer.at/server_config/debootstrap/
http://d-i.alioth.debian.org/tmp/en.i386/apds03.html
http://www.debian.org/releases/stable/amd64/apds03.html.en
https://wiki.archlinux.org/index.php/Convert_a_single_drive_system_to_RAID
http://ariekanarie.nl/archives/211/remote-kernel-upgrade-with-debianubuntu-and-grub2