HowTo Single Disk to RAID
We found ourselves in the need of switching from a single disk root to a root backed by a RAID, here's how we did.
Context
We sat up a new/old server, BitArno, it has a nice /home of ~6TB backed by a RAID and a separate disk for the / partition. As the drive for the / was tested before getting to production resulting in no errors we thought we didn't need a RAID for it. We installed Debian stable 8.1, set up every service, went to bed just to wake up with a hung server refusing to boot.
An on site check was required, we got there and found out that the / drive was failing to reallocate sectors, we fsck'ed it and it booted. "Great!" that same evening it was failing again and so we decided to switch from a single disk root to a RAID1 backed root especially because we do use commodity hardware taken from old computers.
The general idea
We wanted a way to easily switch from a drive to the RAID without having to back everything up and copy it over again with fewer downtime possible and so we decided to:
- Poweroff the server and attach a new disk of equal size
- Power on the server
- Create a RAID1 array composed by 2 drives but with a drive marked as missing (the new drive is the present one, the old one will be attached to the cluster later)
- Set LVM up on top of it (as the old drive had LVM (it's a 120GB drive just for the `/` parition, we wanted to have some space available if we'll ever find in the need)
- Boot a live distro and copy /
- Install the bootloader on the new drive adding the necessary modules for RAID and LVM
- Reboot the server from the RAID-member disk
- Check if everything is OK
- Add the old disk to the RAID and let mdadm resync the contents.
Step by step guide
Here I'll try to cover step by step the command we used, it has been a while since we did it so I'm not 100% sure I'll remember everything
First a layout of the partitions when it was single-disk
sda 8:0 0 114.5G 0 disk ├─sda1 8:1 0 243M 0 part /boot ├─sda2 8:2 0 1K 0 part └─sda5 8:5 0 114.3G 0 part ├─bitarno--system-root 253:0 0 30G 0 lvm / └─bitarno--system-swap 253:3 0 4G 0 lvm [SWAP]
And after the switch (newly created LVM has different volume group name but same size and structure)
sda 8:0 0 114.5G 0 disk
├─sda1 8:1 0 243M 0 part
│ └─md1 9:1 0 242.8M 0 raid1 /boot
├─sda2 8:2 0 1K 0 part
└─sda5 8:5 0 114.3G 0 part
└─md2 9:2 0 114.2G 0 raid1
├─ba--system-root 253:0 0 30G 0 lvm /
└─ba--system-swap 253:3 0 4G 0 lvm [SWAP]
sde 8:64 0 114.5G 0 disk
├─sde1 8:65 0 243M 0 part
│ └─md1 9:1 0 242.8M 0 raid1 /boot
├─sde2 8:66 0 1K 0 part
└─sde5 8:69 0 114.3G 0 part
└─md2 9:2 0 114.2G 0 raid1
├─ba--system-root 253:0 0 30G 0 lvm /
└─ba--system-swap 253:3 0 4G 0 lvm [SWAP]
The sdX2 partition is an extended one which contains the RAID one
In this way we have two arrays, one for the boot and one for the LVM which contains the root and the swap
First of all we need to wipe out every possible header from the new disk as it will become a RAID member
dd if=/dev/zero of=/dev/sde bs=100M count=1
Then we created the partitions with sfdisk about the same size as the first disk. It's not needed for them to be the same as it wasn't possible to copy the partitions with dd
We need:
- one parition for the boot marked as Linux raid autodetect
- the other for LVM marked as Linux raid autodetect that takes the free space left
We then proceeded to create the 2 RAIDS, /dev/sde is the new disk
mdadm --create /dev/md1 --level=1 --raid-devices=2 /dev/sde1 missing mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sde5 missing
We made the filesystem on the newly created /dev/md1
mkfs.ext2 /dev/md1
And copied the files
mount /dev/md1 /mnt rsync -avPh /boot /mnt sync && umount /mnt
We were done with /boot, and we needed to create the LVM volumes:
First we marked /dev/md2 as LVM member with fdisk then we told LVM /dev/md2 was a device it could use
pvcreate /dev/md2
We then created the virtual group that was going to host the logical volumes
vgcreate ba-system /dev/md2
And created the logical volumes over it
lvcreate -L30G -n root ba-system lvcreate -L4G -n swap ba-system
We created the filesystem for root
mkfs.ext4 /dev/mapper/ba--system-root
And for swap
mkswap /dev/mapper/ba--system-swap
Then it was time to poweroff the server and boot from a live, we choose ArchLinux as it's lightweight and provides all the packages we need without forcefully bringing up an installation procedure
We mounted the root partitions
mkdir /mnt/old /mnt/new mount /dev/mapper/ba--system-root /mnt/new mount /dev/mapper/bitarno--system-root /mnt/old
And copied the data over:
rsync -avPh /mnt/old/ /mnt/new
We then proceeded to mount the boot partition
mount /dev/md1 /mnt/new/boot
And arch-chroot'ed into it. Arch-chroot is a wonderful piece of software that saves you from bind mounting /dev /proc and /sys in the chroot
arch-chroot /mnt/new
It was time to install grub on the new disk and regenerate grub configuration and the initram image to contain LVM and RAID modules, this was the hardest part as we wrongly supposed all the modules were already present.
We added "domdadm" to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, the line should look something like
cat /etc/default/grub| grep -i cmdli GRUB_CMDLINE_LINUX_DEFAULT="quiet domdadm"
And we added the modules needed to GRUB_PRELOAD_MODULES in the same file, if the line don't exist create it
GRUB_PRELOAD_MODULES="lvm diskfilter mdraid1x"
Then it was time to install grub on both HDD in the MBR, note that it wasn't needed to specify the boot partition as we were in a chroot
grub-install /dev/sda grub-install /dev/sde
And just to be sure
update-grub
We also had to update the initramfs to actually contain the raid module ans so we edited
/etc/initramfs-tools/modules
appending
raid1
to the list and executed
update-initramfs
to re-generate the initramfs
We exited from the chroot, unmounted everything, rebooted and kept our finger crossed till the password prompt of the root-on-RAID booted system.
This last step I simplified it, we made several errors and had to boot into live several times, as we kept forgetting to add the needed modules and command lines arguments
To be sure to boot from the raid root you can edit the grub entry before booting, make sure it's like the following
insmod mdraid1x
Have to be present inside the menuentry section of your os and the linux entry looks similar to
linux /vmlinuz-3.16.0-4-amd64 root=/dev/mapper/ba--system-root ro quiet domdadm
As we failed to achieve everything on the first try we edited the grub command line and the performed grub install from the booted system.
Once we managed to boot we had to finalize the process and add the old drive to the RAID array
We backed up the partition table of the new disk
sfdisk -d /dev/sde > /tmp/savetablesde
Unmounted the automatically-mounted /dev/sda[1,2,5] deactivating the swap and LVM
swapoff /dev/mapper/bitarno-system-swap vgchange -an bitarno--system
Wiped out all the old drive first 100M
dd if=/dev/zero of=/dev/sda bs=100M count=1 sync
And in the end we copied the partition table on the old disk
sfdisk /dev/sda < /tmp/savetablesde
We added the partitions to the array
mdadm /dev/md1 --add /dev/sda1 mdadm /dev/md2 --add /dev/sda5
DONE!
We then checked the status of the resync of mdadm with
cat /proc/mdstat
Issues
We (of course) had some issues in the process:
- Thought the drive was of the same model the capacity was a little different and so we couldnt copy the partition scheme as we tougth we could do
- It's very important to generate the grub image inserting the modules and the boot parameter needed for lvm and RAID
- The initramfs must be rebuilt with the right raid module present in order for the kernel to find the cluster
- It's impossible to backup and restore the lvm too as for a while we found ourselves with the 2 LVMs disk together and names were conflicting.