RAID 10 FTW
From time to time I have to put on my sysadmin hat to build, manage and repair systems. I'm not an expert, but I do my best to search for expert advice or RTFM. If you have any suggestions to my implementation let me know.
At home I built a system for the purpose of being a development workstation, with similar hardware specs as a production server. That means the box is running Ubuntu desktop on a drive with four additional drives for the raid array, my personal cloud.
Last week I had a drive failure that I replaced so I want to share my experience and recipes.
Let's start from the beginning. Setting up a RAID 10 array, also known as RAID 1+0: creates a striped set from a series of mirrored drives. This way data loss can only occur if a drive fails after already losing all drives in a mirror. Anticipating typical and saddening hard drive failure, it's simple enough to make a replacement allowing the array to heal itself. It's unlikely for simultaneous drives to all fail at once without a notification and the array shutting down.
In my case I'm using 4 drives (sda, sdb, sdc & sdd) separate from the operating file system. The plan is to create a single array (md0) from the drives and then logical volume and mount points. My use case is a mongodb data store, so I followed the guide to setup three volumes mounted at /cloud/data, /cloud/log and /cloud/journal.
Here are the basic steps for what I described above:
Start by entering sudo interactive mode since most of these commands require root privileges.
You can run the following to test mdadm is able to reach you.
The following helped me grow logical volumes hosting my database.
I wasn't immediately able to get the array up and running for a couple of reasons. Using the ubuntu disk utility gave me a visual insight as to what was going on and ability to add a drive, except that it's using fdisk utils that don't support drives greater than something like 2TB, which includes me.
Also, after trying a bunch of the typical procedures documented online, I realized that I wanted to add an entire disk /dev/sdd, to the array as a replacement, and not a partition like /dev/sdd1 which many guides and utilities seem to impose.
Here's what ended up working for me:
Begin by stopping the failed array.
Now you can add the new replacement disk
Then you can sit back and watch the recovery status of your new drive
Hopefully this helps anyone reading this, and will definitely serve as documentation for me when it's time to put the sysadmin hack back on!
At home I built a system for the purpose of being a development workstation, with similar hardware specs as a production server. That means the box is running Ubuntu desktop on a drive with four additional drives for the raid array, my personal cloud.
Last week I had a drive failure that I replaced so I want to share my experience and recipes.
Let's start from the beginning. Setting up a RAID 10 array, also known as RAID 1+0: creates a striped set from a series of mirrored drives. This way data loss can only occur if a drive fails after already losing all drives in a mirror. Anticipating typical and saddening hard drive failure, it's simple enough to make a replacement allowing the array to heal itself. It's unlikely for simultaneous drives to all fail at once without a notification and the array shutting down.
In my case I'm using 4 drives (sda, sdb, sdc & sdd) separate from the operating file system. The plan is to create a single array (md0) from the drives and then logical volume and mount points. My use case is a mongodb data store, so I followed the guide to setup three volumes mounted at /cloud/data, /cloud/log and /cloud/journal.
Here are the basic steps for what I described above:
Start by entering sudo interactive mode since most of these commands require root privileges.
$ sudo -i
Install mdadm if you don't have it already and scope out your /dev
$ apt-get install mdadm
$ mdadm
$ ls -la /dev/
Create the array from 4 physical devices. Note this is different than using partitions like sda1 for instance, which is also possible, but I didn't, per the guide I was following.
$ mdadm --verbose --create /dev/md0 --level=10 --chunk=256 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
$ 'DEVICE /dev/sda /dev/sdb /dev/sdc /dev/sdd' | sudo tee -a /etc/mdadm.conf
$ mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
$ blockdev --setra 128 /dev/md0
$ blockdev --setra 128 /dev/sda
$ blockdev --setra 128 /dev/sdb
$ blockdev --setra 128 /dev/sdc
$ blockdev --setra 128 /dev/sdd
$ dd if=/dev/zero of=/dev/md0 bs=512 count=1
Setup logical volumes from the free space that we can then mount to the file system
$ apt-get install pvcreate
$ apt-get install lvm2
$ pvcreate /dev/md0
$ vgcreate vg0 /dev/md0
Setup our three logical volumes, 98% of the available space given to data, 1% to log and journal each, formatted as ext4 file systems.
$ lvcreate -l 98%vg -n data vg0
$ lvcreate -l 1%vg -n log vg0
$ lvcreate -l 1%vg -n journal vg0
$ mke2fs -t ext4 -F /dev/vg0/data
$ mke2fs -t ext4 -F /dev/vg0/log
$ mke2fs -t ext4 -F /dev/vg0/journal
Create our mount points
$ mkdir /cloud
$ mkdir /cloud/data
$ mkdir /cloud/log
$ mkdir /cloud/journal
Now we want to add entries to fstab to mount on startup
$ echo '/dev/vg0/data /cloud/data ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab
$ echo '/dev/vg0/log /cloud/log ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab
$ echo '/dev/vg0/journal /cloud/journal ext4 defaults,auto,noatime,noexec 0 0' | tee -a /etc/fstab
$ mount /cloud/data/
$ mount /cloud/log/
$ mount /cloud/journal
That should get you up and running. It's also a good idea to take a minute to check out your /etc/mdadm.conf configuration and make sure you're getting email notifications.
You can run the following to test mdadm is able to reach you.
$ sudo mdadm --monitor --scan --test --oneshot
Using logical volumes has the advantage of being able to resize easily regardless of physical devices. On my AWS cloud version of this setup it was fairly simple to create another raid array and add it to the logical volume.
The following helped me grow logical volumes hosting my database.
$ vgdisplay
$ lvdisplay
$ df -hk
$ vgextend vg0 /dev/md1
$ lvresize -L +391GB /dev/vg0/data
$ lvresize -L +8GB /dev/vg0/journal
$ lvdisplay /dev/vg0
$ resize2fs -p /dev/mapper/vg0-journal
Then one day I was having trouble with my server, foolishly not looking at the email from mdadm to see the failure, I decided to reboot the problem away, only to find a drive failure preventing
me from even booting up. I'm not sure why the failed drive was behaving so poorly in the bios boot, there might be a setting I recall reading about related to the reacting time during boot. However, once I took the bad drive out, and booted,
I was able to begin diagnosing my failed RAID array. I did find an email in my inbox giving me details about the failure and array status.
This is an automatically generated mail message from mdadm running on localhostWith one of four drives missing from the array, it was in a failed state, unable to operate. I've since realized that I could add a spare which should keep it running after a single failure. Hence I ordered 2 replacement drives and a couple days later included a replacement for the failed drive (no spare yet until I free a drive slot).
A Fail event had been detected on md device /dev/md0.
It could be related to component device /dev/sdd.
Faithfully yours, etc.
P.S. The /proc/mdstat file currently contains the following:
Personalities : [raid10] [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4]
md0 : active raid10 sdd[4] sdc[2] sda[0](F) sdb[1](F)
5860532736 blocks super 1.0 256K chunks 2 near-copies [4/2] [__UU]
I wasn't immediately able to get the array up and running for a couple of reasons. Using the ubuntu disk utility gave me a visual insight as to what was going on and ability to add a drive, except that it's using fdisk utils that don't support drives greater than something like 2TB, which includes me.
Also, after trying a bunch of the typical procedures documented online, I realized that I wanted to add an entire disk /dev/sdd, to the array as a replacement, and not a partition like /dev/sdd1 which many guides and utilities seem to impose.
Here's what ended up working for me:
Begin by stopping the failed array.
$ sudo mdadm --manage /dev/md0 --stop
Then assemble the array without the failed disk (sdd in this case)
$ mdadm --assemble --run --force /dev/md0 /dev/sda /dev/sdb /dev/sdc
Now you can add the new replacement disk
$ mdadm --manage /dev/md0 --add /dev/sdd
Then you can sit back and watch the recovery status of your new drive
$ watch -n 1 cat /proc/mdstatHopefully this helps anyone reading this, and will definitely serve as documentation for me when it's time to put the sysadmin hack back on!