Goal Using Ubuntu to building a drobo like NAS storage, which can
use some different sized drives together as one large storage tank with 1 drive fault tolerance
expand total capacity by adding drives with data stored.
Idea Basic idea, BeyondRAID , key technology of Drobo
is described in wikipedia
and following figure is suitable to understand.
Drives
| 100 GB | 200 GB | 400 GB | 500 GB |
----------
| x | unusable space (100 GB)
----------
-------------------
| A1 | A1 | RAID 1 set (2× 100 GB)
-------------------
-------------------
| B1 | B1 | RAID 1 set (2× 100 GB)
-------------------
----------------------------
| C1 | C2 | Cp | RAID 5 array (3× 100 GB)
----------------------------
-------------------------------------
| D1 | D2 | D3 | Dp | RAID 5 array (4× 100 GB)
-------------------------------------
We have two open source technology, mdadm and LVM. We can
- Make partitions from drives. These partitions make role as A1, B1, C1,C2,Cp D1,D2,D3,Dp.
- Build RAID array/set with mdadm. We can get md devices.
- Concatinate md devices to one large storage with LVM.
the original idea is
ZFS_and_FreeNAS_expansion.
I've succeeded with ZFS/FreeNAS, but It seems to be more flexible
when building with mdadm and LVM.
Prepare
- Three or more disks which can be different size and interface. I have 80GB,160GB,500GB SATA disks.
- PC server. I have proliant Microserver.
- Installed Linux distro. I installed "Ubuntu 10.10 amd 64 desktop edition" to USB stick memory.
How to build On Ubuntu 10.10, when I use apt-get to install mdadm, the installed binary is not up to date,
so I installed from source code. Parted is also necessary to make partition.
$ sudo apt-get update
$ sudo apt-get install lvm2
$ wget http://www.kernel.org/pub/linux/utils/raid/mdadm/mdadm-3.1.4.tar.gz
$ tar xzf mdadm-3.1.4.tar.gz
$ cd mdadm-3.1.4
$ make
$ sudo make install
At first, I confirmed my drives. Drive sde is boot disk.
$ sudo parted /dev/sda print devices
/dev/sda (80.0GB)
/dev/sdb (160GB)/dev/sdc (500GB)
/dev/sde (16.0GB)
There are three drives, we will finaly get following structure.
Drives
| 80 GB | 160 GB | 500 GB |
----------
| x | unusable space (340 GB)
----------
-------------------
| sdb2 | sdc2 | RAID 1 (2× 80 GB) -> md2 (80GB)
-------------------
----------------------------
| sda1 | sdb1 | sdc1 | RAID 5 (3× 80 GB) -> md1 (160GB)
----------------------------
Total: 240 GB
Initializing drive as GPT label.
$ sudo parted -s /dev/sda mklabel gpt
$ sudo parted -s /dev/sdb mklabel gpt
$ sudo parted -s /dev/sdc mklabel gpt
Making partitions.
$ sudo parted -s /dev/sda mkpart md1 2048s 80GB
$ sudo parted -s /dev/sdb mkpart md1 2048s 80GB
$ sudo parted -s /dev/sdc mkpart md1 2048s 80GB
Building RAID5 array
$ sudo mdadm --create --assume-clean /dev/md1 \
--level=5 --metadata=1 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1
Repeat on md2
$ sudo parted -s /dev/sdb mkpart md2 80GB 160GB
$ sudo parted -s /dev/sdc mkpart md2 80GB 160GB
$ sudo mdadm --create --assume-clean /dev/md2 \
--level=1 --metadata=1 --raid-devices=2 /dev/sdb2 /dev/sdc2
Make md devices as PV. Makeing VG from PV
$ sudo pvcreate -Z y -M2 /dev/md1
$ sudo pvcreate -Z y -M2 /dev/md2
$ sudo vgcreate Storage /dev/md1 /dev/md2
Confirm how many PEs we have. Build LV with all PEs.
$ sudo vgdisplay Storage | grep Free
Free PE / Size 57218 / 223.51 GiB
$sudo lvcreate -l 57218 Storage
Logical volume "lvol0" created
Making fs on LV and mount.
$ sudo mkfs -t ext4 -T largefile /dev/Storage/lvol0
$ sudo mkdir -p /Storage
$ sudo mount -t ext4 /dev/Storage/lvol0 /Storage
$ df -h
/dev/mapper/Storage-lvol0
224G 188M 213G 1% /Storage
Now we can share /Storage with samba. It looks Drobo FS, isn't it?
Adding a drive to expand capacityProliant MicroServer I have can drive four internal drives,
I am going to add a 1TB drive and expanding /Storage as following:
Drives
| 80 GB | 160 GB | 500 GB | 1TB |
----------
| x | 500GB unusable space
----------
-------------------
| sdc3 | sdd3 | RAID 1 (2× 340 GB) -> md3 (340 GB)
-------------------
----------------------------
| sdb2 | sdc2 | sdd2 | RAID 5 (3× 80 GB) -> md2 (160 GB)
----------------------------
-------------------------------------
| sda1 | sdb1 | sdc1 | sdd1 | RAID 5 (4× 80 GB) -> md1 (240 GB)
-------------------------------------
Total: 740 GB
Creating partition and adding to md1
$ sudo parted -s /dev/sdd mklabel gpt
$ sudo parted -s /dev/sdd mkpart md1 2048s 80GB
$ sudo mdadm --add /dev/md1 /dev/sdd1
When adding partiton, its role is "spare".
So expanding md1 using all drives.
$sudo mdadm --grow --level=5 --raid-devices=4 /dev/md1
Above operation take large time.
Confirming when it finish.
$ cat /proc/mdstat
md2 : active raid1 sdc2[1] sdb2[0]
78123960 blocks super 1.2 [2/2] [UU]
md1 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0]
156244992 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] reshape = 0.6% (505856/78122496) finish=66.4min speed=19456K/sec
While rehaping, repeat to md2.
$ sudo parted -s /dev/sdd mkpart md2 80GB 160GB
$ sudo mdadm --add /dev/md2 /dev/sdd2
$ sudo mdadm --grow --level=5 --raid-devices=3 /dev/md2
Confirming how reshapring. md2 is DELAYED. When finishing md1, md2's resharpe will work.
$ cat /proc/mdstat
md2 : active raid5 sdd2[2] sdc2[1] sdb2[0]
78123960 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]
resync=DELAYED
md1 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0]
156244992 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] reshape = 2.5% (1958912/78122496) finish=76.5min speed=16571K/sec
While rehaping, making md3. It's same way when we make md2.
$ sudo parted -s /dev/sdc mkpart md3 160GB 500GB
$ sudo parted -s /dev/sdd mkpart md3 160GB 500GB
$ sudo mdadm --create --assume-clean /dev/md3 \
--level=1 --metadata=1 --raid-devices=2 /dev/sdc3 /dev/sdd3
$ sudo pvcreate -Z y -M2 /dev/md3
Now we already have VG, so created PV should be add to the VG
$ sudo vgextend Storage /dev/md3
Confirming we got new free space
$ sudo vgdisplay Storage | grep Free
Free PE / Size 19073 / 74.50 GiB
Few hour later, all resharpes have finished.
$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md3 : active raid1 sdc3[0] sdd3[1]
332029816 blocks super 1.2 [2/2] [UU]
md2 : active raid5 sdc2[1] sdd2[2] sdb2[0]
156247920 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]
md1 : active raid5 sda1[0] sdc1[2] sdd1[3] sdb1[1]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
Telling LVM md1 and md2 are expanded.
$ sudo pvdisplay /dev/md1 | grep "PV Size"
PV Size 74.50 GiB / not usable 3.93 MiB
$ sudo pvresize /dev/md1
Physical volume "/dev/md1" changed
1 physical volume(s) resized / 0 physical volume(s) not resized
$ sudo pvdisplay /dev/md1 | grep Size
PV Size 149.01 GiB / not usable 3.67 MiB
$ sudo pvresize /dev/md2
Expanding LV
$ sudo lvdisplay | grep "LV Size"
LV Size 223.51 GiB
$ sudo vgdisplay Storage | grep Free
Free PE / Size 119207 / 465.65 GiB
$ sudo lvextend -l +119207 /dev/Storage/lvol0
$ sudo lvdisplay | grep "LV Size"
LV Size 689.16 GiB
Finaly, expanding filesyste.
$ df -h
/dev/mapper/Storage-lvol0
224G 188M 213G 1% /Storage
$ sudo resize2fs /dev/Storage/lvol0
$ df -h
/dev/mapper/Storage-lvol0
689G 197M 655G 1% /Storage
Now we got expanded capacity.
Replacing drive to larger oneI am going to replace 80GB drive to 2TB drive. Arrays will be following:
Drives
| 2TB | 160 GB | 500 GB | 1TB |
----------
| x | ----------------------------> unusable space 1TB
----------
---------- ----------
| sda4 | | sdd4 | RAID 1 (2x 500 GB) -> md4 (500 GB)
---------- ----------
---------- -------------------
| sda3 | | sdc3 | sdd3 | RAID 5 (3× 340 GB) -> md3 (680 GB)
---------- -------------------
--------------------------------------
| sda2 | sdb2 | sdc2 | sdd2 | RAID 5 (4× 80 GB) -> md2 (240 GB)
-------------------------------------
-------------------------------------
| sda1 | sdb1 | sdc1 | sdd1 | RAID 5 (4× 80 GB) -> md1 (240 GB)
-------------------------------------
total: 1.6 TB
At first detach sda from md devices.
now only md1 contains sda, sda1 should be detached.
$ sudo mdadm /dev/md1 --fail /dev/sda1
$ cat /proc/mdstat
md1 : active raid5 sda1[0](F) sdc1[2] sdd1[3] sdb1[1]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
Shuting down server and replacing sda as 2TB drive.
Already md1 device has no sda.
$ cat /proc/mdstat
md1 : active raid5 sdd1[2] sdc1[1] sde1[3]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
$ sudo parted -s /dev/sda print devices
/dev/sda (2000GB)
/dev/sdb (16.0GB)
/dev/sdc (160GB)
/dev/sdd (500GB)
/dev/sde (1000GB)
/dev/mapper/Storage-lvol0 (740GB)
/dev/md1 (240GB)
/dev/md2 (160GB)
/dev/md3 (340GB)
Now we have new drive sda.
Checking 2th large drive to decide the new (largest) drive's partition size.
$ sudo parted -s /dev/sde print free
モデル: ATA SAMSUNG HD103UJ (scsi)
ディスク /dev/sde: 1000GB
セクタサイズ (論理/物理): 512B/512B
パーティションテーブル: gpt
番号 開始 終了 サイズ ファイルシステム 名前 フラグ
17.4kB 1049kB 1031kB 空き容量
1 1049kB 80.0GB 80.0GB md1
2 80.0GB 160GB 80.0GB md2
3 160GB 500GB 340GB md3
500GB 1000GB 500GB 空き容量
Just make partions same size of 2th large drive to new drive.
$ sudo parted -s /dev/sda mklabel gpt
$ sudo parted -s /dev/sda mkpart md1 2048s 80GB
$ sudo parted -s /dev/sda mkpart md2 80GB 160GB
$ sudo parted -s /dev/sda mkpart md3 160GB 500GB
Adding partitions to md devices.
$ sudo mdadm --add /dev/md1 /dev/sda1
$ sudo mdadm --add /dev/md2 /dev/sda2
$ sudo mdadm --add /dev/md3 /dev/sda3
md1 is degrade mode, so as soon as adding, rebuild is started.
$ cat /proc/mdstat
md3 : active raid1 sda3[2](S) sdd3[0] sde3[1]
332029816 blocks super 1.2 [2/2] [UU]
md2 : active raid5 sda2[3](S) sdd2[1] sde2[2] sdc2[0]
156247920 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]
md1 : active raid5 sda1[4] sdd1[2] sdc1[1] sde1[3]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 4.3% (3367748/78122496) finish=19.6min speed=63416K/sec
Expanding md2 and md2 like when we add a drive.
$ sudo mdadm --grow --level=5 --raid-devices=4 /dev/md2
$ sudo mdadm --grow --level=5 --raid-devices=3 /dev/md3
When resharp has finished, telling lvm md's were expanded.
$ sudo pvresize /dev/md2
$ sudo pvresize /dev/md3
As we have unused capacity on sdd and sda, they should be used as mirrored device.
$ sudo parted -s /dev/sdd mkpart md4 500GB 1000GB
$ sudo parted -s /dev/sda mkpart md4 500GB 1000GB
$ sudo mdadm --create --assume-clean /dev/md4 --level=1 --metadata=1 --raid-devices=2 /dev/sda4 /dev/sdd4
$ sudo pvcreate -Z y -M2 /dev/md4
$ sudo vgextend Storage /dev/md4
Expanding LV as we have new free PEs.
$ sudo vgdisplay Storage | grep Free
Free PE / Size 119208 / 465.66 GiB
$ sudo lvextend -l +119208 /dev/Storage/lvol0
Finally, expanding file system.
$ sudo umount /Storage
$ sudo resize2fs /dev/Storage/lvol0
$ df -h
/dev/mapper/Storage-lvol0
1.6T 197M 1.5T 1% /Storage
conclusion I know above operation is quite complex than Drobo.
If you want easy storage, you should buy Drobo. It may be good storage.
But you can also get Drobo-like storage cheaper.
And it is important the storage far from blackbox and vendor-lock-in.
