FC2ブログ

Building DroboFS like storage with Ubuntu


Goal
Using Ubuntu to building a drobo like NAS storage, which can
use some different sized drives together as one large storage tank with 1 drive fault tolerance
expand total capacity by adding drives with data stored.

Idea

Basic idea, BeyondRAID , key technology of Drobo
is described in wikipedia
and following figure is suitable to understand.


Drives
| 100 GB | 200 GB | 400 GB | 500 GB |

----------
| x | unusable space (100 GB)
----------
-------------------
| A1 | A1 | RAID 1 set (2× 100 GB)
-------------------
-------------------
| B1 | B1 | RAID 1 set (2× 100 GB)
-------------------
----------------------------
| C1 | C2 | Cp | RAID 5 array (3× 100 GB)
----------------------------
-------------------------------------
| D1 | D2 | D3 | Dp | RAID 5 array (4× 100 GB)
-------------------------------------




We have two open source technology, mdadm and LVM. We can

  1. Make partitions from drives. These partitions make role as A1, B1, C1,C2,Cp D1,D2,D3,Dp.
  2. Build RAID array/set with mdadm. We can get md devices.
  3. Concatinate md devices to one large storage with LVM.


the original idea is ZFS_and_FreeNAS_expansion.
I've succeeded with ZFS/FreeNAS, but It seems to be more flexible
when building with mdadm and LVM.




Prepare

  1. Three or more disks which can be different size and interface. I have 80GB,160GB,500GB SATA disks.

  2. PC server. I have proliant Microserver.

  3. Installed Linux distro. I installed "Ubuntu 10.10 amd 64 desktop edition" to USB stick memory.


How to build

On Ubuntu 10.10, when I use apt-get to install mdadm, the installed binary is not up to date,
so I installed from source code. Parted is also necessary to make partition.

$ sudo apt-get update
$ sudo apt-get install lvm2
$ wget http://www.kernel.org/pub/linux/utils/raid/mdadm/mdadm-3.1.4.tar.gz
$ tar xzf mdadm-3.1.4.tar.gz
$ cd mdadm-3.1.4
$ make
$ sudo make install


At first, I confirmed my drives. Drive sde is boot disk.

$ sudo parted /dev/sda print devices
/dev/sda (80.0GB)
/dev/sdb (160GB)/dev/sdc (500GB)
/dev/sde (16.0GB)



There are three drives, we will finaly get following structure.


Drives
| 80 GB | 160 GB | 500 GB |
----------
| x | unusable space (340 GB)
----------
-------------------
| sdb2 | sdc2 | RAID 1 (2× 80 GB) -> md2 (80GB)
-------------------
----------------------------
| sda1 | sdb1 | sdc1 | RAID 5 (3× 80 GB) -> md1 (160GB)
----------------------------
Total: 240 GB





Initializing drive as GPT label.

$ sudo parted -s /dev/sda mklabel gpt
$ sudo parted -s /dev/sdb mklabel gpt
$ sudo parted -s /dev/sdc mklabel gpt


Making partitions.

$ sudo parted -s /dev/sda mkpart md1 2048s 80GB
$ sudo parted -s /dev/sdb mkpart md1 2048s 80GB
$ sudo parted -s /dev/sdc mkpart md1 2048s 80GB



Building RAID5 array

$ sudo mdadm --create --assume-clean /dev/md1 \
--level=5 --metadata=1 --raid-devices=3 /dev/sda1 /dev/sdb1 /dev/sdc1



Repeat on md2

$ sudo parted -s /dev/sdb mkpart md2 80GB 160GB
$ sudo parted -s /dev/sdc mkpart md2 80GB 160GB
$ sudo mdadm --create --assume-clean /dev/md2 \
--level=1 --metadata=1 --raid-devices=2 /dev/sdb2 /dev/sdc2


Make md devices as PV. Makeing VG from PV

$ sudo pvcreate -Z y -M2 /dev/md1
$ sudo pvcreate -Z y -M2 /dev/md2
$ sudo vgcreate Storage /dev/md1 /dev/md2



Confirm how many PEs we have. Build LV with all PEs.

$ sudo vgdisplay Storage | grep Free
Free PE / Size 57218 / 223.51 GiB
$sudo lvcreate -l 57218 Storage
Logical volume "lvol0" created


Making fs on LV and mount.

$ sudo mkfs -t ext4 -T largefile /dev/Storage/lvol0
$ sudo mkdir -p /Storage
$ sudo mount -t ext4 /dev/Storage/lvol0 /Storage
$ df -h
/dev/mapper/Storage-lvol0
224G 188M 213G 1% /Storage


Now we can share /Storage with samba. It looks Drobo FS, isn't it?


Adding a drive to expand capacity

Proliant MicroServer I have can drive four internal drives,
I am going to add a 1TB drive and expanding /Storage as following:


Drives
| 80 GB | 160 GB | 500 GB | 1TB |
----------
| x | 500GB unusable space
----------
-------------------
| sdc3 | sdd3 | RAID 1 (2× 340 GB) -> md3 (340 GB)
-------------------
----------------------------
| sdb2 | sdc2 | sdd2 | RAID 5 (3× 80 GB) -> md2 (160 GB)
----------------------------
-------------------------------------
| sda1 | sdb1 | sdc1 | sdd1 | RAID 5 (4× 80 GB) -> md1 (240 GB)
-------------------------------------
Total: 740 GB




Creating partition and adding to md1

$ sudo parted -s /dev/sdd mklabel gpt
$ sudo parted -s /dev/sdd mkpart md1 2048s 80GB
$ sudo mdadm --add /dev/md1 /dev/sdd1


When adding partiton, its role is "spare".
So expanding md1 using all drives.

$sudo mdadm --grow --level=5 --raid-devices=4 /dev/md1



Above operation take large time.
Confirming when it finish.

$ cat /proc/mdstat
md2 : active raid1 sdc2[1] sdb2[0]
78123960 blocks super 1.2 [2/2] [UU]

md1 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0]
156244992 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] reshape = 0.6% (505856/78122496) finish=66.4min speed=19456K/sec


While rehaping, repeat to md2.

$ sudo parted -s /dev/sdd mkpart md2 80GB 160GB
$ sudo mdadm --add /dev/md2 /dev/sdd2
$ sudo mdadm --grow --level=5 --raid-devices=3 /dev/md2



Confirming how reshapring. md2 is DELAYED. When finishing md1, md2's resharpe will work.

$ cat /proc/mdstat
md2 : active raid5 sdd2[2] sdc2[1] sdb2[0]
78123960 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]
resync=DELAYED

md1 : active raid5 sdd1[3] sdc1[2] sdb1[1] sda1[0]
156244992 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
[>....................] reshape = 2.5% (1958912/78122496) finish=76.5min speed=16571K/sec


While rehaping, making md3. It's same way when we make md2.

$ sudo parted -s /dev/sdc mkpart md3 160GB 500GB
$ sudo parted -s /dev/sdd mkpart md3 160GB 500GB
$ sudo mdadm --create --assume-clean /dev/md3 \
--level=1 --metadata=1 --raid-devices=2 /dev/sdc3 /dev/sdd3
$ sudo pvcreate -Z y -M2 /dev/md3



Now we already have VG, so created PV should be add to the VG

$ sudo vgextend Storage /dev/md3


Confirming we got new free space

$ sudo vgdisplay Storage | grep Free
Free PE / Size 19073 / 74.50 GiB


Few hour later, all resharpes have finished.

$ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] [raid1]
md3 : active raid1 sdc3[0] sdd3[1]
332029816 blocks super 1.2 [2/2] [UU]

md2 : active raid5 sdc2[1] sdd2[2] sdb2[0]
156247920 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid5 sda1[0] sdc1[2] sdd1[3] sdb1[1]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]


Telling LVM md1 and md2 are expanded.

$ sudo pvdisplay /dev/md1 | grep "PV Size"
PV Size 74.50 GiB / not usable 3.93 MiB

$ sudo pvresize /dev/md1
Physical volume "/dev/md1" changed
1 physical volume(s) resized / 0 physical volume(s) not resized

$ sudo pvdisplay /dev/md1 | grep Size
PV Size 149.01 GiB / not usable 3.67 MiB

$ sudo pvresize /dev/md2


Expanding LV

$ sudo lvdisplay | grep "LV Size"
LV Size 223.51 GiB

$ sudo vgdisplay Storage | grep Free
Free PE / Size 119207 / 465.65 GiB

$ sudo lvextend -l +119207 /dev/Storage/lvol0

$ sudo lvdisplay | grep "LV Size"
LV Size 689.16 GiB


Finaly, expanding filesyste.

$ df -h
/dev/mapper/Storage-lvol0
224G 188M 213G 1% /Storage

$ sudo resize2fs /dev/Storage/lvol0

$ df -h
/dev/mapper/Storage-lvol0
689G 197M 655G 1% /Storage


Now we got expanded capacity.


Replacing drive to larger one

I am going to replace 80GB drive to 2TB drive. Arrays will be following:


Drives
| 2TB | 160 GB | 500 GB | 1TB |

----------
| x | ----------------------------> unusable space 1TB
----------
---------- ----------
| sda4 | | sdd4 | RAID 1 (2x 500 GB) -> md4 (500 GB)
---------- ----------
---------- -------------------
| sda3 | | sdc3 | sdd3 | RAID 5 (3× 340 GB) -> md3 (680 GB)
---------- -------------------
--------------------------------------
| sda2 | sdb2 | sdc2 | sdd2 | RAID 5 (4× 80 GB) -> md2 (240 GB)
-------------------------------------
-------------------------------------
| sda1 | sdb1 | sdc1 | sdd1 | RAID 5 (4× 80 GB) -> md1 (240 GB)
-------------------------------------
total: 1.6 TB




At first detach sda from md devices.
now only md1 contains sda, sda1 should be detached.

$ sudo mdadm /dev/md1 --fail /dev/sda1

$ cat /proc/mdstat
md1 : active raid5 sda1[0](F) sdc1[2] sdd1[3] sdb1[1]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]


Shuting down server and replacing sda as 2TB drive.
Already md1 device has no sda.

$ cat /proc/mdstat
md1 : active raid5 sdd1[2] sdc1[1] sde1[3]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
$ sudo parted -s /dev/sda print devices
/dev/sda (2000GB)
/dev/sdb (16.0GB)
/dev/sdc (160GB)
/dev/sdd (500GB)
/dev/sde (1000GB)
/dev/mapper/Storage-lvol0 (740GB)
/dev/md1 (240GB)
/dev/md2 (160GB)
/dev/md3 (340GB)


Now we have new drive sda.
Checking 2th large drive to decide the new (largest) drive's partition size.


$ sudo parted -s /dev/sde print free
モデル: ATA SAMSUNG HD103UJ (scsi)
ディスク /dev/sde: 1000GB
セクタサイズ (論理/物理): 512B/512B
パーティションテーブル: gpt

番号 開始 終了 サイズ ファイルシステム 名前 フラグ
17.4kB 1049kB 1031kB 空き容量
1 1049kB 80.0GB 80.0GB md1
2 80.0GB 160GB 80.0GB md2
3 160GB 500GB 340GB md3
500GB 1000GB 500GB 空き容量


Just make partions same size of 2th large drive to new drive.


$ sudo parted -s /dev/sda mklabel gpt
$ sudo parted -s /dev/sda mkpart md1 2048s 80GB
$ sudo parted -s /dev/sda mkpart md2 80GB 160GB
$ sudo parted -s /dev/sda mkpart md3 160GB 500GB


Adding partitions to md devices.

$ sudo mdadm --add /dev/md1 /dev/sda1
$ sudo mdadm --add /dev/md2 /dev/sda2
$ sudo mdadm --add /dev/md3 /dev/sda3


md1 is degrade mode, so as soon as adding, rebuild is started.

$ cat /proc/mdstat
md3 : active raid1 sda3[2](S) sdd3[0] sde3[1]
332029816 blocks super 1.2 [2/2] [UU]

md2 : active raid5 sda2[3](S) sdd2[1] sde2[2] sdc2[0]
156247920 blocks super 1.2 level 5, 8k chunk, algorithm 2 [3/3] [UUU]

md1 : active raid5 sda1[4] sdd1[2] sdc1[1] sde1[3]
234367488 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
[>....................] recovery = 4.3% (3367748/78122496) finish=19.6min speed=63416K/sec


Expanding md2 and md2 like when we add a drive.

$ sudo mdadm --grow --level=5 --raid-devices=4 /dev/md2
$ sudo mdadm --grow --level=5 --raid-devices=3 /dev/md3


When resharp has finished, telling lvm md's were expanded.

$ sudo pvresize /dev/md2
$ sudo pvresize /dev/md3


As we have unused capacity on sdd and sda, they should be used as mirrored device.

$ sudo parted -s /dev/sdd mkpart md4 500GB 1000GB
$ sudo parted -s /dev/sda mkpart md4 500GB 1000GB
$ sudo mdadm --create --assume-clean /dev/md4 --level=1 --metadata=1 --raid-devices=2 /dev/sda4 /dev/sdd4
$ sudo pvcreate -Z y -M2 /dev/md4
$ sudo vgextend Storage /dev/md4


Expanding LV as we have new free PEs.

$ sudo vgdisplay Storage | grep Free
Free PE / Size 119208 / 465.66 GiB
$ sudo lvextend -l +119208 /dev/Storage/lvol0


Finally, expanding file system.

$ sudo umount /Storage
$ sudo resize2fs /dev/Storage/lvol0
$ df -h
/dev/mapper/Storage-lvol0
1.6T 197M 1.5T 1% /Storage



conclusion

I know above operation is quite complex than Drobo.
If you want easy storage, you should buy Drobo. It may be good storage.
But you can also get Drobo-like storage cheaper.
And it is important the storage far from blackbox and vendor-lock-in.

スポンサーサイト



コメントの投稿

非公開コメント

プロフィール

nakanote

Author:nakanote
FC2ブログへようこそ!

最新記事
最新コメント
最新トラックバック
月別アーカイブ
カテゴリ
検索フォーム
RSSリンクの表示
リンク
ブロとも申請フォーム

この人とブロともになる

QRコード
QR