■EXT3+DRBDの導入
GFSで管理しないパターン。
lvm2は今回の手順では使わない。必要なら。
$ echo "cman drbd8-utils drbdlinks lvm2" | \
sudo apt-get install -y `xargs`
■モジュールのロード
$ sudo modprobe drbd
$ lsmod | grep drbd
drbd 193891 0
lru_cache 12969 1 drbd
■procファイル
$ cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
■定義された資源はないよ。ってそりゃそうだ。
$ sudo drbdadm dstate `uname -n`
--== Thank you for participating in the global usage survey ==--
The server's response is:
you are the 783th user to install this version
no resources defined!
■まだ使っていない同じ容量の仮想ディスクがあります。
$ echo "01" >/dev/null;sudo fdisk -l /dev/sdb
Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/sdb doesn't contain a valid partition table
■パーティションの作成
空のディスクなので、そのまま。
$ sudo fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0xd1a5a0e2.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): p
Disk /dev/sdb: 21.5 GB, 21474836480 bytes
255 heads, 63 sectors/track, 2610 cylinders, total 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xd1a5a0e2
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-41943039, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-41943039, default 41943039):
Using default value 41943039
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
■確認
$ sudo fdisk -l /dev/sdb | grep ^/
/dev/sdb1 2048 41943039 20970496 83 Linux
■DRBDのサンプル設定を確認。
としたいところだが、リソースの設定が長すぎる。
「*.res」はリソースの略かな。。。
$ cat /etc/drbd.conf
# You can find an example in /usr/share/doc/drbd.../drbd.conf.example
include "drbd.d/global_common.conf";
include "drbd.d/*.res";
$ lv -s /usr/share/doc/drbd8-utils/examples/drbd.conf.example.gz | \
grep -v "^#\|^\$\|^ *#" | grep -A 39 "^resource r0" >/dev/null
■ディスクがエラーの時は。。。
$ sudo grep -A 3 "[ \t]*disk " /etc/drbd.d/global_common.conf
disk {
# on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
# no-disk-drain no-md-flushes max-bio-bvecs
}
■ハンドラは「pass_on/ call-local-io-error/detach」なので、
pass_onを選択するのだが、後でリソースの設定に記述する。
※globalには記述しない。
$ man drbd.conf | grep -A 13 "on-io-error handler"
on-io-error handler
is taken, if the lower level device reports io-errors to the upper
layers.
handler may be pass_on, call-local-io-error or detach.
pass_on: The node downgrades the disk status to inconsistent, marks
the erroneous block as inconsistent in the bitmap and retries the
IO on the remote node.
call-local-io-error: Call the handler script local-io-error.
detach: The node drops its low level device, and continues in
diskless mode.
■同期のレートはデフォルトが「250 KiB/sec」
今回は後で変更するが、最初の同期が一番時間がかかるので、先にやっておいた方が良い。
$ sudo grep -A 2 "[ \t]*syncer " /etc/drbd.d/global_common.conf
syncer {
# rate after al-extents use-rle cpu-mask verify-alg csums-alg
}
$ man drbdsetup | grep -A 8 "^ *syncer"
syncer
Changes the synchronization daemon parameters of device at runtime.
-r, --rate rate
To ensure smooth operation of the application on top of DRBD, it is
possible to limit the bandwidth that may be used by background
synchronization. The default is 250 KiB/sec, the default unit is
KiB/sec.
■プロトコルはデフォルトのCで。
どの時点で完了とみなすかが異なる。
Aの書き込みはローカルディスクとTCPバッファに届いた時点
Bの書き込みはローカルディスクとリモートのバッファキャッシュに届いた時点
Cの書き込みはローカルディスクとリモートディスクに届いた時点
$ grep protocol /etc/drbd.d/global_common.conf
protocol C;
$ man drbd.conf | grep -A 12 "protocol prot-id"
protocol prot-id
On the TCP/IP link the specified protocol is used. Valid protocol
specifiers are A, B, and C.
Protocol A: write IO is reported as completed, if it has reached
local disk and local TCP send buffer.
Protocol B: write IO is reported as completed, if it has reached
local disk and remote buffer cache.
Protocol C: write IO is reported as completed, if it has reached
both local and remote disk.
■リソースを設定
$ cat /etc/drbd.d/drbd0.res
resource drbd0 {
protocol C;
meta-disk internal;
on xen-debian2 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.152.92:7789;
}
on xen-debian1 {
device /dev/drbd0;
disk /dev/sdb1;
address 192.168.152.91:7789;
}
}
■drbdプロセスの再起動
アクティブ、スタンバイの両方に同じリソースファイルを設定して行う。
--== Thank you for participating in the global usage survey ==--
The server's response is:
you are the 51444th user to install this version
[....] Starting DRBD resources:[
drbd0
no suitable meta data found :(
Command '/sbin/drbdmeta 0 v08 /dev/sdb1 internal check-resize' terminated with exit code 255
drbdadm check-resize drbd0: exited with code 255
d(drbd0) 0: Failure: (119) No valid meta-data signature found.
==> Use 'drbdadm create-md res' to initialize meta-data area. <==
[drbd0] cmd /sbin/drbdsetup 0 disk /dev/sdb1 /dev/sdb1 internal --set-defaults --create-device --on-io-error=pass_on failed - continuing!
[ ok d0) ].
■この状態でのprocでアクティブ、スタンバイ共に同じ下記となる。
$ cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:Unconfigured
■「drbdsetup」でスタンバイ側を設定
$ sudo dd if=/dev/zero bs=1M count=1 of=/dev/sdb1; sync
1+0 レコード入力
1+0 レコード出力
1048576 バイト (1.0 MB) コピーされました、 0.00714007 秒、 147 MB/秒
$ sudo drbdadm create-md drbd0
--== Thank you for participating in the global usage survey ==--
The server's response is:
node already registered
Writing meta data...
initializing activity log
NOT initialized bitmap
New drbd meta data block successfully created.
success
$ sudo drbdsetup /dev/drbd0 disk /dev/sdb1 internal -1
Can not open device 'internal': No such file or directory
$ ls -l /dev/drbd0
brw-rw---T 1 root disk 147, 0 7月 22 23:43 /dev/drbd0
$ sudo drbdsetup /dev/drbd0 net 192.168.152.92 192.168.152.91 C
■アクティブ側ではIP順を逆にする。
$ sudo dd if=/dev/zero bs=1M count=1 of=/dev/sdb1; sync
$ sudo drbdadm create-md drbd0
$ sudo drbdsetup /dev/drbd0 disk /dev/sdb1 internal -1
Can not open device 'internal': No such file or directory
$ sudo drbdsetup /dev/drbd0 net 192.168.152.91 192.168.152.92 C
■アクティブ/スタンバイ共に以下のようなデバイスが出来る。
$ sudo find /dev/drbd* -print
/dev/drbd
/dev/drbd/by-res
/dev/drbd/by-res/drbd0
/dev/drbd/by-disk
/dev/drbd/by-disk/sdb1
/dev/drbd0
■アクティブ/スタンバイ共にSecondary/Secondaryであること。
$ cat /proc/drbd
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:Connected ro:Secondary/Secondary ds:Inconsistent/Inconsistent C r-----
ns:0 nr:0 dw:0 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20969820
■アクティブ側で以下を実施。
$ sudo drbdadm -- --overwrite-data-of-peer primary drbd0
--== Thank you for participating in the global usage survey ==--
The server's response is:
node already registered
■同期が始まったことを確認出来る。
(プライマリ)
$ cat /proc/drbd version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent C r-----
ns:34432 nr:0 dw:0 dr:35104 al:0 bm:2 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:20935388
[>....................] sync'ed: 0.2% (20444/20476)Mfinish: 4:20:23 speed: 1,328 (1,324) K/sec
■以下の方法でも確認出来る。
(セカンダリ)
$ sudo /etc/init.d/drbd status
drbd driver loaded OK; device status:
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
m:res cs ro ds p mounted fstype
0:drbd0 SyncTarget Secondary/Primary Inconsistent/UpToDate C
... sync'ed: 0.5% (20380/20476)Mfinish: 4:20:54 1,312 (1,288) want: 250 K/sec
■プライマリ側はro:Primary/Secondaryで、
セカンダリ側はro:Secondary/Primaryで見える。
■プライマリ側でDRBDボリュームのマウント
$ sudo mke2fs -j /dev/drbd0
mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1310720 inodes, 5242455 blocks
262122 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
160 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
$ sudo mkdir /media/drbd0
$ sudo mount -t ext3 /dev/drbd0 /media/drbd0
■DRBDの動作確認とsyncスピードの変更
アンマウント前に同期が完了していること。
プライマリ側でしかマウントしていない。
プライマリをアンマウント、セカンダリに降格。
$ sudo touch /media/drbd0/test
$ sudo umount /media/drbd0
$ sudo drbdadm secondary dbdb0
■セカンダリ側をプライマリに昇格
$ sudo drbdadm primary drbd0
$ sudo mount -t ext3 /dev/drbd0 /media/drbd0
$ ls /media/drbd0
lost+found test
■プライマリのキャッシュの読み込み、デバイスの読み込み速度を確認。
$ sudo hdparm -Tt /dev/sdb1
/dev/sdb1:
Timing cached reads: 8326 MB in 1.99 seconds = 4183.64 MB/sec
Timing buffered disk reads: 928 MB in 3.00 seconds = 309.12 MB/sec
■書き込み速度の平均値を取得。50MBまではいけそう。
$ for m in `seq 1 10`;do \
for n in `seq 1 100`;do \
sudo dd if=/dev/zero bs=512 count=100 of=/media/drbd0/write 2>&1 ; \
done | grep MB | sed s/".*秒、 "//g | \
awk '{sum+=$1}END{print sum/100"MB"}'; \
done
69.807MB
74.735MB
78.3MB
73.5MB
72.906MB
72.489MB
69.918MB
73.562MB
66.584MB
68.111MB
■syncerの設定
$ grep -A 2 syncer /etc/drbd.d/drbd0.res
syncer {
rate 50M;
}
$ sudo /etc/init.d/drbd restart
■その他
$ sudo drbdadm dstate drbd0
UpToDate/UpToDate
$ sudo drbdadm cstate drbd0
Connected
■プライマリからの結果
$ sudo drbdadm state drbd0
'drbdadm state' is deprecated, use 'drbdadm role' instead.
Primary/Secondary
$ sudo drbdadm show-gi drbd0
+--< Current data generation UUID >-
| +--< Bitmap's base data generation UUID >-
| | +--< younger history UUID >-
| | | +-< older history >-
V V V V
19D1FE4951F2490B:0000000000000000:0004000000000004:0003000000000004:1:1:1:1:0:0:0
^ ^ ^ ^ ^ ^ ^
-< Data consistency flag >--+ | | | | | |
-< Data was/is currently up-to-date >--+ | | | | |
-< Node was/is currently primary >--+ | | | |
-< Node was/is currently connected >--+ | | |
-< Node was in the progress of setting all bits in the bitmap >--+ | |
-< The peer's disk was out-dated or inconsistent >--+ |
-< This node was a crashed primary, and has not seen its peer since >--+
flags: Primary, Connected, UpToDate
■セカンダリからの結果
$ sudo drbdadm state drbd0
'drbdadm state' is deprecated, use 'drbdadm role' instead.
Secondary/Primary
$ sudo drbdadm show-gi drbd0
+--< Current data generation UUID >-
| +--< Bitmap's base data generation UUID >-
| | +--< younger history UUID >-
| | | +-< older history >-
V V V V
19D1FE4951F2490A:0000000000000000:0004000000000004:0003000000000004:1:1:0:1:0:0:0
^ ^ ^ ^ ^ ^ ^
-< Data consistency flag >--+ | | | | | |
-< Data was/is currently up-to-date >--+ | | | | |
-< Node was/is currently primary >--+ | | | |
-< Node was/is currently connected >--+ | | |
-< Node was in the progress of setting all bits in the bitmap >--+ | |
-< The peer's disk was out-dated or inconsistent >--+ |
-< This node was a crashed primary, and has not seen its peer since >--+
flags: Secondary, Connected, UpToDate
■設定を出力
$ sudo drbdadm dump
common {
protocol C;
}
resource drbd0 {
protocol C;
on xen-debian2 {
device /dev/drbd0 minor 0;
disk /dev/sdb1;
address ipv4 192.168.152.92:7789;
meta-disk internal;
}
on xen-debian1 {
device /dev/drbd0 minor 0;
disk /dev/sdb1;
address ipv4 192.168.152.91:7789;
meta-disk internal;
}
disk {
on-io-error pass_on;
}
syncer {
rate 50M;
}
}
■秒単位で同期が終わってしまう程度の書き込みでは同期中の状態も見れないので、
こんな感じでテスト。
■セカンダリ
$ sudo watch -d -n 1 'cat /proc/drbd'
■プライマリ
$ sudo dd if=/dev/zero of=/media/drbd0/test bs=512 count=100000
version: 8.3.11 (api:88/proto:86-96)
srcversion: F937DCB2E5D83C6CCE4A6C9
0: cs:Connected ro:Secondary/Primary ds:UpToDate/UpToDate C r-----
ns:0 nr:55780 dw:55780 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0
