HA-DRBD Heartbeat 建置 MySQL 高可用性

 #yum -y install drbd83 kmod-drbd83
 #yum -y install heartbeat
 #yum -y install heartbeat heartbeat-ldirectord heartbeat-pils heartbeat-stonith
 #yum -y install mysql-server

安裝好後修改 mysql 設定檔，將 mysql 資料庫存放路徑由預設的 /var/lib/mysql 修改至屆時會執行硬碟資料同步的 /db。(Node1 及 Node2 都必須設定)

Modifying mysql configuration file & #xff0c when installed; modifying mysql repository path by default /var/lib/mysql will execute hard disk data synchronization/ db by the end of the term. (Node1 and Node2 have to be set)

 #vi /etc/my.cnf
 [mysqld]
 datadir=/var/lib/mysql                      //預設值
 datadir=/db                                 //修改後
 socket=/var/lib/mysql/mysql.sock
 user=mysql
 ...略...

步驟5.修改 drbd 設定檔

修改 drbd 設定檔，設定檔內容中若您的 CentOS 為 32 bit 則安裝 heartbeat 後路徑為 /usr/lib/heartbeat，若 CentOS 為 64 bit 則 heartbeat 安裝後路徑為 /usr/lib64/heartbeat。(Node1 及 Node2 都必須設定)

Modify the drbd configuration file & #xff0c; install the heartbeat after / usr/ lib/ heartbeat& #xff0c in the configuration file content if your CentOS is 32 bit; if the DentOS is 64 bit after / usr/ lib64/ heartbeat. (Node1 and Node2 have to be set)

 #vi /etc/drbd.conf
 global {
   minor-count 64;
   usage-count yes;
 }
 common {
   syncer { rate 1000M; }
 }
 resource ha {
   protocol C;
   handlers {
     pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
     pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
     local-io-error "/usr/lib/drbd/notify-local-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
     fence-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
     pri-lost "/usr/lib/drbd/notify-pri-lost.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
     split-brain "/usr/lib/drbd/notify-split-brain.sh root";
     out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
   }
   startup {
     wfc-timeout 60;
     degr-wfc-timeout 120;
     outdated-wfc-timeout 2;
   }
   disk {
     on-io-error detach;
     fencing resource-only;
   }
   syncer {
     rate 1000M;
   }
   on node1.weithenn.org {
     device /dev/drbd0;
     disk /dev/hdb1;
     address 192.168.1.1:7788;
     meta-disk internal;
   }
   on node2.weithenn.org {
     device /dev/drbd0;
     disk /dev/hdb1;
     address 192.168.1.2:7788;
     meta-disk internal;
   }
 }

完成後請調整 drbd 執行檔權限以便後續指令能順利執行，若此步驟省略的話後續執行過程會出現錯誤。(Node1 及 Node2 都必須設定)

After completion, adjust the permissions of the drbd executable so that the subsequent command can run xff0c without delay; there will be an error in running the process after this step has been omitted. (Node1 and Node2 have to be set)

 #chgrp haclient /sbin/drbdsetup
 #chmod o-x /sbin/drbdsetup
 #chmod u+s /sbin/drbdsetup
 #chgrp haclient /sbin/drbdmeta
 #chmod o-x /sbin/drbdmeta
 #chmod u+s /sbin/drbdmeta

步驟6.載入 drbd 模組並建立 resource

使用指令 modprobe 指令來載入 drbd 模組，載入完成後使用指令 drbdadm create-md 來建立 drbd resource，由於我們在 drbd 設定檔中指令 resource 名稱為 ha 所以指令便輸入 ha。(Node1 及 Node2 都必須設定)

Use the command modprobe command to load the drbd module & #xff0c; use the command drbdadm Create-md to create drbd resources， after loading; since we command the source name ha in the drbd configuration file, the command must be set. (Node1 and Node2 must be set)

 #modprobe drbd                                    //載入 drbd 模組
 #lsmod|grep drbd                                  //確認 drbd 模組是否載入
 drbd                  228528  0
 #dd if=/dev/zero of=/dev/hdb1 bs=1M count=100     //把一些資料塞到 hdb 內 (否則 create-md 時有可能會出現錯誤)
 #drbdadm create-md ha                             //建立 drbd resource
 #service drbd start                               //啟動 drbd 服務
 #chkconfig drbd on                                //設定 drbd 開機時自動啟動

啟動完成後可以使用 service drbd status 指令來查看目前 drbd 的狀態，當 node1 啟動 drbd 服務而 node2 尚未啟動時會發現狀態為 Secondary/Unknown，當 node2 啟動 drbd 服務後則會發現狀態為 Secondary/Secondary，而 ds 狀態為 Inconsistent 表示二台主機資料尚未同步。

The current status of drbd & #xff0c can be viewed using the service drbd status command after startup has been completed; when node1 starts drbd service and node2 is not started, the status is Second/ Unknown， when node2 starts the drbd service, the status is Condary/Secondary， and ds is Inconsistent indicates that the host data has not yet been synchronized.

 #service drbd status
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                   ds                         p  mounted  fstype
 0:ha   Connected  Secondary/Secondary  Inconsistent/Inconsistent  C 

步驟1.初始化二台主機 hdb 資料

確定 Node1 及 Node2 主機都可偵測到對方 (Secondary/Secondary) 後我們設定 Node1 為 Primary Node，並使二台主機開始同步 hdb 硬碟資料 (屆時的 /dev/drbd0)，此時查看 drbd 狀態可發現同步的百分比及進度。 (僅 Node1 主機執行)

Make sure that Node1 and Node2 hosts can detect each other (Secondary/Secondary) and we set Node1 as Prime Node， and allow the two hosts to start synchdb hard disk data (end /dev/drbd0) & #xff0c; then check the percentage and progress of drbd status to detect synchronization. (Node1 host only)

 #drbdadm -- --overwrite-data-of-peer primary ha
 #service drbd status
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs          ro                 ds                     p             mounted  fstype
 ...    sync'ed:    5.7%               (9660/10236)M          delay_probe:
 0:ha   SyncSource  Primary/Secondary  UpToDate/Inconsistent  C

當同步完成後可發現二台主機的 ds 狀態都為 UpToDate 表示同步完成，二台主機擁有相同且最新的資料。

When synchronisation is completed, the ds status of the two hosts is found to be UpToDate for synchronization & #xff0c; the two hosts have the same and up-to-date data.

在 Node1 主機執行看到下列狀態 (此台為 Primary Node)

sees the following state on the Node1 host (this is Primary Node)

 #service drbd status
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                 ds                 p  mounted  fstype
 0:ha   Connected  Primary/Secondary  UpToDate/UpToDate  C

 #service drbd status
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                 ds                 p  mounted  fstype
 0:ha   Connected  Secondary/Primary  UpToDate/UpToDate  C

 #mkfs.ext3 /dev/drbd0
 mke2fs 1.39 (29-May-2006)
 Filesystem label=
 OS type: Linux
 Block size=4096 (log=2)
 Fragment size=4096 (log=2)
 1310720 inodes, 2621333 blocks
 131066 blocks (5.00%) reserved for the super user
 First data block=0
 Maximum filesystem blocks=2684354560
 80 block groups
 32768 blocks per group, 32768 fragments per group
 16384 inodes per group
 Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632
 Writing inode tables: done
 Creating journal (32768 blocks): done
 Writing superblocks and filesystem accounting information: done
 This filesystem will be automatically checked every 30 mounts or
 180 days, whichever comes first.  Use tune2fs -c or -i to override.
 #mount /dev/drbd0 /db
 #df -h
 Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
                       19G  2.7G   15G  16% /
 /dev/hda1              99M   12M   82M  13% /boot
 tmpfs                 252M     0  252M   0% /dev/shm
 /dev/drbd0            9.9G  151M  9.2G   2% /db

 #service mysqld start
 #ll /db
 total 20544
 -rw-rw---- 1 mysql mysql 10485760 Nov  1 14:48 ibdata1
 -rw-rw---- 1 mysql mysql  5242880 Nov  1 14:48 ib_logfile0
 -rw-rw---- 1 mysql mysql  5242880 Nov  1 14:48 ib_logfile1
 drwx------ 2 mysql mysql    16384 Nov  1 14:46 lost+found
 drwx------ 2 mysql mysql     4096 Nov  1 14:48 mysql             //預設資料庫
 drwx------ 2 mysql mysql     4096 Nov  1 14:48 test              //測試資料庫

 #service mysqld stop                                             //停止 mysql 服務
 #umount /dev/drbd0                                               //卸載 /dev/drbd0
 #drbdadm secondary ha                                            //在 node1 主機執行將其降為 Secondary Node
 #service drbd status                                             //確認二台主機都為 Secondary Node
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                   ds                 p  mounted  fstype
 0:ha   Connected  Secondary/Secondary  UpToDate/UpToDate  C

 #vi /etc/ha.d/ha.cf
 debugfile /var/log/ha-debug                   //heartbeat 除錯記錄檔
 logfile /var/log/ha-log                       //heartbeat 記錄檔
 logfacility local0                            //記錄檔的記錄等級
 autojoin none
 ucast eth0 10.10.25.122                       //此為 Node1 主機的設定，若是 Node2 主機請設定為 10.10.25.121
 ucast eth1 192.168.1.2                        //此為 Node1 主機的設定，若是 Node2 主機請設定為 192.168.1.1
 ping 10.10.25.254                             //IP 連線偵測用，設定區網內的 Gateway (當網路或 Heartbeat 失效時測試用)
 respawn hacluster /usr/lib/heartbeat/ipfail
 respawn hacluster /usr/lib/heartbeat/dopd
 apiauth dopd gid=haclient uid=hacluster
 udpport 694                                   //使用 UDP Protocol 及 694 Port 來通訊
 warntime 5                                    //網路發生狀況時告警時間 5 秒
 deadtime 15                                   //網路斷線狀況發生 15 秒則判定網路出問題 (Secondary Node 準備接手 Primary Node 的服務)
 initdead 60
 keepalive 2                                   //每 2 秒 Node 互相偵測一次 (使用 ping)
 node node1.weithenn.org
 node node2.weithenn.org
 auto_failback off                             //當 Primary Node 發生問題 Secondary Node 接手後若本來的 Primary Node 修復後是否要把目前的 Primary Node 搶回

 #vi /etc/ha.d/haresources
 node1.weithenn.org 10.10.25.120 drbddisk::ha Filesystem::/dev/drbd0::/db::ext3 mysql

 #( echo -ne "auth 1
1 sha1 "; dd if=/dev/urandom bs=512 count=1 | openssl md5) > /etc/ha.d/authkeys
 #cat /etc/ha.d/authkeys
 auth 1
 1 sha1 71461fc5e160d7846c2f4b524f952128
 #chmod 600 /etc/ha.d/authkeys
 #scp /etc/ha.d/authkeys node2:/etc/

 #vi /etc/ha.d/resource.d/mysql                    //內容如下
 #!/bin/bash
 . /etc/ha.d/shellfuncs
 case "$1" in
 start)
   res=`/etc/init.d/mysqld start`
   ret=$?
   ha_log $res
   exit $ret
   ;;
 stop)
   res=`/etc/init.d/mysqld stop`
   ret=$?
   ha_log $res
   exit $ret
   ;;
 status)
   if [[ `ps -ef | grep '[m]ysqld'` > 1 ]]; then
      echo "running"
   else
      echo "stopped"
   fi
   ;;
 *)
   echo "Usage: mysqld {start|stop|status}"
   exit 1
   ;;
 esac
 exit 0
 #chmod 755 /etc/ha.d/resource.d/mysql           //將此檔案權限設定為可執行

 #chkconfig --add heartbeat
 #ll /etc/rc3.d/S7*
 lrwxrwxrwx 1 root root 14 Nov  1 14:25 /etc/rc3.d/S70drbd -> https://blog.csdn.net/liaoyuanzi/article/init.d/drbd
 lrwxrwxrwx 1 root root 19 Nov  1 15:01 /etc/rc3.d/S75heartbeat -> https://blog.csdn.net/liaoyuanzi/article/init.d/heartbeat
 #chkconfig heartbeat on

 #service heartbeat start

 * 掛載 /dev/drbd0 至 /db  (Node2 主機此時無法掛載 /dev/drbd0 至 /db)
 * 啟動 mysqld 服務
 * 擁有 Cluster IP 10.10.25.120 (eth0:0)

 watch -n 1 service drbd status

 #mysqladmin -u root password 'weithenn'                 //設定 MySQL 管理者密碼為 weithenn
 #mysqladmin -p create testdb1                           //建立名稱為 testdb1 的資料庫 (會要求輸入 MySQL 管理者密碼)
 #mysqlshow -p                                           //顯示資料庫 (會要求輸入 MySQL 管理者密碼)

Primary Node for Node1 host & #xff0c; reboot for startbeat service

 [root@node1 ~]#service heartbeat restart
 Stopping High-Availability services:
                                                           [  OK  ]
 Waiting to allow resource takeover to complete:
                                                           [  OK  ]
 Starting High-Availability services:
 2010/11/03_11:31:41 INFO:  Resource is stopped
                                                           [  OK  ]

當上述指令執行時您會看到 Node1 主機中 DRBD 狀態變化為 Primary/Secondary >> Secondary/Secondary >> Secondary/Primary，當上述指令執行完成後狀態如下

When this command is executed, you will see a change in the DRBD status in Node1 host to Primary/Secondary & gt; > Security/Secondary & gt; > Security/Primary& #xff0c; when the command is executed as follows:

 [root@node1 ~]#service drbd status
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                 ds                 p  mounted  fstype
 0:ha   Connected  Secondary/Primary  UpToDate/UpToDate  C

此時在 Node2 主機您可看到接手 Primary Node 角色 (掛載 /dev/drbd0 至 /db、接手 mysqld 服務、啟動 Cluster IP 10.10.25.120 及網卡 eth0:0)

On the Node2 host you can see the assumed Primary Node role (mount / dev/ drd0 to / db, take over mysqld service, start Cluster IP 10.10.25.120 and eth0:0)

 [root@node2 ~]#service drbd status                                         //接手 /dev/drbd0
 drbd driver loaded OK; device status:
 version: 8.3.8 (api:88/proto:86-94)
 GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16
 m:res  cs         ro                 ds                 p  mounted  fstype
 0:ha   Connected  Primary/Secondary  UpToDate/UpToDate  C  /db      ext3
 [root@node2 ~]# service mysqld status                                      //接手 mysqld 服務
 mysqld (pid 10101) is running...
 [root@node2 ~]# ifconfig eth0:0                                            //接手 Cluster IP
 eth0:0    Link encap:Ethernet  HWaddr 00:03:FF:1E:7B:82
           inet addr:10.10.25.120  Bcast:10.10.25.255  Mask:255.255.255.0
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           Interrupt:11 Base address:0xc000

測試2.當主機網路斷線

模擬當網路斷線時是否會自動切換 Node，您可將插在 eth0 網路線拔掉或在 Primary Node 上執行 ifdown eth0 指令去關閉 eth0 網卡來進行 Node 切換測試。

Simulates whether Node， is automatically switched when the network is disconnected; you can deactivate the eth0 network or execute the edown eth0 command to close the eth0 card for the Node switch test.

1. 拔除 eth0 上的網路線或執行 ifdown eth0 指令來關閉網卡連線
2. Primary Node 切換為 Secondary Node，而原 Secondary Node 接手為 Primary Node

測試3.當網路卡及交換器損壞

若您有多片網卡時可設定網卡 Bonding (NIC Teaming) 功能並接在不同的實體網路交換器上，如此一來即可預防網路卡損壞或網路交換器損壞而造成的斷線問題。

If you have more than one Internet card, you can configure the web card Bonding (NIC Teaming) functionality and connect it to a different entity network exchanger & #xff0c; so that you can pre-empt the disconnection problems caused by the loss of the Internet card or the loss of the network exchanger.

機器運作日子一久難免會遇到機器損壞或其它因素而需要停機，當然單台停機時我們可利用停止 Heartbeat 服務來達成切換 Node 的目的進而維修硬體，若遇到特殊情況需要將二台主機都停機的情況下正確的開機/關機流程是必要的，以下便是說明正確的開機/關機流程。

The machine runs for a long time with machine damage or other factors requiring a shutdown & #xff0c; of course, when a single station stops, we can use the Heartbeat service to switch Node to hardware & #xff0c; the correct start/off process is necessary if special circumstances require both hosts to be shut down & #xff0c; the following is the correct start/off process.

正確開機流程

當二台 Node 主機都關機而預設 Node1 為 Primary Node 的情況下，正確的開機流程為：

When both Node hosts are switched off and Node1 is set to be Primary Node xff0c; the correct startup process is xff1a;

1. Power On Node1 主機
2. 過約 1 分鐘後再 Power On Node2 主機
3. 當 Node1 主機開機流程運作到 DRBD 程序時會等待 60 秒 (等待 Secondary Node 回應)，在數秒期間 Node2 主機開機流程也運作到 DRBD 程序時會去找 Node1 主機進行溝通
4. Node1 順利開機完成，Node2 也開機完成
5. 登入 Node1、Node2 主機查看 DRBD 狀態及相關資訊 (掛載 /dev/drbd0 至 /db、接手 mysqld 服務、啟動 Cluster IP 10.10.25.120 及網卡 eth0:0)

正確關機流程

當遇到不可抗力因素必須要二台主機都關機且 Node1 為 Primary Node 的情況下，正確的關機流程為：

When a force majeure factor is encountered, both hosts must be shut down and Node1 must be Primary Node xff0c; the correct shutdown process is xff1a;

1. 在 Node1 主機執行 service heartbeat stop 指令將 Heartbeat 服務關閉
2. 確認 Node2 主機接手成為 Primary Node 且相關服務運作正常
3. 將 Node1 主機關機 (此時在 Node2 主機看到的 DRBD 狀態為 Primary/Unknown)
4. 在 Node2 主機上執行 service mysqld stop 指令將 MySQL 服務關閉
5. MySQL 服務關閉完成後即可將 Node2 主機關機
6. 之後開機時再遵照上述的?正確開機流程?步驟即可順利讓 Cluster 再度運作

因為出現不同的設定需求 (加裝網卡所以要設定網卡為 Bonding 模式、更換 IP Address、伺服器要換地方)，因此除了上述的正常開機/關機流程是不夠的，順便記錄一下整個流程：

xff0c due to different configuration needs (install the webcards as Bonding mode, change IP Address, change the server to a different location); therefore, the normal start-up/cut-off process mentioned above is not enough xff0c; log the entire process xff1a;

1. 先將 Node1、Node2 開機啟動 drbd、heartbeat 服務關閉
2. 於 Node1 主機將 heartbeat 服務停止，確認 Node2 主機接手 HA 服務後即可將 Node1 主機關機
3. 於 Node2 主機將 mysqld 服務停止後即可關機
4. 將 Node1、Node2 退出機櫃後搬移至新地點，上架前先安裝擴充的網卡此時因為開機沒有啟動 HA 服務所以沒有開機順序的問題
5. 開機完成後設定 Node1、Node2 的網卡 Bonding 模式及修改 IP Address 並測試容錯機制是否運作
6. 修改 drbd、heartbeat 相關設定檔內容 (IP Address)
7. 將 Node1、Node2 開機啟動 drbd、heartbeat 服務啟用後先將 Node1 重新開機約 1 分鐘後再將 Node2 重新開機
8. 開機完成後確定 HA 機制是否運作並測試 HA 機制
9. 完成改設定及移機要求