labunix's blog

labunixのラボUnix

vSRXのシャーシクラスタ(HA構成/A-P)を試してみる。

■vSRXのシャーシクラスタ(HA構成/A-P)を試してみる。
 Active/Passive方式の冗長化構成。

■vSRX 15.1X49-D15は以下。
 2GBのダウンロードに2時間位かかりそうなので、
 今回は以前のバージョンで試す。

 Start your 60-day free trial today.
 http://www.juniper.net/us/en/dm/free-vsrx-trial/

■vSRX 12.1X47-D20.7は以下でダウンロード済み。

 vmplayerでvSRXを試す。
 http://labunix.hateblo.jp/entry/20150412/1428767862

 vSRXの電源OFFを行うタイミングについて確認してみる。
 http://labunix.hateblo.jp/entry/20150830/1440878778

 【小休憩】この先に進む前にvSRXの簡単な正常性確認をしてみる。
 http://labunix.hateblo.jp/entry/20150902/1441197271

■Chassis Cluster(HA冗長化機能/JSRP)
 [JUNOS Services Redundancy Protocol]の略称

$ echo -e "\
    [ge-0/0/0] -- Management/Fixed(fxp0)    --> [ge-7/0/0] \n\
    [ge-0/0/1] -- Control Link/Fixed(fxp1)  --> [ge-7/0/1] \n\
    [ge-0/0/2] -- (fab0)Data Link/Any(fab1) --> [ge-7/0/2] \n\
    [ge-0/0/3] -- Untrusti(reth0)           --> [ge-7/0/3] \n\
    [ge-0/0/4] -- Trust(reth1)              --> [ge-7/0/4] \n\
          " | sort -r | graph-easy --dot | dot -Tpng -o vSRX-HA.png

f:id:labunix:20150919032033p:plain

■HA構成の概要
 ※今回は冗長化構成が目的なので、Untrust/Trustを明確に分けていない。

 IPを持たない仮想マシン同士が通信出きるようプロミスキャスモードを設定
 OVAをVMXに変換し、仮想マシンを起動
 オペレーションモードでクラスタIDとノードIDを設定し、再起動
 ホスト名とマネージメントポート(fxp0)を設定
 RedundancyGroup(reth0,reth1)の設定、プライオリティを付与
 ファブリックリンク(fab0/fab1)を設定
 コントロールリンク(fxp1)の設定は不要
 J-Webアクセス設定

■IPを持たない仮想マシン同士が通信出きるようプロミスキャスモードを設定
 ESXiの場合は検索すれば沢山ある。
 今回はVMPlayerでvmnet*をプロミスキャスモードにする。
 ※サービス再起動のために仮想マシンはすべて停止していること。

 VMware WorkStation 10のvmnet*をプロミスキャスモードにする。
 http://labunix.hateblo.jp/entry/20150607/1433634463

$ sudo cp /etc/init.d/{vmware,vmware.bak}
$ diff -c /etc/init.d/{vmware,vmware.bak}
*** /etc/init.d/vmware	2015-09-18 23:10:28.152210702 +0900
--- /etc/init.d/vmware.bak	2015-09-18 23:09:29.719250472 +0900
***************
*** 156,163 ****
  vmwareStartVmnet() {
     vmwareLoadModule $vnet
     "$BINDIR"/vmware-networks --start >> $VNETLIB_LOG 2>&1
-    chgrp promiscuous /dev/vmnet*
-    chmod g+rw /dev/vmnet*
  }
  
  # Stop the virtual ethernet kernel service
--- 156,161 ----

$ USER=`whoami`; \
  grep promiscuous /etc/group > /dev/null || sudo groupadd promiscuous; \
  id -u $USER | grep promiscuous > /dev/null || sudo usermod -a -G promiscuous $USER; \
  grep "promiscuous" /etc/group; \
  ls -l /dev/vmnet*
promiscuous:x:1001:labunix
crw-rw---- 1 root promiscuous 119, 0  917 00:08 /dev/vmnet0
crw-rw---- 1 root promiscuous 119, 1  917 00:08 /dev/vmnet1
crw-rw---- 1 root promiscuous 119, 2  917 00:08 /dev/vmnet2
crw-rw---- 1 root promiscuous 119, 8  917 00:08 /dev/vmnet8

$ sudo /etc/init.d/vmware restart
Stopping VMware services:
   VMware Authentication Daemon                                        done
   VM communication interface socket family                            done
   Virtual machine communication interface                             done
   Virtual machine monitor                                             done
   Blocking file system                                                done
Starting VMware services:
   Virtual machine monitor                                             done
   Virtual machine communication interface                             done
   VM communication interface socket family                            done
   Blocking file system                                                done
   Virtual ethernet                                                    done
   VMware Authentication Daemon                                        done
   Shared Memory Available                                             done

■OVAをVMXに変換し、仮想マシンを起動
 毎度お馴染みちり紙交換ですね。
 NICはデフォルト2つ+HA構成用に3つの計5つを起動前に用意します。

$ ls vSRX_node*/*.vmx | awk '{print "vmrun -T player start "$1" &"}' | sh
Opening OVA source: junos-vsrx-12.1X47-D20.7-domestic.ova
The manifest validates
Opening VMX target: vSRX_node0/vSRX_node0.vmx
Writing VMX file: vSRX_node0/vSRX_node0.vmx
Transfer Completed                    
Source is signed but could not verify certificate (possibly self-signed) 
Completed successfully
Opening OVA source: junos-vsrx-12.1X47-D20.7-domestic.ova
The manifest validates
Opening VMX target: vSRX_node1/vSRX_node1.vmx
Writing VMX file: vSRX_node1/vSRX_node1.vmx
Transfer Completed                    
Source is signed but could not verify certificate (possibly self-signed) 
Completed successfully

$ grep ^ethernet[0-9]*.connectionType vSRX_node*/*.vmx
vSRX_node0/vSRX_node0.vmx:ethernet0.connectionType = "hostonly"
vSRX_node0/vSRX_node0.vmx:ethernet1.connectionType = "hostonly"
vSRX_node0/vSRX_node0.vmx:ethernet2.connectionType = "nat"
vSRX_node0/vSRX_node0.vmx:ethernet3.connectionType = "nat"
vSRX_node0/vSRX_node0.vmx:ethernet4.connectionType = "nat"
vSRX_node1/vSRX_node1.vmx:ethernet0.connectionType = "hostonly"
vSRX_node1/vSRX_node1.vmx:ethernet1.connectionType = "hostonly"
vSRX_node1/vSRX_node1.vmx:ethernet2.connectionType = "nat"
vSRX_node1/vSRX_node1.vmx:ethernet3.connectionType = "nat"
vSRX_node1/vSRX_node1.vmx:ethernet4.connectionType = "nat"

$ echo "仮想マシンのスタート"; \
  ls vSRX_node*/*.vmx | \
    awk '{print "vmrun -T player start "$1" & sleep 2;"}' | sh

■オペレーションモードでクラスタIDとノードIDを設定し、再起動
 シャーシクラスタの設定構文の最後に「reboot」を付けると再起動も行われる。
 起動後にきちんとお互いのノードを認識したことを確認。

#node0
login:root
root@% cli
root> configure
root# delete interfaces
root# delete security
root# set system root-authentication plain-text-password
root# commit and-quit
root> set chassis cluster cluster-id 1 node 0
warning: A reboot is required for chassis cluster to be enabled
root> request system reboot
Reboot the system ? [yes,no] (no) yes

#node1
delete interfaces
delete security
set system root-authentication plain-text-password
commit and-quit
set chassis cluster cluster-id 1 node 1

> show chassis cluster status | match node[01]
node0  1        primary         no      no        None
node0  1        secondary       no      no        None

■ホスト名とマネージメントポート(fxp0)を設定

$ ip a list vmnet1 | awk '/inet /{print $2}'
172.16.76.1/24

{primary:node0}
root> configure
root# set groups node0 system host-name vSRX-node0
root# set groups node0 interface fxp0 unit 0 family inet address 172.16.76.110/24
root# set groups node1 system host-name vSRX-node1
root# set groups node1 interface fxp0 unit 0 family inet address 172.16.76.111/24
root# set apply-groups ${node}
root# commit and-quit
root@% show configuration | display set | match groups
set groups node0 system host-name vSRX-node0
set groups node0 interface fxp0 unit 0 family inet address 172.16.76.110/24
set groups node1 system host-name vSRX-node1
set groups node1 interface fxp0 unit 0 family inet address 172.16.76.111/24
set apply-groups ${node}
commit and-quit

$ ssh root@172.16.76.110
root@vSRX-node0> show chassis cluster status 
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring              
    SP  SPU monitoring              SM  Schedule monitoring
    CF  Config Sync monitoring
 
Cluster ID: 1
Node   Priority Status         Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 1
node0  1        primary        no      no       None           
node1  1        secondary      no      no       None           

{primary:node0}

■RedundancyGroup(reth0,reth1)の設定、プライオリティを付与

$ ip a list vmnet8 | awk '/inet /{print $2}'
192.168.152.1/24

root@vSRX-node0> configure 
{primary:node0}[edit]
root@vSRX-node0# set chassis cluster reth-count 2
root@vSRX-node0# set chassis cluster redundancy-group 0 node 0 priority 100
root@vSRX-node0# set chassis cluster redundancy-group 0 node 1 priority 99
root@vSRX-node0# set chassis cluster redundancy-group 1 node 0 priority 100
root@vSRX-node0# set chassis cluster redundancy-group 1 node 1 priority 99
root@vSRX-node0# commit and-quit

root@vSRX-node0> configure 
{primary:node0}[edit]

root@vSRX-node0# set interfaces reth0.0 family inet address 192.168.152.110/24 
root@vSRX-node0# set interfaces reth0 redundant-ether-options redundancy-group 1 
root@vSRX-node0# set interfaces ge-0/0/3 gigether-options redundant-parent reth0
root@vSRX-node0# set interfaces ge-7/0/3 gigether-options redundant-parent reth0
 
root@vSRX-node0# set interfaces reth1.0 family inet address 192.168.152.111/24
root@vSRX-node0# set interfaces reth1 redundant-ether-options redundancy-group 1
root@vSRX-node0# set interfaces ge-0/0/4 gigether-options redundant-parent reth1 
root@vSRX-node0# set interfaces ge-7/0/4 gigether-options redundant-parent reth1
 
root@vSRX-node0# set routing-options static route 0/0 next-hop 192.168.152.1
root@vSRX-node0# commit and-quit

root@vSRX-node0> show configuration | display set | match reth 
set chassis cluster reth-count 2
set interfaces ge-0/0/3 gigether-options redundant-parent reth0
set interfaces ge-0/0/4 gigether-options redundant-parent reth1
set interfaces ge-7/0/3 gigether-options redundant-parent reth0
set interfaces ge-7/0/4 gigether-options redundant-parent reth1
set security zones security-zone external interfaces reth0.0
set security zones security-zone internal interfaces reth1.0
set security zones security-zone internal host-inbound-traffic system-services all

root@vSRX-node0> configure
{primary:node0}[edit]
root@vSRX-node0# set security policies from-zone internal to-zone external policy allow-all-internal match source-address any
root@vSRX-node0# set security policies from-zone internal to-zone external policy allow-all-internal match destination-address any
root@vSRX-node0# set security policies from-zone internal to-zone external policy allow-all-internal match application any
root@vSRX-node0# set security policies from-zone internal to-zone external policy allow-all-internal then permit
root@vSRX-node0# commit and-quit

root@vSRX-node0> show configuration | display set | match policies
set security policies from-zone internal to-zone external policy allow-all-internal match source-address any
set security policies from-zone internal to-zone external policy allow-all-internal match destination-address any
set security policies from-zone internal to-zone external policy allow-all-internal match application any
set security policies from-zone internal to-zone external policy allow-all-internal then permit

■ファブリックリンク(fab0/fab1)を設定

root@vSRX-node0> configure
{primary:node0}[edit]
root@vSRX-node0# set interfaces fab0 fabric-options member-interfaces ge-0/0/2
root@vSRX-node0# set interfaces fab1 fabric-options member-interfaces ge-7/0/2
root@vSRX-node0# commit and-quit

root@vSRX-node0> show configuration | display set | match fab 
set interfaces fab0 fabric-options member-interfaces ge-0/0/2
set interfaces fab1 fabric-options member-interfaces ge-7/0/2

■コントロールリンク(fxp1)の設定は不要

root@vSRX-node0> show interfaces fxp1 terse 
Interface               Admin Link Proto    Local                 Remote
fxp1                    up    up  
fxp1.0                  up    up   inet     129.16.0.1/2    
                                   tnp      0x1100001       

■J-Webアクセス設定

{primary:node0}
root@vSRX-node0> configure
root@vSRX-node0# set system services web-management http interface fxp0.0
root@vSRX-node0# commit and-quit
root@vSRX-node0> show configuration | display set | match http    
set system services web-management http interface ge-0/0/0.0
set system services web-management http interface fxp0.0
set system license autoupdate url https://ae1.juniper.net/junos/key_retrieval

{primary:node0}

■JUNOSバージョンの確認

root@vSRX-node0> show version 
node0:
--------------------------------------------------------------------------
Hostname: vSRX-node0
Model: firefly-perimeter
JUNOS Software Release [12.1X47-D20.7]

node1:
--------------------------------------------------------------------------
Hostname: vSRX-node1
Model: firefly-perimeter
JUNOS Software Release [12.1X47-D20.7]

■インターフェイスの確認

root@vSRX-node0> show interfaces terse | match "up.*up.*inet" 
fab0.0                  up    up   inet     30.17.0.200/24  
fab1.0                  up    up   inet     30.18.0.200/24  
fxp0.0                  up    up   inet     172.16.76.110/24
fxp1.0                  up    up   inet     129.16.0.1/2    
lo0.16384               up    up   inet     127.0.0.1           --> 0/0
lo0.16385               up    up   inet     10.0.0.1            --> 0/0
reth0.0                 up    up   inet     192.168.152.110/24
reth1.0                 up    up   inet     192.168.152.111/24

■シャーシクラスタの状態確認

root@vSRX-node0> show chassis cluster status 
Monitor Failure codes:
    CS  Cold Sync monitoring        FL  Fabric Connection monitoring
    GR  GRES monitoring             HW  Hardware monitoring
    IF  Interface monitoring        IP  IP monitoring
    LB  Loopback monitoring         MB  Mbuf monitoring
    NH  Nexthop monitoring          NP  NPC monitoring              
    SP  SPU monitoring              SM  Schedule monitoring
    CF  Config Sync monitoring
 
Cluster ID: 1
Node   Priority Status         Preempt Manual   Monitor-failures

Redundancy group: 0 , Failover count: 1
node0  100      primary        no      no       None           
node1  99       secondary      no      no       None           

Redundancy group: 1 , Failover count: 1
node0  100      primary        no      no       None           
node1  99       secondary      no      no       None           

■シャーシクラスタの情報

root@vSRX-node0> show chassis cluster information 
node0:
--------------------------------------------------------------------------
Redundancy Group Information:

    Redundancy Group 0 , Current State: primary, Weight: 255

        Time            From           To             Reason
        Sep 19 02:08:40 hold           secondary      Hold timer expired
        Sep 19 02:08:56 secondary      primary        Control & Fabric links down

    Redundancy Group 1 , Current State: primary, Weight: 255

        Time            From           To             Reason
        Sep 19 02:36:18 hold           secondary      Hold timer expired
        Sep 19 02:36:19 secondary      primary        Remote yield (0/0)

Chassis cluster LED information:
    Current LED color: Green
    Last LED change reason: No failures

node1:
--------------------------------------------------------------------------
Redundancy Group Information:           

    Redundancy Group 0 , Current State: secondary, Weight: 255

        Time            From           To             Reason
        Sep 19 02:08:56 hold           secondary      Hold timer expired

    Redundancy Group 1 , Current State: secondary, Weight: 255

        Time            From           To             Reason
        Sep 19 02:36:15 hold           secondary      Hold timer expired

Chassis cluster LED information:
    Current LED color: Green
    Last LED change reason: No failures

■シャーシクラスタのインターフェイス

root@vSRX-node0> show chassis cluster interfaces       
Control link status: Up

Control interfaces: 
    Index   Interface   Monitored-Status   Internal-SA
    0       fxp1        Up                 Disabled   

Fabric link status: Up

Fabric interfaces: 
    Name    Child-interface    Status
                               (Physical/Monitored)
    fab0    ge-0/0/2           Up   / Up  
    fab0   
    fab1    ge-7/0/2           Up   / Up  
    fab1   

Redundant-ethernet Information:     
    Name         Status      Redundancy-group
    reth0        Up          1                
    reth1        Up          1                
   
Redundant-pseudo-interface Information:
    Name         Status      Redundancy-group
    lo0          Up          0                

■コントロールプレーンの統計情報確認

root@vSRX-node0> show chassis cluster control-plane statistics 
Control link statistics:
    Control link 0:
        Heartbeat packets sent: 3771
        Heartbeat packets received: 3738
        Heartbeat packet errors: 0
Fabric link statistics:
    Child link 0
        Probes sent: 978
        Probes received: 977
    Child link 1
        Probes sent: 0
        Probes received: 0

■データプレーンのインターフェイスと統計

root@vSRX-node0> show chassis cluster data-plane interfaces    
fab0:

    Name               Status      
                       (Physical/Monitored)
    ge-0/0/2           Up   / Up        
fab1:

    Name               Status      
                       (Physical/Monitored)
    ge-7/0/2           Up   / Up        

root@vSRX-node0> show chassis cluster data-plane statistics | match TCP   
    JSF TCP STACK                             0            0