HCL Cluster Network

Switch Management

The cluster is connected via two Cisco Catalyst 3560G switches. The ingress bandwidth on any physical port can be configured to any value between 8kb/s and 1Gb/s. The switches are connected to each other via a gigabit sfp cable.

As the <a href="Specs.html">Cluster Specifications</a> show, each node has two Network Interfaces, each with its own IP Address. Each eth0 is connected to switch 1, and each eth1 is connected to switch 2. Which topology you wish to use will determine which IP address you should use when referring to each machine.

 

Switch Access

The switches can be accessed through telnet from any machine on the cluster network. Type
telnet 192.168.21.252 for switch1 or telnet 192.168.21.253 for switch2, and the switch should prompt for a password. For the password, email brett becker.

 

Node Configuration

Here[1] is an excellent description how to introduce packet delays for outgoing traffic, which is a way to control latency.

 

Switch Configuration

For example, we will demonstrate how to limit the bandwidth of the connection between hcl01 and hcl02 to 100Mbps:
-----------------------------------
hcl02 $> telnet 192.168.21.252
     Trying 192.168.21.252...
     Connected to 192.168.21.252 (192.168.21.252).
     Escape character is '^]'.

    User Access Verification

Password: <enter password>
hclswitch1>enable
     Password: <enter password>
hclswitch1#configure terminal
     Enter configuration commands, one per line. End with CNTL/Z.
hclswitch1(config)#policy-map example
hclswitch1(config-pmap)#class ipclass1
hclswitch1(config-pmap-c)#police 100000000 800000 exceed-action drop
hclswitch1(config-pmap-c)#exit
hclswitch1(config-pmap)#exit
hclswitch1(config)#interface gigabitEthernet0/3
hclswitch1(config-if)#service-policy input example
hclswitch1(config-if)#exit
hclswitch1(config)#interface gigabitEthernet0/4
hclswitch1(config-if)#service-policy input example
hclswitch1(config-if)#exit
hclswitch1#show policy-map
   Policy Map example
    Class ipclass1
     police 100000000 800000 exceed-action drop
hclswitch1#exit
     Connection closed by foreign host.

hcl02 $>
----------------------------------------

In the above example, we telnet to the switch and enter the password. We then type “enable” and enter the password again. Then we type “configure terminal” to bring us into config mode. We then create a new policy-map named “example” We bind this policy-map to class “ipclass1”. ipclass1 is a class that incorporates all ports on the switch. PLEASE DO NOT REMOVE THIS CLASS! We then enter the key statement police 100000000 800000 exceed-action drop. This tells the switch that this policy-map is to limit the bandwidth to 100Mbps, with a bucket burst of 80000, and if the rate is exceeded packets are to be dropped. We then exit policy-map configuration and enter interface gigabitEthernet0/3. This is the port that is connected to hcl03. We then attach the policy-map “example” to this interface. We then do the same for port gigabitEthernet0/4 which is connected to hcl04. The policy-map is then viewed using the show command to ensure that it is correct.

This example demonstrates an important fact about the cluster. hcl01’s eth0 (192.168.21.3) is connected to port gigabitEthernet0/1 on switch1. Similarly, hclx’s eth0 (192.168.21.x+2) is connected to gigabitEthernet0.x on switch1.
Further, hcl01’s eth1 (192.168.21.103) is connected to gigabitEthernet0/1 on switch2. Similarly, hclx’s eth1 (192.168.21.100+x+2) is connected to gigabitEthernet0/x on switch2.
The table below shows this comprehensively.

Machine	IP Address	Switch1 Port	Switch2 Port
hcl01	192.168.21.3	gigabitEthernet0/1	N/A
hcl01_eth1	192.168.21.103	N/A	gigabitEthernet0/1
hcl02	192.168.21.4	gigabitEthernet0/2	N/A
hcl02_eth1	192.168.21.104	N/A	gigabitEthernet0/2
hcl03	192.168.21.5	gigabitEthernet0/3	N/A
hcl03_eth1	192.168.21.105	N/A	gigabitEthernet0/3
hcl04	192.168.21.6	gigabitEthernet0/4	N/A
hcl04_eth1	192.168.21.106	N/A	gigabitEthernet0/4
hcl05	192.168.21.7	gigabitEthernet0/5	N/A
hcl05_eth1	192.168.21.107	N/A	gigabitEthernet0/5
hcl06	192.168.21.8	gigabitEthernet0/6	N/A
hcl06_eth1	192.168.21.108	N/A	gigabitEthernet0/6
hcl07	192.168.21.9	gigabitEthernet0/7	N/A
hcl07_eth1	192.168.21.109	N/A	gigabitEthernet0/7
hcl08	192.168.21.10	gigabitEthernet0/8	N/A
hcl08_eth1	192.168.21.110	N/A	gigabitEthernet0/8
hcl09	192.168.21.11	gigabitEthernet0/9	N/A
hcl09_eth1	192.168.21.111	N/A	gigabitEthernet0/9
hcl10	192.168.21.12	gigabitEthernet0/10	N/A
hcl10_eth1	192.168.21.112	N/A	gigabitEthernet0/10
hcl11	192.168.21.13	gigabitEthernet0/11	N/A
hcl11_eth1	192.168.21.113	N/A	gigabitEthernet0/11
hcl12	192.168.21.14	gigabitEthernet0/12	N/A
hcl12_eth1	192.168.21.114	N/A	gigabitEthernet0/12
hcl13	192.168.21.15	gigabitEthernet0/13	N/A
hcl13_eth1	192.168.21.115	N/A	gigabitEthernet0/13
hcl14	192.168.21.16	gigabitEthernet0/14	N/A
hcl14_eth1	192.168.21.116	N/A	gigabitEthernet0/14
hcl15	192.168.21.17	gigabitEthernet0/15	N/A
hcl15_eth1	192.168.21.117	N/A	gigabitEthernet0/15
hcl16	192.168.21.18	gigabitEthernet0/16	N/A
hcl16_eth1	192.168.21.118	N/A	gigabitEthernet0/16

Limiting the bandwidth between the two switches is similar to the above example, but on switch1, Interface gigabitEthernet0/25 should be assigned the desired policy-map. Then on switch2, the same should be done.

 

Simulating Two Clusters

If you want to simulate two clusters with the cluster, the following example should be considered.

Say we want Cluster A to be comprised of hcl01 and hcl02, and Cluster B, to comprised of hcl03 and hcl04. We want hcl01 and hcl02 to “talk” to each other at 10Mbps, and hcl03 and hcl04 to “talk” at 1Gbps. Additionally, we want to restrict the link between Clusters A and B to a bandwidth of 250Mbps.
To accomplish this we perform the following steps:

1.) Log onto hcl01 and make sure that eth0 is active and eth1 is not. This means that the machine is connected to switch1.

2.) Do the same for hcl02

3.) Log onto switch1, and create a policy-map to limit the bandwidth to 10Mbps (10000000bps). Attach this policy-map to gigabitEthernet0/1 and gigabitEthernet0/2.

4.) Log onto hcl03 and perform the following steps:

hcl03 $> /sbin/ifconfig
This will list the active network devices. If the user before you cleaned up after him/her self, only “lo” (the loopback interface) and “eth0” should be listed.
hcl03 $> /sbin/ifup eth1 (debian command is sudo /sbin/ifup eth1)
hcl03 $> /sbin/ifdown eth0 (debian command is sudo /sbin/ifdown eth0)

This connects hcl03 to switch2, as eth1 is connected to switch2. Then hcl03 is disconnected from switch1, as eth0 is connected to switch1. MAKE SURE YOU ALWAYS BRING ONE INTERFACE UP BEFORE YOU BRING ANOTHER DOWN. OTHERWISE THE MACHINE WILL BE ISOLATED WITH NO ACTIVE NETWORK DEVICES. To see what devices are currently active, use the /sbin/ifconfig command.

5.) The same should be done for hcl04.

6.) Since we want these machines to talk at 1Gbps, we should log onto switch2 and make sure that there are no policy-maps existing.

7.) Log onto switch1 and create a new policy-map, limit it to 250000000bps and attach it to gigabitEthernet0/25. Do the same for switch2.

Done!

The following example shows how to delete a policy-map. YOU SHOULD ALWAYS REMEMBER TO DELETE YOUR POLICY-MAPS WHEN YOUR JOBS ARE DONE SO OTHER USER’S JOBS DON’T GET MESSED UP!

--------------------------------------------------------------
hcl0x $> telnet 192.168.21.252
     Trying 192.168.21.252...
     Connected to 192.168.21.252 (192.168.21.252).
     Escape character is '^]'.

     User Access Verification
Password: <enter password>
hclswitch1>enable
Password:
hclswitch1#show policy-map
   Policy Map example
    Class ipclass1
     police 100000000 800000 exceed-action drop
hclswitch1#config t
     Enter configuration commands, one per line. End with CNTL/Z.
hclswitch1(config)#no policy-map example
hclswitch1(config)#exit
hclswitch1#show policy-map

hclswitch1#exit
     Connection closed by foreign host.

hcl0x $>
---------------------------------------------------------------