HCL Cluster Network
Switch Management
The cluster is connected via two Cisco Catalyst 3560G switches. The ingress bandwidth on any physical port can be configured to any value between 8kb/s and 1Gb/s. The switches are connected to each other via a gigabit sfp cable.
As the <a href="Specs.html">Cluster Specifications</a> show, each node has two Network Interfaces, each with its own IP Address. Each eth0 is connected to switch 1, and each eth1 is connected to switch 2. Which topology you wish to use will determine which IP address you should use when referring to each machine.
Switch Access
The switches can be accessed through telnet from any machine
on the cluster network. Type
telnet 192.168.21.252 for switch1 or telnet 192.168.21.253 for switch2, and
the switch should prompt for a password. For the password, email brett becker.
Node Configuration
Here[1] is an excellent description how to introduce packet delays for outgoing traffic, which is a way to control latency.
Switch Configuration
For example, we will demonstrate how to limit the bandwidth of the connection
between hcl01 and hcl02 to 100Mbps:
-----------------------------------
hcl02 $> telnet 192.168.21.252
Trying 192.168.21.252...
Connected to 192.168.21.252 (192.168.21.252).
Escape character is '^]'.
User Access Verification
Password: <enter password>
hclswitch1>enable
Password: <enter password>
hclswitch1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
hclswitch1(config)#policy-map example
hclswitch1(config-pmap)#class ipclass1
hclswitch1(config-pmap-c)#police 100000000 800000 exceed-action drop
hclswitch1(config-pmap-c)#exit
hclswitch1(config-pmap)#exit
hclswitch1(config)#interface gigabitEthernet0/3
hclswitch1(config-if)#service-policy input example
hclswitch1(config-if)#exit
hclswitch1(config)#interface gigabitEthernet0/4
hclswitch1(config-if)#service-policy input example
hclswitch1(config-if)#exit
hclswitch1#show policy-map
Policy Map example
Class ipclass1
police 100000000 800000 exceed-action drop
hclswitch1#exit
Connection closed by foreign host.
hcl02 $>
----------------------------------------
In the above example, we telnet to the switch and enter the password. We then type “enable” and enter the password again. Then we type “configure terminal” to bring us into config mode. We then create a new policy-map named “example” We bind this policy-map to class “ipclass1”. ipclass1 is a class that incorporates all ports on the switch. PLEASE DO NOT REMOVE THIS CLASS! We then enter the key statement police 100000000 800000 exceed-action drop. This tells the switch that this policy-map is to limit the bandwidth to 100Mbps, with a bucket burst of 80000, and if the rate is exceeded packets are to be dropped. We then exit policy-map configuration and enter interface gigabitEthernet0/3. This is the port that is connected to hcl03. We then attach the policy-map “example” to this interface. We then do the same for port gigabitEthernet0/4 which is connected to hcl04. The policy-map is then viewed using the show command to ensure that it is correct.
This example demonstrates an important fact about the cluster.
hcl01’s eth0 (192.168.21.3) is connected to port gigabitEthernet0/1 on
switch1. Similarly, hclx’s eth0 (192.168.21.x+2) is connected to gigabitEthernet0.x
on switch1.
Further, hcl01’s eth1 (192.168.21.103) is connected to gigabitEthernet0/1
on switch2. Similarly, hclx’s eth1 (192.168.21.100+x+2) is connected to
gigabitEthernet0/x on switch2.
The table below shows this comprehensively.
| Machine | IP Address | Switch1 Port | Switch2 Port |
| hcl01 | 192.168.21.3 | gigabitEthernet0/1 | N/A |
| hcl01_eth1 | 192.168.21.103 | N/A | gigabitEthernet0/1 |
| hcl02 | 192.168.21.4 | gigabitEthernet0/2 | N/A |
| hcl02_eth1 | 192.168.21.104 | N/A | gigabitEthernet0/2 |
| hcl03 | 192.168.21.5 | gigabitEthernet0/3 | N/A |
| hcl03_eth1 | 192.168.21.105 |
N/A | gigabitEthernet0/3 |
| hcl04 | 192.168.21.6 | gigabitEthernet0/4 | N/A |
| hcl04_eth1 | 192.168.21.106 | N/A | gigabitEthernet0/4 |
| hcl05 | 192.168.21.7 | gigabitEthernet0/5 | N/A |
| hcl05_eth1 | 192.168.21.107 | N/A | gigabitEthernet0/5 |
| hcl06 | 192.168.21.8 | gigabitEthernet0/6 | N/A |
| hcl06_eth1 | 192.168.21.108 | N/A | gigabitEthernet0/6 |
| hcl07 | 192.168.21.9 | gigabitEthernet0/7 | N/A |
| hcl07_eth1 | 192.168.21.109 | N/A | gigabitEthernet0/7 |
| hcl08 | 192.168.21.10 | gigabitEthernet0/8 | N/A |
| hcl08_eth1 | 192.168.21.110 | N/A | gigabitEthernet0/8 |
| hcl09 | 192.168.21.11 | gigabitEthernet0/9 | N/A |
| hcl09_eth1 | 192.168.21.111 | N/A | gigabitEthernet0/9 |
| hcl10 | 192.168.21.12 | gigabitEthernet0/10 | N/A |
| hcl10_eth1 | 192.168.21.112 | N/A | gigabitEthernet0/10 |
| hcl11 | 192.168.21.13 | gigabitEthernet0/11 | N/A |
| hcl11_eth1 | 192.168.21.113 | N/A | gigabitEthernet0/11 |
| hcl12 | 192.168.21.14 | gigabitEthernet0/12 | N/A |
| hcl12_eth1 | 192.168.21.114 | N/A | gigabitEthernet0/12 |
| hcl13 | 192.168.21.15 | gigabitEthernet0/13 | N/A |
| hcl13_eth1 | 192.168.21.115 | N/A | gigabitEthernet0/13 |
| hcl14 | 192.168.21.16 | gigabitEthernet0/14 | N/A |
| hcl14_eth1 | 192.168.21.116 | N/A | gigabitEthernet0/14 |
| hcl15 | 192.168.21.17 | gigabitEthernet0/15 | N/A |
| hcl15_eth1 | 192.168.21.117 | N/A | gigabitEthernet0/15 |
| hcl16 | 192.168.21.18 | gigabitEthernet0/16 | N/A |
| hcl16_eth1 | 192.168.21.118 | N/A | gigabitEthernet0/16 |
Limiting the bandwidth between the two switches is similar to the above example, but on switch1, Interface gigabitEthernet0/25 should be assigned the desired policy-map. Then on switch2, the same should be done.
Simulating Two Clusters
If you want to simulate two clusters with the cluster, the following example should be considered.
Say we want Cluster A to be comprised of hcl01 and hcl02, and
Cluster B, to comprised of hcl03 and hcl04. We want hcl01 and hcl02 to “talk”
to each other at 10Mbps, and hcl03 and hcl04 to “talk” at 1Gbps.
Additionally, we want to restrict the link between Clusters A and B to a bandwidth
of 250Mbps.
To accomplish this we perform the following steps:
1.) Log onto hcl01 and make sure that eth0 is active and eth1 is not. This means that the machine is connected to switch1.
2.) Do the same for hcl02
3.) Log onto switch1, and create a policy-map to limit the bandwidth to 10Mbps (10000000bps). Attach this policy-map to gigabitEthernet0/1 and gigabitEthernet0/2.
4.) Log onto hcl03 and perform the following steps:
hcl03 $> /sbin/ifconfig
This will list the active network devices. If the user before you cleaned up
after him/her self, only “lo” (the loopback interface) and “eth0”
should be listed.
hcl03 $> /sbin/ifup eth1 (debian command is sudo /sbin/ifup eth1)
hcl03 $> /sbin/ifdown eth0 (debian command is sudo /sbin/ifdown eth0)
This connects hcl03 to switch2, as eth1 is connected to switch2. Then hcl03 is disconnected from switch1, as eth0 is connected to switch1. MAKE SURE YOU ALWAYS BRING ONE INTERFACE UP BEFORE YOU BRING ANOTHER DOWN. OTHERWISE THE MACHINE WILL BE ISOLATED WITH NO ACTIVE NETWORK DEVICES. To see what devices are currently active, use the /sbin/ifconfig command.
5.) The same should be done for hcl04.
6.) Since we want these machines to talk at 1Gbps, we should log onto switch2 and make sure that there are no policy-maps existing.
7.) Log onto switch1 and create a new policy-map, limit it to 250000000bps and attach it to gigabitEthernet0/25. Do the same for switch2.
Done!
The following example shows how to delete a policy-map. YOU SHOULD ALWAYS REMEMBER TO DELETE YOUR POLICY-MAPS WHEN YOUR JOBS ARE DONE SO OTHER USER’S JOBS DON’T GET MESSED UP!
--------------------------------------------------------------
hcl0x $> telnet 192.168.21.252
Trying 192.168.21.252...
Connected to 192.168.21.252 (192.168.21.252).
Escape character is '^]'.
User Access Verification
Password: <enter password>
hclswitch1>enable
Password:
hclswitch1#show policy-map
Policy Map example
Class ipclass1
police 100000000 800000 exceed-action drop
hclswitch1#config t
Enter configuration commands, one per line. End with CNTL/Z.
hclswitch1(config)#no policy-map example
hclswitch1(config)#exit
hclswitch1#show policy-map
hclswitch1#exit
Connection closed by foreign host.
hcl0x $>
---------------------------------------------------------------
