https://hcl.ucd.ie/wiki/api.php?action=feedcontributions&user=Rhiggins&feedformat=atomHCL - User contributions [en]2024-03-28T16:43:23ZUser contributionsMediaWiki 1.27.1https://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=547HCL cluster2010-12-01T15:32:23Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>== General Information ==<br />
[[Image:Cluster.jpg|right|thumbnail||HCL Cluster]]<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram shows a schematic of the cluster.<br />
<br />
=== Detailed Cluster Specification ===<br />
* [[HCL Cluster Specifications]]<br />
* [[Old HCL Cluster Specifications]] (pre May 2010)<br />
<br />
=== Documentation ===<br />
* [[media:PE750.tgz|Dell Poweredge 750 Documentation]]<br />
* [[media:SC1425.tgz|Dell Poweredge SC1425 Documentation]]<br />
* [[media:X306.pdf|IBM x-Series 306 Documentation]]<br />
* [[media:E326.pdf|IBM e-Series 326 Documentation ]]<br />
* [[media:Proliant100SeriesGuide.pdf|HP Proliant DL-140 G2 Documentation]]<br />
* [[media:ProliantDL320G3Guide.pdf|HP Proliant DL-320 G3 Documentation]]<br />
* [[media:Cisco3560Specs.pdf|Cisco Catalyst 3560 Specifications]]<br />
* [[media:Cisco3560Guide.pdf|Cisco Catalyst 3560 User Guide]]<br />
* [[HCL Cluster Network]]<br />
<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `seq -w 1 16`; do root_ssh hcl$i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `seq -w 1 16`; do screen -L -d -m root_ssh hcl$i apt-get update \&\& apt-get -y upgrade; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* gcc<br />
* ctags<br />
* cg-vg<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
* colorgcc<br />
<br />
[[HCL_cluster/hcl_node_install_configuration_log|new hcl node install & configuration log]]<br />
<br />
[[HCL_cluster/heterogeneous.ucd.ie_install_log|new heterogeneous.ucd.ie install log]]<br />
<br />
===APT===<br />
To do unattended updates on cluster machines you need to specify some environment variables and switches to apt-get:<br />
<br />
export DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade<br />
<br />
NOTE: on hcl01 and hcl02 any updates to grub will force a prompt, despite the switches above. This happens because there are two disks on these machines and grub asks which it should install itself on.<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16) that you are running a pbs job on.<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
=== Creating new user accounts ===<br />
As root on heterogeneous run:<br />
adduser <username><br />
make -C /var/yp<br />
<br />
=== Access to the nodes is controlled by Torque PBS.===<br />
Use qsub to submit a job, -I is for an interactive session, walltime is time required.<br />
qsub -I -l walltime=1:00 \\ Reserve 1 node for 1 hour<br />
qsub -l nodes=hcl01+hcl07,walltime=1:00 myscript.sh<br />
<br />
Example Script:<br />
#!/bin/sh<br />
#General Script<br />
#<br />
#<br />
#These commands set up the Grid Environment for your job:<br />
#PBS -N JOBNAME<br />
#PBS -l walltime=48:00:00<br />
#PBS -l nodes=16<br />
#PBS -m abe<br />
#PBS -k eo<br />
#PBS -V<br />
echo foo<br />
<br />
So see the queue<br />
qstat -n<br />
showq<br />
<br />
To remove your job <br />
qdel JOBNUM<br />
<br />
More info: [http://www.clusterresources.com/products/torque/docs/]<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
For this reason overcommit has been disabled on the cluster.<br />
cat /proc/sys/vm/overcommit_memory <br />
2<br />
cat /proc/sys/vm/overcommit_ratio <br />
100<br />
<br />
To restore to default overcommit<br />
# echo 0 > /proc/sys/vm/overcommit_memory<br />
# echo 50 > /proc/sys/vm/overcommit_ratio<br />
<br />
== Manually Limit the Memory on the OS level ==<br />
<br />
as root edit /etc/default/grub<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet mem=128M"<br />
then run the command<br />
update-grub<br />
reboot</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=544HCL cluster2010-11-15T19:07:18Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>== General Information ==<br />
[[Image:Cluster.jpg|right|thumbnail||HCL Cluster]]<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram shows a schematic of the cluster.<br />
<br />
=== Detailed Cluster Specification ===<br />
* [[HCL Cluster Specifications]]<br />
* [[Old HCL Cluster Specifications]] (pre May 2010)<br />
<br />
=== Documentation ===<br />
* [[media:PE750.tgz|Dell Poweredge 750 Documentation]]<br />
* [[media:SC1425.tgz|Dell Poweredge SC1425 Documentation]]<br />
* [[media:X306.pdf|IBM x-Series 306 Documentation]]<br />
* [[media:E326.pdf|IBM e-Series 326 Documentation ]]<br />
* [[media:Proliant100SeriesGuide.pdf|HP Proliant DL-140 G2 Documentation]]<br />
* [[media:ProliantDL320G3Guide.pdf|HP Proliant DL-320 G3 Documentation]]<br />
* [[media:Cisco3560Specs.pdf|Cisco Catalyst 3560 Specifications]]<br />
* [[media:Cisco3560Guide.pdf|Cisco Catalyst 3560 User Guide]]<br />
* [[HCL Cluster Network]]<br />
<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `seq -w 1 16`; do root_ssh hcl$i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `seq -w 1 16`; do screen -L -d -m root_ssh hcl$i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* gcc<br />
* ctags<br />
* cg-vg<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
* colorgcc<br />
<br />
[[HCL_cluster/hcl_node_install_configuration_log|new hcl node install & configuration log]]<br />
<br />
[[HCL_cluster/heterogeneous.ucd.ie_install_log|new heterogeneous.ucd.ie install log]]<br />
<br />
===APT===<br />
To do unattended updates on cluster machines you need to specify some environment variables and switches to apt-get:<br />
<br />
export DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade<br />
<br />
NOTE: on hcl01 and hcl02 any updates to grub will force a prompt, despite the switches above. This happens because there are two disks on these machines and grub asks which it should install itself on.<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16) that you are running a pbs job on.<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
=== Creating new user accounts ===<br />
As root on heterogeneous run:<br />
adduser <username><br />
make -C /var/yp<br />
<br />
=== Access to the nodes is controlled by Torque PBS.===<br />
Use qsub to submit a job, -I is for an interactive session, walltime is time required.<br />
qsub -I -l walltime=1:00 \\ Reserve 1 node for 1 hour<br />
qsub -l nodes=hcl01+hcl07,walltime=1:00 myscript.sh<br />
<br />
Example Script:<br />
#!/bin/sh<br />
#General Script<br />
#<br />
#<br />
#These commands set up the Grid Environment for your job:<br />
#PBS -N JOBNAME<br />
#PBS -l walltime=48:00:00<br />
#PBS -l nodes=16<br />
#PBS -m abe<br />
#PBS -k eo<br />
#PBS -V<br />
echo foo<br />
<br />
So see the queue<br />
qstat -n<br />
showq<br />
<br />
To remove your job <br />
qdel JOBNUM<br />
<br />
More info: [http://www.clusterresources.com/products/torque/docs/]<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
For this reason overcommit has been disabled on the cluster.<br />
cat /proc/sys/vm/overcommit_memory <br />
2<br />
cat /proc/sys/vm/overcommit_ratio <br />
100<br />
<br />
To restore to default overcommit<br />
# echo 0 > /proc/sys/vm/overcommit_memory<br />
# echo 50 > /proc/sys/vm/overcommit_ratio<br />
<br />
== Manually Limit the Memory on the OS level ==<br />
<br />
as root edit /etc/default/grub<br />
GRUB_CMDLINE_LINUX_DEFAULT="quiet mem=128M"<br />
then run the command<br />
update-grub<br />
reboot</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=472HCL Cluster Specifications2010-08-22T11:31:04Z<p>Rhiggins: </p>
<hr />
<div>==Cluster Specifications==<br />
<br />
{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}<br />
<br />
==Cluster Benchmarks==<br />
===Stream===<br />
====Cluster Performance====<br />
----------------------------------------------<br />
Double precision appears to have 16 digits of accuracy<br />
Assuming 8 bytes per DOUBLE PRECISION word<br />
----------------------------------------------<br />
Number of processors = 18<br />
Array size = 2000000<br />
Offset = 0<br />
The total memory requirement is 824.0 MB ( 45.8MB/task)<br />
You are running each test 10 times<br />
--<br />
The *best* time for each test is used<br />
*EXCLUDING* the first and last iterations<br />
----------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond<br />
Your clock granularity/precision appears to be 1 microseconds<br />
----------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 24589.1750 0.0235 0.0234 0.0237<br />
Scale: 24493.9786 0.0237 0.0235 0.0245<br />
Add: 27594.1797 0.0314 0.0313 0.0315<br />
Triad: 27695.7938 0.0313 0.0312 0.0315<br />
-----------------------------------------------<br />
Solution Validates!<br />
-----------------------------------------------<br />
<br />
====Individual Node Performance====<br />
[[Image:stream_bench_results.png]]<br />
<br />
hcl01.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8386 microseconds.<br />
(= 8386 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2615.4617 0.0125 0.0122 0.0132<br />
Scale: 2609.2783 0.0125 0.0123 0.0133<br />
Add: 3046.4707 0.0161 0.0158 0.0168<br />
Triad: 3064.7322 0.0160 0.0157 0.0166<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl02.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 7790 microseconds.<br />
(= 7790 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2740.6840 0.0117 0.0117 0.0120<br />
Scale: 2745.3930 0.0117 0.0117 0.0117<br />
Add: 3063.1599 0.0157 0.0157 0.0157<br />
Triad: 3075.3572 0.0156 0.0156 0.0159<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl03.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8382 microseconds.<br />
(= 8382 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2788.6728 0.0115 0.0115 0.0115<br />
Scale: 2722.0144 0.0118 0.0118 0.0121<br />
Add: 3266.6166 0.0148 0.0147 0.0150<br />
Triad: 3287.4249 0.0146 0.0146 0.0147<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl04.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8378 microseconds.<br />
(= 8378 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2815.4071 0.0114 0.0114 0.0114<br />
Scale: 2751.5270 0.0116 0.0116 0.0117<br />
Add: 3260.1979 0.0148 0.0147 0.0150<br />
Triad: 3274.2183 0.0147 0.0147 0.0150<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl05.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14916 microseconds.<br />
(= 14916 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1581.8068 0.0203 0.0202 0.0204<br />
Scale: 1557.1081 0.0207 0.0206 0.0215<br />
Add: 1807.7015 0.0266 0.0266 0.0266<br />
Triad: 1832.9646 0.0263 0.0262 0.0265<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl06.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14824 microseconds.<br />
(= 14824 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1558.3835 0.0206 0.0205 0.0206<br />
Scale: 1550.3951 0.0207 0.0206 0.0209<br />
Add: 1863.4239 0.0258 0.0258 0.0259<br />
Triad: 1885.3916 0.0255 0.0255 0.0259<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl07.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13642 microseconds.<br />
(= 13642 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1759.4014 0.0183 0.0182 0.0186<br />
Scale: 1740.2731 0.0184 0.0184 0.0185<br />
Add: 2036.4962 0.0236 0.0236 0.0238<br />
Triad: 2045.5920 0.0235 0.0235 0.0235<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl08.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13476 microseconds.<br />
(= 13476 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1809.6454 0.0177 0.0177 0.0177<br />
Scale: 1784.8271 0.0179 0.0179 0.0179<br />
Add: 2085.3320 0.0231 0.0230 0.0232<br />
Triad: 2095.7094 0.0231 0.0229 0.0239<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl09.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7890 microseconds.<br />
(= 3945 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2838.6204 0.0114 0.0113 0.0115<br />
Scale: 2774.6396 0.0116 0.0115 0.0117<br />
Add: 3144.0396 0.0155 0.0153 0.0156<br />
Triad: 3160.7905 0.0153 0.0152 0.0156<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl10.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7959 microseconds.<br />
(= 3979 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2864.2732 0.0113 0.0112 0.0114<br />
Scale: 2784.2952 0.0116 0.0115 0.0118<br />
Add: 3162.2553 0.0153 0.0152 0.0155<br />
Triad: 3223.8430 0.0151 0.0149 0.0153<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl11.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 1 microseconds.<br />
Each test below will take on the order of 13082 microseconds.<br />
(= 13082 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1842.8984 0.0174 0.0174 0.0174<br />
Scale: 1818.1816 0.0177 0.0176 0.0179<br />
Add: 2109.8918 0.0228 0.0227 0.0231<br />
Triad: 2119.0161 0.0227 0.0227 0.0227<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl12.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12621 microseconds.<br />
(= 12621 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1805.0657 0.0178 0.0177 0.0179<br />
Scale: 1795.5385 0.0179 0.0178 0.0182<br />
Add: 2089.9636 0.0230 0.0230 0.0230<br />
Triad: 2097.5313 0.0229 0.0229 0.0231<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl13.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12203 microseconds.<br />
(= 12203 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1768.9387 0.0181 0.0181 0.0182<br />
Scale: 1765.9100 0.0182 0.0181 0.0184<br />
Add: 2000.5875 0.0241 0.0240 0.0244<br />
Triad: 2001.0897 0.0240 0.0240 0.0241<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl14.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9576 microseconds.<br />
(= 9576 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2254.3285 0.0142 0.0142 0.0142<br />
Scale: 2291.6040 0.0140 0.0140 0.0140<br />
Add: 2779.5354 0.0173 0.0173 0.0173<br />
Triad: 2804.8951 0.0177 0.0171 0.0215<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl15.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9788 microseconds.<br />
(= 9788 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2223.0043 0.0146 0.0144 0.0161<br />
Scale: 2264.6703 0.0142 0.0141 0.0142<br />
Add: 2740.5171 0.0176 0.0175 0.0178<br />
Triad: 2762.2584 0.0174 0.0174 0.0174<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl16.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9590 microseconds.<br />
(= 9590 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2285.3882 0.0140 0.0140 0.0143<br />
Scale: 2294.5687 0.0140 0.0139 0.0140<br />
Add: 2800.9568 0.0172 0.0171 0.0172<br />
Triad: 2823.2048 0.0170 0.0170 0.0171<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=471HCL Cluster Specifications2010-08-22T11:29:54Z<p>Rhiggins: </p>
<hr />
<div><br />
{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}<br />
<br />
==Cluster Benchmarks==<br />
===Stream===<br />
====Cluster Performance====<br />
----------------------------------------------<br />
Double precision appears to have 16 digits of accuracy<br />
Assuming 8 bytes per DOUBLE PRECISION word<br />
----------------------------------------------<br />
Number of processors = 18<br />
Array size = 2000000<br />
Offset = 0<br />
The total memory requirement is 824.0 MB ( 45.8MB/task)<br />
You are running each test 10 times<br />
--<br />
The *best* time for each test is used<br />
*EXCLUDING* the first and last iterations<br />
----------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond<br />
Your clock granularity/precision appears to be 1 microseconds<br />
----------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 24589.1750 0.0235 0.0234 0.0237<br />
Scale: 24493.9786 0.0237 0.0235 0.0245<br />
Add: 27594.1797 0.0314 0.0313 0.0315<br />
Triad: 27695.7938 0.0313 0.0312 0.0315<br />
-----------------------------------------------<br />
Solution Validates!<br />
-----------------------------------------------<br />
<br />
====Individual Node Performance====<br />
[[Image:stream_bench_results.png]]<br />
<br />
hcl01.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8386 microseconds.<br />
(= 8386 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2615.4617 0.0125 0.0122 0.0132<br />
Scale: 2609.2783 0.0125 0.0123 0.0133<br />
Add: 3046.4707 0.0161 0.0158 0.0168<br />
Triad: 3064.7322 0.0160 0.0157 0.0166<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl02.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 7790 microseconds.<br />
(= 7790 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2740.6840 0.0117 0.0117 0.0120<br />
Scale: 2745.3930 0.0117 0.0117 0.0117<br />
Add: 3063.1599 0.0157 0.0157 0.0157<br />
Triad: 3075.3572 0.0156 0.0156 0.0159<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl03.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8382 microseconds.<br />
(= 8382 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2788.6728 0.0115 0.0115 0.0115<br />
Scale: 2722.0144 0.0118 0.0118 0.0121<br />
Add: 3266.6166 0.0148 0.0147 0.0150<br />
Triad: 3287.4249 0.0146 0.0146 0.0147<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl04.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8378 microseconds.<br />
(= 8378 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2815.4071 0.0114 0.0114 0.0114<br />
Scale: 2751.5270 0.0116 0.0116 0.0117<br />
Add: 3260.1979 0.0148 0.0147 0.0150<br />
Triad: 3274.2183 0.0147 0.0147 0.0150<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl05.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14916 microseconds.<br />
(= 14916 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1581.8068 0.0203 0.0202 0.0204<br />
Scale: 1557.1081 0.0207 0.0206 0.0215<br />
Add: 1807.7015 0.0266 0.0266 0.0266<br />
Triad: 1832.9646 0.0263 0.0262 0.0265<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl06.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14824 microseconds.<br />
(= 14824 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1558.3835 0.0206 0.0205 0.0206<br />
Scale: 1550.3951 0.0207 0.0206 0.0209<br />
Add: 1863.4239 0.0258 0.0258 0.0259<br />
Triad: 1885.3916 0.0255 0.0255 0.0259<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl07.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13642 microseconds.<br />
(= 13642 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1759.4014 0.0183 0.0182 0.0186<br />
Scale: 1740.2731 0.0184 0.0184 0.0185<br />
Add: 2036.4962 0.0236 0.0236 0.0238<br />
Triad: 2045.5920 0.0235 0.0235 0.0235<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl08.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13476 microseconds.<br />
(= 13476 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1809.6454 0.0177 0.0177 0.0177<br />
Scale: 1784.8271 0.0179 0.0179 0.0179<br />
Add: 2085.3320 0.0231 0.0230 0.0232<br />
Triad: 2095.7094 0.0231 0.0229 0.0239<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl09.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7890 microseconds.<br />
(= 3945 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2838.6204 0.0114 0.0113 0.0115<br />
Scale: 2774.6396 0.0116 0.0115 0.0117<br />
Add: 3144.0396 0.0155 0.0153 0.0156<br />
Triad: 3160.7905 0.0153 0.0152 0.0156<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl10.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7959 microseconds.<br />
(= 3979 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2864.2732 0.0113 0.0112 0.0114<br />
Scale: 2784.2952 0.0116 0.0115 0.0118<br />
Add: 3162.2553 0.0153 0.0152 0.0155<br />
Triad: 3223.8430 0.0151 0.0149 0.0153<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl11.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 1 microseconds.<br />
Each test below will take on the order of 13082 microseconds.<br />
(= 13082 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1842.8984 0.0174 0.0174 0.0174<br />
Scale: 1818.1816 0.0177 0.0176 0.0179<br />
Add: 2109.8918 0.0228 0.0227 0.0231<br />
Triad: 2119.0161 0.0227 0.0227 0.0227<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl12.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12621 microseconds.<br />
(= 12621 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1805.0657 0.0178 0.0177 0.0179<br />
Scale: 1795.5385 0.0179 0.0178 0.0182<br />
Add: 2089.9636 0.0230 0.0230 0.0230<br />
Triad: 2097.5313 0.0229 0.0229 0.0231<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl13.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12203 microseconds.<br />
(= 12203 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1768.9387 0.0181 0.0181 0.0182<br />
Scale: 1765.9100 0.0182 0.0181 0.0184<br />
Add: 2000.5875 0.0241 0.0240 0.0244<br />
Triad: 2001.0897 0.0240 0.0240 0.0241<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl14.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9576 microseconds.<br />
(= 9576 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2254.3285 0.0142 0.0142 0.0142<br />
Scale: 2291.6040 0.0140 0.0140 0.0140<br />
Add: 2779.5354 0.0173 0.0173 0.0173<br />
Triad: 2804.8951 0.0177 0.0171 0.0215<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl15.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9788 microseconds.<br />
(= 9788 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2223.0043 0.0146 0.0144 0.0161<br />
Scale: 2264.6703 0.0142 0.0141 0.0142<br />
Add: 2740.5171 0.0176 0.0175 0.0178<br />
Triad: 2762.2584 0.0174 0.0174 0.0174<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl16.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9590 microseconds.<br />
(= 9590 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2285.3882 0.0140 0.0140 0.0143<br />
Scale: 2294.5687 0.0140 0.0139 0.0140<br />
Add: 2800.9568 0.0172 0.0171 0.0172<br />
Triad: 2823.2048 0.0170 0.0170 0.0171<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=470HCL Cluster Specifications2010-08-22T11:29:02Z<p>Rhiggins: </p>
<hr />
<div>{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}<br />
<br />
==Cluster Benchmarks==<br />
===Stream===<br />
===Cluster Performance===<br />
----------------------------------------------<br />
Double precision appears to have 16 digits of accuracy<br />
Assuming 8 bytes per DOUBLE PRECISION word<br />
----------------------------------------------<br />
Number of processors = 18<br />
Array size = 2000000<br />
Offset = 0<br />
The total memory requirement is 824.0 MB ( 45.8MB/task)<br />
You are running each test 10 times<br />
--<br />
The *best* time for each test is used<br />
*EXCLUDING* the first and last iterations<br />
----------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond<br />
Your clock granularity/precision appears to be 1 microseconds<br />
----------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 24589.1750 0.0235 0.0234 0.0237<br />
Scale: 24493.9786 0.0237 0.0235 0.0245<br />
Add: 27594.1797 0.0314 0.0313 0.0315<br />
Triad: 27695.7938 0.0313 0.0312 0.0315<br />
-----------------------------------------------<br />
Solution Validates!<br />
-----------------------------------------------<br />
<br />
===Individual Node Performance===<br />
[[Image:stream_bench_results.png]]<br />
<br />
hcl01.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8386 microseconds.<br />
(= 8386 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2615.4617 0.0125 0.0122 0.0132<br />
Scale: 2609.2783 0.0125 0.0123 0.0133<br />
Add: 3046.4707 0.0161 0.0158 0.0168<br />
Triad: 3064.7322 0.0160 0.0157 0.0166<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl02.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 7790 microseconds.<br />
(= 7790 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2740.6840 0.0117 0.0117 0.0120<br />
Scale: 2745.3930 0.0117 0.0117 0.0117<br />
Add: 3063.1599 0.0157 0.0157 0.0157<br />
Triad: 3075.3572 0.0156 0.0156 0.0159<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl03.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8382 microseconds.<br />
(= 8382 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2788.6728 0.0115 0.0115 0.0115<br />
Scale: 2722.0144 0.0118 0.0118 0.0121<br />
Add: 3266.6166 0.0148 0.0147 0.0150<br />
Triad: 3287.4249 0.0146 0.0146 0.0147<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl04.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8378 microseconds.<br />
(= 8378 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2815.4071 0.0114 0.0114 0.0114<br />
Scale: 2751.5270 0.0116 0.0116 0.0117<br />
Add: 3260.1979 0.0148 0.0147 0.0150<br />
Triad: 3274.2183 0.0147 0.0147 0.0150<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl05.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14916 microseconds.<br />
(= 14916 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1581.8068 0.0203 0.0202 0.0204<br />
Scale: 1557.1081 0.0207 0.0206 0.0215<br />
Add: 1807.7015 0.0266 0.0266 0.0266<br />
Triad: 1832.9646 0.0263 0.0262 0.0265<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl06.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14824 microseconds.<br />
(= 14824 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1558.3835 0.0206 0.0205 0.0206<br />
Scale: 1550.3951 0.0207 0.0206 0.0209<br />
Add: 1863.4239 0.0258 0.0258 0.0259<br />
Triad: 1885.3916 0.0255 0.0255 0.0259<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl07.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13642 microseconds.<br />
(= 13642 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1759.4014 0.0183 0.0182 0.0186<br />
Scale: 1740.2731 0.0184 0.0184 0.0185<br />
Add: 2036.4962 0.0236 0.0236 0.0238<br />
Triad: 2045.5920 0.0235 0.0235 0.0235<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl08.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13476 microseconds.<br />
(= 13476 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1809.6454 0.0177 0.0177 0.0177<br />
Scale: 1784.8271 0.0179 0.0179 0.0179<br />
Add: 2085.3320 0.0231 0.0230 0.0232<br />
Triad: 2095.7094 0.0231 0.0229 0.0239<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl09.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7890 microseconds.<br />
(= 3945 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2838.6204 0.0114 0.0113 0.0115<br />
Scale: 2774.6396 0.0116 0.0115 0.0117<br />
Add: 3144.0396 0.0155 0.0153 0.0156<br />
Triad: 3160.7905 0.0153 0.0152 0.0156<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl10.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7959 microseconds.<br />
(= 3979 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2864.2732 0.0113 0.0112 0.0114<br />
Scale: 2784.2952 0.0116 0.0115 0.0118<br />
Add: 3162.2553 0.0153 0.0152 0.0155<br />
Triad: 3223.8430 0.0151 0.0149 0.0153<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl11.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 1 microseconds.<br />
Each test below will take on the order of 13082 microseconds.<br />
(= 13082 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1842.8984 0.0174 0.0174 0.0174<br />
Scale: 1818.1816 0.0177 0.0176 0.0179<br />
Add: 2109.8918 0.0228 0.0227 0.0231<br />
Triad: 2119.0161 0.0227 0.0227 0.0227<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl12.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12621 microseconds.<br />
(= 12621 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1805.0657 0.0178 0.0177 0.0179<br />
Scale: 1795.5385 0.0179 0.0178 0.0182<br />
Add: 2089.9636 0.0230 0.0230 0.0230<br />
Triad: 2097.5313 0.0229 0.0229 0.0231<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl13.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12203 microseconds.<br />
(= 12203 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1768.9387 0.0181 0.0181 0.0182<br />
Scale: 1765.9100 0.0182 0.0181 0.0184<br />
Add: 2000.5875 0.0241 0.0240 0.0244<br />
Triad: 2001.0897 0.0240 0.0240 0.0241<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl14.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9576 microseconds.<br />
(= 9576 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2254.3285 0.0142 0.0142 0.0142<br />
Scale: 2291.6040 0.0140 0.0140 0.0140<br />
Add: 2779.5354 0.0173 0.0173 0.0173<br />
Triad: 2804.8951 0.0177 0.0171 0.0215<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl15.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9788 microseconds.<br />
(= 9788 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2223.0043 0.0146 0.0144 0.0161<br />
Scale: 2264.6703 0.0142 0.0141 0.0142<br />
Add: 2740.5171 0.0176 0.0175 0.0178<br />
Triad: 2762.2584 0.0174 0.0174 0.0174<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl16.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9590 microseconds.<br />
(= 9590 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2285.3882 0.0140 0.0140 0.0143<br />
Scale: 2294.5687 0.0140 0.0139 0.0140<br />
Add: 2800.9568 0.0172 0.0171 0.0172<br />
Triad: 2823.2048 0.0170 0.0170 0.0171<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=File:Stream_bench_results.png&diff=455File:Stream bench results.png2010-07-29T13:29:00Z<p>Rhiggins: Plots of STREAM benchmark for HCL Cluster</p>
<hr />
<div>Plots of STREAM benchmark for HCL Cluster</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=454HCL Cluster Specifications2010-07-29T13:28:38Z<p>Rhiggins: /* Stream */</p>
<hr />
<div>{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}<br />
<br />
==Cluster Benchmarks==<br />
===Stream===<br />
<br />
[[Image:stream_bench_results.png]]<br />
<br />
hcl01.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8386 microseconds.<br />
(= 8386 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2615.4617 0.0125 0.0122 0.0132<br />
Scale: 2609.2783 0.0125 0.0123 0.0133<br />
Add: 3046.4707 0.0161 0.0158 0.0168<br />
Triad: 3064.7322 0.0160 0.0157 0.0166<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl02.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 7790 microseconds.<br />
(= 7790 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2740.6840 0.0117 0.0117 0.0120<br />
Scale: 2745.3930 0.0117 0.0117 0.0117<br />
Add: 3063.1599 0.0157 0.0157 0.0157<br />
Triad: 3075.3572 0.0156 0.0156 0.0159<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl03.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8382 microseconds.<br />
(= 8382 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2788.6728 0.0115 0.0115 0.0115<br />
Scale: 2722.0144 0.0118 0.0118 0.0121<br />
Add: 3266.6166 0.0148 0.0147 0.0150<br />
Triad: 3287.4249 0.0146 0.0146 0.0147<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl04.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8378 microseconds.<br />
(= 8378 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2815.4071 0.0114 0.0114 0.0114<br />
Scale: 2751.5270 0.0116 0.0116 0.0117<br />
Add: 3260.1979 0.0148 0.0147 0.0150<br />
Triad: 3274.2183 0.0147 0.0147 0.0150<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl05.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14916 microseconds.<br />
(= 14916 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1581.8068 0.0203 0.0202 0.0204<br />
Scale: 1557.1081 0.0207 0.0206 0.0215<br />
Add: 1807.7015 0.0266 0.0266 0.0266<br />
Triad: 1832.9646 0.0263 0.0262 0.0265<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl06.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14824 microseconds.<br />
(= 14824 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1558.3835 0.0206 0.0205 0.0206<br />
Scale: 1550.3951 0.0207 0.0206 0.0209<br />
Add: 1863.4239 0.0258 0.0258 0.0259<br />
Triad: 1885.3916 0.0255 0.0255 0.0259<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl07.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13642 microseconds.<br />
(= 13642 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1759.4014 0.0183 0.0182 0.0186<br />
Scale: 1740.2731 0.0184 0.0184 0.0185<br />
Add: 2036.4962 0.0236 0.0236 0.0238<br />
Triad: 2045.5920 0.0235 0.0235 0.0235<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl08.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13476 microseconds.<br />
(= 13476 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1809.6454 0.0177 0.0177 0.0177<br />
Scale: 1784.8271 0.0179 0.0179 0.0179<br />
Add: 2085.3320 0.0231 0.0230 0.0232<br />
Triad: 2095.7094 0.0231 0.0229 0.0239<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl09.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7890 microseconds.<br />
(= 3945 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2838.6204 0.0114 0.0113 0.0115<br />
Scale: 2774.6396 0.0116 0.0115 0.0117<br />
Add: 3144.0396 0.0155 0.0153 0.0156<br />
Triad: 3160.7905 0.0153 0.0152 0.0156<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl10.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7959 microseconds.<br />
(= 3979 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2864.2732 0.0113 0.0112 0.0114<br />
Scale: 2784.2952 0.0116 0.0115 0.0118<br />
Add: 3162.2553 0.0153 0.0152 0.0155<br />
Triad: 3223.8430 0.0151 0.0149 0.0153<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl11.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 1 microseconds.<br />
Each test below will take on the order of 13082 microseconds.<br />
(= 13082 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1842.8984 0.0174 0.0174 0.0174<br />
Scale: 1818.1816 0.0177 0.0176 0.0179<br />
Add: 2109.8918 0.0228 0.0227 0.0231<br />
Triad: 2119.0161 0.0227 0.0227 0.0227<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl12.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12621 microseconds.<br />
(= 12621 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1805.0657 0.0178 0.0177 0.0179<br />
Scale: 1795.5385 0.0179 0.0178 0.0182<br />
Add: 2089.9636 0.0230 0.0230 0.0230<br />
Triad: 2097.5313 0.0229 0.0229 0.0231<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl13.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12203 microseconds.<br />
(= 12203 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1768.9387 0.0181 0.0181 0.0182<br />
Scale: 1765.9100 0.0182 0.0181 0.0184<br />
Add: 2000.5875 0.0241 0.0240 0.0244<br />
Triad: 2001.0897 0.0240 0.0240 0.0241<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl14.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9576 microseconds.<br />
(= 9576 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2254.3285 0.0142 0.0142 0.0142<br />
Scale: 2291.6040 0.0140 0.0140 0.0140<br />
Add: 2779.5354 0.0173 0.0173 0.0173<br />
Triad: 2804.8951 0.0177 0.0171 0.0215<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl15.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9788 microseconds.<br />
(= 9788 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2223.0043 0.0146 0.0144 0.0161<br />
Scale: 2264.6703 0.0142 0.0141 0.0142<br />
Add: 2740.5171 0.0176 0.0175 0.0178<br />
Triad: 2762.2584 0.0174 0.0174 0.0174<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl16.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9590 microseconds.<br />
(= 9590 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2285.3882 0.0140 0.0140 0.0143<br />
Scale: 2294.5687 0.0140 0.0139 0.0140<br />
Add: 2800.9568 0.0172 0.0171 0.0172<br />
Triad: 2823.2048 0.0170 0.0170 0.0171<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=453HCL Cluster Specifications2010-07-29T12:13:14Z<p>Rhiggins: </p>
<hr />
<div>{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}<br />
<br />
==Cluster Benchmarks==<br />
===Stream===<br />
hcl01.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8386 microseconds.<br />
(= 8386 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2615.4617 0.0125 0.0122 0.0132<br />
Scale: 2609.2783 0.0125 0.0123 0.0133<br />
Add: 3046.4707 0.0161 0.0158 0.0168<br />
Triad: 3064.7322 0.0160 0.0157 0.0166<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl02.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 7790 microseconds.<br />
(= 7790 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2740.6840 0.0117 0.0117 0.0120<br />
Scale: 2745.3930 0.0117 0.0117 0.0117<br />
Add: 3063.1599 0.0157 0.0157 0.0157<br />
Triad: 3075.3572 0.0156 0.0156 0.0159<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl03.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8382 microseconds.<br />
(= 8382 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2788.6728 0.0115 0.0115 0.0115<br />
Scale: 2722.0144 0.0118 0.0118 0.0121<br />
Add: 3266.6166 0.0148 0.0147 0.0150<br />
Triad: 3287.4249 0.0146 0.0146 0.0147<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl04.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 8378 microseconds.<br />
(= 8378 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2815.4071 0.0114 0.0114 0.0114<br />
Scale: 2751.5270 0.0116 0.0116 0.0117<br />
Add: 3260.1979 0.0148 0.0147 0.0150<br />
Triad: 3274.2183 0.0147 0.0147 0.0150<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl05.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14916 microseconds.<br />
(= 14916 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1581.8068 0.0203 0.0202 0.0204<br />
Scale: 1557.1081 0.0207 0.0206 0.0215<br />
Add: 1807.7015 0.0266 0.0266 0.0266<br />
Triad: 1832.9646 0.0263 0.0262 0.0265<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl06.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 14824 microseconds.<br />
(= 14824 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1558.3835 0.0206 0.0205 0.0206<br />
Scale: 1550.3951 0.0207 0.0206 0.0209<br />
Add: 1863.4239 0.0258 0.0258 0.0259<br />
Triad: 1885.3916 0.0255 0.0255 0.0259<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl07.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13642 microseconds.<br />
(= 13642 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1759.4014 0.0183 0.0182 0.0186<br />
Scale: 1740.2731 0.0184 0.0184 0.0185<br />
Add: 2036.4962 0.0236 0.0236 0.0238<br />
Triad: 2045.5920 0.0235 0.0235 0.0235<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl08.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 13476 microseconds.<br />
(= 13476 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1809.6454 0.0177 0.0177 0.0177<br />
Scale: 1784.8271 0.0179 0.0179 0.0179<br />
Add: 2085.3320 0.0231 0.0230 0.0232<br />
Triad: 2095.7094 0.0231 0.0229 0.0239<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl09.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7890 microseconds.<br />
(= 3945 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2838.6204 0.0114 0.0113 0.0115<br />
Scale: 2774.6396 0.0116 0.0115 0.0117<br />
Add: 3144.0396 0.0155 0.0153 0.0156<br />
Triad: 3160.7905 0.0153 0.0152 0.0156<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl10.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 2<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 2 microseconds.<br />
Each test below will take on the order of 7959 microseconds.<br />
(= 3979 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2864.2732 0.0113 0.0112 0.0114<br />
Scale: 2784.2952 0.0116 0.0115 0.0118<br />
Add: 3162.2553 0.0153 0.0152 0.0155<br />
Triad: 3223.8430 0.0151 0.0149 0.0153<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl11.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity/precision appears to be 1 microseconds.<br />
Each test below will take on the order of 13082 microseconds.<br />
(= 13082 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1842.8984 0.0174 0.0174 0.0174<br />
Scale: 1818.1816 0.0177 0.0176 0.0179<br />
Add: 2109.8918 0.0228 0.0227 0.0231<br />
Triad: 2119.0161 0.0227 0.0227 0.0227<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl12.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12621 microseconds.<br />
(= 12621 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1805.0657 0.0178 0.0177 0.0179<br />
Scale: 1795.5385 0.0179 0.0178 0.0182<br />
Add: 2089.9636 0.0230 0.0230 0.0230<br />
Triad: 2097.5313 0.0229 0.0229 0.0231<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl13.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 12203 microseconds.<br />
(= 12203 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 1768.9387 0.0181 0.0181 0.0182<br />
Scale: 1765.9100 0.0182 0.0181 0.0184<br />
Add: 2000.5875 0.0241 0.0240 0.0244<br />
Triad: 2001.0897 0.0240 0.0240 0.0241<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl14.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9576 microseconds.<br />
(= 9576 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2254.3285 0.0142 0.0142 0.0142<br />
Scale: 2291.6040 0.0140 0.0140 0.0140<br />
Add: 2779.5354 0.0173 0.0173 0.0173<br />
Triad: 2804.8951 0.0177 0.0171 0.0215<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl15.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9788 microseconds.<br />
(= 9788 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2223.0043 0.0146 0.0144 0.0161<br />
Scale: 2264.6703 0.0142 0.0141 0.0142<br />
Add: 2740.5171 0.0176 0.0175 0.0178<br />
Triad: 2762.2584 0.0174 0.0174 0.0174<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------<br />
hcl16.heterogeneous.ucd.ie<br />
-------------------------------------------------------------<br />
STREAM version $Revision: 5.9 $<br />
-------------------------------------------------------------<br />
This system uses 8 bytes per DOUBLE PRECISION word.<br />
-------------------------------------------------------------<br />
Array size = 2000000, Offset = 0<br />
Total memory required = 45.8 MB.<br />
Each test is run 10 times, but only<br />
the *best* time for each is used.<br />
-------------------------------------------------------------<br />
Number of Threads requested = 1<br />
-------------------------------------------------------------<br />
Printing one line per active thread....<br />
-------------------------------------------------------------<br />
Your clock granularity appears to be less than one microsecond.<br />
Each test below will take on the order of 9590 microseconds.<br />
(= 9590 clock ticks)<br />
Increase the size of the arrays if this shows that<br />
you are not getting at least 20 clock ticks per test.<br />
-------------------------------------------------------------<br />
WARNING -- The above is only a rough guideline.<br />
For best results, please be sure you know the<br />
precision of your system timer.<br />
-------------------------------------------------------------<br />
Function Rate (MB/s) Avg time Min time Max time<br />
Copy: 2285.3882 0.0140 0.0140 0.0143<br />
Scale: 2294.5687 0.0140 0.0139 0.0140<br />
Add: 2800.9568 0.0172 0.0171 0.0172<br />
Triad: 2823.2048 0.0170 0.0170 0.0171<br />
-------------------------------------------------------------<br />
Solution Validates<br />
-------------------------------------------------------------</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=452HCL cluster2010-07-28T13:04:11Z<p>Rhiggins: /* APT */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
* colorgcc<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
===APT===<br />
To do unattended updates on cluster machines you need to specify some environment variables and switches to apt-get:<br />
<br />
export DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade<br />
<br />
NOTE: on hcl01 and hcl02 any updates to grub will force a prompt, despite the switches above. This happens because there are two disks on these machines and grub asks which it should install itself on.<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
For this reason overcommit has been disabled on the cluster.<br />
cat /proc/sys/vm/overcommit_memory <br />
2<br />
cat /proc/sys/vm/overcommit_ratio <br />
100<br />
<br />
To restore to default overcommit<br />
# echo 0 > /proc/sys/vm/overcommit_memory<br />
# echo 50 > /proc/sys/vm/overcommit_ratio</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=451HCL cluster2010-07-28T11:54:01Z<p>Rhiggins: /* APT */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
* colorgcc<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
===APT===<br />
To do unattended updates on cluster machines you need to specify some environment variables and switches to apt-get:<br />
<br />
export DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
For this reason overcommit has been disabled on the cluster.<br />
cat /proc/sys/vm/overcommit_memory <br />
2<br />
cat /proc/sys/vm/overcommit_ratio <br />
100<br />
<br />
To restore to default overcommit<br />
# echo 0 > /proc/sys/vm/overcommit_memory<br />
# echo 50 > /proc/sys/vm/overcommit_ratio</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=450HCL cluster2010-07-28T11:53:43Z<p>Rhiggins: /* Software packages available on HCL Cluster 2.0 */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
* colorgcc<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
===APT===<br />
To do unattended updates on cluster machines you need to specify some environment variables and switches to apt-get:<br />
<br />
* export DEBIAN_FRONTEND=noninteractive apt-get -q -y upgrade<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
For this reason overcommit has been disabled on the cluster.<br />
cat /proc/sys/vm/overcommit_memory <br />
2<br />
cat /proc/sys/vm/overcommit_ratio <br />
100<br />
<br />
To restore to default overcommit<br />
# echo 0 > /proc/sys/vm/overcommit_memory<br />
# echo 50 > /proc/sys/vm/overcommit_ratio</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=449HCL Cluster Specifications2010-07-25T14:33:37Z<p>Rhiggins: </p>
<hr />
<div>{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | [[Hcl01]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | [[Hcl02]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | [[Hcl03]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | [[Hcl04]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | [[Hcl05]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | [[Hcl06]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | [[Hcl07]](NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | [[Hcl08]] (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | [[Hcl09]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | [[Hcl10]] (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | [[Hcl11]] (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | [[Hcl12]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | [[Hcl13]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | [[Hcl14]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | [[Hcl15]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | [[Hcl16]] (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=445HCL cluster2010-07-05T11:30:48Z<p>Rhiggins: </p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1 (see also [http://hcl.ucd.ie/wiki/index.php/OpenMPI] ):<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
== Paging and the OOM-Killer ==<br />
Due to the nature of experiments run on the cluster, we often induce heavy paging and complete exhaustion of available memory on certain nodes. Linux has a pair of strategies to deal with heavy memory use. First, is overcommitting. This is where a process is allowed allocate or fork even when there is no more memory available. You can seem some interesting numbers here:[http://opsmonkey.blogspot.com/2007/01/linux-memory-overcommit.html]. The assumption is that processes may not use all memory that they allocate and failing on allocation is worse than failing at a later date when the memory use is actually required. More processes may be supported by allowing them to allocate memory (provided they do not use it all). The second part of the strategy is the Out-of-Memory killer (OOM Killer). When memory has been exhausted and a process tries to use some 'overcommitted' part of memory, the OOM killer is invoked. It's job is to rank all processes in terms of their memory use, priority, privilege and some other parameters, and then select a process to kill based on the ranks.<br />
<br />
The argument for using overcommital+OOM Killer is that rather than failing to allocate memory for some random unlucky process, which as a result would probably terminate, the kernel can instead allow the unlucky process to continue executing and then make a some-what-informed decision on which process to kill. Unfortunately, the behaviour of the OOM-killer sometimes causes problems which grind the machine to a complete halt, particularly when it decides to kill system processes. There is a good discussion on the OOM-killer here: [http://lwn.net/Articles/104179/]<br />
<br />
If you notice a machine has gone down (particularly hcl11) on the cluster, it is likely to be as a result of this. We can consider disabling overcommit which will probably result in the termination of the types of experiments that cause the heavy paging, rather than some other system process. This would probably be most desirable but it is up for discussion.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_Cluster_Specifications&diff=394HCL Cluster Specifications2010-06-15T16:08:26Z<p>Rhiggins: </p>
<hr />
<div>{| border="1" cellspacing="1" cellpadding="5" <br />
| Name<br />
| Make/Model<br />
| IP<br />
| Processor<br />
| Front Side Bus<br />
| L2 Cache<br />
| RAM<br />
| HDD 1<br />
| HDD 2<br />
| NIC<br />
| Rack<br />
|- <br />
| ALIGN=LEFT | hclswitch1 <br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.252<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 42<br />
|- <br />
| ALIGN=LEFT | hclswitch2<br />
| ALIGN=LEFT | Cisco Catalyst 3560G<br />
| ALIGN=LEFT | 192.168.21.253<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 24 x Gigabit<br />
| 41<br />
|- <br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | APC Smart UPS 1500<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | N/A<br />
| 1 – 2<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.3<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl01 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.103<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 3<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.4<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | 250GB SATA<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl02 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.104<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 4<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.5<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl03 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.105<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 5<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.6<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl04 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.106<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 6<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.7<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl05 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.107<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 7<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge SC1425<br />
| ALIGN=LEFT | 192.168.21.8<br />
| ALIGN=LEFT | 3.0 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl06 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.108<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 8<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.9<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl07 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.109<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 9<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC1)<br />
| ALIGN=LEFT | Dell Poweredge 750<br />
| ALIGN=LEFT | 192.168.21.10<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 256MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl08 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.110<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 10<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.11<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl09 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.111<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 11<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC1)<br />
| ALIGN=LEFT | IBM E-server 326<br />
| ALIGN=LEFT | 192.168.21.12<br />
| ALIGN=LEFT | 1.8 AMD Opteron<br />
| ALIGN=LEFT | 1GHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl10 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.112<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 12<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC1)<br />
| ALIGN=LEFT | IBM X-Series 306<br />
| ALIGN=LEFT | 192.168.21.13<br />
| ALIGN=LEFT | 3.2 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl11 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.113<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 13<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.14<br />
| ALIGN=LEFT | 3.4 P4<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 512MB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl12 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.114<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 14<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 320 G3<br />
| ALIGN=LEFT | 192.168.21.15<br />
| ALIGN=LEFT | 2.9 Celeron<br />
| ALIGN=LEFT | 533MHz<br />
| ALIGN=LEFT | 256KB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl13 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.115<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 15<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.16<br />
| ALIGN=LEFT | 3.4 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl14 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.116<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 16<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.17<br />
| ALIGN=LEFT | 2.8 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 1MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl15 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.117<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 17<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC1)<br />
| ALIGN=LEFT | HP Proliant DL 140 G2<br />
| ALIGN=LEFT | 192.168.21.18<br />
| ALIGN=LEFT | 3.6 Xeon<br />
| ALIGN=LEFT | 800MHz<br />
| ALIGN=LEFT | 2MB<br />
| ALIGN=LEFT | 1GB<br />
| ALIGN=LEFT | 80GB SATA<br />
| ALIGN=LEFT | N/A<br />
| ALIGN=LEFT | 2 x Gigabit<br />
| 18<br />
|- <br />
| ALIGN=LEFT | Hcl16 (NIC2)<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | 192.168.21.118<br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| ALIGN=LEFT | <BR><br />
| 18<br />
|}</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=Other_UCD_Resources&diff=389Other UCD Resources2010-05-19T11:00:14Z<p>Rhiggins: </p>
<hr />
<div>CSI has a number of systems available for use, you will need login credentials to view this page:<br />
<br />
http://www.csi.ucd.ie/content/accounts-and-systems<br />
<br />
AIX/PowerPC, Solaris/x86 and Linux/x86 systems are available and can be handy for testing reliability of your code across many platforms.<br />
<br />
UCD has a Pay-for-Use cluster named Phaeton, information on that may be viewed here:<br />
<br />
https://login.ucd.ie/</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=388HCL cluster2010-05-11T15:12:22Z<p>Rhiggins: /* Some networking issues on HCL cluster (unsolved) */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1:<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected, but does not effect the return path of the ping response packet.<br />
<br />
<br />
In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=387HCL cluster2010-05-11T15:11:41Z<p>Rhiggins: /* Some networking issues on HCL cluster (unsolved) */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1:<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.<br />
<br />
Further, after a number of ping packets have been sent in sequence (50 to 100), pings from A, though the -i eth0 switch is specified, begin to appear on both A-eth0 and A-eth1. This behaviour is unexpected.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=386HCL cluster2010-05-11T15:09:33Z<p>Rhiggins: /* Some networking issues on HCL cluster (unsolved) */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1:<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0. In order to get a symmetric behaviour, where a packet leaves A-eth0, travels via the switch bridge to B-eth1 and returns back from B-eth1 to A-eth0, one must ensure the routing table of B contains no eth0 entries.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=385HCL cluster2010-05-11T15:06:36Z<p>Rhiggins: /* Some networking issues on HCL cluster (unsolved) */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)<br />
<br />
== Some networking issues on HCL cluster (unsolved) ==<br />
<br />
"/sbin/route" should give:<br />
<br />
Kernel IP routing table<br />
Destination Gateway Genmask Flags Metric Ref Use Iface<br />
239.2.11.72 * 255.255.255.255 UH 0 0 0 eth0<br />
heterogeneous.u * 255.255.255.255 UH 0 0 0 eth0<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
192.168.20.0 * 255.255.255.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth1<br />
default heterogeneous.u 0.0.0.0 UG 0 0 0 eth0<br />
<br />
<br />
For reasons unclear, sometimes many machines miss the entry:<br />
<br />
192.168.21.0 * 255.255.255.0 U 0 0 0 eth1<br />
<br />
For Open MPI, this leads to inability to do a system sockets "connect" call to any 192.*.21.* address (hangup).<br />
For this case, you can <br />
<br />
* switch off eth1:<br />
<br />
mpirun --mca btl_tcp_if_exclude lo,eth1 ...<br />
<br />
or<br />
<br />
* you can restore the above table on all nodes by running "sh /etc/network/if-up.d/00routes" as root<br />
<br />
It is not yet clear why without this entry the connection to the "21" addresses can't be connected. We expect that in this case following rule should be matched (because of the mask):<br />
192.168.20.0 * 255.255.254.0 U 0 0 0 eth0<br />
<br />
The packets leave over the eth0 network interface then and should go over switch1 to switch2 and eth1 interface of the corresponding node<br />
<br />
<br />
* If one attempts a ping from one node A, via its eth0 interface, to the address of another node's (B) eth1 interface, the following is observed:<br />
** outgoing ping packets appear only on the eth0 interface of the first node A.<br />
** incoming ping packets appear only on eth1 interface of the second node B.<br />
** outgoing ping response packets appear on the eth0 interface of the second node B, never on the eth1 interface despite pinging the eth1 address specifically.<br />
What explains this? With the routing tables as they are above, or in the damaged case, the ping may arrive to the correct interface, but the response from B is routed to A-eth0 via B-eth0.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=376HCL cluster/hcl node install configuration log2010-05-11T14:04:02Z<p>Rhiggins: /* Complications */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
==Networking==<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
===Routing Tables===<br />
Add an executable script named <code>00routes</code> to the directory <code>/etc/network/if-up.d</code>. This script will be called after the interfaces listed as <code>auto</code> are brought up on boot (or networking service restart). Note: the DHCP process for eth1 is backgrounded so that the startup of other services can continue. Our routing script adds routes on this interface even though it is not yet fully up. The script outputs some errors but the routing entries remain none-the-less. The script should read as follows:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
# Static Routes<br />
<br />
# route ganglia broadcast t<br />
route add -host 239.2.11.72 dev eth0<br />
<br />
# all traffic to heterogeneous gate goes through eth0<br />
route add -host 192.168.20.254 dev eth0<br />
<br />
# all subnet traffic goes through specific interface<br />
route add -net 192.168.20.0 netmask 255.255.255.0 dev eth0<br />
route add -net 192.168.21.0 netmask 255.255.255.0 dev eth1<br />
</source><br />
<br />
The naming of the script is important, we want our routes in place before other scripts in the <code>/etc/network/if-up.d</code> directory are executed, the order in which they are executed is alphabetical.<br />
<br />
===Hosts===<br />
Change the hosts file so that it does not list the node's hostname, otherwise this would confuse nodes that are cloned from this image.<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
==NTP==<br />
Install NTP software:<br />
apt-get install ntp<br />
Edit configuration and make <code>server heterogeneous.ucd.ie</code> the sole server entry. Comment out any other servers. Restart the NTP service.<br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source><br />
<br />
==Sysstat==<br />
[http://pagesperso-orange.fr/sebastien.godard/ Sysstat] is a useful suite of tools for measuring performance of different system components. Unfortunately it adds some un-useful cron entries for collecting a historic set of system performance data. Though these cron entries point to disabled scripts, we will disable them none the less. Edit the file <code>/etc/cron.d/sysstat</code> and comment out all lines.<br />
<source lang="text"><br />
# The first element of the path is a directory where the debian-sa1<br />
# script is located<br />
#PATH=/usr/lib/sysstat:/usr/sbin:/usr/sbin:/usr/bin:/sbin:/bin<br />
<br />
# Activity reports every 10 minutes everyday<br />
#5-55/10 * * * * root command -v debian-sa1 > /dev/null && debian-sa1 1 1<br />
<br />
# Additional run at 23:59 to rotate the statistics file<br />
#59 23 * * * root command -v debian-sa1 > /dev/null && debian-sa1 60 2<br />
</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=MPI&diff=375MPI2010-05-07T18:08:01Z<p>Rhiggins: /* Tips & Tricks */</p>
<hr />
<div>== Documentation ==<br />
* http://www.mpi-forum.org/docs/docs.html<br />
<br />
== Implementations ==<br />
* [[LAM]]<br />
* [[MPICH]]<br />
* [[OpenMPI]]<br />
* [[MPICH2]]<br />
<br />
== Manual installation ==<br />
Install in separate subfolder <code>$HOME/SUBDIR</code>, because you may need some MPI implementations (see [[Libraries]])<br />
<br />
== Tips & Tricks ==<br />
* For safe consecutive communications create new context, for example:<br />
<source lang="C"><br />
int communication_operation(MPI_Comm comm) {<br />
MPI_Comm newcomm;<br />
MPI_Comm_dup(comm, &newcomm);<br />
... // work with newcomm<br />
MPI_Comm_free(&newcomm);<br />
}<br />
</source><br />
Mind the overhead of <code>MPI_Comm_dup</code> and <code>MPI_Comm_free</code>.<br />
<br />
* If you are having trouble with the multi-homed nature of the HCL Cluster, check [http://www.open-mpi.org/faq/?category=tcp#tcp-selection here]<br />
<br />
== Debugging ==<br />
* Add the following code:<br />
<source lang="C"><br />
int rank;<br />
MPI_Comm_rank(MPI_COMM_WORLD, &rank);<br />
if (!rank)<br />
getc(stdin);<br />
MPI_Barrier(MPI_COMM_WORLD);<br />
</source><br />
* Compile your code with <code>-g</code> option<br />
* Run parallel application<br />
* Attach to process(es) from [[GDB]]<br />
** MPICH-1 runs a background process for each application process: 0, 0b, 1, 1b, ..., therefore, attach to the first ones.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=371HCL cluster/heterogeneous.ucd.ie install log2010-05-05T18:05:10Z<p>Rhiggins: /* Queue Setup */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==NTP Daemon==<br />
Nodes in the cluster should use the Network Time Protocol to set their clocks, heterogeneous.ucd.ie should provide them that service. Install ntpd with the following command:<br />
apt-get install ntp<br />
Configure the daemon with the following line in <code>/etc/ntp.conf</code>:<br />
restrict 192.168.20.0 mask 255.255.254.0 nomodify notrap<br />
<br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make ... TODO<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contains:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Open <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all lines after the first instance of DEVICESCAN are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.<br />
<br />
= Torque - PBS =<br />
<br />
Download Torque from [http://www.clusterresources.com/pages/products/torque-resource-manager.php here]. Extract archive and configure:<br />
./configure --prefix=/usr/local<br />
Run <code>make && make install</code>. Torque will have installed files under <code>/usr/local</code> and <code>/var/spool/torque</code>. Edit the Torque config file <code>/var/spool/torque/torque.cfg</code> adding the line:<br />
SERVERHOST=192.168.20.254<br />
Add the compute nodes of the cluster to the file <code>/var/spool/torque/server_priv/nodes</code><br />
<source lang="text"><br />
hcl01<br />
hcl02<br />
hcl03<br />
hcl04<br />
hcl05<br />
hcl06<br />
hcl07<br />
hcl08<br />
hcl09<br />
hcl10<br />
hcl11<br />
hcl12<br />
hcl13<br />
hcl14<br />
hcl15<br />
hcl16<br />
</source><br />
<br />
Now return to the extracted source distribution, find the <code>contrib/init.d</code> folder and copy the <code>debian.pbs_server</code> to <code>/etc/init.d/pbs_server</code>. Edit the inplace file so that the header reads:<br />
<source lang="bash"><br />
#!/bin/sh<br />
### BEGIN INIT INFO<br />
# Provides: pbs_server<br />
# Required-Start: $local_fs $named<br />
# Should-Start:<br />
# Required-Stop:<br />
</source><br />
This will ensure that important services are started before the PBS/Torque daemon.<br />
<br />
===Queues===<br />
We will configure 4 queues, with varying priority and preemption policies.<br />
*The first is the <b>normal</b> queue. This will be where most jobs, which should not be interrupted, will be placed. It has the highest priority.<br />
*The second queue is <b>lowpri</b>, it is for running jobs which may run for extended periods but are interruptible. Should a job <i>A</i> be placed on the normal queue while a lowpri job <i>B</i> is running, <i>B</i> will be sent a kill signal, so it may shut down cleanly, and then it will be requeued on the system, so it can resume running after <i>A</i>b has finished.<br />
*The third queue is for running of service jobs like the homedir backup. It is named <b>service</b>. This will have lower priority than the above queues and jobs running on this queue will be preemptable.<br />
*The final queue is the <b>volunteer</b> queue. This has the lowest priority and is for executing volunteer computing jobs during otherwise idle periods. These jobs will also be preemptable.<br />
<br />
====Queue Setup====<br />
Enter the queue manager program <code>qmgr</code> as the root user on heterogeneous. Send it the following commands:<br />
<br />
Allow all users to see all queued jobs:<br />
set server query_other_jobs=TRUE'<br />
Create the default <b>normal</b> queue<br />
create queue normal<br />
set queue normal queue_type = Execution<br />
set queue normal Priority = 10000<br />
set queue normal enabled = True<br />
set queue normal started = True<br />
Create <b>lowpri</b> queue<br />
create queue lowpri<br />
set queue lowpri queue_type = Execution<br />
set queue lowpri Priority = 1000<br />
set queue lowpri enabled = True<br />
set queue lowpri started = True<br />
Create <b>service</b> queue<br />
set queue service queue_type = Execution<br />
set queue service Priority = 100<br />
set queue service enabled = True<br />
set queue service started = True<br />
Create <b>volunteer</b> queue<br />
set queue volunteer queue_type = Execution<br />
set queue volunteer Priority = 0<br />
set queue volunteer enabled = True<br />
set queue volunteer started = True<br />
Set some server settings<br />
set server default_queue=normal<br />
set server resources_default.nodes = 1<br />
set server resources_default.walltime = 12:00:00<br />
<br />
Now edit the maui configuration for these queues:<br />
<source lang="text"><br />
SERVERHOST heterogeneous<br />
ADMIN1 root<br />
<br />
RMPOLLINTERVAL 00:00:01<br />
DEFERTIME 0<br />
DEFERCOUNT 86400<br />
<br />
PREEMPTIONPOLICY REQUEUE<br />
<br />
QUEUETIMEWEIGHT 1<br />
QOSWEIGHT 1<br />
SYSCFG QLIST=normal,lowpri,service,volunteer<br />
QOSCFG[normal] QFLAGS=PREEMPTOR<br />
QOSCFG[lowpri] QFLAGS=PREEMPTEE,PREEMPTOR<br />
QOSCFG[service] QFLAGS=PREEMPTEE,PREEMPTOR<br />
QOSCFG[volunteer] QFLAGS=PREEMPTEE<br />
<br />
CREDWEIGHT 1<br />
CLASSWEIGHT 1<br />
<br />
CLASSCFG[normal] QDEF=normal PRIORITY=10000<br />
CLASSCFG[lowpri] QDEF=lowpri PRIORITY=1000<br />
CLASSCFG[service] QDEF=service PRIORITY=100<br />
CLASSCFG[volunteer] QDEF=volunteer PRIORITY=0<br />
</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=370HCL cluster/heterogeneous.ucd.ie install log2010-05-05T15:30:45Z<p>Rhiggins: /* Torque - PBS */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==NTP Daemon==<br />
Nodes in the cluster should use the Network Time Protocol to set their clocks, heterogeneous.ucd.ie should provide them that service. Install ntpd with the following command:<br />
apt-get install ntp<br />
Configure the daemon with the following line in <code>/etc/ntp.conf</code>:<br />
restrict 192.168.20.0 mask 255.255.254.0 nomodify notrap<br />
<br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make ... TODO<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contains:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Open <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all lines after the first instance of DEVICESCAN are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.<br />
<br />
= Torque - PBS =<br />
<br />
Download Torque from [http://www.clusterresources.com/pages/products/torque-resource-manager.php here]. Extract archive and configure:<br />
./configure --prefix=/usr/local<br />
Run <code>make && make install</code>. Torque will have installed files under <code>/usr/local</code> and <code>/var/spool/torque</code>. Edit the Torque config file <code>/var/spool/torque/torque.cfg</code> adding the line:<br />
SERVERHOST=192.168.20.254<br />
Add the compute nodes of the cluster to the file <code>/var/spool/torque/server_priv/nodes</code><br />
<source lang="text"><br />
hcl01<br />
hcl02<br />
hcl03<br />
hcl04<br />
hcl05<br />
hcl06<br />
hcl07<br />
hcl08<br />
hcl09<br />
hcl10<br />
hcl11<br />
hcl12<br />
hcl13<br />
hcl14<br />
hcl15<br />
hcl16<br />
</source><br />
<br />
Now return to the extracted source distribution, find the <code>contrib/init.d</code> folder and copy the <code>debian.pbs_server</code> to <code>/etc/init.d/pbs_server</code>. Edit the inplace file so that the header reads:<br />
<source lang="bash"><br />
#!/bin/sh<br />
### BEGIN INIT INFO<br />
# Provides: pbs_server<br />
# Required-Start: $local_fs $named<br />
# Should-Start:<br />
# Required-Stop:<br />
</source><br />
This will ensure that important services are started before the PBS/Torque daemon.<br />
<br />
===Queues===<br />
We will configure 4 queues, with varying priority and preemption policies.<br />
*The first is the <b>normal</b> queue. This will be where most jobs, which should not be interrupted, will be placed. It has the highest priority.<br />
*The second queue is <b>lowpri</b>, it is for running jobs which may run for extended periods but are interruptible. Should a job <i>A</i> be placed on the normal queue while a lowpri job <i>B</i> is running, <i>B</i> will be sent a kill signal, so it may shut down cleanly, and then it will be requeued on the system, so it can resume running after <i>A</i>b has finished.<br />
*The third queue is for running of service jobs like the homedir backup. It is named <b>service</b>. This will have lower priority than the above queues and jobs running on this queue will be preemptable.<br />
*The final queue is the <b>volunteer</b> queue. This has the lowest priority and is for executing volunteer computing jobs during otherwise idle periods. These jobs will also be preemptable.<br />
<br />
====Queue Setup====<br />
Enter the queue manager program <code>qmgr</code> as the root user on heterogeneous. Send it the following commands:<br />
<br />
Allow all users to see all queued jobs:<br />
set server query_other_jobs=TRUE'<br />
Create the default <b>normal</b> queue<br />
create queue normal<br />
set queue normal queue_type = Execution<br />
set queue normal Priority = 10000<br />
set queue normal enabled = True<br />
set queue normal started = True<br />
Create <b>lowpri</b> queue<br />
create queue lowpri<br />
set queue lowpri queue_type = Execution<br />
set queue lowpri Priority = 1000<br />
set queue lowpri enabled = True<br />
set queue lowpri started = True<br />
Create <b>service</b> queue<br />
set queue service queue_type = Execution<br />
set queue service Priority = 100<br />
set queue service enabled = True<br />
set queue service started = True<br />
Create <b>volunteer</b> queue<br />
set queue volunteer queue_type = Execution<br />
set queue volunteer Priority = 0<br />
set queue volunteer enabled = True<br />
set queue volunteer started = True<br />
Set some server settings<br />
set server default_queue=normal<br />
set server resources_default.nodes = 1<br />
set server resources_default.walltime = 12:00:00<br />
<br />
Now edit the maui configuration for these queues:<br />
<source><br />
QOSWEIGHT 1<br />
SYSCFG QLIST=normal,lowpri,service,volunteer<br />
QOSCFG[normal] PRIORITY=10000 QFLAGS=PREEMPTEE<br />
QOSCFG[lowpri] PRIORITY=1000 QFLAGS=PREEMPTEE<br />
QOSCFG[service] PRIORITY=100 QFLAGS=PREEMPTOR<br />
QOSCFG[volunteer] PRIORTIY=0 QFLAGS=PREEMPTEE</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=369HCL cluster/heterogeneous.ucd.ie install log2010-05-03T16:31:59Z<p>Rhiggins: /* Torque - PBS */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==NTP Daemon==<br />
Nodes in the cluster should use the Network Time Protocol to set their clocks, heterogeneous.ucd.ie should provide them that service. Install ntpd with the following command:<br />
apt-get install ntp<br />
Configure the daemon with the following line in <code>/etc/ntp.conf</code>:<br />
restrict 192.168.20.0 mask 255.255.254.0 nomodify notrap<br />
<br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make ... TODO<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contains:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Open <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all lines after the first instance of DEVICESCAN are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.<br />
<br />
= Torque - PBS =<br />
<br />
Download Torque from [http://www.clusterresources.com/pages/products/torque-resource-manager.php here]. Extract archive and configure:<br />
./configure --prefix=/usr/local<br />
Run <code>make && make install</code>. Torque will have installed files under <code>/usr/local</code> and <code>/var/spool/torque</code>. Edit the Torque config file <code>/var/spool/torque/torque.cfg</code> adding the line:<br />
SERVERHOST=192.168.20.254<br />
Add the compute nodes of the cluster to the file <code>/var/spool/torque/server_priv/nodes</code><br />
<source lang="text"><br />
hcl01<br />
hcl02<br />
hcl03<br />
hcl04<br />
hcl05<br />
hcl06<br />
hcl07<br />
hcl08<br />
hcl09<br />
hcl10<br />
hcl11<br />
hcl12<br />
hcl13<br />
hcl14<br />
hcl15<br />
hcl16<br />
</source><br />
<br />
Now return to the extracted source distribution, find the <code>contrib/init.d</code> folder and copy the <code>debian.pbs_server</code> to <code>/etc/init.d/pbs_server</code>. Edit the inplace file so that the header reads:<br />
<source lang="bash"><br />
#!/bin/sh<br />
### BEGIN INIT INFO<br />
# Provides: pbs_server<br />
# Required-Start: $local_fs $named<br />
# Should-Start:<br />
# Required-Stop:<br />
</source><br />
This will ensure that important services are started before the PBS/Torque daemon.<br />
<br />
Allow all users to see all queued jobs:<br />
<code>qmgr -c 'set server query_other_jobs=TRUE'</code></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=368HCL cluster2010-05-03T11:23:58Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>). <code>root_ssh</code> will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh [user@]<host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=367HCL cluster2010-05-03T11:23:19Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>. <code>root_ssh</code> is will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh <host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy. Sometimes all logs are sent to screenlog.0, not sure why.<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=366HCL cluster/hcl node install configuration log2010-05-02T16:57:47Z<p>Rhiggins: /* Routing Tables */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
==Networking==<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
===Routing Tables===<br />
Add an executable script named <code>00routes</code> to the directory <code>/etc/network/if-up.d</code>. This script will be called after the interfaces listed as <code>auto</code> are brought up on boot (or networking service restart). Note: the DHCP process for eth1 is backgrounded so that the startup of other services can continue. Our routing script adds routes on this interface even though it is not yet fully up. The script outputs some errors but the routing entries remain none-the-less. The script should read as follows:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
# Static Routes<br />
<br />
# route ganglia broadcast t<br />
route add -host 239.2.11.72 dev eth0<br />
<br />
# all traffic to heterogeneous gate goes through eth0<br />
route add -host 192.168.20.254 dev eth0<br />
<br />
# all subnet traffic goes through specific interface<br />
route add -net 192.168.20.0 netmask 255.255.255.0 dev eth0<br />
route add -net 192.168.21.0 netmask 255.255.255.0 dev eth1<br />
</source><br />
<br />
The naming of the script is important, we want our routes in place before other scripts in the <code>/etc/network/if-up.d</code> directory are executed, the order in which they are executed is alphabetical.<br />
<br />
===Hosts===<br />
Change the hosts file so that it does not list the node's hostname, otherwise this would confuse nodes that are cloned from this image.<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
==NTP==<br />
Install NTP software:<br />
apt-get install ntp<br />
Edit configuration and make <code>server heterogeneous.ucd.ie</code> the sole server entry. Comment out any other servers. Restart the NTP service.<br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=365HCL cluster/hcl node install configuration log2010-05-02T16:57:26Z<p>Rhiggins: /* General Installation */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
==Networking==<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
===Routing Tables===<br />
Add an executable script named <code>00routes</code> to the directory <code>/etc/network/if-up.d</code>. This script will be called after the interfaces listed as <code>auto</code> are brought up on boot (or networking service restart). Note: the DHCP process for eth1 is backgrounded so that the startup of other services can continue. Our routing script adds routes on this interface even though it is not yet fully up. The script outputs some errors but the routing entries remain none-the-less. The script should read as follows:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
# Static Routes<br />
<br />
# route ganglia broadcast t<br />
route add -host 239.2.11.72 dev eth0<br />
<br />
# all traffic to heterogeneous gate goes through eth0<br />
route add -host 192.168.20.254 dev eth0<br />
<br />
# all subnet traffic goes through specific interface<br />
route add -net 192.168.20.0 netmask 255.255.255.0 dev eth0<br />
route add -net 192.168.21.0 netmask 255.255.255.0 dev eth1<br />
</code><br />
<br />
The naming of the script is important, we want our routes in place before other scripts in the <code>/etc/network/if-up.d</code> directory are executed, the order in which they are executed is alphabetical.<br />
<br />
===Hosts===<br />
Change the hosts file so that it does not list the node's hostname, otherwise this would confuse nodes that are cloned from this image.<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
==NTP==<br />
Install NTP software:<br />
apt-get install ntp<br />
Edit configuration and make <code>server heterogeneous.ucd.ie</code> the sole server entry. Comment out any other servers. Restart the NTP service.<br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=364HCL cluster/hcl node install configuration log2010-05-02T16:51:53Z<p>Rhiggins: /* General Installation */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth1 eth0<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
Change the hosts file so that it does not list the node's hostname, otherwise this would confuse nodes that are cloned from this image.<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
==NTP==<br />
Install NTP software:<br />
apt-get install ntp<br />
Edit configuration and make <code>server heterogeneous.ucd.ie</code> the sole server entry. Comment out any other servers. Restart the NTP service.<br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=363HCL cluster/heterogeneous.ucd.ie install log2010-05-02T16:50:21Z<p>Rhiggins: </p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==NTP Daemon==<br />
Nodes in the cluster should use the Network Time Protocol to set their clocks, heterogeneous.ucd.ie should provide them that service. Install ntpd with the following command:<br />
apt-get install ntp<br />
Configure the daemon with the following line in <code>/etc/ntp.conf</code>:<br />
restrict 192.168.20.0 mask 255.255.254.0 nomodify notrap<br />
<br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make ... TODO<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contains:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Open <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all lines after the first instance of DEVICESCAN are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.<br />
<br />
= Torque - PBS =<br />
<br />
Allow all users to see all queued jobs:<br />
<code>qmgr -c 'set server query_other_jobs=TRUE'</code></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=362HCL cluster/heterogeneous.ucd.ie install log2010-04-30T08:52:30Z<p>Rhiggins: /* Disk Monitoring */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contains:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Open <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all lines after the first instance of DEVICESCAN are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.<br />
<br />
= Torque - PBS =<br />
<br />
Allow all users to see all queued jobs:<br />
<code>qmgr -c 'set server query_other_jobs=TRUE'</code></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=357HCL cluster/heterogeneous.ucd.ie install log2010-04-29T16:02:18Z<p>Rhiggins: </p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
Firstly, Clonezilla is probably going to pollute a lot of your server configuration when it sets itself up. Be prepared to loose your IPtables configuration, NFS (if any) and DHCP settings. Maybe more.<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
==Install DHCP==<br />
Install the DHCP server package with <code>apt-get install dhcp3-server</code>. When you install Clonezilla it will probably pollute your DHCP server setup, so make<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=356HCL cluster/heterogeneous.ucd.ie install log2010-04-29T15:57:52Z<p>Rhiggins: /* Installing Ganglia Frontend */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" 192.168.20.1 192.168.20.16<br />
<br />
This means that the gmetad collector connect to hcl01 and hcl16 on the .20 subnet to gather data for the frontend to use.<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://heterogeneous.ucd.ie/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=355HCL cluster/heterogeneous.ucd.ie install log2010-04-29T15:55:27Z<p>Rhiggins: /* Install NIS */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnets .20 and .21<br />
255.255.254.0 192.168.20.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=354HCL cluster/heterogeneous.ucd.ie install log2010-04-29T15:53:54Z<p>Rhiggins: /* IP Tables */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory, named <code>00iptables</code>. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=353HCL cluster/heterogeneous.ucd.ie install log2010-04-29T15:51:07Z<p>Rhiggins: /* DNS / BIND */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9 (<code>apt-get install bind9</code>). Edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse). We have two subdomains where reverse lookups will have to be specified 192.168.20 and 192.168.21<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
};<br />
<br />
zone "20.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.20";<br />
};<br />
</source><br />
<br />
Also edit the options file: <code>/etc/bind/named.conf.options</code>, note the subnet we define in the allow sections, 192.168.20/23, it will permit access from 192.168.20.* and 192.168.21.* addresses.<br />
<br />
<source lang="text"><br />
options {<br />
directory "/var/cache/bind";<br />
<br />
// If there is a firewall between you and nameservers you want<br />
// to talk to, you may need to fix the firewall to allow multiple<br />
// ports to talk. See http://www.kb.cert.org/vuls/id/800113<br />
<br />
// If your ISP provided one or more IP addresses for stable<br />
// nameservers, you probably want to use them as forwarders.<br />
// Uncomment the following block, and insert the addresses replacing<br />
// the all-0's placeholder.<br />
<br />
forwarders {<br />
137.43.116.19;<br />
137.43.116.17;<br />
137.43.105.22;<br />
};<br />
<br />
recursion yes;<br />
<br />
version "REFUSED";<br />
<br />
allow-recursion {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
allow-query {<br />
127.0.0.1;<br />
192.168.20.0/23;<br />
};<br />
<br />
auth-nxdomain no; # conform to RFC1035<br />
listen-on-v6 { any; };<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> & <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=352HCL cluster2010-04-28T19:56:50Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster (in <code>/root/scripts</code>. <code>root_ssh</code> is will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# root_ssh<br />
usage: root_ssh <host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do root_ssh $i ps ax \| grep pbs; done<br />
<br />
The above is sequential. To run parallel jobs, for example: <code>apt-get update && apt-get -y upgrade</code>, try the following trick with [http://www.gnu.org/software/screen/ screen]:<br />
# for i in `cat /etc/dsh/machines.list`; do screen -L -d -m root_ssh $i apt-get update \&\& apt-get -y upgrade'; done<br />
You can check the screenlog.* files for errors and delete them when you are happy<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=351HCL cluster2010-04-28T15:08:40Z<p>Rhiggins: /* Useful Tools */</p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|thumbnail||Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster. <code>root_login.expect</code> is will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# ./root_login.expect <br />
usage: root_ssh.expect <host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do ./root_login.expect $i ps ax \| grep pbs; done<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster&diff=348HCL cluster2010-04-28T15:04:18Z<p>Rhiggins: </p>
<hr />
<div>http://hcl.ucd.ie/Hardware<br />
<br />
==General Information==<br />
[[Image:network.jpg|right|Layout of the Cluster]]<br />
The hcl cluster is heterogeneous in computing hardware & network ability.<br />
<br />
Nodes are from Dell, IBM, and HP, with Celeron, Pentium 4, Xeon, and AMD processors ranging in speeds from 1.8 to 3.6Ghz. Accordingly architectures and parameters such as Front Side Bus, Cache, and Main Memory all vary.<br />
<br />
Operating System used is Debian “squeeze” with Linux kernel 2.6.32.<br />
<br />
The network hardware consists of two Cisco 24+4 port Gigabit switches. Each node has two Gigabit ethernet ports - each eth0 is connected to the first switch, and each eth1 is connected to the second switch. The switches are also connected to each other. The bandwidth of each port can be configured to meet any value between 8Kb/s and 1Gb/s. This allows testing on a very large number of network topologies, As the bandwidth on the link connecting the two switches can also be configured, the cluster can actually act as two separate clusters connected via one link.<br />
<br />
The diagram below shows a schematic of the cluster.<br />
<br />
<br />
== Detailed Cluster Specification ==<br />
A table of hardware configuration is available here: [[Cluster Specification]]<br />
== Cluster Administration ==<br />
===Useful Tools===<br />
<code>root</code> on <code>heterogeneous.ucd.ie</code> has a number of [http://expect.nist.gov/ Expect] scripts to automate administration on the cluster. <code>root_login.expect</code> is will automatically log into a host, provide the root password and either return a shell to the user or execute a command that is passed as a second argument. Command syntax is as follows:<br />
<br />
<source lang="text"><br />
# ./root_login.expect <br />
usage: root_login.expect <host> [command]<br />
</source><br />
<br />
Example usage, to login and execute a command on each node in the cluster (note the file <code>/etc/dsh/machines.list</code> contains the hostnames of all compute nodes of the cluster):<br />
# for i in `cat /etc/dsh/machines.list`; do ./root_login.expect $i ps ax \| grep pbs; done<br />
<br />
== Software packages available on HCL Cluster 2.0 ==<br />
<br />
Wit a fresh installation of operating systems on HCL Cluster the follow list of packages are avalible:<br />
* autoconf<br />
* automake<br />
* fftw2<br />
* git<br />
* gfortran<br />
* gnuplot<br />
* libtool<br />
* netperf<br />
* octave3.2<br />
* qhull<br />
* subversion<br />
* valgrind<br />
* gsl-dev<br />
* vim<br />
* python<br />
* mc<br />
* openmpi-bin <br />
* openmpi-dev<br />
* evince<br />
* libboost-graph-dev<br />
* libboost-serialization-dev<br />
* r-cran-strucchange<br />
* graphviz<br />
* doxygen<br />
<br />
[[new hcl node install & configuration log]]<br />
<br />
[[new heterogeneous.ucd.ie install log]]<br />
<br />
== Access and Security ==<br />
All access and security for the cluster is handled by the gateway machine (heterogeneous.ucd.ie). This machine is not considered a compute node and should not be used as such. The only new incoming connections allowed are ssh, other incoming packets such as http that are responding to requests from inside the cluster (established or related) are also allowed. Incoming ssh packets are only accepted if they are originating from designated IP addresses. These IP's must be registered ucd IP's. csserver.ucd.ie is allowed, as is hclgate.ucd.ie, on which all users have accounts. Thus to gain access to the cluster you can ssh from csserver, hclgate or other allowed machines to heterogeneous. From there you can ssh to any of the nodes (hcl01-hcl16).<br />
<br />
Access from outside the UCD network is only allowed once you have gained entry to a server that allows outside connections (such as csserver.ucd.ie)</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=338HCL cluster/heterogeneous.ucd.ie install log2010-04-27T23:51:28Z<p>Rhiggins: </p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.<br />
=Hardware Monitoring & Backup=<br />
==Disk Monitoring==<br />
Install smartmontools as per [http://www.howtoforge.com/checking-hard-disk-sanity-with-smartmontools-debian-ubuntu here]. Briefly:<br />
apt-get install smartmontools<br />
Edit <code>/etc/defaults/smartmontools</code> so that it contails:<br />
<source lang="text"><br />
# List of devices you want to explicitly enable S.M.A.R.T. for<br />
# Not needed (and not recommended) if the device is monitored by smartd<br />
enable_smart="/dev/sda"<br />
<br />
# uncomment to start smartd on system startup<br />
start_smartd=yes<br />
<br />
# uncomment to pass additional options to smartd on startup<br />
smartd_opts="--interval=1800"<br />
</source><br />
Edit <code>/etc/smartd.conf</code> and edit the first line that begins with DEVICESCAN (all other lines are ignored). Have it read something like:<br />
DEVICESCAN -d removable -n standby -m root -m robert_higgins@iol.ie -M exec /usr/share/smartmontools/smartd-runner<br />
Then start the service <code>/etc/init.d/smartmontools start</code><br />
Note, consider installing this on all nodes, as it would be interesting to have prior notice of any failing disks.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=336HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:58:19Z<p>Rhiggins: /* DNS / BIND */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
</source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code> Populate them with all nodes of the cluster.<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=335HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:57:43Z<p>Rhiggins: /* Interfaces */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). This will allow Gigabit operation on eth0 with the tg3 hardware (I think). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux:<br />
<br />
<source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code>:<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=334HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:56:27Z<p>Rhiggins: /* Networking */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux: <source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code>:<br />
<br />
===IP Tables===<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=333HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:54:07Z<p>Rhiggins: /* DNS / BIND */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 127.0.0.1<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code>:<br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux: <source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=332HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:53:35Z<p>Rhiggins: /* Networking */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
===Interfaces===<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
===DNS / BIND===<br />
We will run our own DNS server for the cluster. First set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 192.168.21.254<br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
Now install bind9, edit <code>/etc/bind/named.conf.local</code> and set the domain zones for the cluster (forwards and reverse).<br />
<br />
<source lang="text"><br />
//<br />
// Do any local configuration here<br />
//<br />
<br />
// Consider adding the 1918 zones here, if they are not used in your<br />
// organization<br />
//include "/etc/bind/zones.rfc1918";<br />
<br />
include "/etc/bind/rndc.key";<br />
<br />
controls {<br />
inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };<br />
};<br />
<br />
zone "heterogeneous.ucd.ie" {<br />
type master;<br />
file "db.heterogeneous.ucd.ie";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<br />
zone "21.168.192.in-addr.arpa" {<br />
type master;<br />
file "db.192.168.21";<br />
allow-update { key "rndc-key"; };<br />
notify yes;<br />
};<br />
<source><br />
<br />
Now work on the zone files specified <code>db.heterogneneous.ucd.ie</code> and the reverse maps <code>db.192.168.21</code>:<br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux: <source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=331HCL cluster/heterogeneous.ucd.ie install log2010-04-27T11:38:40Z<p>Rhiggins: /* Networking */</p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
* set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux: <source lang="text">apt-get update && apt-get install firmware-linux</source> You probably need to reboot now.<br />
<br />
* Set up <code>iptables</code>. We want to implement NAT between the internal network (<code>eth0</code>) and external one (<code>eth1</code>). Add a script to <code>/etc/network/if-up.d</code> directory. All scripts in this directory will be executed after network interfaces are brought up, so this will persist:<br />
<br />
<source lang="bash"><br />
#!/bin/sh<br />
<br />
PATH=/usr/sbin:/sbin:/bin:/usr/bin<br />
<br />
IF_INT=eth0<br />
IF_EXT=eth1<br />
<br />
#<br />
# delete all existing rules.<br />
#<br />
iptables -F<br />
iptables -t nat -F<br />
iptables -t mangle -F<br />
iptables -X<br />
<br />
# Always accept loopback traffic<br />
iptables -A INPUT -i lo -j ACCEPT<br />
<br />
<br />
# Allow established connections, and those not coming from the outside<br />
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
iptables -A INPUT -m state --state NEW ! -i $IF_EXT -j ACCEPT<br />
iptables -A FORWARD -i $IF_EXT -o $IF_INT -m state --state ESTABLISHED,RELATED -j ACCEPT<br />
<br />
# Allow outgoing connections from the LAN side.<br />
iptables -A FORWARD -i $IF_INT -o $IF_EXT -j ACCEPT<br />
<br />
# Masquerade.<br />
iptables -t nat -A POSTROUTING -o $IF_EXT -j MASQUERADE<br />
<br />
# Don't forward from the outside to the inside.<br />
iptables -A FORWARD -i $IF_EXT -o $IF_EXT -j REJECT<br />
<br />
# Enable routing.<br />
echo 1 > /proc/sys/net/ipv4/ip_forward<br />
</source><br />
<br />
<br />
==Clonezilla==<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/heterogeneous.ucd.ie_install_log&diff=330HCL cluster/heterogeneous.ucd.ie install log2010-04-27T10:34:34Z<p>Rhiggins: </p>
<hr />
<div>* Basic installation of Debian Squeeze<br />
<br />
==Networking==<br />
* edit <code>/etc/networks/interfaces</code> Note that at some point eth1 should be configured by DHCP, it is on the UCD LAN and must be registered correctly (update MAC address with services). <code>eth0</code> is the internal network.<br />
<br />
<source lang="text"><br />
# The loopback network interface<br />
auto lo eth0 eth1<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet static<br />
address 192.168.21.254<br />
netmask 255.255.255.0<br />
gateway 192.168.21.1<br />
<br />
iface eth1 inet static<br />
address 193.1.132.124<br />
netmask 255.255.252.0<br />
gateway 193.1.132.1<br />
</source><br />
<br />
* set resolv.conf:<br />
<br />
<source lang="text"><br />
nameserver 137.43.116.19<br />
nameserver 137.43.116.17<br />
nameserver 137.43.105.22<br />
domain ucd.ie<br />
search ucd.ie<br />
</source><br />
<br />
<br />
* Install non-free linux firmware for network interface (eth0). Edit <code>/etc/apt/sources.list</code> including the lines:<br />
<br />
<source lang="text">deb http://ftp.ie.debian.org/debian/ squeeze main contrib non-free<br />
deb-src http://ftp.ie.debian.org/debian/ squeeze main contrib non-free</source><br />
<br />
* Install firmware-linux: <source lang="text">apt-get update && apt-get install firmware-linux</source><br />
<br />
<br />
* follow the guide to installing Clonezilla [http://www.howtoforge.com/cloning-linux-systems-with-clonezilla-server-edition-clonezilla-se-p2 here]. Essentially:<br />
** add repository key <source lang="text">wget -q http://drbl.sourceforge.net/GPG-KEY-DRBL -O- | apt-key add -</source><br />
** the line add to /etc/apt/sources.list: <source lang="text">deb http://drbl.sourceforge.net/drbl-core drbl stable</source><br />
** run: <source lang="text">apt-get update && apt-get install drbl && /opt/drbl/sbin/drbl4imp</source><br />
** accept default options to drbl4imp.<br />
<br />
* After Clonezilla has installed edit <code>/etc/dhcpd3/dhcpd.conf</code>, adding all entries for test nodes <code>hcl07</code> and <code>hcl03</code>. Also ensure these nodes have been removed from the inplace heterogeneous.ucd.ie server so that they are only served by one machine.<br />
<source lang="text"><br />
default-lease-time 300;<br />
max-lease-time 300;<br />
option subnet-mask 255.255.255.0;<br />
option domain-name-servers 137.43.116.19,137.43.116.17,137.43.105.22;<br />
option domain-name "ucd.ie"; <br />
ddns-update-style none; # brett had ad-hoc ...?<br />
server-name drbl;<br />
filename = "pxelinux.0";<br />
<br />
<br />
subnet 192.168.21.0 netmask 255.255.255.0 {<br />
option subnet-mask 255.255.255.0;<br />
option routers 192.168.21.1;<br />
next-server 192.168.21.254;<br />
<br />
pool {<br />
# allow members of "DRBL-Client";<br />
range 192.168.21.200 192.168.21.212;<br />
}<br />
<br />
host hcl03 {<br />
option host-name "hcl03.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6C;<br />
fixed-address 192.168.21.5;<br />
}<br />
host hcl03_eth1 {<br />
option host-name "hcl03_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:22:6D;<br />
fixed-address 192.168.21.105;<br />
}<br />
host hcl07 {<br />
option host-name "hcl07.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E2;<br />
fixed-address 192.168.21.9;<br />
}<br />
host hcl07_eth1 {<br />
option host-name "hcl07_eth1.ucd.ie";<br />
hardware ethernet 00:14:22:0A:20:E3;<br />
fixed-address 192.168.21.109;<br />
}<br />
<br />
default-lease-time 21600; <br />
max-lease-time 43200;<br />
}<br />
</source><br />
<br />
==Install NIS==<br />
Copy users from <code>passwd</code>, <code>groups</code> and <code>shadow</code> from <code>/etc</code> on <code>hcl01</code>.<br />
<br />
Install nis.<br />
<br />
Edit <code>/etc/defaultdomain</code> so that it contains:<br />
<br />
heterogeneous.ucd.ie<br />
<br />
Edit <code>/etc/defaults/nis</code> so that it contains:<br />
<br />
# Are we a NIS server and if so what kind (values: false, slave, master)<br />
NISSERVER=master<br />
<br />
Edit <code>/etc/ypserv.securenets</code> so that is contains:<br />
<br />
# allow connects from local<br />
255.0.0.0 127.0.0.0<br />
# allow connections from heterogeneous subnet<br />
255.255.255.0 192.168.21.0<br />
<br />
The NIS host is also a client of itself, so do the client set up as follows:<br />
<br />
Edit <code>/etc/hosts</code> end ensure the NIS Master is listed<br />
<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
<br />
Edit <code>/etc/yp.conf</code> and ensure that it contains:<br />
<br />
domain heterogeneous.ucd.ie server localhost<br />
<br />
Edit <code>/etc/passwd</code> adding a line to the end that reads: <code>+::::::</code>. Edit <code>/etc/group</code> with a line <code>+:::</code> at the line.<br />
<br />
The NIS Makefile will not pull userid and groupids that are lower than a certain value, we must set this to 500 in <code>/var/yp/Makefile</code><br />
<br />
MINUID=500<br />
MINGID=500<br />
<br />
Start the <code>ypbind</code> and <code>yppasswd</code> services. Then initialise the NIS database:<br />
/usr/lib/yp/ypinit -m<br />
Accept defaults at prompts.<br />
<br />
Now start other NIS services<br />
service nis start<br />
<br />
==Installing Ganglia Frontend==<br />
Install the packages gmetad and ganglia-webfrontend.<br />
<br />
Configure the front end by appending to <code>/etc/apache2/apache2.conf</code>, the following:<br />
Include /etc/ganglia-webfrontend/apache.conf<br />
<br />
Configure gmetad by adding to the <code>/etc/ganglia/gmetad.conf</code>, the following line:<br />
data_source "HCL Cluster" hcl07<br />
<br />
After all packages are configured execute:<br />
<source lang="text"><br />
service apache2 restart<br />
service gmetad restart<br />
</source><br />
Pointing your browser to [http://192.168.21.254/ganglia/index.php here] should display the monitoring page for HCL Cluster. <code>gmond</code> must also be installed and configured on the cluster nodes.</div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=300HCL cluster/hcl node install configuration log2010-04-25T17:51:26Z<p>Rhiggins: /* General Installation */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth1 eth0<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
Change the hosts file so that it does not list the node's hostname, otherwise this would confuse nodes that are cloned from this image.<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=299HCL cluster/hcl node install configuration log2010-04-25T17:50:47Z<p>Rhiggins: /* NIS Client */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth1 eth0<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
Change the hosts file so that it does not list the node's hostname<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code>, this is because DNS may not be active when the NIS client is starting and we want to ensure that it connects to the server successful.<br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadow</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhigginshttps://hcl.ucd.ie/wiki/index.php?title=HCL_cluster/hcl_node_install_configuration_log&diff=298HCL cluster/hcl node install configuration log2010-04-25T17:48:38Z<p>Rhiggins: /* NIS Client */</p>
<hr />
<div>HCL Nodes will be installed from a clone of a root node, <code>hcl07</code>. The general installation of the root is documented here. There are a number of complications as a result of the cloning process. Solutions to these complications are also explained.<br />
<br />
=General Installation=<br />
Partition filesystem with swap at the end of the disk, size 1GB, equal to maximum of the installed memory on cluster nodes. Root file system occupies the remainder of the disk, EXT4 format.<br />
<br />
Install long list of packages.<br />
<br />
Configure network interface as follows:<br />
<br />
<source lang="text"># This file describes the network interfaces available on your system<br />
# and how to activate them. For more information, see interfaces(5).<br />
<br />
# The loopback network interface<br />
auto lo eth1 eth0<br />
<br />
iface lo inet loopback<br />
<br />
# The primary network interface<br />
allow-hotplug eth0<br />
iface eth0 inet dhcp<br />
<br />
allow-hotplug eth1<br />
iface eth1 inet dhcp<br />
</source><br />
<br />
Change the hosts file so that it does not list the node's hostname<br />
<source lang="text"><br />
127.0.0.1 localhost<br />
<br />
# The following lines are desirable for IPv6 capable hosts<br />
::1 localhost ip6-localhost ip6-loopback<br />
fe00::0 ip6-localnet<br />
ff00::0 ip6-mcastprefix<br />
ff02::1 ip6-allnodes<br />
ff02::2 ip6-allrouters<br />
</source><br />
<br />
==Ganglia==<br />
Install the ganglia-monitor package. <br />
<br />
Configure ganglia monitor by editing <code>/etc/ganglia/gmond.conf</code> so that it contains:<br />
<br />
<source lang="text">cluster {<br />
name = "HCL Cluster"<br />
owner = "University College Dublin"<br />
latlong = "unspecified"<br />
url = "http://hcl.ucd.ie/"<br />
}<br />
</source><br />
And ...<br />
<source lang="text"><br />
/* Feel free to specify as many udp_send_channels as you like. Gmond<br />
used to only support having a single channel */<br />
udp_send_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
ttl = 1<br />
}<br />
<br />
/* You can specify as many udp_recv_channels as you like as well. */<br />
udp_recv_channel {<br />
mcast_join = 239.2.11.72<br />
port = 8649<br />
bind = 239.2.11.72<br />
}<br />
</source><br />
After all packages are complete execute:<br />
<source lang="text"><br />
service ganglia-monitor restart<br />
</source><br />
<br />
==NIS Client==<br />
<br />
Install nis package.<br />
Set <code>/etc/defaultdomain</code> to contain <code>heterogeneous.ucd.ie</code><br />
Make sure the NIS Server has an entry in <code>/etc/hosts</code><br />
192.168.21.254 heterogeneous.ucd.ie heterogeneous<br />
Make sure the file <code>/etc/nsswitch.conf</code> contains:<br />
passwd: compat<br />
group: compat<br />
shadow: compat<br />
Append to the file: <code>/etc/passwd</code> the line <code>+::::::</code><br />
Append to the file: <code>/etc/group</code> the line <code>+:::</code><br />
Append to the file: <code>/etc/shadown</code> the line <code>+::::::::</code><br />
Start the nis service:<br />
service nis start<br />
Check that nis is operating correctly by running the following command:<br />
ypcat passwd<br />
<br />
==NFS==<br />
Add the line<br />
192.168.21.3:/home /home nfs soft,retrans=6 0 0<br />
to <code>/etc/fstab</code><br />
<br />
=Complications=<br />
<br />
==Hostnames==<br />
Debian does not pull the hostname from the DHCP Server. Without intervention cloned nodes will keep the hostname stored on the image of the root node. A bug describing the setting of a hostname via DHCP is described [https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/90388 here]<br />
<br />
The solution we will use is to add the file <code>/etc/dhcp3/dhclient-exit-hooks.d/hostname</code> with the contents:<br />
<br />
<source lang="bash"><br />
if [[ -n $new_host_name ]]; then<br />
echo "$new_host_name" > /etc/hostname<br />
/bin/hostname $new_host_name<br />
fi<br />
</source><br />
<br />
The effect of this is to set the hostname of the machine after an interface is configured using dhclient (DHCP Client). Note, the hostname of the machine will be set by the last interface that is configured via DHCP, in the current configuration that will be <code>eth0</code>. If an interface is reconfigured using dhclient, the hostname will be reset to the name belonging to that interface.<br />
<br />
Further, the current hostnames for the second interface on nodes <code>eth1</code> are '''invalid'''. They follow the format hcl??_eth1.ucd.ie, however the '_' character is not permitted in hostnames and attempting to set such a hostname fails.<br />
<br />
==udev and Network Interfaces==<br />
<br />
The udev system attempts to keep network interface names consistent regardless of changing hardware. This may be useful for laptops with wirless cards that a plugged in and out, but it causes problems when trying to install our root node image across all machines in the cluster. A description of the problem can be read [http://www.ducea.com/2008/09/01/remove-debian-udev-persistent-net-rules/ here].<br />
<br />
The solution is to remove the udev rules for persistent network interfaces, and disable the generator script for these rules. On the root cloning node do the following<br />
<br />
#remove the file <code>/etc/udev/rules.d/70-persistent-net.rules</code><br />
#and to the top of the file: <code>/lib/udev/rules.d/75-persistent-net-generator.rules</code>, the following lines:<br />
<source lang="text"># skip generation of persistent network interfaces<br />
ACTION=="*", GOTO="persistent_net_generator_end"</source></div>Rhiggins