HCL - User contributions [en]

BitTorrent (B. Cohen's version)

2012-09-14T12:27:39Z

Kiril:

The modified bittorrent tarball is currently under the MPIBlib repository:

svn co https://hcl.ucd.ie/repos/CPM/trunk/MPIBlib/tests/bittorrent/bittorrent.tar.gz

* Extract this under your home directory
** You might need
mv $HOME/lib/python $HOME/lib/python2.6

* export following variables to tweak the way Python paths are searched:
export PYTHONPATH=/usr/lib/python2.6/:<local-installation-path-of-Python-libs>
* Modify logfile path in /home/kdichev/lib/python2.6/BitTorrent/StorageWrapper.py and create according directory structure
* Create a file of any size consisting of "s" characters only.
* Create torrent file:
btmakemetafile myfile.ext http://<frontend on G5K>:6969/announce
* Stick the torrent file into $HOME/public at the frontend.
** It is then available from within G5K under http://public.lille.grid5000.fr/~kdichev/{torrent file}
* Start the tracker on the frontend:
bttrack --port 6969 --dfile dstate
* Book nodes interactively:
qsub -I -lnodes=...
* Start one client that has the full copy
ssh <one-node> btdownloadheadless --url http://public.lille.grid5000.fr/~kdichev/{torrent file} &

* On all other nodes, launch the client at the same time:
script-per-process.sh
#!/bin/bash
cd /tmp ; btdownloadheadless --display_interval 100 --url http://public.lille.grid5000.fr/~kdichev/{torrent file}
run.sh
mpirun -n 14 --machinefile hostfile $PWD/script-per-process.sh

OpenMPI

2011-12-15T15:10:55Z

Kiril: /* PERUSE */

http://www.open-mpi.org/faq/

== MCA parameter files ==
If you want to permanently use some MCA parameter settings, you can create a file $HOME/.openmpi/mca-params.conf, e.g.:

cat $HOME/.openmpi/mca-params.conf
btl_tcp_if_exclude = lo,eth1

== Running applications on Multiprocessors/Multicores ==
Process can be bound to specific sockets and cores on nodes by choosing right options of mpirun.
* [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect9 Process binding]
* [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect10 Rankfiles]

== Debugging applications on Multiprocessors/Multicores ==

* [http://www.open-mpi.org/faq/?category=debugging#serial-debuggers Serial Debugger (Gdb)]
** 1. Attach to individual MPI processes after they are running. For example, launch your MPI application as normal with mpirun. Then login to the node(s) where your application is running and use the --pid option to gdb to attach to your application.
** 2. Use mpirun to launch xterms (or equivalent) with serial debuggers.
shell$ mpirun -np 4 xterm -e gdb my_mpi_application

* [http://www.open-mpi.org/faq/?category=debugging#parallel-debuggers Parallel Debugers]
** [http://www.open-mpi.org/faq/?category=running#run-with-tv TotalView]
** [http://www.open-mpi.org/faq/?category=running#run-with-ddt DDT]

== PERUSE ==
[[Media:current_peruse_spec.pdf|PERUSE Specification]]

OpenMPI

2011-12-15T15:09:40Z

Kiril: /* PERUSE */

File:Current peruse spec.pdf

2011-12-15T15:08:06Z

Kiril: PERUSE specification

PERUSE specification

2011-12-13T11:42:45Z

Kiril: Created page with "The modified bittorrent tarball is currently under the MPIBlib repository: svn co https://hcl.ucd.ie/repos/CPM/trunk/MPIBlib/tests/bittorrent/bittorrent.tar.gz * Create a file…"

Main Page

2011-12-13T11:23:44Z

Kiril: /* Libraries */

This site is set up for sharing ideas, findings and experience in heterogeneous computing. Please, log in and create new or edit existing pages. How to format wiki-pages read [[Help:Editing|here]].

== HCL software for heterogeneous computing ==
* Extensions for [[MPI]]: [http://hcl.ucd.ie/project/mpC mpC] [http://hcl.ucd.ie/project/HeteroMPI HeteroMPI] [http://hcl.ucd.ie/project/libELC libELC]
* Extensions for [[GridRPC]]: [http://hcl.ucd.ie/project/SmartGridSolve SmartGridSolve] [http://hcl.ucd.ie/project/NI-Connect NI-Connect]
* Computation benchmarking, modeling, dynamic load balancing: [http://hcl.ucd.ie/project/fupermod FuPerMod] [http://hcl.ucd.ie/project/pmm PMM]
* Communication benchmarking, modeling, optimization: [http://hcl.ucd.ie/project/cpm CPM] [http://hcl.ucd.ie/project/mpiblib MPIBlib]

== Heterogeneous mathematical software ==
* [http://hcl.ucd.ie/project/HeteroScaLAPACK HeteroScaLAPACK]
* [http://hcl.ucd.ie/project/Hydropad Hydropad]

== Operating systems ==
* [[Linux]]
* [[Windows]]

== Development tools ==
* [[C/C++]], [[Python]], [[UML]]
* [[Autotools]]
* [[GDB]], [[OProfile]], [[Valgrind]]
* [[Doxygen]]
* [[ChangeLog]], [[Subversion]]
* [[Eclipse]]

== [[Libraries]] ==
* [[GNU C Library]]
* [[MPI]]
* [[STL]], [[Boost]]
* [[GSL]]
* [[BLAS LAPACK ScaLAPACK]]
* [[NLOPT]]
* [[BitTorrent (B. Cohen's version)]]

== Data processing ==
* [[gnuplot]]
* [[Graphviz]]
* [[Octave]], [[R]]
* [[G3DViewer]]

== Paper & Presentation Tools ==
* [[Dia]]
* [[LaTeX]]
* [[JabRef]]

== Hardware ==
* [[HCL cluster]]
* [[Other UCD Resources]]
* [[UTK multicores + GPU]]
* [[Grid5000]]

[[SSH|How to connect to cluster via SSH]]

[[hwloc|How to find information about the hardware]]

== Mathematics ==
* [http://en.wikipedia.org/wiki/Confidence_interval Confidence interval (Statistics)], [http://en.wikipedia.org/wiki/Student's_t-distribution Student's t-distribution] (implemented in [[GSL]])
* [http://en.wikipedia.org/wiki/Linear_regression Linear regression] (implemented in [[GSL]])
* [http://en.wikipedia.org/wiki/Binomial_tree#Binomial_tree Binomial tree] (use [[Graphviz]] to visualize trees)
* [http://en.wikipedia.org/wiki/Spline_interpolation Spline interpolation], [http://en.wikipedia.org/wiki/B-spline Spline approximation] (implemented in [[GSL]])

SSH

2011-01-27T16:46:38Z

Kiril: /* Better than automatically saying "yes" */

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script automates typing "yes" when asked by SSH if a host should be added to known_hosts

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
set timeout 2
spawn ssh $arg1
expect "yes/no" {
send "yes\n"
}
send "exit\n"
send "\r"

You can include it in a bash script to iterate over all nodes doing this:

for i in `uniq hostfile` ; do
./say-yes.exp $i
done

== Better than automatically saying "yes" ==

Remark: It turns out there is a more ellegant way to do this task: using a tool called ''ssh-keyscan''.

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous instead and use "qsub" [[HCL_cluster#Access_and_Security]]

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>
or add the following line to your .ssh/ssh_config file
ForwardX11 yes

SSH

2011-01-27T16:45:39Z

Kiril: /* Automatically saying "yes" */

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script automates typing "yes" when asked by SSH if a host should be added to known_hosts

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
set timeout 2
spawn ssh $arg1
expect "yes/no" {
send "yes\n"
}
send "exit\n"
send "\r"

You can include it in a bash script to iterate over all nodes doing this:

for i in `uniq hostfile` ; do
./say-yes.exp $i
done

== Better than automatically saying "yes" ==

Remark: It turns out there is a more ellegant way to do this task: using a tool called ''ssh-add''.

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous instead and use "qsub" [[HCL_cluster#Access_and_Security]]

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>
or add the following line to your .ssh/ssh_config file
ForwardX11 yes

HCL Cluster Network

2011-01-27T13:23:20Z

Kiril:

Switch Management
The cluster is connected via two Cisco Catalyst 3560G switches.
The ingress bandwidth on any physical port can be configured to any value between
8kb/s and 1Gb/s. The switches are connected to each other via a gigabit sfp
cable.
As the <a href="Specs.html">Cluster Specifications</a>
show, each node has two Network Interfaces, each with its own IP Address. Each
eth0 is connected to switch 1, and each eth1 is connected to switch 2. Which
topology you wish to use will determine which IP address you should use when
referring to each machine. 
 
Switch Access
The switches can be accessed through telnet from any machine
on the cluster network. Type 
telnet 192.168.21.252 for switch1 or telnet 192.168.21.253 for switch2, and
the switch should prompt for a password. For the password, email brett becker.
 

Node Configuration

Here[http://www.linuxfoundation.org/collaborate/workgroups/networking/netem] is an excellent description how to introduce packet delays for outgoing traffic, which is a way to control latency.


 

Switch Configuration
 
For example, we will demonstrate how to limit the bandwidth of the connection
between hcl01 and hcl02 to 100Mbps: 
----------------------------------- 
hcl02 $> telnet 192.168.21.252 
     Trying 192.168.21.252... 
     Connected to 192.168.21.252 (192.168.21.252). 
     Escape character is '^]'.
    User Access Verification
Password: <enter password> 
hclswitch1>enable 
     Password: <enter password> 
hclswitch1#configure terminal 
     Enter configuration commands, one per line. End with CNTL/Z. 
hclswitch1(config)#policy-map example 
hclswitch1(config-pmap)#class ipclass1 
hclswitch1(config-pmap-c)#police 100000000 800000 exceed-action drop 
hclswitch1(config-pmap-c)#exit 
hclswitch1(config-pmap)#exit 
hclswitch1(config)#interface gigabitEthernet0/3 
hclswitch1(config-if)#service-policy input example 
hclswitch1(config-if)#exit 
hclswitch1(config)#interface gigabitEthernet0/4 
hclswitch1(config-if)#service-policy input example 
hclswitch1(config-if)#exit 
hclswitch1#show policy-map 
   Policy Map example 
    Class ipclass1 
     police 100000000 800000 exceed-action drop 
hclswitch1#exit 
     Connection closed by foreign host.
hcl02 $> 
----------------------------------------
In the above example, we telnet to the switch and enter the
password. We then type “enable” and enter the password again. Then
we type “configure terminal” to bring us into config mode. We then
create a new policy-map named “example” We bind this policy-map
to class “ipclass1”. ipclass1 is a class that incorporates all ports
on the switch. PLEASE DO NOT REMOVE THIS CLASS! We then enter the key statement
police 100000000 800000 exceed-action drop. This tells the switch that this
policy-map is to limit the bandwidth to 100Mbps, with a bucket burst of 80000,
and if the rate is exceeded packets are to be dropped. We then exit policy-map
configuration and enter interface gigabitEthernet0/3. This is the port that
is connected to hcl03. We then attach the policy-map “example” to
this interface. We then do the same for port gigabitEthernet0/4 which is connected
to hcl04. The policy-map is then viewed using the show command to ensure that
it is correct. 
This example demonstrates an important fact about the cluster.
hcl01’s eth0 (192.168.21.3) is connected to port gigabitEthernet0/1 on
switch1. Similarly, hclx’s eth0 (192.168.21.x+2) is connected to gigabitEthernet0.x
on switch1. 
Further, hcl01’s eth1 (192.168.21.103) is connected to gigabitEthernet0/1
on switch2. Similarly, hclx’s eth1 (192.168.21.100+x+2) is connected to
gigabitEthernet0/x on switch2. 
The table below shows this comprehensively. 
<div align="center">
<table width="53%" border="1" cellpadding="0">
<tr>
<td width="20%">Machine</td>
<td width="23%">IP Address</td>
<td width="26%">Switch1 Port</td>
<td width="31%">Switch2 Port</td>
</tr>
<tr>
<td>hcl01</td>
<td>192.168.21.3</td>
<td>gigabitEthernet0/1</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl01_eth1</td>
<td>192.168.21.103</td>
<td>N/A</td>
<td>gigabitEthernet0/1</td>
</tr>
<tr>
<td>hcl02</td>
<td>192.168.21.4</td>
<td>gigabitEthernet0/2</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl02_eth1</td>
<td>192.168.21.104</td>
<td>N/A</td>
<td>gigabitEthernet0/2</td>
</tr>
<tr>
<td>hcl03</td>
<td>192.168.21.5</td>
<td>gigabitEthernet0/3</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl03_eth1</td>
<td><div align="left">192.168.21.105</div></td>
<td>N/A</td>
<td>gigabitEthernet0/3</td>
</tr>
<tr>
<td>hcl04</td>
<td>192.168.21.6</td>
<td>gigabitEthernet0/4</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl04_eth1</td>
<td>192.168.21.106</td>
<td>N/A</td>
<td>gigabitEthernet0/4</td>
</tr>
<tr>
<td>hcl05</td>
<td>192.168.21.7</td>
<td>gigabitEthernet0/5</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl05_eth1</td>
<td>192.168.21.107</td>
<td>N/A</td>
<td>gigabitEthernet0/5</td>
</tr>
<tr>
<td>hcl06</td>
<td>192.168.21.8</td>
<td>gigabitEthernet0/6</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl06_eth1</td>
<td>192.168.21.108</td>
<td>N/A</td>
<td>gigabitEthernet0/6</td>
</tr>
<tr>
<td>hcl07</td>
<td>192.168.21.9</td>
<td>gigabitEthernet0/7</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl07_eth1</td>
<td>192.168.21.109</td>
<td>N/A</td>
<td>gigabitEthernet0/7</td>
</tr>
<tr>
<td>hcl08</td>
<td>192.168.21.10</td>
<td>gigabitEthernet0/8</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl08_eth1</td>
<td>192.168.21.110</td>
<td>N/A</td>
<td>gigabitEthernet0/8</td>
</tr>
<tr>
<td>hcl09</td>
<td>192.168.21.11</td>
<td>gigabitEthernet0/9</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl09_eth1</td>
<td>192.168.21.111</td>
<td>N/A</td>
<td>gigabitEthernet0/9</td>
</tr>
<tr>
<td>hcl10</td>
<td>192.168.21.12</td>
<td>gigabitEthernet0/10</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl10_eth1</td>
<td>192.168.21.112</td>
<td>N/A</td>
<td>gigabitEthernet0/10</td>
</tr>
<tr>
<td>hcl11</td>
<td>192.168.21.13</td>
<td>gigabitEthernet0/11</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl11_eth1</td>
<td>192.168.21.113</td>
<td>N/A</td>
<td>gigabitEthernet0/11</td>
</tr>
<tr>
<td>hcl12</td>
<td>192.168.21.14</td>
<td>gigabitEthernet0/12</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl12_eth1</td>
<td>192.168.21.114</td>
<td>N/A</td>
<td>gigabitEthernet0/12</td>
</tr>
<tr>
<td>hcl13</td>
<td>192.168.21.15</td>
<td>gigabitEthernet0/13</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl13_eth1</td>
<td>192.168.21.115</td>
<td>N/A</td>
<td>gigabitEthernet0/13</td>
</tr>
<tr>
<td>hcl14</td>
<td>192.168.21.16</td>
<td>gigabitEthernet0/14</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl14_eth1</td>
<td>192.168.21.116</td>
<td>N/A</td>
<td>gigabitEthernet0/14</td>
</tr>
<tr>
<td>hcl15</td>
<td>192.168.21.17</td>
<td>gigabitEthernet0/15</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl15_eth1</td>
<td>192.168.21.117</td>
<td>N/A</td>
<td>gigabitEthernet0/15</td>
</tr>
<tr>
<td>hcl16</td>
<td>192.168.21.18</td>
<td>gigabitEthernet0/16</td>
<td>N/A</td>
</tr>
<tr>
<td>hcl16_eth1</td>
<td>192.168.21.118</td>
<td>N/A</td>
<td>gigabitEthernet0/16</td>
</tr>
</table>
</div>

 
Limiting the bandwidth between the two switches is similar to
the above example, but on switch1, Interface gigabitEthernet0/25 should be assigned
the desired policy-map. Then on switch2, the same should be done. 
 
Simulating Two Clusters
If you want to simulate two clusters with the cluster, the following
example should be considered.
Say we want Cluster A to be comprised of hcl01 and hcl02, and
Cluster B, to comprised of hcl03 and hcl04. We want hcl01 and hcl02 to “talk”
to each other at 10Mbps, and hcl03 and hcl04 to “talk” at 1Gbps.
Additionally, we want to restrict the link between Clusters A and B to a bandwidth
of 250Mbps. 
To accomplish this we perform the following steps:
1.) Log onto hcl01 and make sure that eth0 is active and eth1
is not. This means that the machine is connected to switch1.
2.) Do the same for hcl02 
3.) Log onto switch1, and create a policy-map to limit the bandwidth
to 10Mbps (10000000bps). Attach this policy-map to gigabitEthernet0/1 and gigabitEthernet0/2.

4.) Log onto hcl03 and perform the following steps:
 hcl03 $> /sbin/ifconfig 
This will list the active network devices. If the user before you cleaned up
after him/her self, only “lo” (the loopback interface) and “eth0”
should be listed. 
hcl03 $> /sbin/ifup eth1 (debian command is sudo /sbin/ifup eth1) 
hcl03 $> /sbin/ifdown eth0 (debian command is sudo /sbin/ifdown eth0)
This connects hcl03 to switch2, as eth1 is connected to switch2.
Then hcl03 is disconnected from switch1, as eth0 is connected to switch1. MAKE
SURE YOU ALWAYS BRING ONE INTERFACE UP BEFORE YOU BRING ANOTHER DOWN. OTHERWISE
THE MACHINE WILL BE ISOLATED WITH NO ACTIVE NETWORK DEVICES. To see what devices
are currently active, use the /sbin/ifconfig command. 
5.) The same should be done for hcl04. 
6.) Since we want these machines to talk at 1Gbps, we should
log onto switch2 and make sure that there are no policy-maps existing. 
7.) Log onto switch1 and create a new policy-map, limit it to
250000000bps and attach it to gigabitEthernet0/25. Do the same for switch2.
Done!
 
The following example shows how to delete a policy-map. YOU
SHOULD ALWAYS REMEMBER TO DELETE YOUR POLICY-MAPS WHEN YOUR JOBS ARE DONE SO
OTHER USER’S JOBS DON’T GET MESSED UP! 
-------------------------------------------------------------- 
hcl0x $> telnet 192.168.21.252 
     Trying 192.168.21.252... 
     Connected to 192.168.21.252 (192.168.21.252). 
     Escape character is '^]'.
     User Access Verification 
Password: <enter password> 
hclswitch1>enable 
Password: 
hclswitch1#show policy-map 
   Policy Map example 
    Class ipclass1 
     police 100000000 800000 exceed-action drop 
hclswitch1#config t 
     Enter configuration commands, one per line. End with CNTL/Z. 
hclswitch1(config)#no policy-map example 
hclswitch1(config)#exit 
hclswitch1#show policy-map 
 
hclswitch1#exit 
     Connection closed by foreign host.
hcl0x $> 
---------------------------------------------------------------

HCL Cluster Network

2011-01-27T13:21:34Z

2010-06-25T22:46:03Z

Kiril: /* Automate the inclusion of hostname to "known_host" */

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script says "yes" when asked if

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
spawn ssh $arg1
expect "yes"
send "yes\r"
send "exit\r"
expect eof

You can include it in a bash script to iterate over all nodes doing this:

cat yes-everywhere.sh
for i in `cat hostfile` ; do
./say-yes.exp $i
done
~
~

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous
instead and use "qsub"

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>

2010-05-11T14:09:08Z

Kiril: /* Some networking issues on HCL cluster (unsolved) */

HCL cluster

2010-05-11T14:06:40Z

Kiril: /* Some networking issues on HCL cluster (unsolved) */

HCL cluster

2010-05-11T14:05:33Z

Kiril: /* Access and Security */

SSH

2010-05-07T11:01:29Z

Kiril: /* Making a cascade of SSH connections easy */

SSH

2010-05-07T10:59:33Z

Kiril: /* Making a cascade of SSH connections easy */