HCL - User contributions [en]

BlueGene/P

2013-03-22T18:01:18Z

Xalid:

BlueGene/P

2013-03-22T18:00:16Z

Xalid:

Some members of HCL have access to Shaheen BlueGene/P at King Abdullah University of Science and Technology ([http://www2.hpc.kaust.edu.sa/documentation/shaheen/]) . In addition, some members has access to BleGene/P at West University of Timisoara, Romania ([http://hpc.uvt.ro/infrastructure/bluegenep/]).

===== Fupermod on Shaheen BlueGene/P =====

In order to compile fupermod on the BG/P the following commands should be run to load some libraries:

#module load bluegene
#module load essl
#module load gsl

Then, configure command can be executed as follows:

/fupermod/configure --with-gsl-dir=/opt/share/math_libraries/gsl/ppc64/IBM --with-blas=essl CFLAGS="-O3 -qarch=450 -qtune=450" --with-essl-dir=/opt/share/ibmmath/essl/4.4/

Somehow, on BG/P autotools didn't see LD_LIBRARY_PATH. Therefore, the following hardcoded path was added into configure.ac.



if test "$with_essl_dir" != ""; then       CPPFLAGS="$CPPFLAGS -I$with_essl_dir/include"        LDFLAGS="$LDFLAGS -L$with_essl_dir/lib '''-L/opt/ibmcmp/xlf/bg/11.1/lib'''" fi 

 

-->

BlueGene/P

2013-03-22T17:59:29Z

Xalid:

Some members of HCL have access to Shaheen BlueGene/P at King Abdullah University of Science and Technology ([http://www2.hpc.kaust.edu.sa/documentation/shaheen/]) . In addition, some members has access to BleGene/P at West University of Timisoara, Romania ([http://hpc.uvt.ro/infrastructure/bluegenep/]).

===== Fupermod on Shaheen BlueGene/P =====

In order to compile fupermod on the BG/P the following commands should be run to load some libraries:

#module load bluegene
#module load essl
#module load gsl

Then, configure command can be executed as follows:

/fupermod/configure --with-gsl-dir=/opt/share/math_libraries/gsl/ppc64/IBM --with-blas=essl CFLAGS="-O3 -qarch=450 -qtune=450" --with-essl-dir=/opt/share/ibmmath/essl/4.4/

Somehow, on BG/P autotools didn't see LD_LIBRARY_PATH. Therefore, the following hardcoded path was added into configure.ac.

BlueGene/P

2013-03-22T17:57:36Z

Xalid:

Some members of HCL have access to Shaheen BlueGene/P at King Abdullah University of Science and Technology ([http://www2.hpc.kaust.edu.sa/documentation/shaheen/]) . In addition, some members has access to BleGene/P at West University of Timisoara, Romania ([http://hpc.uvt.ro/infrastructure/bluegenep/]).

===== Fupermod on Shaheen BlueGene/P =====

In order to compile fupermod on the BG/P the following commands should be run to load some libraries:

#module load bluegene
#module load essl
#module load gsl

Then, configure command can be executed as follows:

/fupermod/configure --with-gsl-dir=/opt/share/math_libraries/gsl/ppc64/IBM --with-blas=essl CFLAGS="-O3 -qarch=450 -qtune=450" --with-essl-dir=/opt/share/ibmmath/essl/4.4/

Somehow, on BG/P autotools didn't see LD_LIBRARY_PATH. Therefore, the following hardcoded path was added into configure.ac.

if test "$with_essl_dir" != ""; then       CPPFLAGS="$CPPFLAGS -I$with_essl_dir/include"        LDFLAGS="$LDFLAGS -L$with_essl_dir/lib '''-L/opt/ibmcmp/xlf/bg/11.1/lib'''" fi

BlueGene/P

2013-03-22T17:56:33Z

Xalid:

Some members of HCL have access to Shaheen BlueGene/P at King Abdullah University of Science and Technology ([http://www2.hpc.kaust.edu.sa/documentation/shaheen/]) . In addition, some members has access to BleGene/P at West University of Timisoara, Romania ([http://hpc.uvt.ro/infrastructure/bluegenep/]).

====== Fupermod on Shaheen BlueGene/P ======

In order to compile fupermod on the BG/P the following commands should be run to load some libraries:

#module load bluegene
#module load essl
#module load gsl

Then, configure command can be executed as follows:

/fupermod/configure --with-gsl-dir=/opt/share/math_libraries/gsl/ppc64/IBM --with-blas=essl CFLAGS="-O3 -qarch=450 -qtune=450" --with-essl-dir=/opt/share/ibmmath/essl/4.4/

Somehow, on BG/P autotools didn't see LD_LIBRARY_PATH. Therefore, the following hardcoded path was added into configure.ac.

if test "$with_essl_dir" != ""; then       CPPFLAGS="$CPPFLAGS -I$with_essl_dir/include"        LDFLAGS="$LDFLAGS -L$with_essl_dir/lib '''-L/opt/ibmcmp/xlf/bg/11.1/lib'''" fi

BlueGene/P

2013-03-19T15:02:10Z

Xalid:

BlueGene/P

2013-03-19T15:00:15Z

Xalid: Created page with "Some members of HCL have access to Shaheen BlueGene/P at King Abdullah University of Science and Technology ([http://www2.hpc.kaust.edu.sa/documentation/shaheen/]) . In addition,…"

Main Page

2013-03-19T14:48:39Z

Xalid: /* Hardware */

This site is set up for sharing ideas, findings and experience in heterogeneous computing. Please, log in and create new or edit existing pages. How to format wiki-pages read [[Help:Editing|here]].

== HCL software for heterogeneous computing ==
* Extensions for [[MPI]]: [http://hcl.ucd.ie/project/mpC mpC] [http://hcl.ucd.ie/project/HeteroMPI HeteroMPI] [http://hcl.ucd.ie/project/libELC libELC]
* Extensions for [http://en.wikipedia.org/wiki/GridRPC GridRPC]: [http://hcl.ucd.ie/project/SmartGridSolve SmartGridSolve] [http://hcl.ucd.ie/project/NI-Connect NI-Connect]
* Computation benchmarking, modeling, dynamic load balancing: [http://hcl.ucd.ie/project/fupermod FuPerMod] [http://hcl.ucd.ie/project/pmm PMM]
* Communication benchmarking, modeling, optimization: [http://hcl.ucd.ie/project/cpm CPM] [http://hcl.ucd.ie/project/mpiblib MPIBlib]

== Heterogeneous mathematical software ==
* [http://hcl.ucd.ie/project/HeteroScaLAPACK HeteroScaLAPACK]
* [http://hcl.ucd.ie/project/Hydropad Hydropad]

== Operating systems ==
* [[Linux]]
* [[Windows]]

== Development tools ==
* [[C/C++]], [[Python]], [[UML]], [[FORTRAN]]
* [[Autotools]]
* [[GDB]], [[OProfile]], [[Valgrind]]
* [[Doxygen]]
* [[ChangeLog]], [[Subversion]]
* [[Eclipse]]
* [[Bash Scripts]]

== [[Libraries]] ==
* [[GNU C Library]]
* [[MPI]]
* [[STL]], [[Boost]]
* [[GSL]]
* [[BLAS LAPACK ScaLAPACK]]
* [[NLOPT]]
* [[BitTorrent (B. Cohen's version)]]
* [[CUDA SDK]]

== Data processing ==
* [[gnuplot]], [[pgfplot]], [[matplotlib]]
* [[Graphviz]]
* [[Octave]], [[R]]
* [[G3DViewer]]

== Paper & Presentation Tools ==
* [[Dia]], [[PGF/Tikz]], [[pgfplot]]
* [[LaTeX]], [[Beamer]]
* [[BibTeX]], [[JabRef]]

== Hardware ==
* [[HCL cluster]]
* [[Other UCD Resources]]
* [[UTK multicores + GPU]]
* [[Grid5000]]
* [[BlueGene/P]]
* [[Desktop Backup]]

[[SSH|How to connect to cluster via SSH]]

[[hwloc|How to find information about the hardware]]

== Mathematics ==
* [http://en.wikipedia.org/wiki/Confidence_interval Confidence interval (Statistics)], [http://en.wikipedia.org/wiki/Student's_t-distribution Student's t-distribution] (implemented in [[GSL]])
* [http://en.wikipedia.org/wiki/Linear_regression Linear regression] (implemented in [[GSL]])
* [http://en.wikipedia.org/wiki/Binomial_tree#Binomial_tree Binomial tree] (use [[Graphviz]] to visualize trees)
* [http://en.wikipedia.org/wiki/Spline_interpolation Spline interpolation], [http://en.wikipedia.org/wiki/B-spline Spline approximation] (implemented in [[GSL]])

Grid5000

2012-08-24T15:41:54Z

Xalid: /* Login, job submission, deployment of image */

https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home

[https://www.grid5000.fr/mediawiki/index.php/Grid5000:UserCharter USAGE POLICY]  - Very important, after booking nodes (oarsub ...) run the command: <source lang="">outofchart</source> This will check that you haven't booked too many resources and therefore get in trouble with grid5000 admin.

== Login, job submission, deployment of image ==

*Select sites and clusters for experiments, using information on the [https://www.grid5000.fr/mediawiki/index.php/Grid5000:Network#Grid.275000_Sites Grid5000 network] and the [https://www.grid5000.fr/mediawiki/index.php/Status Status page]
*Access is provided via access nodes '''access.SITE.grid5000.fr''' marked [https://www.grid5000.fr/mediawiki/index.php/External_access here] as ''accessible from '''everywhere''' via ssh with '''keyboard-interactive''' authentication method''. As soon as you are on one of the sites, you can directly ssh frontend node of any other site:

<source lang="bash">
access_$ ssh frontend.SITE2
</source>

*There is no access to Internet from computing nodes (external IPs should be registered on proxy), therefore, download/update your stuff at the access nodes. Several revision control clients are available.
*Each site has a separate NFS, therefore, to run an application on several sites at once, you need to copy it '''scp, sftp, rsync''' between access or frontend nodes.
*Jobs are run from the frondend nodes, using a [http://en.wikipedia.org/wiki/OpenPBS PBS]-like system [https://www.grid5000.fr/mediawiki/index.php/Cluster_experiment-OAR2 OAR]. Basic commands:
**'''oarstat''' - queue status
**'''oarsub''' - job submission
**'''oardel''' - job removal

Interactive job on deployed images: <source lang="bash">
fontend_$ oarsub -I -t deploy -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source> Batch job on installed images: <source lang="bash">
fontend_$ oarsub BATCH_FILE -t allow_classic_ssh -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source>
Specifying cluster name to reserve: <source lang="bash">
oarsub -r 'YYYY-MM-dd HH:mm:ss' -l nodes=2,walltime=1 -p "cluster='Genepi'"
</source> If the resources are available two nodes from the cluster "Genepi" will be reserved for the specified time.

*The image to deploy can be created and loaded with help of a [http://wiki.systemimager.org/index.php/Main_Page Systemimager]-like system [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2 Kadeploy]. Creating: [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2#Tune_an_environment_to_build_another_one:_customize_authentification_parameters described here]

Loading: <source lang="bash">
fontend_$ kadeploy3 -a PATH_TO_PRIVATE_IMAGE_DESC -f $OAR_FILE_NODES
</source> A Linux distribution lenny-x64-nfs-2.1 with mc, subversion, autotools, doxygen, MPICH2, GSL, Boost, R, gnuplot, graphviz, X11, evince is available at Orsay /home/nancy/alastovetsky/grid5000.

== Compiling and running MPI applications ==

*Compilation should be done on one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)
*Running MPI applications is described [https://www.grid5000.fr/mediawiki/index.php/Run_MPI_On_Grid%275000 here]
**mpirun/mpiexec should be run from one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)

== Setting up new deploy image ==

List available images

kaenv3 -l

Then book node and launch:

oarsub -I -t deploy -l nodes=1,walltime=12
kadeploy3 -e squeeze-x64-big -f $OAR_FILE_NODES -k
ssh root@`head -n 1 $OAR_NODEFILE`

default password: grid5000

edit /etc/apt/sources.list

apt-get update
apt-get upgrade

apt-get install libtool autoconf automake mc colorgcc ctags libboost-serialization-dev libboost-graph-dev
libatlas-base-dev gfortran vim gdb valgrind screen subversion iperf bc gsl-bin libgsl0-dev

Possibly also install (for using extrae):

libxml2-dev binutils-dev libunwind7-dev

 Compiled for sources by us:

*<strike>gsl-1.14 (download: ftp://ftp.gnu.org/gnu/gsl/) </strike> ''Now with squeeze it is in repository.''

<strike>./configure && make && make install</strike>

*mpich2 (download: http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downloads)

./configure --enable-shared --enable-sharedlibs=gcc --with-pm=mpd
make && make install

Mpich2 installed to:

Installing MPE2 include files to /usr/local/include
Installing MPE2 libraries to /usr/local/lib
Installing MPE2 utility programs to /usr/local/bin
Installing MPE2 configuration files to /usr/local/etc
Installing MPE2 system utility programs to /usr/local/sbin
Installing MPE2 man to /usr/local/share/man
Installing MPE2 html to /usr/local/share/doc/
Installed MPE2 in /usr/local

*hwloc (and lstopo) (download: http://www.open-mpi.org/software/hwloc/v1.2/)

compile from sources. To get xml support install libxml2-dev and pkg-config

apt-get install libxml2-dev pkg-config
tar -xzvf hwloc-1.1.1.tar.gz
cd hwloc-1.1.1
./configure && make && make install

Change root password.

rm sources from root dir.

Edit the "message of the day"

vi /etc/motd.tail

echo 90 > /proc/sys/vm/overcommit_ratio
echo 2 > /proc/sys/vm/overcommit_memory
date >> release

Cleanup

apt-get clean
rm /etc/udev/rules.d/*-persistent-net.rules

Make image

ssh root@'''node''' tgz-g5k > $HOME/grid5000/'''imagename'''.tgz

make appropriate .env file.

kaenv3 -p lenny-x64-nfs -u deploy > lenny-x64-custom-2.3.env

 

== GotoBLAS2 ==

When compiling gotoblas on a node without direct internet access get this error: <source lang="">wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
--2011-05-19 03:11:03-- http://www.netlib.org/lapack/lapack-3.1.1.tgz
Resolving www.netlib.org... 160.36.58.108
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.

--2011-05-19 03:14:13-- (try: 2) http://www.netlib.org/lapack/lapack-3.1.1.tgz
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.
...</source>

Fix by downloading http://www.netlib.org/lapack/lapack-3.1.1.tgz to the GotoBLAS2 source directory and editing this line in the Makefile

184c184
< -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
---
> # -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz

 GotoBLAS needs to be compiled individualy for each unique machine - ie each cluster. Add the following to .bashrc

export CLUSTER=`hostname |sed 's/$[a-z]*$.*/\1/'`
LD_LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LIBRARY_PATH

Run the following script once on each cluster:

<source lang="bash">#! /bin/bash
echo "Compiling gotoblas for cluster: $CLUSTER"
cd $HOME/src
if [ ! -d "$CLUSTER" ]; then
mkdir $CLUSTER
fi
cd $CLUSTER
tar -xzf ../Goto*.tar.gz
cd Goto*
make &> m.log

if [ ! -d "$HOME/lib/$CLUSTER" ]; then
mkdir $HOME/lib/$CLUSTER
fi

cp libgoto2.so $HOME/lib/$CLUSTER

echo results
ls -d $HOME/src/$CLUSTER
ls $HOME/src/$CLUSTER

ls -d $HOME/lib/$CLUSTER
ls $HOME/lib/$CLUSTER</source>

note: for newer processors this may fail. If it is a NEHALEM processor try:

make clean
make TARGET=NEHALEM

== Paging and the OOM-Killer ==

When doing exhaustion of available memory experiments, problems can occur with over-commit. See [[HCL cluster#Paging_and_the_OOM-Killer]] for more detail.

== Example of experiment setup across several sites ==

Sources of all files mentioned below is available at: [[Grid5000:sources]].

Pick one head node as the main head node (I use grenoble, but any will do). Setup sources

cd dave/fupermod-1.1.0
make clean
./configure --with-cblas=goto --prefix=/usr/local/

Reserve 2 nodes from all clusters on a 3 cluster site:

oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=3/nodes=2,walltime=11:59:00

Automate with:

for a in 2 3 4; do for i in `cat sites.$a`; do echo $a $i; ssh $i oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=$a/nodes=2,walltime=11:59:00; done; done

Then on each site, where xxx is site name:

kadeploy3 -a $HOME/grid5000/lenny-dave.env -f $OAR_NODE_FILE --output-ok-nodes deployed.xxx

Gather deployed files to a head node:

for i in `cat ~/sites `; do echo $i; scp $i:deployed* . ; done
cat deployed.* > deployed.all

Copy cluster specific libs to each deployed node /usr/local/lib dir with script

copy_local_libs.sh deployed.all

Copy source files to root dir of each deployed node. Then make install each (node ssh -f does this in parallel)

for i in `cat ~/deployed.all`; do echo $i; rsync -aP ~/dave/fupermod-1.1.0 root@$i: ; done
for i in `cat ~/deployed.all`; do echo $i; ssh -f root@$i "cd fupermod-1.1.0 ; make all install" ; done

ssh to the first node

ssh `head -n1 deployed.all`
n=$(cat deployed.all |wc -l)
mpdboot --totalnum=$n --file=$HOME/deployed.all
mpdtrace

cd dave/data/
mpirun -n $n /usr/local/bin/partitioner -l /usr/local/lib/libmxm_col.so -a0 -D10000 -o N=100

Cleanup after:

for i in `cat ~/sites `; do echo $i; ssh $i rm deployed.* ; done

== Check network speed ==

apt-get install iperf

== Choose which network interface to use ==

mpirun --mca btl self,openib ...

or

mpirun --mca btl self,tcp ...

Grid5000

2012-08-24T15:41:03Z

Xalid: /* Login, job submission, deployment of image */

https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home

[https://www.grid5000.fr/mediawiki/index.php/Grid5000:UserCharter USAGE POLICY]  - Very important, after booking nodes (oarsub ...) run the command: <source lang="">outofchart</source> This will check that you haven't booked too many resources and therefore get in trouble with grid5000 admin.

== Login, job submission, deployment of image ==

*Select sites and clusters for experiments, using information on the [https://www.grid5000.fr/mediawiki/index.php/Grid5000:Network#Grid.275000_Sites Grid5000 network] and the [https://www.grid5000.fr/mediawiki/index.php/Status Status page]
*Access is provided via access nodes '''access.SITE.grid5000.fr''' marked [https://www.grid5000.fr/mediawiki/index.php/External_access here] as ''accessible from '''everywhere''' via ssh with '''keyboard-interactive''' authentication method''. As soon as you are on one of the sites, you can directly ssh frontend node of any other site:

<source lang="bash">
access_$ ssh frontend.SITE2
</source>

*There is no access to Internet from computing nodes (external IPs should be registered on proxy), therefore, download/update your stuff at the access nodes. Several revision control clients are available.
*Each site has a separate NFS, therefore, to run an application on several sites at once, you need to copy it '''scp, sftp, rsync''' between access or frontend nodes.
*Jobs are run from the frondend nodes, using a [http://en.wikipedia.org/wiki/OpenPBS PBS]-like system [https://www.grid5000.fr/mediawiki/index.php/Cluster_experiment-OAR2 OAR]. Basic commands:
**'''oarstat''' - queue status
**'''oarsub''' - job submission
**'''oardel''' - job removal

Interactive job on deployed images: <source lang="bash">
fontend_$ oarsub -I -t deploy -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source> Batch job on installed images: <source lang="bash">
fontend_$ oarsub BATCH_FILE -t allow_classic_ssh -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source>
Specifying cluster name to reserve: <source lang="bash">
oarsub -r '2012-08-24 19:30:00' -l nodes=2,walltime=1 -p "cluster='Genepi'"
</source> If the resources are available two nodes from the cluster "Genepi" will be reserved for the specified time.

*The image to deploy can be created and loaded with help of a [http://wiki.systemimager.org/index.php/Main_Page Systemimager]-like system [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2 Kadeploy]. Creating: [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2#Tune_an_environment_to_build_another_one:_customize_authentification_parameters described here]

Loading: <source lang="bash">
fontend_$ kadeploy3 -a PATH_TO_PRIVATE_IMAGE_DESC -f $OAR_FILE_NODES
</source> A Linux distribution lenny-x64-nfs-2.1 with mc, subversion, autotools, doxygen, MPICH2, GSL, Boost, R, gnuplot, graphviz, X11, evince is available at Orsay /home/nancy/alastovetsky/grid5000.

== Compiling and running MPI applications ==

*Compilation should be done on one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)
*Running MPI applications is described [https://www.grid5000.fr/mediawiki/index.php/Run_MPI_On_Grid%275000 here]
**mpirun/mpiexec should be run from one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)

== Setting up new deploy image ==

List available images

kaenv3 -l

Then book node and launch:

oarsub -I -t deploy -l nodes=1,walltime=12
kadeploy3 -e squeeze-x64-big -f $OAR_FILE_NODES -k
ssh root@`head -n 1 $OAR_NODEFILE`

default password: grid5000

edit /etc/apt/sources.list

apt-get update
apt-get upgrade

apt-get install libtool autoconf automake mc colorgcc ctags libboost-serialization-dev libboost-graph-dev
libatlas-base-dev gfortran vim gdb valgrind screen subversion iperf bc gsl-bin libgsl0-dev

Possibly also install (for using extrae):

libxml2-dev binutils-dev libunwind7-dev

 Compiled for sources by us:

*<strike>gsl-1.14 (download: ftp://ftp.gnu.org/gnu/gsl/) </strike> ''Now with squeeze it is in repository.''

<strike>./configure && make && make install</strike>

*mpich2 (download: http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downloads)

./configure --enable-shared --enable-sharedlibs=gcc --with-pm=mpd
make && make install

Mpich2 installed to:

Installing MPE2 include files to /usr/local/include
Installing MPE2 libraries to /usr/local/lib
Installing MPE2 utility programs to /usr/local/bin
Installing MPE2 configuration files to /usr/local/etc
Installing MPE2 system utility programs to /usr/local/sbin
Installing MPE2 man to /usr/local/share/man
Installing MPE2 html to /usr/local/share/doc/
Installed MPE2 in /usr/local

*hwloc (and lstopo) (download: http://www.open-mpi.org/software/hwloc/v1.2/)

compile from sources. To get xml support install libxml2-dev and pkg-config

apt-get install libxml2-dev pkg-config
tar -xzvf hwloc-1.1.1.tar.gz
cd hwloc-1.1.1
./configure && make && make install

Change root password.

rm sources from root dir.

Edit the "message of the day"

vi /etc/motd.tail

echo 90 > /proc/sys/vm/overcommit_ratio
echo 2 > /proc/sys/vm/overcommit_memory
date >> release

Cleanup

apt-get clean
rm /etc/udev/rules.d/*-persistent-net.rules

Make image

ssh root@'''node''' tgz-g5k > $HOME/grid5000/'''imagename'''.tgz

make appropriate .env file.

kaenv3 -p lenny-x64-nfs -u deploy > lenny-x64-custom-2.3.env

 

== GotoBLAS2 ==

When compiling gotoblas on a node without direct internet access get this error: <source lang="">wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
--2011-05-19 03:11:03-- http://www.netlib.org/lapack/lapack-3.1.1.tgz
Resolving www.netlib.org... 160.36.58.108
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.

--2011-05-19 03:14:13-- (try: 2) http://www.netlib.org/lapack/lapack-3.1.1.tgz
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.
...</source>

Fix by downloading http://www.netlib.org/lapack/lapack-3.1.1.tgz to the GotoBLAS2 source directory and editing this line in the Makefile

184c184
< -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
---
> # -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz

 GotoBLAS needs to be compiled individualy for each unique machine - ie each cluster. Add the following to .bashrc

export CLUSTER=`hostname |sed 's/$[a-z]*$.*/\1/'`
LD_LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LIBRARY_PATH

Run the following script once on each cluster:

<source lang="bash">#! /bin/bash
echo "Compiling gotoblas for cluster: $CLUSTER"
cd $HOME/src
if [ ! -d "$CLUSTER" ]; then
mkdir $CLUSTER
fi
cd $CLUSTER
tar -xzf ../Goto*.tar.gz
cd Goto*
make &> m.log

if [ ! -d "$HOME/lib/$CLUSTER" ]; then
mkdir $HOME/lib/$CLUSTER
fi

cp libgoto2.so $HOME/lib/$CLUSTER

echo results
ls -d $HOME/src/$CLUSTER
ls $HOME/src/$CLUSTER

ls -d $HOME/lib/$CLUSTER
ls $HOME/lib/$CLUSTER</source>

note: for newer processors this may fail. If it is a NEHALEM processor try:

make clean
make TARGET=NEHALEM

== Paging and the OOM-Killer ==

When doing exhaustion of available memory experiments, problems can occur with over-commit. See [[HCL cluster#Paging_and_the_OOM-Killer]] for more detail.

== Example of experiment setup across several sites ==

Sources of all files mentioned below is available at: [[Grid5000:sources]].

Pick one head node as the main head node (I use grenoble, but any will do). Setup sources

cd dave/fupermod-1.1.0
make clean
./configure --with-cblas=goto --prefix=/usr/local/

Reserve 2 nodes from all clusters on a 3 cluster site:

oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=3/nodes=2,walltime=11:59:00

Automate with:

for a in 2 3 4; do for i in `cat sites.$a`; do echo $a $i; ssh $i oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=$a/nodes=2,walltime=11:59:00; done; done

Then on each site, where xxx is site name:

kadeploy3 -a $HOME/grid5000/lenny-dave.env -f $OAR_NODE_FILE --output-ok-nodes deployed.xxx

Gather deployed files to a head node:

for i in `cat ~/sites `; do echo $i; scp $i:deployed* . ; done
cat deployed.* > deployed.all

Copy cluster specific libs to each deployed node /usr/local/lib dir with script

copy_local_libs.sh deployed.all

Copy source files to root dir of each deployed node. Then make install each (node ssh -f does this in parallel)

for i in `cat ~/deployed.all`; do echo $i; rsync -aP ~/dave/fupermod-1.1.0 root@$i: ; done
for i in `cat ~/deployed.all`; do echo $i; ssh -f root@$i "cd fupermod-1.1.0 ; make all install" ; done

ssh to the first node

ssh `head -n1 deployed.all`
n=$(cat deployed.all |wc -l)
mpdboot --totalnum=$n --file=$HOME/deployed.all
mpdtrace

cd dave/data/
mpirun -n $n /usr/local/bin/partitioner -l /usr/local/lib/libmxm_col.so -a0 -D10000 -o N=100

Cleanup after:

for i in `cat ~/sites `; do echo $i; ssh $i rm deployed.* ; done

== Check network speed ==

apt-get install iperf

== Choose which network interface to use ==

mpirun --mca btl self,openib ...

or

mpirun --mca btl self,tcp ...

Grid5000

2012-08-24T15:38:28Z

Xalid: /* Login, job submission, deployment of image */

https://www.grid5000.fr/mediawiki/index.php/Grid5000:Home

[https://www.grid5000.fr/mediawiki/index.php/Grid5000:UserCharter USAGE POLICY]  - Very important, after booking nodes (oarsub ...) run the command: <source lang="">outofchart</source> This will check that you haven't booked too many resources and therefore get in trouble with grid5000 admin.

== Login, job submission, deployment of image ==

*Select sites and clusters for experiments, using information on the [https://www.grid5000.fr/mediawiki/index.php/Grid5000:Network#Grid.275000_Sites Grid5000 network] and the [https://www.grid5000.fr/mediawiki/index.php/Status Status page]
*Access is provided via access nodes '''access.SITE.grid5000.fr''' marked [https://www.grid5000.fr/mediawiki/index.php/External_access here] as ''accessible from '''everywhere''' via ssh with '''keyboard-interactive''' authentication method''. As soon as you are on one of the sites, you can directly ssh frontend node of any other site:

<source lang="bash">
access_$ ssh frontend.SITE2
</source>

*There is no access to Internet from computing nodes (external IPs should be registered on proxy), therefore, download/update your stuff at the access nodes. Several revision control clients are available.
*Each site has a separate NFS, therefore, to run an application on several sites at once, you need to copy it '''scp, sftp, rsync''' between access or frontend nodes.
*Jobs are run from the frondend nodes, using a [http://en.wikipedia.org/wiki/OpenPBS PBS]-like system [https://www.grid5000.fr/mediawiki/index.php/Cluster_experiment-OAR2 OAR]. Basic commands:
**'''oarstat''' - queue status
**'''oarsub''' - job submission
**'''oardel''' - job removal

Interactive job on deployed images: <source lang="bash">
fontend_$ oarsub -I -t deploy -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source> Batch job on installed images: <source lang="bash">
fontend_$ oarsub BATCH_FILE -t allow_classic_ssh -l [/cluster=N/]nodes=N,walltime=HH[:MM[:SS]] [-p 'PROPERTY="VALUE"']
</source>
Specifying cluster name to reserve: <source lang="bash">
oarsub -r '2012-08-24 19:30:00' -l nodes=16,walltime=12 -p "cluster='Genepi'"
</source>

*The image to deploy can be created and loaded with help of a [http://wiki.systemimager.org/index.php/Main_Page Systemimager]-like system [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2 Kadeploy]. Creating: [https://www.grid5000.fr/mediawiki/index.php/Deploy_environment-OAR2#Tune_an_environment_to_build_another_one:_customize_authentification_parameters described here]

Loading: <source lang="bash">
fontend_$ kadeploy3 -a PATH_TO_PRIVATE_IMAGE_DESC -f $OAR_FILE_NODES
</source> A Linux distribution lenny-x64-nfs-2.1 with mc, subversion, autotools, doxygen, MPICH2, GSL, Boost, R, gnuplot, graphviz, X11, evince is available at Orsay /home/nancy/alastovetsky/grid5000.

== Compiling and running MPI applications ==

*Compilation should be done on one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)
*Running MPI applications is described [https://www.grid5000.fr/mediawiki/index.php/Run_MPI_On_Grid%275000 here]
**mpirun/mpiexec should be run from one of the reserved nodes (e.g. ssh `head -n 1 $OAR_NODEFILE`)

== Setting up new deploy image ==

List available images

kaenv3 -l

Then book node and launch:

oarsub -I -t deploy -l nodes=1,walltime=12
kadeploy3 -e squeeze-x64-big -f $OAR_FILE_NODES -k
ssh root@`head -n 1 $OAR_NODEFILE`

default password: grid5000

edit /etc/apt/sources.list

apt-get update
apt-get upgrade

apt-get install libtool autoconf automake mc colorgcc ctags libboost-serialization-dev libboost-graph-dev
libatlas-base-dev gfortran vim gdb valgrind screen subversion iperf bc gsl-bin libgsl0-dev

Possibly also install (for using extrae):

libxml2-dev binutils-dev libunwind7-dev

 Compiled for sources by us:

*<strike>gsl-1.14 (download: ftp://ftp.gnu.org/gnu/gsl/) </strike> ''Now with squeeze it is in repository.''

<strike>./configure && make && make install</strike>

*mpich2 (download: http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downloads)

./configure --enable-shared --enable-sharedlibs=gcc --with-pm=mpd
make && make install

Mpich2 installed to:

Installing MPE2 include files to /usr/local/include
Installing MPE2 libraries to /usr/local/lib
Installing MPE2 utility programs to /usr/local/bin
Installing MPE2 configuration files to /usr/local/etc
Installing MPE2 system utility programs to /usr/local/sbin
Installing MPE2 man to /usr/local/share/man
Installing MPE2 html to /usr/local/share/doc/
Installed MPE2 in /usr/local

*hwloc (and lstopo) (download: http://www.open-mpi.org/software/hwloc/v1.2/)

compile from sources. To get xml support install libxml2-dev and pkg-config

apt-get install libxml2-dev pkg-config
tar -xzvf hwloc-1.1.1.tar.gz
cd hwloc-1.1.1
./configure && make && make install

Change root password.

rm sources from root dir.

Edit the "message of the day"

vi /etc/motd.tail

echo 90 > /proc/sys/vm/overcommit_ratio
echo 2 > /proc/sys/vm/overcommit_memory
date >> release

Cleanup

apt-get clean
rm /etc/udev/rules.d/*-persistent-net.rules

Make image

ssh root@'''node''' tgz-g5k > $HOME/grid5000/'''imagename'''.tgz

make appropriate .env file.

kaenv3 -p lenny-x64-nfs -u deploy > lenny-x64-custom-2.3.env

 

== GotoBLAS2 ==

When compiling gotoblas on a node without direct internet access get this error: <source lang="">wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
--2011-05-19 03:11:03-- http://www.netlib.org/lapack/lapack-3.1.1.tgz
Resolving www.netlib.org... 160.36.58.108
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.

--2011-05-19 03:14:13-- (try: 2) http://www.netlib.org/lapack/lapack-3.1.1.tgz
Connecting to www.netlib.org|160.36.58.108|:80... failed: Connection timed out.
Retrying.
...</source>

Fix by downloading http://www.netlib.org/lapack/lapack-3.1.1.tgz to the GotoBLAS2 source directory and editing this line in the Makefile

184c184
< -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz
---
> # -wget http://www.netlib.org/lapack/lapack-3.1.1.tgz

 GotoBLAS needs to be compiled individualy for each unique machine - ie each cluster. Add the following to .bashrc

export CLUSTER=`hostname |sed 's/$[a-z]*$.*/\1/'`
LD_LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LD_LIBRARY_PATH
export LIBRARY_PATH=$HOME/lib/$CLUSTER:$HOME/lib:/usr/local/lib:$LIBRARY_PATH

Run the following script once on each cluster:

<source lang="bash">#! /bin/bash
echo "Compiling gotoblas for cluster: $CLUSTER"
cd $HOME/src
if [ ! -d "$CLUSTER" ]; then
mkdir $CLUSTER
fi
cd $CLUSTER
tar -xzf ../Goto*.tar.gz
cd Goto*
make &> m.log

if [ ! -d "$HOME/lib/$CLUSTER" ]; then
mkdir $HOME/lib/$CLUSTER
fi

cp libgoto2.so $HOME/lib/$CLUSTER

echo results
ls -d $HOME/src/$CLUSTER
ls $HOME/src/$CLUSTER

ls -d $HOME/lib/$CLUSTER
ls $HOME/lib/$CLUSTER</source>

note: for newer processors this may fail. If it is a NEHALEM processor try:

make clean
make TARGET=NEHALEM

== Paging and the OOM-Killer ==

When doing exhaustion of available memory experiments, problems can occur with over-commit. See [[HCL cluster#Paging_and_the_OOM-Killer]] for more detail.

== Example of experiment setup across several sites ==

Sources of all files mentioned below is available at: [[Grid5000:sources]].

Pick one head node as the main head node (I use grenoble, but any will do). Setup sources

cd dave/fupermod-1.1.0
make clean
./configure --with-cblas=goto --prefix=/usr/local/

Reserve 2 nodes from all clusters on a 3 cluster site:

oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=3/nodes=2,walltime=11:59:00

Automate with:

for a in 2 3 4; do for i in `cat sites.$a`; do echo $a $i; ssh $i oarsub -r "2011-07-25 11:01:01" -t deploy -l cluster=$a/nodes=2,walltime=11:59:00; done; done

Then on each site, where xxx is site name:

kadeploy3 -a $HOME/grid5000/lenny-dave.env -f $OAR_NODE_FILE --output-ok-nodes deployed.xxx

Gather deployed files to a head node:

for i in `cat ~/sites `; do echo $i; scp $i:deployed* . ; done
cat deployed.* > deployed.all

Copy cluster specific libs to each deployed node /usr/local/lib dir with script

copy_local_libs.sh deployed.all

Copy source files to root dir of each deployed node. Then make install each (node ssh -f does this in parallel)

for i in `cat ~/deployed.all`; do echo $i; rsync -aP ~/dave/fupermod-1.1.0 root@$i: ; done
for i in `cat ~/deployed.all`; do echo $i; ssh -f root@$i "cd fupermod-1.1.0 ; make all install" ; done

ssh to the first node

ssh `head -n1 deployed.all`
n=$(cat deployed.all |wc -l)
mpdboot --totalnum=$n --file=$HOME/deployed.all
mpdtrace

cd dave/data/
mpirun -n $n /usr/local/bin/partitioner -l /usr/local/lib/libmxm_col.so -a0 -D10000 -o N=100

Cleanup after:

for i in `cat ~/sites `; do echo $i; ssh $i rm deployed.* ; done

== Check network speed ==

apt-get install iperf

== Choose which network interface to use ==

mpirun --mca btl self,openib ...

or

mpirun --mca btl self,tcp ...

OpenMPI

2012-08-22T10:45:46Z

Xalid: /* MCA parameter files */

http://www.open-mpi.org/faq/

== MCA parameter files ==
If you want to permanently use some MCA parameter settings, you can create a file $HOME/.openmpi/mca-params.conf, e.g.:

cat $HOME/.openmpi/mca-params.conf
btl_tcp_if_exclude = lo,eth1

== Handling SSH key issues ==

This trick avoids a confirmation message asking "yes" when asked by SSH if a host should be added to known_hosts:

ssh -q -o StrictHostKeyChecking=no

So with OpenMPI it can be used as

mpirun --mca plm_rsh_agent "ssh -q -o StrictHostKeyChecking=no"

== Running applications on Multiprocessors/Multicores ==
Process can be bound to specific sockets and cores on nodes by choosing right options of mpirun.
* [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect9 Process binding]
* [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect10 Rankfile]

== PERUSE ==
[[Media:current_peruse_spec.pdf|PERUSE Specification]]

SSH

2012-08-22T10:43:29Z

Xalid: /* The best way is saying no "YES" */

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script automates typing "yes" when asked by SSH if a host should be added to known_hosts

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
set timeout 2
spawn ssh $arg1
expect "yes/no" {
send "yes\n"
}
send "exit\n"
send "\r"

You can include it in a bash script to iterate over all nodes doing this:

for i in `uniq hostfile` ; do
./say-yes.exp $i
done

== Better than automatically saying "yes" ==

Remark: It turns out there is a more ellegant way to do this task: using a tool called ''ssh-keyscan''.

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous instead and use "qsub" [[HCL_cluster#Access_and_Security]]

== The best way is saying no "YES" ==

This trick avoids a confirmation message asking "yes" when asked by SSH if a host should be added to known_hosts:

ssh -q -o StrictHostKeyChecking=no

So with OpenMPI it can be used as

mpirun --mca plm_rsh_agent "ssh -q -o StrictHostKeyChecking=no"

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>
or add the following line to your .ssh/ssh_config file
ForwardX11 yes

SSH

2012-08-22T10:43:04Z

Xalid: /* The best way is saying no "YES" */

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script automates typing "yes" when asked by SSH if a host should be added to known_hosts

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
set timeout 2
spawn ssh $arg1
expect "yes/no" {
send "yes\n"
}
send "exit\n"
send "\r"

You can include it in a bash script to iterate over all nodes doing this:

for i in `uniq hostfile` ; do
./say-yes.exp $i
done

== Better than automatically saying "yes" ==

Remark: It turns out there is a more ellegant way to do this task: using a tool called ''ssh-keyscan''.

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous instead and use "qsub" [[HCL_cluster#Access_and_Security]]

== The best way is saying no "YES" ==

This ssh trick avoids a confirmation message asking "yes" when asked by SSH if a host should be added to known_hosts:

ssh -q -o StrictHostKeyChecking=no

So with OpenMPI it can be used as

mpirun --mca plm_rsh_agent "ssh -q -o StrictHostKeyChecking=no"

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>
or add the following line to your .ssh/ssh_config file
ForwardX11 yes

SSH

2012-08-22T10:41:01Z

Xalid:

== Passwordless SSH ==
To set up passwordless SSH, there are three main things to do:
* generate a pair of public/private keys on your local computer
* copy the public key from the source computer to the target computer's authorized_keys file
* check the permissions.

You can repeat that transitively for "A->B->C". You can use the initial pair of keys everywhere.

See here for details:

http://www.stearns.org/doc/ssh-techniques.current.html

== Automatically saying "yes" ==

This expect script automates typing "yes" when asked by SSH if a host should be added to known_hosts

#!/usr/bin/expect -f
set arg1 [lindex $argv 0]
set timeout 2
spawn ssh $arg1
expect "yes/no" {
send "yes\n"
}
send "exit\n"
send "\r"

You can include it in a bash script to iterate over all nodes doing this:

for i in `uniq hostfile` ; do
./say-yes.exp $i
done

== Better than automatically saying "yes" ==

Remark: It turns out there is a more ellegant way to do this task: using a tool called ''ssh-keyscan''.

== Making a cascade of SSH connections easy ==
Here is a very convenient way to set up the access to any machine directly instead of doing a cascade of SSH calls. If you can not directly access e.g. the machine "heterogeneous", but you can log into "csserver" and then to "heterogeneous", you can put this into your .ssh/config file :
Host csserver
User kdichev
Hostname csserver.ucd.ie
Host heterogeneous
User kiril
Hostname heterogeneous.ucd.ie
ProxyCommand ssh -qax csserver nc %h %p

Since the installation of a new PBS system, you can not directly log into a hclXX node. You can do
ssh heterogeneous instead and use "qsub" [[HCL_cluster#Access_and_Security]]

== The best way is saying no "YES" ==

This ssh trick avoids typing "yes" when asked by SSH if a host should be added to known_hosts:

ssh -q -o StrictHostKeyChecking=no

So with OpenMPI it can be used as

mpirun --mca plm_rsh_agent "ssh -q -o StrictHostKeyChecking=no"

== X11 forwarding ==
<code lang="bash">
ssh -X hostname
</code>
or add the following line to your .ssh/ssh_config file
ForwardX11 yes