Computing Infrastructure

HiFiMagnet is available on various computing ressources. Here we describe in more details how to:

connect to these ressources,
launch HiFiMagnet on these ressources,
transfer (if needed) data to/from these ressources.

Available ressources are:

Lncmi
Gricad
Cemosis
other ressources

1. Lncmi

To use Lncmi ressources you need a Lncmi account. To run HiFiMagnet you can use either:

workstation: only for CAD/Meshing and eventually simulation of one helix; need MobaXterm if running on Windows 10
servers: for CAD/Meshing of an insert, static simulation of an insert except for fully 3D coupled,

For large simulation you shall consider Gricad or Cemosis.

1.1. workstation

1.1.1. On Linux workstation

On Linux workstation you can use singularity images without problem. However keep in mind that you will be limited by the RAM memory installed on the machine you are using.

1.1.2. On Windows 10 workstation

On Windows 10 worstation you have to install first WSL2, then Ubuntu 18.04 from Microsoft store. Finally to run applications in grapic mode, you will need MobaXterm.

To install HiFiMagnet, you can use:

WSL2
Docker

To run HiFiMagnet when using WSL:

eventually start MobaXterm to start the X11 server
launch Ubuntu
Within Ubuntu LTS just run HiFiMagnet app you want

1.2. servers

On Lncmi servers, we recommend to use singularity containers images which shall be store on /home/singularity directory. Depending on the server singularity version, use either a .sif or .simg image.

When connecting to the serveur, Do not forget to add -Y ssh option when you want to run graphical apps like Salome.

1.2.1. Connect

ssh [-Y] stokes

When connecting to the serveur, Do not forget to add -Y ssh option when you want to run graphical apps like Salome in GUI mode.

1.2.2. Run HiFiMagnet

As HiFiMagnet apps can run for a while, you are adviced to use screen. That is:

1st start screen then run the app within screen.
Then you can detach the screen process with Ctrl-A Ctrl-D.
To latter re-attach the screen process, run
- screen -ls to get the screen-idyour screen session
- and re-attached it with screen -rd screen-id.

Running HiFiMagnet on Lncmi server relies on singularity. Depending on the server, singularity version is different. Please check which version is installed before running any singularity command.

To run HiFiMagnet in parallel on stokes, you have to proceed as follows:

singularity exec image.sif mpirun -np jobs ...

where jobs corresponds to the number of core you plan to use, and image.sif to the singularity image you use.

On stokes, jobs is limited to 32.

1.2.3. Retreive/Send Data

Use either scp or rsync over ssh.

2. Gricad

Gricad is the Grenoble MesoCentre for HPC. {dahu} is the platform to be used for HPC.

2.1. connect

To connect to {dahu}, you need to get an account on Perseus. Follow the instructions on this page to proceed.

Once you’ve got a Perseus account, you still have to be registered as a user of the pr-hifimagnet project. Contact your administrator <christophe.trophime@lncmi.cnrs.fr> to be registered as a member of this project.

To setup your ssh access to {dahu}, proceed as follow:

setup your ssh config:

Add the following lines to your $HOME/.ssh/config

Host access-rr-ciment
   #LogLevel DEBUG3
   Hostname access-rr-ciment.imag.fr
   ProxyCommand nc -q0 %h %p
Host *.ciment
  User username (1)
  ProxyCommand ssh username@access-rr-ciment "nc -w 60 `basename %h .ciment` %p"
Host dahu
  User username
  ProxyCommand ssh username@access-rr-ciment.imag.fr "nc dahu %p"

with <1> your actual username on Perseus.

test your connection to {dahu}:

ssh -Y dahu

setup key authentification: to be done

2.2. run HiFiMagnet in parallel

Please read the dahu documentation before proceed.

only run interactive simu on small problem to check everything is ok
can only use XXXX

To start an interactive session on 2 nodes:

oarsub -I -l /nodes=2 --project hifimagnet

…

2.3. submit HiFiMagnet jobs

To submit jobs on {dahu}, you need to use the Oar scheduler. To do so, you need to create an Oar script like that:

#!/bin/bash
#OAR -n HL-31 (1)
#OAR -l /nodes=14,walltime=8:00:00 (2)
#OAR -O HL-31_%jobid%.out (3)
#OAR -E HL-31_%jobid%.err
#OAR --project hifimagnet (4)
#OAR --notify mail:youremailadress (5)

# Setup environment (6)
source /applis/site/nix.sh
export NIX_PATH="nixpkgs=$HOME/.nix-defexpr/channels/nixpkgs"
nix-env --switch-profile $NIX_USER_PROFILE_DIR/nur-openmpi4
echo "Load nix-env:" $(nix-env -q)

echo "Singularity:" $(which singularity)
SVERSION=$(singularity --version)
echo "Singularity Version:" ${SVERSION}

nbcores=$(cat $OAR_NODE_FILE|wc -l)
nbnodes=$(cat $OAR_NODE_FILE|sort|uniq|wc -l)
njobs=$(cat $OAR_FILE_NODES | wc -l)

echo "nbcores=" $nbcores
echo "nbnodes=" $nbnodes
echo "np=" $njobs
echo "OAR_WORKDIR=" ${OAR_WORKDIR}

# Exec env (TODO param SVERSION)
IMGS_DIR=/bettik/$USER/singularity (7)
HIFIMAGNET=${IMGS_DIR}/hifimagnet-thermobox-debianpack.sif (8)
if [ ! -f $HIFIMAGNET ]; then
	echo "Cannot find ${HIFIMAGNET}"
fi

# Sim setup (9)
SIMDIR=/bettik/$USER/HL-31
CFG=M19061901-full.cfg
LOG=M19061901-full.log
if [ ! -f $CFG ]; then
	echo "Cannot find ${CFG}"
fi

OUTDIR=${SIMDIR}/full
mkdir -p ${OUTDIR}
echo "create ${OUTDIR}"

# Check if mesh is correctly partitionned: aka nparts==njobs

# Run the program (9)
mpirun -np ${njobs} \
       -machinefile $OAR_NODEFILE \
       -mca plm_rsh_agent "oarsh" -mca btl_openib_allow_ib true \
       --prefix $HOME/.nix-profile \
     singularity exec -H ${OAR_WORKDIR} -B $OUTDIR:/feel $HIFIMAGNET \
       feelpp_hfm_coupledcartmodel_3DP1N1 --config-file ${CFG}

As a rule of the thumb, a mesh of 1 Go would mean to use 2 T Ram for fully coupled simulations. Thus you will have to adapt the number of nodes you request on dahu for your needs. On dahu a node contains n cores. Each core has xxx Ram.

To submit the job, from the directory holding the oar script just type:

oarsub -S ./M19061901_full.oar

To check the status of your job:

oarstat -u usename

Upon job completion, you will receive an email from {dahu} on the email adress you have defined in the Oar script.

For best performance, the job shall be create on bettik - namely in the mounted directory /bettik/username. All the files mentionned in the Oar script must exist. In peculiar the mesh, cfg and json files.

The results will be exported to /bettik/username as defined in the Oar script and cfg file.

2.4. retreive/send data

In this paragraph, we explain how to send data to {dahu} and how to recover simulation results from {dahu}

Use either scp or rsync over ssh. For large files, you must use rsync over ssh.

3. Cemosis

Cemosis is hosted by the Institute of Advanced Mathematical Research (IRMA).