Computing clusters: Local Guide

Configuration     – Home and scratch   – Quotas   – Backups

Access from VSCHT   – VPN and X-windows   – Shell   – File transfer Access from outside   – Shell   – File transfer   – X-windows graphics

Getting started   – Managing files   – Editing text files Managing jobs   – Queues   – Submit jobs   – List jobs   – Remove jobs

Configuration

We have three clusters:

• Selenium = as67-1.vscht.cz (old),
• Argon = 403-a324-01.vscht.cz (middle age), and
• Neon = 403-as67-01.vscht.cz (new, sponsored by KvaLab)

Home and scratch

User home directory is /home/USER.

On nodes (clients), a user can use scratch disk space in directory /scratch/USER/.
On Selenium (as67-1), scratch disks are periodically cleared. Your files will not be erased if:

• you run a job (on that node), or
• there is a file named .keepscratch.all in your home, or
• there is a file named .keepscratch.NODE (e.g., .keepscratch.s22)

Quotas

User disk space (/home) is limited by quotas. To check your quota status, use command:

quota -s

Backups

Backups are performed daily at night. All user home directories (/home/*/) are periodically backed up. Four daily backups are kept, and weekly backups (Saturday to Sunday) up to the backup server capacity. Older files are lost.

Only root can access the backup server. Ask jiri.polach(at)marge.uochb.cas.cz in case you need to restore a file.

Access from the VSCHT domain

Shell access

From Linux

ssh USER@as67-1.vscht.cz
ssh USER@403-a324-01.vscht.cz
(answer yes for adding the computer to your list of trusted hosts)

From Windows

File transfer

From Linux

One option is scp:

to copy file /home/USER/MYFILE/ from cluster to your local ./:
scp USER@a324-2.vscht.cz:MYFILE .

to copy your local file file ./MYFILE to /home/USER/MYFILE on the cluster:
scp MYFILE USER@a324-2.vscht.cz:MYFILE

From Windows

One option is WinSCP, which provides a Windows-commander-like or Explorer-like interface to transfering files. Start WinSCP.exe, enter the host, then your name and the password. The TCP port should be 22.

X-windows (X11) graphics

To use graphical applications like gnuplot, xxgdb, etc., on the cluster, you need an X11 server running on your machine.
Note: People are often confused by the client/server model of X11. A client (running on a remote machine, i.e., the cluster [server or node]) asks the server (running, e.g., on your M$ Windoze PC) to display graphics (e.g., to draw a rectangle).

From Linux

ssh -X USER@403-a324-01.vscht.cz

In case of problems: You may need to add the client computer to the list of allowed hosts on your computer. Thus, on your computer, run:

xhost +a324-2.vscht.cz

From Windows

• Enable X forwarding in yout Putty session by selecting
  Connection → Tunneling → X11 → Enable X11 forwarding
• Sometimes it is necessary to select "Disable server access control" while starting XMing.

setenv DISPLAY NAME:0.0     # in csh, tcsh
export DISPLAY=NAME:0.0     # in sh, bash

Access from outside

VPN and X-windows (X11)

The recommended way how to connect to a cluster from home is to establish a VPN session to VSCHT. Then you may connect (incl. X-forwarding) and access remote files directly from your PC. You need to install a VPN client to do this, see computer center info.

From Linux

VPN client

The Computer Center recommends the Cisco client – see instructions there. However, I experienced incompatibility with the Ubuntu/Debian update system (some time ago, may be fixed now).

The openconnect package available via the Synaptic package manager (or similar) works, too. On older systems, the connect script vpnc-script is missing and must be installed separately. To connect to a cluster, first start a VPN session by:

sudo openconnect -s /etc/vpnc/vpnc-script vpn.vscht.cz

As soon as you are connected by VPN to VSCHT, the usage is the same as if you are in the domain (ssh -X USER@as67-1.vscht.cz).

Back connection

scp REMOTEFILE LOCALUSER@OFFICE_MODE_IP:LOCALFILE
scp LOCALUSER@OFFICE_MODE_IP:LOCALFILE REMOTEFILE

From Windows

As soon as you are connected by VPN to VSCHT, the usage is the same as if you are in the domain (Putty and XMing).

Shell access

To get a remote shell, you may use the SSH gateway: use any ssh connection to ftpin.vscht.cz and log in as sshgw (mnemonics: SSH GateWay) with passwd=sshgw. Then, type the name of the target computer (as67-1; newer ones are likely not supported), your user ID, and password. X11 forwarding is not supported (unless you create a tunnel); for connecting incl. graphics, see above.

From Linux

ssh sshgw@ftpin.vscht.cz
sshgw@ftpin.vscht.cz's password: sshgw

Zadejte adresu systemu ke kteremu se chcete pripojit
>a324-2
Zadejte jmeno uzivatele, pod kterym se chcete pripojit
>USER
Probiha pripojovani... USER@a324-2's password: PASSWORD

From Windows

File transfer from outside

One option is to use the ftp server ftpin.vscht.cz, otherwise see above. You need a (temporary) account on ftpin.vscht.cz. From a shell at the cluster, run

telnet ftpin
login: ftpman
Password: ftpman
Zadejte nove uzivatelske jmeno: USER (=new user name)
Zadejte uzivatelske heslo:PASSWORD
Zadejte uzivatelske heslo znovu(kontrola):PASSWORD (once more to check)
Kolik dni chcete ponechat ucet aktivni(maximum 7)?[1] DAYS_ACTIVE (account active max 7 days)
Zmacknete ENTER pro zalozeni uzivatele nebo ukoncete spojeni. ENTER

Hint: Consider putting your ftpin login data into file .netrc, e.g.

machine ftpin.vscht.cz login USER
password PASSWORD
macdef init
binary
 

Getting started

A user is normally logged to the server where (s)he can manage and edit files, compile and debug programs, submit jobs to be run on the client nodes, and analyse the results (incl. X11 graphics). No lengthy calculations are allowed directly on the server!

If necessary (e.g., lengthy interactive debugging), you may also use ssh to connect directly to machines inside the cluster. It is not allowed to jump the queue of submitted jobs in this way! The most important commands to survive are:

passwd	Change your password
man COMMAND	Get the manual page of COMMAND
xman	Manual pages browsing tool, requires X11
info	Comprehensive manual of GNU software
info COMMAND	Info on COMMAND (often more up-to-date than the man-page)

Managing files

Your shell is tcsh or bash. (To figure out which one, execute command ps.) To get help, use

man tcsh     or     man bash

It's pretty long, isn't it? A few basic commands, common for both shells, are listed below.

ls ls DIRECTORY/	List files
ls -l ls -l DIRECTORY/	List files with verbose info
cat FILE	Print a (short) text file
less FILE	View a (long) text file; use arrows, PgUp/PgDn or u/space, quit by q
cp -i FILE1 FILE2 cp FILE(s) DIRECTORY/	Copy files
mv -i FILE1 FILE2 mv FILE(s) DIRECTORY/	Rename or move files
rm -i FILE(s)	Remove files

(Option -i in the above commands ensures confirmation if a file is about to be overwritten or erased; based on your environment, you may have aliased the above commands so that -i does not have to be used.)

Another possibility is to replace this shell by the Midnight Commander, a clone of the popular Norton/Windows/Total Commander. It is started by:

mc

Editing text files

Another possibility is to edit files locally (on your Windows) and to move them using WinSCP.

Managing jobs

A user batch job (binary executable or a script) is submitted on a server to a queue. As soon as there are resources available, the job is started on a client.

Queues

On Selenium (as67-1), there are two instances of queues:

• Normal queues (max 25 cores/user). Now running are:
  • sq-8-16 (24 nodes by 8 cores, 16 GiB RAM)
  • sq-32-128 (node s29 of 32 cores, 128 GiB RAM)
  The tasks are started with normal (maximum) priority (nice 0). More jobs than the limit will wait in a queue. Basic commands to control jobs are qsub, qdel, qstat.
• Nice queues (max 35 cores/user) Similarly as above:
  • mq-8-16 (24 nodes by 8 cores, 16 GiB RAM)
  • mq-32-128 (node s29 of 32 cores, 128 GiB RAM)
  The tasks are started with minimum priority (nice -19). Basic commands to control jobs are qsubsec, qdelsec, qstatsec; any command for the nice queue can be also obtained by prefix sgesec, e.g., the following two commands are equivalent:

  sgesec qsub nicescript.sh
  qsubsec nicescript.sh

• aq (14 nodes by 12 cores, 48 GiB RAM)
• aq-gpu-gtx690 (3 nodes with GPUs)
• aq-gpu-m2090 (1 node with GPU)

Submit jobs

Examples:

• Submit parallel (no GPU) program using two threads on 403-a324-01, accepting two arguments:

  qsub -q aq -cwd -b y -pe shm 2 -N MyParJob MyExe MyParm1 MyParm2

  • -b y: Myexe is a binary executable (not script), MyParm1 MyParm2 are its arguments.
  • -cwd: It is started from the current directory
  • -q aq: The job is started on standard nodes (queue aq)
  • -pe shm 2: Parallel job, two cores requested
  • Standard output = MyParJob.o# (stdout) and MyParJob.e# (stderr), where # denotes job number; in case of a parallel job, there are also MyParJob.po# and MyParJob.pe#
     
• Submit script using 64 GPUs (and 1 CPU)

  qsub -q aq-gpu-gtx690 -l num_gpu=64 MyScript.sh

  • -l num_gpu=64: Use 64 GPUs (in case of more CPUs requested, this is per one CPU).
   
• Submit program a.out on a67-1 which reads file fort.11 and writes file fort.22; however, your input file is myin.dat and output should be myout.dat. All these files are in the current directory (option -cwd). The standard output (stdout) and error output (stderr) are concatenated (option -j y). The name of the job should be MYJOB.
  • Create script myscript.sh:

    #!/bin/bash
    #$ -cwd
    #$ -q "sq-8-16"
    #$ -j y
    ln -s fort.11 myin.dat
    a.out
    mv fort.22 myout.dat

  • then make it executable and submit it:

    chmod +x myscript.sh
    qsub -N MYJOB myscript.sh

• Submit a time-intensive calculation (script simul.sh) as a nice job on any client of Selenium (as67-1)

  qsubsec -q "mq-*-*" simul.sh

• A shortcut command to submit a job (binary executable or script) with parameters and in the current environment is (example)

  jsub a.out

  Run jsub without arguments to get help.

List jobs

• Edited listing for the current user:

  jstat

• Basic listing commands:

  qstat
  qstatsec

• Detailed listing:

  qstat -f
  qstatsec -f

Remove jobs

• Remove a normal job waiting in a queue (use qstat or jstat to get JOB-ID):

  qdel JOB-ID

• Remove a nice job waiting in a queue (use qstatsec or jstat to get JOB-ID), applies for Selenium only:

  qdelsec JOB-ID

• Kill (interrupt, signal) a running job
  • Use qstat/qstatsec/jstat to determine on which client the job is running
  • ssh to the client (e.g., a77) and determine the process ID:

    ssh a77
    ps x

  • Signal the running process:

    kill -2 ID	# SIGINT, the same as Ctrl-C interactively; # many applications finish (some) work and close files
    kill ID	# try harder (signal SIGTERM)
    kill -9 ID	# kill unconditionally (nothing saved)

  • Alternatively, run top and kill the process by hot key k.