Shell Framework (shf3)

Advanced Shell Scripting

The Bash shell is omnipresent and a very natural way to interact with a UNIX system. Even vanilla OS X comes with such a shell. Even though, it lacks "modern" programming language features it is a very powerful tool. In order to construct a modular Bash framework we made some non-intrusive extensions. As a result arbitrarily complex shell programs can be built by mapping libraries to a directory structure. This structure also promotes a clean programming practice without name collision. Libraries can be reused or extended for future needs without rewrite the bulk of the code. The current framework was developed for Bash version 3.

Extensions

Libraries / Namespaces

Libraries are kept under the shf3/lib/ directory. Each subdirectory is a namespace. Each namespace (directory) contains the following files in load order:

 kernel - function definitions
 [config] - variables
 [config.$OSTYPE] - OS specific variables  
 [config.$USER] - user specific variables

In a script a namespace is imported by:

 import lib1/lib2

Function and variable names are prefixed by the namespace:
 a function in shf3/lib/lib1/lib2 is function lib1/lib2/function() {...
 a variable in shf3/lib/lib1/lib2 is lib1_lib2_variable=...

Meta ID (MID)

Meta ID is a data object (bag) like a json or yml file. These files contains key=value pairs.

Directories

The most important directories are:

 bin - scripts 
 lib - libraries
 mid - mid files
 key - key files

Operators

String values can have operators at the 1st character. Default operators are: < + - . Operator clean value of a variable can be obtained by:

 ns/op/value $var

Framework Scripts

Framework scripts should contain the following header after the shebang:

 source $(dirname ${BASH_SOURCE})/../lib/header

Extensions Scripts

Extension script should refer the framework header more directly. Eg. the a script in shf3.ext/bin would do:

 source $(dirname ${BASH_SOURCE})/../../shf3/lib/header
 or
 source ${HOME}/shf3/lib/header

Libraries under shf3.ext/lib are added to the import path by:

 include $(dirname ${BASH_SOURCE})/../lib

CLI

There is an interactive interface for testing and for the internal help:

 shfcli

To get help:

 help
 help namespace

Donation

Shell Framework is free of charge, if you appreciate our work please donate!

File:Bitcoin accept.png 17mtnUyAoR8yXuRRB57eGSRNjDkgQRaVA2

Install

You can download the framework from the github:

 cd $HOME
 git clone git://github.com/thieringgergo/shf3.git

Edit .profile:

 source $HOME/shf3/bin/shfrc
 shf3/alias yes

Update

Update the framework:

 shfupd

Configure

You can fine tune the shell by setting the following commands in $HOME/.profile .

Command	Feature
shf3/ps1 LABEL	Set the PS1 prompt to LABEL instead of the hostname
shf3/alias yes/no	Load/unload alias commands
shf3/screen yes/no	Load/unload screen theme
shf3/mc/color yes/no	Load/unload color mc theme

Password Manager

This program provides basic password management with random strings and per pair master password. Passwords are stored in an sqlite database and encrypted by GPG.

Create a password with N random characters:

 passmgr [-l N] -u user@host

Retrieve a password:

 passmgr -u user@host

Change the master password [ and the password to M random characters]:

 passmgr [-l M] -p user@host

List passwords:

 passmgr

Search:

 passmgr -s tag

SSH Manager

These programs provide a unified and simple interface for SSH-related tasks. Each login has a MID and optional keys and/or certificate. To create a MID:

 sshmgr -n MID

For GSI-SSH you have to install your key and certificate manually since it is obtained through a request-verify-sign process. SSH keys are found in shf3/key/ssh:

 MID.pub - public key (should be transferred to remote host .ssh/authorized_keys)
 MID.sec - private key, keep it in secret
 MID.crt - GSI-SSH certificate (need to install manually)

Edit a MID:

 sshmgr -e MID

Change MID key password:

 sshmgr -p MID

Show details of a MID:

 sshmgr -i MID

Check connection:

 sshmgr -c MID

List MIDs:

 sshmgr

Currently, PRACE certificates are supported. To update trusted certificates:

 sshmgr -g prace

SSH Hoping

Except GSI-SSH the framework supports 1-step SSH hoping (login via a gateway). To use hoping create a MID for the gateway and the internal machine as well. Connection check does not work for the internal machine. For hoping you have to prefix the MID by the gateway MID (GMID), you can use this notation with the subsequent commands.

 GMID/MID

Login

To login:

 sshin MID
 or with the gateway (login to MID via GMID)
 sshin GMID/MID

If you use proxy redirections and want to clear tunnel lock after a not clean logout (force redirections)

 sshto -f -m MID

File Transfer

With the file transfer programs you can copy between the current working directory and the remote scp directory (mid_ssh_scp_dst). SSH transfer uses rsync over ssh.

Send files to MID:

 sshtx put MID file1 dir1 ...

Recieve files to MID:

 sshtx get MID file1 dir1 ...

Interactive FTP-like shell:

 sshftp
 put MID file1 file2
 get MID file1 file2

SSH Mount

You have to install FUSE with sshfs. Remote machines are mounted into

 $HOME/sshfs

Mount a MID:

 sshmount MID
 or to force
 sshmnt -f -m MID

Unmount a MID:

 sshumount MID

List mounts:

 sshmnt

Bookmark Manager

With the bookmark manager you can save a link of the current working directory:

 book MID

List bookmarks:

 bmkmgr

Change working directory to a bookmark:

 bmkcd MID

Compressor

By default the framework uses serial gzip. You can set a parallel gzip (pigz):

 cd $HOME
 echo "sys_zip_xg_gz=pigz" > shf3/lib/sys/zip/xg/config.$USER

Visualization

You can use VMD as a visual terminal for the interactive shell or for scripts. The framework contains a VMD wrapper with input and output redirection. Start the visual terminal in the background:

 vmdmgr start &

Load a geometry:

 vmdmgr load <GEOMETRY>

Run vmd commands

 vmdmgr run <COMMANDS>

Reset the scene:

 vmdmgr reset

Stop the visual terminal:

 vmdmgr stop

Please mind that the wrapper calls VMD by vmd which should be the VMD binary and not a VMD wrapper script.

Program Wrapper

Program runner helps you to execute various programs in the following single step scheme:

1. Prepare inputs
2. Run program
3. Save outputs

In order to rune a program wrapper you need i) a wrapper library module (PROGRAM) ii) a guide file (GUIDE):

 runprg -p PROGRAM -g GUIDE

Guide File

The program wrapper need some parameters to guide the execution. This guide file is a simple key=value shell file sourced by the runner. The allowed keys are the following:

Key	Value
PRGBIN	path to the program binary
PRGOPT	program options
INPUTDIR	input directory
MAIN	main input file
OTHER	space separated list of other input files
DATADIR	data directory
DATA	space separated list of data files
WORKDIR	working directory
RESULTDIR	result directory
RESULT	space separated list of result patterns
ERROR	error handling: save / clean

Operators

Operators modify interpretation of values.

Operator	Key	Effect
<	MAIN	redirecting input
-	MAIN	exclude input from the command
+	WORKDIR	initiate input stage broadcast (see below)

Programs

General

The general (-p general) module is universal wrapper. Inputs (outputs) will be distributed (saved) without any conversion. It is suitable for unsupported programs or can be used as a template for new modules.

VASP

The VASP (-p vasp) module is wrapper for the VASP ab initio package. It promotes the following input file convention. The wrapper will rename the inputs to the standard names. PREFIX is removed from the OTHER input files. Outputs will renamed to get this prefix as well.

Standard VASP input	Module input
INCAR	PREFIX.cntl
POSCAR	PREFIX.geom
KPOINTS	PREFIX.kpts
QPOINTS	PREFIX.qpts
POTCAR	assembled from the guide

Queue Submit

Queue File

With this module you can create scheduler specific job scripts for single step jobs. You need a queue file containing static and queue specific parameters and a job file containing job specific varying parameters (resource needs). The name of the queue file is the Queue MID.

Common Keys

Key	Value
QSCHED	scheduler
QSETUP	space-separated list of setup scripts
QULIMIT	ulimit setting
QMAIL	mail setting
QMAILTO	email address for notifications
QERR	error output
QOUT	standard output
QEXCL	exclusive queue access (yes,no)
QOPT	options

Queue Specific Keys

Key	Slurm	PBS	SGE	Load Leveller
QSCHED	slurm	pbs	sge	loadl
QMAIL	ALL		abe
QCONST	constraints (--constraint)	constraints (-lnodes)	constraints (-l)	constraints (netwok.MPI)
QACC	account (--account)	account (-A)	account (-A)	account (account_no)
QPART	partition (--partition)	--	--	--
QQOS	qos (--qos)	--	--	job_type
QPE	--	--	Parallel env. (-pe)	--
QQUEUE	--	queue (-q)	queue (-q)	--
QBIND	--	--	binding (-binding)	--
QARCH	--	--	arch. (-l arch)	--

Mail notifications

With QMAIL key you can request the scheduler to send notifications. If you set QMAIL=runprg then the wrapper script will send notifications bypassing the scheduler.

Job File

The job file contains job-specific parameters like resource needs. It also refers to the queue by a Queue MID.

Key	Value
NAME	job name
NODES	# of nodes
SCKTS	# of sockets per node
CORES	# of cores per socket
MODE	parallel mode
QUEUE	queue MID
MODULE	space-sep. list of modules to load
BIND	CPU bind pre-command
BOOST	boost pre-command
PROF	profiler pre-command
RUN	Command to run
VERBOSE	verbose level: 0-3

Parallel Modes

The parallel mode is set by the MODE key. MPI modes have a /MPI modifier to set the MPI subsystem as well.

MODE	Description
omp	OpenMP only
mpi/MPI	MPI-only with the selected MPI subsystem
mpiomp/MPI	MPI-OMP hybris with the selected MPI subsystem

MPI	Description
mpt	SGI MPT
impi	Intel MPI
ompi	Open MPI

Resource Specification

Three types of parallel mode is supported: MPI-only, OMP-only, MPI-OMP hybrid. Shell framework will set environment variables for OMP and parameters for the mpirun according to the following table. Number of OMP threads can be override by THRDS . In case of SGE you can also specify the total number of slots per node by SLTPN .

Paralell Mode	# of MPI processes	# of OMP threads per MPI proc.
MPI-only (mpi)	NODES x SCKTS x CORES	1
OMP-only (omp)	--	SCKTS x CORES
MPI-OMP hybrid (mpiomp)	NODES x SCKTS	CORES

CPU binding

With the BIND key you can bind processes to CPUs. Please refer to the manual of the MPI subsystem. Usually, it is enough to set the parameters in the table below.

MPI	MPI-only	MPI-OMP hybrid
SGI MPT (mpt)	dplace -s 1	omplace -s 1
Intel MPI (impi)	-binding	-binding
Open MPI (ompi)	-bind-to-core -bycore	--