Shell Framework (shf3)

From Nano Group Budapest
Jump to: navigation, search

Advanced Shell Scripting

The Bash shell is omnipresent and a very natural way to interact with a UNIX system. Even vanilla OS X comes with such a shell. Even though, it lacks "modern" programming language features it is a very powerful tool. In order to construct a modular Bash framework we made some non-intrusive extensions. As a result arbitrarily complex shell programs can be built by mapping libraries to a directory structure. This structure also promotes a clean programming practice without name collision. Libraries can be reused or extended for future needs without rewrite the bulk of the code. The current framework was developed for Bash version 3.

Extensions

Libraries / Namespaces

Libraries are kept under the shf3/lib/ directory. Each subdirectory is a namespace. Each namespace (directory) contains the following files in load order:

 kernel - function definitions
 [config] - variables
 [config.$OSTYPE] - OS specific variables  
 [config.$USER] - user specific variables

In a script a namespace is imported by:

 import lib1/lib2
Function and variable names are prefixed by the namespace:
 a function in shf3/lib/lib1/lib2 is function lib1/lib2/function() {...
 a variable in shf3/lib/lib1/lib2 is lib1_lib2_variable=...

Meta ID (MID)

Meta ID is a data object (bag) like a json or yml file. These files contains key=value pairs.

Directories

The most important directories are:

 bin - scripts 
 lib - libraries
 mid - mid files
 key - key files

Operators

String values can have operators at the 1st character. Default operators are: < + - . Operator clean value of a variable can be obtained by:

 ns/op/value $var

Framework Scripts

Framework scripts should contain the following header after the shebang:

 source $(dirname ${BASH_SOURCE})/../lib/header

Extensions Scripts

Extension script should refer the framework header more directly. Eg. the a script in shf3.ext/bin would do:

 source $(dirname ${BASH_SOURCE})/../../shf3/lib/header
 or
 source ${HOME}/shf3/lib/header

Libraries under shf3.ext/lib are added to the import path by:

 include $(dirname ${BASH_SOURCE})/../lib

CLI

There is an interactive interface for testing and for the internal help:

 shfcli

To get help:

 help
 help namespace

Donation

Shell Framework is free of charge, if you appreciate our work please donate!

File:Bitcoin accept.png 17mtnUyAoR8yXuRRB57eGSRNjDkgQRaVA2

Install

You can download the framework from the github:

 cd $HOME
 git clone git://github.com/thieringgergo/shf3.git

Edit .profile:

 source $HOME/shf3/bin/shfrc
 shf3/alias yes

Update

Update the framework:

 shfupd

Configure

You can fine tune the shell by setting the following commands in $HOME/.profile .

Command Feature
shf3/ps1 LABEL Set the PS1 prompt to LABEL instead of the hostname
shf3/alias yes/no Load/unload alias commands
shf3/screen yes/no Load/unload screen theme
shf3/mc/color yes/no Load/unload color mc theme

Password Manager

This program provides basic password management with random strings and per pair master password. Passwords are stored in an sqlite database and encrypted by GPG.

Create a password with N random characters:

 passmgr [-l N] -u user@host

Retrieve a password:

 passmgr -u user@host

Change the master password [ and the password to M random characters]:

 passmgr [-l M] -p user@host

List passwords:

 passmgr

Search:

 passmgr -s tag

SSH Manager

These programs provide a unified and simple interface for SSH-related tasks. Each login has a MID and optional keys and/or certificate. To create a MID:

 sshmgr -n MID

For GSI-SSH you have to install your key and certificate manually since it is obtained through a request-verify-sign process. SSH keys are found in shf3/key/ssh:

 MID.pub - public key (should be transferred to remote host .ssh/authorized_keys)
 MID.sec - private key, keep it in secret
 MID.crt - GSI-SSH certificate (need to install manually)

Edit a MID:

 sshmgr -e MID

Change MID key password:

 sshmgr -p MID

Show details of a MID:

 sshmgr -i MID

Check connection:

 sshmgr -c MID

List MIDs:

 sshmgr

Currently, PRACE certificates are supported. To update trusted certificates:

 sshmgr -g prace

SSH Hoping

Except GSI-SSH the framework supports 1-step SSH hoping (login via a gateway). To use hoping create a MID for the gateway and the internal machine as well. Connection check does not work for the internal machine. For hoping you have to prefix the MID by the gateway MID (GMID), you can use this notation with the subsequent commands.

 GMID/MID

Login

To login:

 sshin MID
 or with the gateway (login to MID via GMID)
 sshin GMID/MID

If you use proxy redirections and want to clear tunnel lock after a not clean logout (force redirections)

 sshto -f -m MID

File Transfer

With the file transfer programs you can copy between the current working directory and the remote scp directory (mid_ssh_scp_dst). SSH transfer uses rsync over ssh.

Send files to MID:

 sshtx put MID file1 dir1 ...

Recieve files to MID:

 sshtx get MID file1 dir1 ...

Interactive FTP-like shell:

 sshftp
 put MID file1 file2
 get MID file1 file2

SSH Mount

You have to install FUSE with sshfs. Remote machines are mounted into

 $HOME/sshfs

Mount a MID:

 sshmount MID
 or to force
 sshmnt -f -m MID

Unmount a MID:

 sshumount MID

List mounts:

 sshmnt

Encrypted Directory Manager

Bookmark Manager

With the bookmark manager you can save a link of the current working directory:

 book MID

List bookmarks:

 bmkmgr

Change working directory to a bookmark:

 bmkcd MID

Compressor

By default the framework uses serial gzip. You can set a parallel gzip (pigz):

 cd $HOME
 echo "sys_zip_xg_gz=pigz" > shf3/lib/sys/zip/xg/config.$USER

Visualization

You can use VMD as a visual terminal for the interactive shell or for scripts. The framework contains a VMD wrapper with input and output redirection. Start the visual terminal in the background:

 vmdmgr start &

Load a geometry:

 vmdmgr load <GEOMETRY>

Run vmd commands

 vmdmgr run <COMMANDS>

Reset the scene:

 vmdmgr reset

Stop the visual terminal:

 vmdmgr stop

Please mind that the wrapper calls VMD by vmd which should be the VMD binary and not a VMD wrapper script.

Program Wrapper

Program runner helps you to execute various programs in the following single step scheme:

  1. Prepare inputs
  2. Run program
  3. Save outputs

In order to rune a program wrapper you need i) a wrapper library module (PROGRAM) ii) a guide file (GUIDE):

 runprg -p PROGRAM -g GUIDE

Guide File

The program wrapper need some parameters to guide the execution. This guide file is a simple key=value shell file sourced by the runner. The allowed keys are the following:

Key Value
PRGBIN path to the program binary
PRGOPT program options
INPUTDIR input directory
MAIN main input file
OTHER space separated list of other input files
DATADIR data directory
DATA space separated list of data files
WORKDIR working directory
RESULTDIR result directory
RESULT space separated list of result patterns
ERROR error handling: save / clean

Operators

Operators modify interpretation of values.

Operator Key Effect
< MAIN redirecting input
- MAIN exclude input from the command
+ WORKDIR initiate input stage broadcast (see below)

Programs

General

The general (-p general) module is universal wrapper. Inputs (outputs) will be distributed (saved) without any conversion. It is suitable for unsupported programs or can be used as a template for new modules.

VASP

The VASP (-p vasp) module is wrapper for the VASP ab initio package. It promotes the following input file convention. The wrapper will rename the inputs to the standard names. PREFIX is removed from the OTHER input files. Outputs will renamed to get this prefix as well.

Standard VASP input Module input
INCAR PREFIX.cntl
POSCAR PREFIX.geom
KPOINTS PREFIX.kpts
QPOINTS PREFIX.qpts
POTCAR assembled from the guide

Queue Submit

Queue File

With this module you can create scheduler specific job scripts for single step jobs. You need a queue file containing static and queue specific parameters and a job file containing job specific varying parameters (resource needs). The name of the queue file is the Queue MID.

Common Keys

Key Value
QSCHED scheduler
QSETUP space-separated list of setup scripts
QULIMIT ulimit setting
QMAIL mail setting
QMAILTO email address for notifications
QERR error output
QOUT standard output
QEXCL exclusive queue access (yes,no)
QOPT options

Queue Specific Keys

Key Slurm PBS SGE Load Leveller
QSCHED slurm pbs sge loadl
QMAIL ALL abe
QCONST constraints (--constraint) constraints (-lnodes) constraints (-l) constraints (netwok.MPI)
QACC account (--account) account (-A) account (-A) account (account_no)
QPART partition (--partition) -- -- --
QQOS qos (--qos) -- -- job_type
QPE -- -- Parallel env. (-pe) --
QQUEUE -- queue (-q) queue (-q) --
QBIND -- -- binding (-binding) --
QARCH -- -- arch. (-l arch) --

Mail notifications

With QMAIL key you can request the scheduler to send notifications. If you set QMAIL=runprg then the wrapper script will send notifications bypassing the scheduler.

Job File

The job file contains job-specific parameters like resource needs. It also refers to the queue by a Queue MID.

Key Value
NAME job name
NODES # of nodes
SCKTS # of sockets per node
CORES # of cores per socket
MODE parallel mode
QUEUE queue MID
MODULE space-sep. list of modules to load
BIND CPU bind pre-command
BOOST boost pre-command
PROF profiler pre-command
RUN Command to run
VERBOSE verbose level: 0-3

Parallel Modes

The parallel mode is set by the MODE key. MPI modes have a /MPI modifier to set the MPI subsystem as well.

MODE Description
omp OpenMP only
mpi/MPI MPI-only with the selected MPI subsystem
mpiomp/MPI MPI-OMP hybris with the selected MPI subsystem
MPI Description
mpt SGI MPT
impi Intel MPI
ompi Open MPI

Resource Specification

Three types of parallel mode is supported: MPI-only, OMP-only, MPI-OMP hybrid. Shell framework will set environment variables for OMP and parameters for the mpirun according to the following table. Number of OMP threads can be override by THRDS . In case of SGE you can also specify the total number of slots per node by SLTPN .

Paralell Mode # of MPI processes # of OMP threads per MPI proc.
MPI-only (mpi) NODES x SCKTS x CORES 1
OMP-only (omp) -- SCKTS x CORES
MPI-OMP hybrid (mpiomp) NODES x SCKTS CORES

CPU binding

With the BIND key you can bind processes to CPUs. Please refer to the manual of the MPI subsystem. Usually, it is enough to set the parameters in the table below.

MPI MPI-only MPI-OMP hybrid
SGI MPT (mpt) dplace -s 1 omplace -s 1
Intel MPI (impi) -binding -binding
Open MPI (ompi) -bind-to-core -bycore --