Debrecen2
This system maintained by NIIF.
Detailed info about debrecen2 Nvidia GPU cluseter is at here
Detailed info about debrecen3 Intel Phi cluseter is at here
Specification
| Name | Debrecen2 (Leo) | Debrecen3 Phi (Apollo) |
| Type | HP SL250s | HP Apollo 8000 |
| core / node | 16 | 24 |
| Accelator / node | 3x Nvidia K20x in 68 nodes (K40x in 16 nodes) | 2x Intel(R) Xeon Phi(TM) MIC SE10/7120D |
| Memory / node | 120 GB | 120 GB |
| CPU | 2× Intel Xeon E5-2650 v2 @ 2.60GHz | 2× Intel Xeon E5-2670 v3 @ 2.30GHz |
| Architecture | x86_64 / intel64 / em64t little-endian | x86_64 / intel64 / em64t little-endian |
| Scheduler | Slurm | Slurm |
| MPI | Intel MPI (impi), Open MPI (ompi) | Intel MPI (impi), Open MPI (ompi) |
Logging in
Set up the SSH access from Skynet, and mount its storage on skynet. (log into skynet and type:)
cd ~/shf3/mid/ssh cp niif/debrecen2 debrecen2 cd ~/shf3/key/ssh
Then place your private NIIF key here, and rename it as:
mv <YOUR_NIIF_KEY> debrecen.sec
You might need to export the private key from putty in OPENSSH format, if you used puttygen to generate the keypair.
Precompiled Environment
From Skynet sync up the precompiled environment:
cd /share/niif/debrecen2 sshput -m debrecen2 -s .
Log into debrecen: You can do this with putty, etc if you don't like logging into debrecen from Skynet.
sshto -m debrecen2
Then add the following into your .bash_profile:
PATH=$PATH:$HOME/bin
VASP
We use intel MPI for VASP.
First please transfer the compiled VASP binary and projectors from skynet. Log into "skynet"
cd /share/niif/ sshput -t 2 -m debrecen2 -s vasp/5.4.1.03082016.impi sshput -t 2 -m debrecen2 -s vasp/5.4.1.03082016.impi.7.0cuda sshput -t 2 -m debrecen2 -s vasp/proj
Then add the following into your .bash_profile:
export PATH=$PATH:$HOME/bin export VASP_PROJ_HOME=$HOME/vasp/proj module load intel/compiler/2016.1.150 module load intel/mkl/2016.1.150 module load intel/ipp/2016.1.150 module load intel/daal/2016.1.150 module load intel/tbb/2016.1.150 module load intel/mpi/2016.1.150 module load intel/mpi/4.1.0.027 LD_LIBRARY_PATH=/opt/intel/compilers_and_libraries_2016.1.150/linux/mpi/intel64/lib:$LD_LIBRARY_PATH module unload cuda/6.5 module load cuda/7.0.28
debrecen3 with CPU's only
Log into "debrecen2" You can find a sample job of an ozone molecule in $HOME/jobsamples
cd $HOME/jobsamples/ozone
Please fill your email in the debrecen2 jobfile,
mcedit debrecen2_cpu
then submit it:
sbatch debrecen2_cpu
This job should finish in mere seconds. Please replace the partition to "prod" from "test" for actual large scale runs.
#SBATCH --partition=prod-phi
You can easily increase the number of nodes:
#SBATCH --nodes=4
Since we cannot utilize the Xeon Phi accelators, please use this computer till there are no users with Xeon Phi accelerated programs.
debrecen2 GPU port
Log into "debrecen2" You can find a sample job of an ozone molecule in $HOME/jobsamples
cd $HOME/jobsamples/gpu
Please fill your email in the debrecen2 jobfile,
mcedit debrecen2_gpu
then submit it:
sbatch debrecen2_gpu
This job should finish in a hour.
You can easily increase the number of nodes:
#SBATCH --nodes=4
Tips.: - The GPU port will launch 3 MPI threads per node. One MPI process per physical GPU.
- There is no Gamma only version available yet. You have to use the "standard" version "vasp_gpu" for gamma only calculations. The non-collinear binary is called "vasp_gpu_ncl"
- Use one GPU for each k-point. This improves the performance by ~30%. The connection between the GPU's seems too slow.
- Tipically one GPU node equals with 2-6 nodes approximately with ~20 intel CPU cores each.
- You can use both PBE, and hybrid (HSE06) functionals.
- prod-gpu-k40 queue contains GPU's with 12 GB memory on board, while prod-gpu-k20's have only 6 GB