RMG

From Tau Wiki
Jump to: navigation, search


RMG Quantum Mechanics Code

Background

RMG (Real-space Multi-Grid method) is a quantum mechanics code that employs hybrid MPI + OpenMP + pthreads + GPU capabilities. Co-PIs for this work and code are Shirley Moore (University of Texas at El Paso) and Emil Briggs (North Carolina State University). In the paper "Scaling the RMG Quantum Mechanics Code", the initial scaling of the code using the hybrid model to 128k cores was described, including preliminary TAU performance data (no GPU performance) at 128-cores on ORNL's Jaguar machine (titan-dev).

Preliminary TAU instrumentation of RMG was based on a combination of library wrapping and hybrid statistical sampling (tau_exec -ebs). MPI calls are wrapped, as are invocations to the pthread library. OpenMP pragmas are not handled, but OpenMP threads entities are captured through the side-effect of pthread library wrapping. This is only possible if OpenMP happens to be implemented as pthreads. This happens to be true on the Cray XK6. Hybrid Statistical Sampling reconstructs RMG's internal function-level code details to complete the picture. Traditional instrumentation of the code currently fails with runtime overlaps. It is still unclear why this happens. Apparently, CrayPat and VampirTrace also cannot currently handle RMG for various other reasons.


Platform and Environment

ORNL Jaguar (Titan-dev)

module swap PrgEnv-pgi PrgEnv-gnu

module load cudatoolkit

module load fftw/2.1.5.2

export CRAY_ROOTFS=DSL