Hybrid OPENMP/MPI parallelization of a high-order Discontinuous Galerkin CFD/CAA solver

This paper describes the implementation of an hybrid OpenMP/MPI parallelization strategy in a Discontinuous Galerkin solver used for DNS and LES or CAA computations, to fruitfully exploit the modern massively parallel HPC facilities. It is usually believed that the sheared memory view of OpenMP can easily increase the parallel efficiency of codes dealing with multi-core clusters. The idea consists of running calculations on those machines restricting as much as possible the use of the MPI library to the communications between nodes and exploiting the shared memory paradigm within a node. However, in practice, the achievement of a real parallel performance gain is not straightforward. Moreover, as far as DG solvers are concerned, almost nothing is reported in the current literature about the hybrid MPI/OpenMP implementation. In this work a colouring algorithm has been employed for OpenMP. The resulting hybrid strategy performs quite satisfactory, since generally it is more efficient of the pure MPI implementation. However, the performances are heavily dependent on hardware platforms, as well as on computational details such as the polynomial order of space discretization or the number of computational elements. Several scalability tests have been performed, resulting in the conclusion that the best performance can be achieved only with a proper choice of the number of MPI partition and OpenMP threads to be used within a single node. The reliability of the method was here assessed by solving the Taylor Green vortex problem at Reynolds numbers equal to 800 and 1600 and the Linear Euler acustic scattering from a rigid sphere.