XGC Coding Guidelines

Below are some general guidelines/requirements for writing code in XGC and guidelines for what to look for when reviewing a pull request.

Basics

  1. Keep the code clean – do not add dead/unnecessary code to the repository!

  2. As soon as it becomes clear that a piece of code is no longer needed, remove it!

  3. Comment your code! Comments help other developers understand what a piece of code does.

  4. Avoid generic variable names (a, b, c) where possible and reasonable; instead, try to come up with variable names that convey the role of the variable!

  5. Never use goto!

  6. In C++, use the conventions described here.

Formatting

  1. Indent with spaces (not tabs) for every logical block. 2 spaces in Fortran, 4 spaces in C++.

  2. Indent OpenMP clauses so that they are at the same indentation level as the loops to which they belong.

  3. C preprocessor macros such as #ifdef cannot be indented. Therefore, avoid overly convoluted preprocessor constructs and add a Fortran comment before every #else and #endif to indicate to which #ifdef they belong.

  4. Avoid interrupting short do-loops or if-blocks with #ifdefs where it is reasonable to improve readability.

  5. Break lines of source code after approximately 130 characters or fewer.

Function/Subroutine arguments

  1. In fortran, use the intent property for every dummy variable of a subroutine. This shows other developers what variables are intended for and it helps catch bugs when compiling if a variable is misused in a subroutine.

  2. In C++, arguments should follow the convention described here.

  3. If your function has many arguments, consider whether the arguments should be logically grouped together in a larger common object.

Function/Subroutine

  1. When developing new features/subroutines, ask yourself the following questions:

    1. Is there already a module/source file to which your routine logically belongs? If so, try to be consistent with existing code where it is reasonable to do so.

    2. Does the feature you develop merit its own Fortran module? If you are working on something that includes a group of routines needed to perform a specific task and involve global variables, it might be best to create a Fortran module for this (e.g. field-aligned Fourier transforms).

    3. Are there any conflicts between the code your developing and other XGC capabilities? Always try to write general code that is most likely to work for all typical use cases.

    4. When in doubt, discuss with other XGC developers.

Doxygen comments

Add Doxygen documentation to your recent developments.

  1. Fortran

    Using Doxygen is almost as simple as writing Fortran comments and will help us to comply with future requirements for code documentation. But if you write your documentation using Doxygen format, we can easily create a PDF or HTML documentation of XGC. The interactive HTML documentation is especially helpful.

    Fortran usage examples

    1. Explain a variable:

      logical :: ptb_3db_on=.false. !< Switch 3D perturbed field on (.true.) and off (.false.)

    2. Documentation of a subroutine:

      !> Calculates Fourier decomposition of the change of dA_phi
!! and determines the damping factor for a given iteration
!! @param[in] grid grid data, type(grid_type)
!! @param[out] damp_fac global damping factor, real(8)
subroutine ptb_3db_damping_factor(grid,damp_fac)

    3. You can also add LaTeX comments, e.g.:

      !> \f$(x_1,y_1)\f$

  2. C++

    C++ Doxygen is almost the same as Fortran but replace ! with // or /** ... */. If you add new C++ functions in a header file (*.hpp) and a source file (*.cpp) please include the detailed documentation in the source file.

    Always include a file documentation with the @file tag. If omitted function documentation in the file may not show up when Doxygen generates the documentation pages.

    If your code contains conditional preprocessor directives, e.g.

    #if PETSC_VERSION_GE(3,14,0)

    you probably want to add this macro to the list of PREDEFINED in the Doxygen config files

    PREDEFINED=XGCA PSPLINE \
     FFTW3 FUSION_IO \
     USE_OMP USE_GPU USE_CUDA PETSC_VERSION_GE

    or the documentation inside the condition may not be generated by Doxygen.

    C++ usage examples

    1. File description

      /**
 * @file interpolate.cpp
 * @Version 1.0
 * @Authors  A. Mollen, R. Hager
 * @Date Mar 2021
 *
 *
 * @section DESCRIPTION
 *
 * C++ functions for interpolation.
 */
#include "interpolate.hpp"

    2. Function

      /**
 * Creates the PETSc DM object
 * used for the interpolation.
 *
 * @param[in,out] dm is the PETSc DM object created.
 * @param[in] order is the interpolation order.
 * @param[in] Nspecies is the number of species being interpolated.
 *
 * @return void
 */
void initialize_interpolation(DM &dm, \
                PetscInt order, const PetscInt Nspecies)

Version control (git)

  1. Actively use git version control and our git repositories at GitHub.com

    1. For coordinating the work of all XGC developers, it is essential that everyone uses version control and our git repository. This makes it easy to share your work, discuss problems, and make sure that you do not break the production version of XGC.

    2. When developing a feature, regularly test whether the code compiles and runs. If it does, commit the current state to a branch of the repository. This helps you keep track of changes and find bugs later. It also makes it easy to merge your work with the production version of XGC.

Hardcoding

  1. Do not hard-code numbers “in situ” - use parameter (Fortran) or const (C++) for numerical values

  2. In C++, use enum if you are hardcoding indices (i.e. never write something like species[0], instead have an enum of the types of species and use species[ELECTRON].

Compatibility

  1. All changes should consider future users. Tests and errors should be included such that sets of variables that may be inconsistent, or routines that may be incompatible with each other can’t be accidentally used at the same time by a user unfamiliar with the source code.

Runtime inputs and preprocessor flags

  1. When in doubt, use runtime inputs rather than preprocessor flags when adding a new optional functionality.

  2. Use preprocessor flags in the following situations:

    1. A runtime option would ruin vectorization in a performance-critical code region, and templating is infeasible;

    2. The new option is meant to be a rapid replacement if successful. This should be coordinated with the team.

  3. Add runtime input variables in XGC_core/setup.F90:

    1. If the new input fits logically into existing namelists; otherwise, create a new one;

    2. Set a default value for the variable at declaration.

  4. Add preprocessor flags in CMake/XGCConfigOptions.cmake.

Modularity

  1. NO GLOBAL VARIABLES in the C++ code

  2. Limited number of global parameters (e.g. proton mass) are acceptable, put them in cpp_globals.hpp if they may be of general use

  3. A function should perform a single logical operation Hints that your function might not be performing a single logical operation and needs to be broken up:

    1. It is very large (more than 200 lines)

    2. There are regions in your code where most variables inside that region aren’t used outside of it, and vice versa

    3. You want to time a region of your function. (That indicates that you that region meaningful as its own operation - it should be a separate function)

    4. There are large regions of your function inside a preprocessor flag

    5. Any code repetition

Source file length

  1. No firm rule on file length; they should be a logical unit of related code

  2. One class or struct per file a) Exception if they are extremely similar (e.g. SimdVector and SimdVector2D), in which case consider grouping them in a namespace

Testing

  1. All non-trivial new C++ functions must have an associated test a) Trivial would be e.g. double get_x(){return x;}

  2. We use GoogleTest, so #include <gtest/gtest.h> must be included.

  3. There are multiple “test sets” in our test suite

    1. Each one is its own executable and tests a single part of the code, either a kernel or a class/function.

    2. Each consists of multiple gtests (at least one per function).

  4. Add a new test set in CMakeLists.txt by adding its name to the list, and specifying the source files that need to be compiled, e.g.: set(scatter_SRCS XGC_core/cpp/cpp_scatter.cpp utils/cpp_unit_tests/electron_scatter_test.cpp) set(TESTS “push” “scatter”)

  5. Prefer to have your test check against analytic solutions rather than reference data; however, if the latter is necessary:

  6. If your test requires a lot of objects like grid and magnetic field, you can use the example in SmallExample/ and read it in with cpp_init_test_objects.hpp

  7. If your test requires an additional large object, or needs to check results against saved reference data:

    1. Add that new data to SmallExample

    2. Load your new SmallExample to the kitware site

    3. Update SmallExample.tar.gz.sha512 with the new sha512.

  8. There is some existing infrastructure to generate a correct answer to your test as reference and save it to file:

    1. Build the tests with the UPDATE_TESTS flag. Rather than checking against *.exp.txt with a tolerance of *.tol.txt, it will generate *.exp.txt if on CPU, and generate *.tol.txt (as the diff between the GPU results and the CPU results) if on GPU.

    2. You can also edit these manually.

Pull requests

  1. Pull requests (PRs) should be:

    1. Single topic

    2. Relatively small (<200 lines)

    3. Submitted with a short summary of what the changes do

    4. Submitted after running a Python script to automatically update the preprossor macro and input parameter documention. For example, from the XGC-Devel/Docs directory:

    module load gcc
python update_docs.py

    1. Merged using the Squash and merge option on github.com.