Configuring Caelus Python Library

CPL provides a YAML-based configuration utility that can be used to customize the library depending on the operating system and user’s specific needs. A good example is to provide non-standard install locations for the OpenFOAM or Caelus CML executables, as well as using different versions of OpenFOAM/CML with CPL simultaneously.

The use of configuration file is optional, CPL provides defaults that should work on most systems and will attempt to auto-detect CML installations on standard paths. On Linux/OS X systems, CPL will look at ~/Caelus/caelus-VERSION to determine the installed CML versions and use the VERSION tag to determine the latest version to use. On Window systems, the default search path is C:\Caelus.

Upon invocation, CPL will search and load configuration files from the following locations, if available. The files are loaded in sequence shown below and options found in succeeding files will overwrite configuration options found in preceeding files.

  1. Default configuration supplied with CPL;

  2. The system-wide configuration in file pointed by environment variable CAELUSRC_SYSTEM if it exists;

  3. The per-user configuration file, if available. On Linux/OS X, this is the file ~/.caelus/caelus.yaml, and %APPDATA%/caelus/caelus.yaml on Windows systems;

  4. The per-user configuration file pointed by the environment variable CAELUSRC if it exists;

  5. The file caelus.yaml in the current working directory, if it exists.

While CPL provides a way to auto-discover installed OpenFOAM and CML versions, often it will be necessary to provide at least a system-wide or per-user configuration file to allow CPL to use the right CML executables present in your system. A sample CPL configuration is shown below download caelus.yaml:

# -*- mode: yaml -*-
#
# Sample CPL configuration file
#

# Root CPL configuration node
caelus:
  # Control logging of CPL library
  logging:
    log_to_file: true
    log_file: ~/Caelus/cpl.log

  # Configuration for Caelus CML or OpenFOAM versions
  caelus_cml:
    # Pick the development version of CML available; use "latest" to choose the
    # latest version available.
    default: "v2012"

    # Versions that can be used with CPL
    versions:
      - version: "9.04"
        path: ~/Caelus/caelus-9.04

      - version: "10.04"
        path: ~/Caelus/caelus-10.04

      - version: "dev-gcc"
        path: ~/Caelus/caelus-cml            # Use latest git repository
        mpi_path: /usr/local/openmpi         # Use system OpenMPI
        build_option: "linux64gcc++DPOpt"    # Use the GCC version

      - version: "v2012"
        path: ~/OpenFOAM/OpenFOAM-v2012      # Use OpenFOAM
        mpi_path: /usr/local/openmpi         # Use system OpenMPI

The above configuration would be suitable as as a system-wide or per-user configuration stored in the home directory, and the user can override specific options used for particular runs by using, for example, the following caelus.yaml within the case directory:

# Local CPL settings for this working directory
caelus:
  logging:
    log_file: cpl_dev.log  # Change log file to a local file

  caelus_cml:
    default: "dev-gcc"     # Use the latest dev version for this run

Note that only options that are being overridden need to be specified. Other options are populated from the system-wide or per-user configuration file if they exist.

Checking current configuration

To aid debugging and troubleshooting, CPL provides a command caelus cfg to dump the configuration used by the library based on all available configuration files. A sample usage is shown here:

 1$ caelus -v cfg
 2DEBUG: Loaded configuration from files = ['/home/caelus/.caelus/caelus.yaml']
 3INFO: Caelus Python Library (CPL) v0.1.0
 4# -*- mode: yaml -*-
 5#
 6# Caelus Python Library (CPL) v0.1.0
 7#
 8# Auto-generated on: 2018-04-21 17:03:35 (UTC)
 9#
10
11caelus:
12  cpl:
13    python_env_type: conda
14    python_env_name: caelus
15    conda_settings:
16      conda_bin: ~/anaconda/bin
17  system:
18    job_scheduler: local_mpi
19    always_use_scheduler: false
20    scheduler_defaults:
21      join_outputs: true
22      shell: /bin/bash
23      mail_opts: NONE
24  logging:
25    log_to_file: true
26    log_file: null
27  caelus_cml:
28    default: latest
29    versions: []

The final configuration after parsing all available configuration files is shown in the output. If the user provides -v (verbose) flag, then the command also prints out all the configuration files that were detected and read during the initialization process. Users can also use caelus cfg to create a configuration file with all the current settings using the -f option. Please see caelus command documentation for details.

CPL configuration reference

CPL configuration files are in YAML format and must contain at least one node caelus. Two other optional nodes can be present in the file, caelus_scripts and caelus_user whose purpose is described below.

caelus

The root YAML node containing the core CPL configuration object. This node contains all configuration options used internally by the library.

caelus_scripts

An optional node used to store configuration for CPL CLI apps.

caelus_user

An optional node node reserved for user scripts and applications that will be built upon CPL.

Note

In the following sections, the configuration parameters are documented in the format root_note.sub_node.config_parameter. Please see the sample configuration file above for the exact nesting structure used for caelus.logging.log_file.

Core library configuration

Python environment options

caelus.cpl

This section contains options to configure the python environment (either Anaconda/Conda environment or virtualenv settings).

caelus.cpl.python_env_type

Type of python environment. Currently this can be either conda or virtualenv.

caelus.cpl.python_env_name

The name of the Python environment for use with CPL, e.g., caelus or caelus-dev.

caelus.cpl.conda_settings

Extra information for Conda installation on your system.

System configuration

caelus.system

This section provides CPL with necessary information on the system settings, particularly the queue configuration on HPC systems.

caelus.system.job_scheduler

The type of job-scheduler available on the system and used by CPL when executing CML executables on the system. By default, all parallel jobs will use the job scheduler, user can configure even serial jobs (e.g., mesh generation, domain decomposition and reconstruction) be submitted on queues.

Name

Description

local_mpi

No scheduler, submit locally

slurm

Use SLURM commands to submit jobs
caelus.system.always_use_scheduler

A Boolean flag indicating whether even serial jobs (e.g., mesh generation) should use the queue system. This flag is useful when the user intends to generate large meshes and requires access to the high-memory compute nodes on the HPC system.

caelus.system.scheduler_defaults

This section contains options that are used by default when submitting jobs to an HPC queue system.

Option

Description

queue

Default queue for submitting jobs

account

Account for charging core hours

stdout

Default file pattern for redirecting standard output

stdout

Default file pattern for redirecting standard error

join_outputs

Join stdout and stderr (queue specific)

mail_options

A string indicating mail options for queue

email_address

Address where notifications should be sent

time_limit

Wall clock time limit

machinefile

File used in mpirun -machinefile <FILE>

Note

Currently, these options accept strings and are specific to the queue system (e.g., SLURM or PBS Torque). So the user must consult their queue system manuals for appropriate values to these options.

CPL logging options

caelus.logging

This section of the configuration file controls the logging options for the CPL library. By default, CPL only outputs messages to the standard output. Users can optionally save all messages from CPL into a log file of their choice. This is useful for tracking and troubleshooting, or providing additional information regarding bugs observed by the user.

Internally, CPL uses the logging module. For brevity, messages output to console are usually at log levels INFO or higher. However, all messages DEBUG and above are captured in log files.

caelus.logging.log_to_file

A Boolean value indicating whether CPL should output messages to the log file. The default value is false. If set to true, then the log messages will also be saved to the file indicated by log_file as well as output to the console.

caelus.logging.log_file

Filename where the log messages are saved if log_to_file evaluates to True.

CML version configuration

caelus.caelus_cml

The primary purpose of CPL is to interact with OpenFOAM and CML executables and utilities. This section informs CPL of the various OpenFOAM/CML installations available on a system and the desired version used by CPL when invoking OpenFOAM or CML executables.

caelus.caelus_cml.default

A string parameter indicating default version used when invoking OpenFOAM or CML executables. It must be one of the version entries provided in the file. Alternately, the user can specify latest to indicate that the latest version must be used. If users rely on auto-discovery of Caelus versions in default install locations, then it is recommended that this value be latest so that CPL picks the latest CML version. For example, with the following configuration, CPL will choose version 10.04 when attempting to execute programs like pisoSolver.

caelus:
  caelus_cml:
    default: "latest"

    versions:
      - version: "9.04"
        path: ~/Caelus/caelus-9.04

      - version: "10.04"
        path: ~/Caelus/caelus-10.04

      - version: "v2012"
        path: ~/OpenFOAM/OpenFOAM-v2012
caelus.caelus_cml.versions

A list of configuration mapping listing various versions available for use with CPL. It is recommended that the users only provide version and path entries, the remaining entries are optional. CPL will auto-detect remaining parmeters.

caelus.caelus_cml.versions.version

A unique string identifier that is used to tag this specific instance of OpenFOAM or CML installation. Typically, this is the version number of the OpenFOAM or Caelus CML release, e.g., 10.04. However, as indicated in the example CPL configuration file, users can use any unique tag to identify a specific version. If this identifier does not follow the conventional version number format, then it is recommended that the user provide a specific version in caelus.caelus_cml.default instead of using latest.

caelus.caelus_cml.versions.path

The path to the C++ code/executables install. This is equivalent to the directory pointed by the WM_PROJECT_DIR or CAELUS_PROJECT_DIR environment variable, e.g., /home/caelus_user/projects/caelus/caelus-10.04.

caelus.caelus_cml.versions.build_option

A string parameter identifying the OpenFOAM or Caelus build, if multiple builds are present within a CML install, to be used with CPL. This is an expert only option used by developers who are testing multiple compilers and build options. It is recommended that the normal users let CPL autodetect the build option.

caelus.caelus_cml.versions.mpi_root

Path to the MPI installation used to compile OpenFOAM or Caelus for parallel execution. By default, CPL expects the MPI library to be present within the project directory.

caelus.caelus_cml.versions.mpi_bin_path

Directory containing MPI binaries used for mpiexec when executing in parallel mode. If absent, CPL will assume that the binaries are located within the subdirectory bin in the path pointed by mpi_root.

caelus.caelus_cml.versions.mpi_lib_path

Directory containing MPI libraries used for mpiexec when executing in parallel mode. If absent, CPL will assume that the libraries are located within the subdirectory lib in the path pointed by mpi_root.