2022/05/05 CAM FV3 install on UM GreatLakes system

Reconfiguring the build system to work with the new CMake CIME infrastructure

Unchanged xml machine file:

~/.cime/config_machines.xml

<?xml version="1.0"?>
<!-- This is an ordered list, not all fields are required, optional fields are noted below. -->
<config_machines version="2.0">
<!-- MACH is the name that you will use in machine options -->

  <machine MACH="greatlakes">

    <!-- DESC: a text description of the machine, this field is current not used in code-->
    <DESC>UMich Great Lakes cluster, os is Linux, 36 PES/node, batch system is SLURM, GPFS on /scratch</DESC>

    <!-- NODENAME_REGEX: a regular expression used to identify this machine
	  it must work on compute nodes as well as login nodes, use machine option
	  to create_test or create_newcase if this flag is not available -->
    <NODENAME_REGEX>gl.*\.arc-ts\.umich.edu</NODENAME_REGEX>

    <!-- OS: the operating system of this machine. Passed to cppflags for
	 compiled programs as -DVALUE  recognized are LINUX, AIX, Darwin, CNL -->
    <OS>LINUX</OS>

    <!-- PROXY: optional http proxy for access to the internet-->
    <PROXY>  </PROXY>

    <!-- COMPILERS: compilers supported on this machine, comma seperated list, first is default -->
    <COMPILERS>intel,gnu</COMPILERS>

    <!-- MPILIBS: mpilibs supported on this machine, comma seperated list,
	     first is default, mpi-serial is assumed and not required in this list-->
    <MPILIBS>openmpi</MPILIBS>

    <!-- PROJECT: A project or account number used for batch jobs
         This value is used for directory names. If different from
         actual accounting project id, use CHARGE_ACCOUNT
	 can be overridden in environment or $HOME/.cime/config -->
    <!--PROJECT>my_cesm_project_placeholder</PROJECT-->

    <!-- CHARGE_ACCOUNT: A project or account number used for batch jobs
	 This is the actual project used for cost accounting set in
         the batch script (ex. #PBS -A charge_account). Will default
         to PROJECT if not set.
	 can be overridden in environment or $HOME/.cime/config -->
    <!-- <CHARGE_ACCOUNT>$ENV{MY_SLURM_ACCOUNT}</CHARGE_ACCOUNT> -->

    <!-- SAVE_TIMING_DIR: (Acme only) directory for archiving timing output -->
    <!-- SAVE_TIMING_DIR> </SAVE_TIMING_DIR -->

    <!-- SAVE_TIMING_DIR_PROJECTS: (Acme only) projects whose jobs archive timing output -->
    <!-- SAVE_TIMING_DIR_PROJECTS> </SAVE_TIMING_DIR_PROJECTS -->

    <!-- CIME_OUTPUT_ROOT: Base directory for case output,
	 the case/bld and case/run directories are written below here -->
    <CIME_OUTPUT_ROOT>$ENV{MY_CESM_ROOT}/output/$ENV{CESM_GROUP}</CIME_OUTPUT_ROOT>

    <!-- DIN_LOC_ROOT: location of the inputdata data directory
	 inputdata is downloaded automatically on a case by case basis as
	 long as the user has write access to this directory.   We recommend that
	 all cime model users on a system share an inputdata directory
	 as it can be quite large -->
    <DIN_LOC_ROOT>$ENV{DIN_LOC_ROOT}</DIN_LOC_ROOT>

    <!-- DIN_LOC_ROOT_CLMFORC: override of DIN_LOC_ROOT specific to CLM
	 forcing data -->
    <DIN_LOC_ROOT_CLMFORC>$ENV{MY_CESM_ROOT}/shared_lmwg</DIN_LOC_ROOT_CLMFORC>

    <!-- DOUT_S_ROOT: root directory of short term archive files, short term
      archiving moves model output data out of the run directory, but
      keeps it on disk-->
    <DOUT_S_ROOT>$CIME_OUTPUT_ROOT/archive/$CASE</DOUT_S_ROOT>

    <!-- BASELINE_ROOT:  Root directory for system test baseline files -->
    <BASELINE_ROOT>$ENV{MY_CESM_ROOT}/cesm_baselines</BASELINE_ROOT>

    <!-- CCSM_CPRNC: location of the cprnc tool, compares model output in testing-->
    <CCSM_CPRNC>$ENV{MY_CESM_ROOT}/cesm2.1.3/cime/tools/cprnc/</CCSM_CPRNC>

    <!-- GMAKE: gnu compatible make tool, default is 'gmake' -->
    <GMAKE>gmake</GMAKE>

    <!-- GMAKE_J: optional number of threads to pass to the gmake flag -->
    <GMAKE_J>8</GMAKE_J>

    <!-- BATCH_SYSTEM: batch system used on this machine,
      supported values are: none, cobalt, lsf, pbs, slurm -->
    <BATCH_SYSTEM>slurm</BATCH_SYSTEM>

    <!-- SUPPORTED_BY: contact information for support for this system
      this field is not used in code -->
    <SUPPORTED_BY>arcts-support@umich.edu</SUPPORTED_BY>

    <!-- MAX_TASKS_PER_NODE: maximum number of threads*tasks per
	 shared memory node on this machine,
	 should always be >= MAX_MPITASKS_PER_NODE -->
    <MAX_TASKS_PER_NODE>36</MAX_TASKS_PER_NODE>

    <!-- MAX_MPITASKS_PER_NODE: number of physical PES per shared node on
	 this machine, in practice the MPI tasks per node will not exceed this value -->
    <MAX_MPITASKS_PER_NODE>36</MAX_MPITASKS_PER_NODE>

    <!-- PROJECT_REQUIRED: Does this machine require a project to be specified to
	 the batch system?  See PROJECT above -->
    <PROJECT_REQUIRED>FALSE</PROJECT_REQUIRED>

    <!-- mpirun: The mpi exec to start a job on this machine, supported values
	 are values listed in MPILIBS above, default and mpi-serial -->
    <mpirun mpilib="openmpi">
      <!-- name of the exectuable used to launch mpi jobs -->
      <executable>mpiexec</executable>
      <!-- arguments to the mpiexec command, the name attribute here is ignored-->
      <arguments>
	<arg name="num_tasks">-n </arg>
      </arguments>
    </mpirun>
    <mpirun mpilib="mpi-serial">
      <executable></executable>
    </mpirun>
    <!-- module system: allowed module_system type values are:
	 module  http://www.tacc.utexas.edu/tacc-projects/mclay/lmod
	 soft http://www.mcs.anl.gov/hs/software/systems/softenv/softenv-intro.html
         none
      -->
    <module_system type="module" allow_error="true">
      <init_path lang="perl">/sw/arcts/centos7/lmod/lmod/init/perl</init_path>
      <init_path lang="python">/sw/arcts/centos7/lmod/lmod/init/env_modules_python.py</init_path>
      <init_path lang="csh">/sw/arcts/centos7/lmod/lmod/init/csh</init_path>
      <init_path lang="sh">/sw/arcts/centos7/lmod/lmod/init/sh</init_path>

      <cmd_path lang="perl">/sw/arcts/centos7/lmod/lmod/libexec/lmod perl</cmd_path>
      <cmd_path lang="python">/sw/arcts/centos7/lmod/lmod/libexec/lmod python</cmd_path>
      <cmd_path lang="sh">module</cmd_path>
      <cmd_path lang="csh">module</cmd_path>

      <modules compiler="gnu">
        <command name="purge"></command>
        <command name="load">gcc/8.2.0</command>
        <!-- <command name="load">openmpi/4.0.2</command> --> <!-- Does not work on GL! -->
        <command name="load">openmpi/3.1.4</command> <!-- Has bugs openmpi/3.1.4 but seems to work! -->
        <command name="load">cmake/3.13.2</command>
        <command name="load">hdf5/1.8.21</command>
        <command name="load">netcdf-c/4.6.2</command>
        <command name="load">netcdf-fortran/4.4.5</command>
      </modules>
      <modules compiler="intel">
        <command name="purge"></command>
        <!-- <command name="load">intel/19.1</command> -->
        <command name="load">intel/18.0.5</command>
        <!-- <command name="load">intel/14.0.2</command> -->
        <!-- <command name="load">openmpi/4.0.2</command> --> <!-- Does not work on GL! -->
        <command name="load">openmpi/3.1.4</command> <!-- Has bugs openmpi/3.1.4 but seems to work! -->
        <command name="load">cmake/3.13.2</command>
        <command name="load">hdf5/1.8.21</command>
        <command name="load">netcdf-c/4.6.2</command>
        <command name="load">netcdf-fortran/4.4.5</command>
      </modules>
    </module_system>

    <!-- environment variables, a blank entry will unset a variable -->
    <environment_variables>
      <env name="NETCDF_C_PATH">$ENV{NCDIR}</env>
      <env name="NETCDF_FORTRAN_PATH">$ENV{NFDIR}</env>
      <env name="LD_LIBRARY_PATH">$ENV{LD_LIBRARY_PATH}</env>
      <env name="OMP_STACKSIZE">256M</env>
      <!--env name="MPI_TYPE_DEPTH">16</env-->
    </environment_variables>
    <!-- resource settings as defined in https://docs.python.org/2/library/resource.html -->
    <resource_limits>
      <resource name="RLIMIT_STACK">-1</resource>
    </resource_limits>
  </machine>

</config_machines>

New CMake file:

~/.cime/greatlakes.cmake

string(APPEND CFLAGS " -qno-opt-dynamic-align -fp-model precise -std=gnu99 -L$ENV{NC_ROOT}/lib -lnetcdf -L$ENV{NF_ROOT}/lib -lnetcdff")
if (compile_threaded)
  string(APPEND CFLAGS " -qopenmp")
endif()
if (NOT DEBUG)
  string(APPEND CFLAGS " -O2 -debug minimal -xCORE-AVX2")
endif()
if (DEBUG)
  string(APPEND CFLAGS " -O0 -g")
endif()
string(APPEND CPPDEFS " -DFORTRANUNDERSCORE -DCPRINTEL")
string(APPEND CXX_LDFLAGS " -cxxlib")
set(CXX_LINKER "FORTRAN")
set(FC_AUTO_R8 "-r8")
string(APPEND FFLAGS " -qno-opt-dynamic-align  -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model source -L$ENV{NC_ROOT}/lib -lnetcdf -L$ENV{NF_ROOT}/lib -lnetcdff")
if (compile_threaded)
  string(APPEND FFLAGS " -qopenmp")
endif()
if (DEBUG)
  string(APPEND FFLAGS " -O0 -g -check uninit -check bounds -check pointers -fpe0 -check noarg_temp_created")
endif()
if (NOT DEBUG)
  string(APPEND FFLAGS " -O2 -debug minimal -xCORE-AVX2")
endif()
set(FFLAGS_NOOPT "-O0")
set(FIXEDFLAGS "-fixed")
set(FREEFLAGS "-free")
if (compile_threaded)
  string(APPEND LDFLAGS " -qopenmp")
endif()
set(MPICC "mpicc")
set(MPICXX "mpicxx")
set(MPIFC "mpif90")
set(SCC "icc")
set(SCXX "icpc")
set(SFC "ifort")
set(NETCDF_C_PATH $ENV{NETCDF_C_PATH})
set(NETCDF_FORTRAN_PATH $ENV{NETCDF_FORTRAN_PATH})
if (MPILIB STREQUAL mpich)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL mpich2)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL mvapich)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL mvapich2)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL mpt)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL openmpi)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL impi)
  string(APPEND SLIBS " -mkl=cluster")
endif()
if (MPILIB STREQUAL mpi-serial)
  string(APPEND SLIBS " -mkl")
endif()
set(SUPPORTS_CXX "TRUE")
set(HAS_F2008_CONTIGUOUS "FALSE")

The main stumbling block here is that it relies on the environment variables $NETCDF_C_PATH and $NETCDF_FORTRAN_PATH which are set in the XML file above. Note that these are distinct directories. This will be important later.

Dealing with an unresolved ESMF dependency:

If I follow my usual case creation script, then I do the following

cd ${CASE_DIR} #set in the script above, or use your own
source bash.source
bash xml_change_helper.sh

which all succeed. However, when I run ./case.build I get the following error message:

   Calling /home/owhughes/cesm_src/2022_05_05_FV3/components/cmeps/cime_config/buildnml
ERROR: ESMFMKFILE not found None

There is likely a more elegant way to fix this, but I fixed this by downloading the ESMF 8.1.1 source code and building it using the following script

build_esmf.sh

module load intel/18.0.5
module load openmpi/3.1.4
export ESMF_LIB=`pwd`
export ESMF_CXX=mpicxx
export ESMF_COMM=openmpi
export ESMF_COMPILER=intel
export ESMF_F90=mpif90
make -j8

and run this script in the root directory of the ESMF source (which for me is /home/owhughes/esmf/esmf-ESMF_8_1_1).

Once this is built I run find ~+ -name "esmf.mk" (note: this uses a bash-specific idiom). For me this returns /home/owhughes/esmf/esmf-ESMF_8_1_1/lib/libO/Linux.intel.64.openmpi.default/esmf.mk.

Once I add export ESMFMKFILE="/home/owhughes/esmf/esmf-ESMF_8_1_1/lib/libO/Linux.intel.64.openmpi.default/esmf.mk" to my ${CASE_DIR}/bash.source file, and run source bash.source, this resolves the error for this section.

Fixing a bug in the FMS Makefile.cesm

Once the previous modifications are made, I get the following error messages when I attempt to run ./case.build

Error log

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(316): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(317): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_DATA]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(318): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(319): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF4_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6404: This name does not have a type, and must have an explicit type.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6611: The case-value must be of the same type as the case-expr.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6404: This name does not have a type, and must have an explicit type.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6611: The case-value must be of the same type as the case-expr.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

  errput: ERROR: /home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(316): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(317): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_DATA]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(318): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(319): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF4_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6404: This name does not have a type, and must have an explicit type.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6611: The case-value must be of the same type as the case-expr.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6404: This name does not have a type, and must have an explicit type.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6611: The case-value must be of the same type as the case-expr.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

ERROR: FMS build Failed ERROR: /home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(316): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(317): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_DATA]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(318): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/mpp_io.F90(319): error #7002: Error in opening the compiled module file.  Check INCLUDE paths.   [NETCDF4_NF_INTERFACES]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6404: This name does not have a type, and must have an explicit type.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(653): error #6611: The case-value must be of the same type as the case-expr.   [NF_INT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6404: This name does not have a type, and must have an explicit type.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(655): error #6611: The case-value must be of the same type as the case-expr.   [NF_FLOAT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6404: This name does not have a type, and must have an explicit type.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(657): error #6611: The case-value must be of the same type as the case-expr.   [NF_DOUBLE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_BYTE]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6601: In a CASE statement, the case-value must be a constant expression.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6612: In a CASE statement, the case-value must be of type INTEGER, CHARACTER, or LOGICAL.   [NF_SHORT]

/home/owhughes/cesm_src/2022_05_05_FV3/libraries/FMS/src/mpp/include/mpp_io_util.inc(667): error #6611: The case-value must be of the same type as the case-expr.   [NF_SHORT]

cam built in 3.614217 seconds
ERROR: BUILD FAIL: cam.buildlib failed, cat /scratch/cjablono_root/cjablono1/owhughes/CESM_ROOT/output/test_cases/2022_05_05_FV3.C96_C96_mg17.FADIAB.36.test_cases.jw06/bld/atm.bldlog.220511-151500

This makes it clear that the FMS build infrastructure is unable to locate the include directories for NetCDF-c and NetCDF-fortran. The problem lies in the file ${CESM_SRC}/libraries/FMS/Makefile.cesm on lines 175 and 176:

FFLAGS += -I$(INC_NETCDF)
CFLAGS += -I$(INC_NETCDF)

The rest of this makefile makes it clear that the correct conditionals are used so that the SLIBS variable is set correctly if NETCDF_C_PATH and NETCDF_FORTRAN_PATH are set to distinct locations. Indeed, in the case that the install locations differ the INC_NETCDF_C and INC_NETCDF_FORTRAN variables will be set correctly. However, unlike SLIBS, CFLAGS and FFLAGS will always append INC_NETCDF even if this variable is unset and INC_NETCDF_C and INC_NETCDF_FORTRAN should be used instead.

The simplest solution I've found to this problem is to replace the two problem lines in FMS/Makefile.cesm with the following code:

ifdef INC_NETCDF_FORTRAN
  FFLAGS += -I$(INC_NETCDF_FORTRAN)
else
  FFLAGS += -I$(INC_NETCDF)
endif
ifdef INC_NETCDF_C
  CFLAGS += -I$(INC_NETCDF_C)
else
  CFLAGS += -I$(INC_NETCDF)
endif

The code just above these modified lines handles the case where e.g. NETCDF_C_PATH is set but NETCDF_FORTRAN_PATH and NETCDF_PATH are not set (and throws appropriate errors). I therefore don't bother doing additional error checking here.

Once these modifications are made, the FV3 test case builds and runs without a hitch.

Parent post:

Community Earth System Model lifehacks

Descendent posts:

Case creation script