Skip to content

DEBUG build doesn't disable optimization or enable runtime checks for ICON (Intel, [GNU]) #137

Description

@s-poll

Problem

In cmake/BuildICON.cmake, the DEBUG branch only prepends -g:

if (CMAKE_BUILD_TYPE STREQUAL "DEBUG")
  string(PREPEND ICON_CFLAGS "-g ")
  string(PREPEND ICON_FCFLAGS "-g ")

Under the Intel compilers (ifort/ifx), the default optimization level is -O2, and since no -O flag is ever specified in this branch, -g is added on top of the implicit -O2 rather than replacing it. So a "DEBUG" build gives debug symbols but keeps full optimization, and enables none of Intel's runtime diagnostics (bounds checking, uninitialized-variable checks, floating-point exception trapping).

Proposed change

Add to the DEBUG branch for the Intel/IntelLLVM case:

-O0 -check bounds -check uninit -fpe0 -init=snan

so that CMAKE_BUILD_TYPE=DEBUG actually disables optimization and turns on array-bounds checking, uninitialized-variable checking, floating-point exception trapping, and NaN-initialization of otherwise-uninitialized memory.

Notes / caveats

  • These flags materially slow down execution, so this should stay scoped to the DEBUG build type only (RELEASE is unaffected).
  • Checked whether the GNU env has a comparable gap:
    • The optimization half does not apply to GNU because of gfortran's default (no -O flag) is identical to explicit -O0 (confirmed via gfortran -Q --help=optimizers), so DEBUG's bare -g prepend doesn't silently leave optimization on the way it does under Intel.
    • The runtime-checks half does apply equally: there is no -fcheck=bounds, -ffpe-trap, -finit-real=snan, or -finit-integer=... anywhere in cmake/. The GNU FCFLAGS has warnings (-Wall, -Wconversion, …) and -fbacktrace but no runtime bounds/uninitialized/FPE checking. Worth adding the GNU equivalents (e.g. -fcheck=bounds -ffpe-trap=invalid,zero,overflow -finit-real=snan) to the DEBUG branch alongside the Intel fix.
  • This was found as a side effect of investigating output differences; that investigation showed ICON is not bit-reproducible run-to-run in this MPI/OpenMP configuration even with an unmodified binary.

Activity

  1. kvrigor commented on Aug 7, 2026

    @kvrigor
    Member

    @s-poll I checked those flags you proposed from the Intel Fortran compiler guide:

    • -check bounds: Checks for array subscript and character substring expressions.
    • -check uninit: Checks for uninitialized variables. (Linux only)
    • -fpe0: Floating-point invalid, divide-by-zero, and overflow exceptions are enabled throughout the application when the main program is compiled with this value. If any such exceptions occur, execution is aborted. This option causes subnormal floating-point results to be set to zero. Underflow results will also be set to zero, unless you override this by explicitly specifying option -no-ftz or -fp-model precise (Linux*) or option /Qftz- or /fp:precise (Windows*).
    • -init=snan: Determines whether the compiler initializes to signaling NaN all uninitialized variables of intrinsic type REAL or COMPLEX that are saved, local, automatic, or allocated variables.

    All these flags pertain to program correctness; thus I think it makes more sense to add these flags on both DEBUG and RELEASE builds. This can be done by updating ICON_FCFLAGS:

    set(ICON_FCFLAGS "-gdwarf-4 -march=native -pc64 -fp-model source -traceback -qno-opt-dynamic-align -no-fma")

    Then for DEBUG configuration only -g O0 needs to be added.

  2. s-poll commented on Aug 7, 2026

    @s-poll
    MemberAuthor

    Thanks for adding the overview.

    Especially the check_bounds and the check_uninit produce runtime overhead. That is why i currently just have added these to DEBUG as in PR #138

  3. s-poll commented on Aug 7, 2026

    @s-poll
    MemberAuthor

    I needed to remove `check uninit' flag on JURECA-DC

    Here the reasoning by LLM:

    On ifx 2024.2 (the LLVM-based Intel Fortran compiler used at JURECA default stage, as opposed to classic ifort), -check uninit is implemented via Clang's MemorySanitizer, not the old runtime-check mechanism. Passing it silently changes the link line: it pulls in for_main_msan.o, libclang_rt.msan, libifport_msan, libifcoremt_msan, etc., and wraps these (and neighboring libraries) in -Bstatic ... -Bdynamic toggles.

    ICON's link line mixes:

    static archives (OASIS3-MCT: libpsmile.MPI1, libmct, libmpeu, libscrip)
    dynamic system libraries (HDF5, libxml2, MKL, libz.so, ...)
    The -Bstatic/-Bdynamic toggles injected by the msan instrumentation leave the linker in static mode by the time it reaches libz.so, which has no static (.a) counterpart on this system so the link fails with attempted static link of dynamic object.

    This only breaks configure's minimal "does the Fortran compiler work" test (an empty program main / end linked against the full library set), so it looks like the compiler itself is broken, when actually it's this one debug flag interacting with the mixed static/dynamic library set.

    Confirmed by reproducing the exact link command from config.log locally and bisecting the DEBUG-only flags one at a time -check uninit reproduces the failure; the link succeeds cleanly without it.

  4. kvrigor commented on Aug 10, 2026

    @kvrigor
    Member

    Confirmed by reproducing the exact link command from config.log locally and bisecting the DEBUG-only flags one at a time -check uninit reproduces the failure; the link succeeds cleanly without it.

    Good catch; we've seen the same DEBUG build issues in #72 and #48. I agree dropping uninit and fix the broken DEBUG build flags separately.

  5. s-poll commented on Aug 10, 2026

    @s-poll
    MemberAuthor

    To build ICON-eCLM on JURECA the same issue with uninit is created at the eCLM build. By removing it was successfully (build and run). Should I also use this PR to remove the option form eCLM?

  6. s-poll commented on Aug 10, 2026

    @s-poll
    MemberAuthor

    I just recognized that the changes needs to be done in the eCLM repo; thus could not be part or this PR. But I will also create a PR at the eCLM repo reference to here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    build-scriptsIssues related to CMake and/or build_tsmp2.sh

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions