Cache Windows compilations with sccache - #899
Draft
mjcarroll wants to merge 1 commit into
Draft
Conversation
mjcarroll
force-pushed
the
mjcarroll/windows-compiler-cache-v2
branch
from
September 9, 2026 21:45
2b22f6f to
c49f32d
Compare
The Linux jobs have had ccache since #135, by way of the /usr/lib/ccache symlinks that masquerade as the compilers on PATH. Windows had nothing: there is no equivalent masquerade, so the cache has to be named to CMake through CMAKE_<LANG>_COMPILER_LAUNCHER instead, which is why the Ninja generator had to come first -- CMake accepts that setting under the Visual Studio generator and then ignores it. Export the launchers rather than passing them as -D on the colcon command line. CMake initialises CMAKE_<LANG>_COMPILER_LAUNCHER from the environment variable of the same name, so an exported value also reaches the nested configures that ament_vendor and ExternalProject_Add run. ament_vendor forwards a fixed allow-list of variables to its sub-builds through a -C initial-cache file, and the launchers are not on it, so with -D alone every vendored package would keep compiling uncached -- which is most of the expensive C++ in the workspace. CMAKE_<LANG>_COMPILER_LAUNCHER reaches only the C and C++ compilers, so export RUSTC_WRAPPER too: zenoh_cpp_vendor builds several hundred crates through cargo. CARGO_INCREMENTAL goes off alongside it because sccache refuses to cache incremental compilation. CMAKE_TRY_COMPILE_CONFIGURATION goes to Release because try_compile() otherwise builds Debug, which on MSVC is '/Zi /Od /RTC1', and sccache will not cache /Zi. The cache directory is .sccache beside the workspace -- inside the Jenkins workspace, which is bind mounted into the container from the agent, and of which run() removes only the 'ws' subdirectory. So it outlives the container, and since every Jenkins job has a workspace of its own, no two jobs share one: ccache locks its cache and is safe to share, whereas each sccache server holds its index in memory, so two servers over one directory evict each other's entries. That is deliberately not a mount in ci_job.xml.em. A Jenkins job's build steps live in Jenkins, written there by create_jenkins_job.py, so a template change reaches a running job only once somebody pushes the job configuration; CI_SCRIPTS_BRANCH swaps the checked-out repository, not the job. An earlier attempt at the mount reported Using sccache with SCCACHE_DIR='<unset>' and SCCACHE_CACHE_SIZE='8G' with no cache mount on the docker run line -- the cache was working and then being thrown away with the container on every build. Choosing the directory here instead takes effect as soon as CI_SCRIPTS_BRANCH points at this branch, and wiping the workspace is then a reasonable way to ask for a cold build. sccache is looked up rather than assumed, and a missing one is a warning and an uncached build rather than a failed job. That guard is in _setup_compiler_cache() rather than in pre(), so that it cannot also skip the MAX_PATH work, which Ninja needs whether or not there is a cache. The packaging jobs get no cache. They stay on the Visual Studio generator, and they build RelWithDebInfo, which on MSVC means /Zi, which sccache will not cache in any case. sccache comes from the buildfarm environment in ros2/ros2#1854.
mjcarroll
force-pushed
the
mjcarroll/windows-ninja
branch
from
September 9, 2026 21:57
fb3ad92 to
70e12c9
Compare
mjcarroll
force-pushed
the
mjcarroll/windows-compiler-cache-v2
branch
from
September 9, 2026 21:57
c49f32d to
a929858
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of two, stacked on #900. Review that first — this shows only the cache commit.
Linux has had ccache since #135 via the
/usr/lib/ccachesymlinks onPATH. Windows has no equivalent, so the cache is named to CMake throughCMAKE_<LANG>_COMPILER_LAUNCHER, which works only under Ninja — hence the split.The launchers are exported rather than passed as
-D. CMake initialisesCMAKE_<LANG>_COMPILER_LAUNCHERfrom the environment variable of the same name, so an exported value also reaches the nested configuresament_vendorandExternalProject_Addrun.ament_vendorforwards a fixed allow-list through a-Cinitial-cache file and the launchers are not on it, so with-Dalone every vendored package would compile uncached — most of the expensive C++ in the workspace.RUSTC_WRAPPERcovers cargo inzenoh_cpp_vendor;CARGO_INCREMENTALandCMAKE_TRY_COMPILE_CONFIGURATIONare set because sccache caches neither incremental compilation nor/Zi.The cache lives in
.sccachebeside the workspace — inside the bind-mounted Jenkins workspace, of whichrun()removes onlyws. So it outlives the container, and since each job has its own workspace no two share one: ccache locks its cache and is safe to share, sccache keeps its index in memory and two servers over one directory evict each other. This deliberately is not a mount inci_job.xml.em, because a job's build steps live in Jenkins and a template change only lands when someone pushes the job config;CI_SCRIPTS_BRANCHswaps the repository, not the job.A missing sccache is a warning and an uncached build, not a failure. Packaging gets no cache: Visual Studio generator, and RelWithDebInfo means
/Zi.Blocked on ros2/ros2#1854 for
sccache, and on gazebosim/gz-cmake#586 —gz_math_vendorandgz_utils_vendorare the only packages emitting/Ziunder Release, and under sccache they fail to compile withC1041rather than merely missing. That needs a gz-cmake release and agz_cmake_vendorbump, which is why this stays draft while #900 does not.Each build prints
sccache --show-statsbefore and after; watchNon-cacheable compilationsandCache errorsas well as hit rate.