Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 61 additions & 18 deletions .github/workflows/coverage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,19 +11,30 @@ name: MSL coverage
# skipPhases and broken marks are removed, so every model goes through every
# phase and the documentation shows what really stops each one.
#
# Two MSL versions: 3.2.3 weekly (docs/COVERAGE.md, the README's coverage
# section, the site's root) and 4.1.0 every third day (docs/COVERAGE_4.1.0.md,
# its own README section, the site's msl-4.1.0/). OMFrontend bundles 3.2.3;
# 4.1.0 is installed on the runner from the MSL repository, as OpenModelica's
# package manager lays it out (~/.openmodelica/libraries).
#
# OM.jl and the sibling packages are checked out at the branch this workflow
# runs on (or `line`), as in OM.jl's omjl-ci.yml. The models' experiment
# settings come from models/experiments_Modelica_3.2.3.txt (there is no omc
# on the runner; record_experiments() refreshes the file).
# settings come from models/experiments_Modelica_<version>.txt (there is no
# omc on the runner; record_experiments() refreshes the file).
#
# Manual runs (Actions > MSL coverage > Run workflow) commit the documentation
# back and publish the run on the site when `commit` is set; the weekly run
# always does. A schedule only fires
# back and publish the run on the site when `commit` is set; the scheduled
# runs always do. A schedule only fires
# from the default branch, so it measures that branch's line.

on:
workflow_dispatch:
inputs:
msl:
description: MSL version
type: choice
options: ["3.2.3", "4.1.0"]
default: "3.2.3"
line:
description: OM.jl line to measure (empty for the branch the workflow runs on)
required: false
Expand All @@ -46,7 +57,8 @@ on:
required: false
default: false
schedule:
- cron: "0 2 * * 0" # Sundays 02:00 UTC
- cron: "0 2 * * 0" # MSL 3.2.3: Sundays 02:00 UTC
- cron: "0 14 */3 * *" # MSL 4.1.0: every third day of the month (1st, 4th, ...), 14:00 UTC

permissions:
contents: write # the documentation commit
Expand All @@ -58,12 +70,15 @@ concurrency:

jobs:
coverage:
name: MSL coverage, Julia ${{ inputs.julia || '1.13' }}
name: MSL ${{ inputs.msl || (github.event.schedule == '0 14 */3 * *' && '4.1.0') || '3.2.3' }} coverage, Julia ${{ inputs.julia || '1.13' }}
runs-on: ubuntu-latest
timeout-minutes: 360
env:
JULIA_PKG_PRECOMPILE_AUTO: "0"
LD_LIBRARY_PATH: ${{ github.workspace }}/OM.jl/OMRuntimeExternalC.jl/lib/ext/shared/x86_64-linux-gnu
# The MSL version: the input, 4.1.0 for the every-third-day schedule, else 3.2.3.
MSL: ${{ inputs.msl || (github.event.schedule == '0 14 */3 * *' && '4.1.0') || '3.2.3' }}
OMJL_COVERAGE_MSL: MSL:${{ inputs.msl || (github.event.schedule == '0 14 */3 * *' && '4.1.0') || '3.2.3' }}
steps:
- name: Check out OM.jl
uses: JKRT/OM.jl/.github/actions/checkout-siblings@master
Expand Down Expand Up @@ -106,20 +121,46 @@ jobs:
- name: Set up OMLibraryTesting
run: julia --color=yes --project=OM.jl/OMLibraryTesting.jl -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'

# The MAP-LIB reference results (1.3 GB) change with download_refs.sh only.
- name: The version's documentation
run: |
doc=docs/COVERAGE.md
[ "$MSL" = 3.2.3 ] || doc="docs/COVERAGE_$MSL.md"
echo "COVERAGE_MD=$doc" >> "$GITHUB_ENV"

# OMFrontend bundles MSL 3.2.3 and 4.0.0; another version is the installed
# library: Modelica, ModelicaServices and Complex (a single file in the
# repository, a folder with its package.mo when installed).
- name: Install MSL ${{ env.MSL }}
if: env.MSL != '3.2.3' && env.MSL != '4.0.0'
run: |
src="$RUNNER_TEMP/msl"
git clone -q --depth 1 --branch "v$MSL" --filter=blob:none --sparse \
https://github.com/modelica/ModelicaStandardLibrary.git "$src"
git -C "$src" sparse-checkout set Modelica ModelicaServices
libs="$HOME/.openmodelica/libraries"
mkdir -p "$libs/Complex $MSL"
cp -r "$src/Modelica" "$libs/Modelica $MSL"
cp -r "$src/ModelicaServices" "$libs/ModelicaServices $MSL"
cp "$src/Complex.mo" "$libs/Complex $MSL/package.mo"
ls "$libs"

# The MAP-LIB reference results (1.3 GB for 3.2.3, 2 GB for 4.1.0) change with
# download_refs.sh and the experiment lists only.
- name: Cache the reference results
id: refs
uses: actions/cache@v4
with:
path: |
OM.jl/OMLibraryTesting.jl/reference/csv
OM.jl/OMLibraryTesting.jl/reference/signals
key: msl-refs-3.2.3-${{ hashFiles('OM.jl/OMLibraryTesting.jl/reference/download_refs.sh') }}
OM.jl/OMLibraryTesting.jl/reference/${{ env.MSL }}
key: msl-refs-${{ env.MSL }}-${{ hashFiles('OM.jl/OMLibraryTesting.jl/reference/download_refs.sh', 'OM.jl/OMLibraryTesting.jl/models/experiments_Modelica_*.txt') }}

- name: Download the reference results
if: steps.refs.outputs.cache-hit != 'true'
working-directory: OM.jl/OMLibraryTesting.jl/reference
run: bash download_refs.sh
run: |
if [ "$MSL" = 3.2.3 ]; then bash download_refs.sh; else bash download_refs.sh --msl "$MSL"; fi

- name: Commits measured
run: |
Expand Down Expand Up @@ -148,18 +189,18 @@ jobs:
run: |
results=$(ls -t logs/partial_ci_*.jls | head -1)
julia --color=yes --project=. scripts/publish_coverage.jl "$results" "$GITHUB_WORKSPACE/commits.txt" "$NOTE"
cat docs/COVERAGE.md >> "$GITHUB_STEP_SUMMARY"
cat "$COVERAGE_MD" >> "$GITHUB_STEP_SUMMARY"

- name: Upload the reports
if: ${{ !cancelled() }}
uses: actions/upload-artifact@v4
with:
name: msl-coverage
name: msl-coverage-${{ env.MSL }}
path: |
commits.txt
OM.jl/OMLibraryTesting.jl/reports/
OM.jl/OMLibraryTesting.jl/logs/partial_ci_*.jls
OM.jl/OMLibraryTesting.jl/docs/COVERAGE.md
OM.jl/OMLibraryTesting.jl/${{ env.COVERAGE_MD }}
if-no-files-found: warn

- name: Commit the documentation
Expand All @@ -168,13 +209,14 @@ jobs:
run: |
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git add README.md docs/COVERAGE.md
git add README.md "$COVERAGE_MD"
if git diff --cached --quiet; then echo "Coverage unchanged"; exit 0; fi
git commit -m "Coverage of $(date -u +%Y-%m-%d) [skip ci]"
git commit -m "Coverage of $(date -u +%Y-%m-%d), MSL $MSL [skip ci]"
git push origin "HEAD:${GITHUB_REF_NAME}"

# The run on the coverage site: runs/<date>/, the index and the badges on
# the gh-pages branch (created on the first run), which Pages serves.
# the gh-pages branch (created on the first run), which Pages serves; the
# root for MSL 3.2.3, msl-<version>/ for another (site_folder).
- name: Publish the coverage site
if: steps.publish.outcome == 'success' && (github.event_name == 'schedule' || inputs.commit)
working-directory: OM.jl/OMLibraryTesting.jl
Expand All @@ -192,11 +234,12 @@ jobs:
fi
results=$(ls -t logs/partial_ci_*.jls | head -1)
report=$(ls -t reports/coverage_ci_*.html | head -1)
julia --color=yes --project=. scripts/publish_site.jl "$site" "$results" "$report" docs/COVERAGE.md "$GITHUB_WORKSPACE/commits.txt" "$NOTE"
julia --color=yes --project=. scripts/publish_site.jl "$site" "$results" "$report" "$COVERAGE_MD" "$GITHUB_WORKSPACE/commits.txt" "$NOTE"
cd "$site"
git add -A
if git diff --cached --quiet; then echo "Site unchanged"; exit 0; fi
git -c user.name="github-actions[bot]" -c user.email="41898282+github-actions[bot]@users.noreply.github.com" \
commit -q -m "Coverage of $(date -u +%Y-%m-%d)"
commit -q -m "Coverage of $(date -u +%Y-%m-%d), MSL $MSL"
git push origin gh-pages
echo "Coverage site: $OMJL_SITE_URL" >> "$GITHUB_STEP_SUMMARY"
folder=""; [ "$MSL" = 3.2.3 ] || folder="msl-$MSL/"
echo "Coverage site: $OMJL_SITE_URL$folder" >> "$GITHUB_STEP_SUMMARY"
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ logs/
# Reference CSVs and signals (large binary/text data, fetched by download_refs.sh)
reference/csv/
reference/signals/
reference/*/csv/
reference/*/signals/

# Editor and OS artifacts
.DS_Store
Expand Down
2 changes: 1 addition & 1 deletion Project.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
name = "OMLibraryTesting"
uuid = "4866894a-ca78-4bef-9a5c-0d35f2f81ce1"
version = "0.1.2"
version = "0.2.0"
authors = ["John Tinnerholm <john.tinnerholm@liu.se>"]

[deps]
Expand Down
34 changes: 25 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# OMLibraryTesting.jl

[![MSL coverage](https://img.shields.io/endpoint?url=https%3A%2F%2Fopenmodelica.github.io%2FOMLibraryTesting.jl%2Fbadge.json)](https://openmodelica.github.io/OMLibraryTesting.jl/)
[![MSL 4.1.0 coverage](https://img.shields.io/endpoint?url=https%3A%2F%2Fopenmodelica.github.io%2FOMLibraryTesting.jl%2Fmsl-4.1.0%2Fbadge.json)](https://openmodelica.github.io/OMLibraryTesting.jl/msl-4.1.0/)

A Julia package for running MSL (Modelica Standard Library) coverage tests against [OM.jl](https://github.com/OpenModelica/OM.jl). It auto-discovers all experiment models in the MSL using `omc`, runs each through the OM.jl pipeline (flatten -> translate -> simulate -> validate), and generates HTML coverage reports.

Expand All @@ -23,7 +24,11 @@ Results are compared against Dymola-generated reference trajectories from the MA
Measured 2026-10-11 on 425 MSL example models: frontend 424 (99.8 %), backend 408 (96.0 %), simulate 340 (80.0 %), validate 314 (73.9 %). Per domain and what stops the rest: [docs/COVERAGE.md](docs/COVERAGE.md).
<!-- coverage:end -->

Every published run, by date, with the trend: the [coverage site](https://openmodelica.github.io/OMLibraryTesting.jl/).
<!-- coverage-4.1.0:begin (scripts/publish_coverage.jl) -->
Measured 2026-10-12 on 534 MSL 4.1.0 example models: frontend 534 (100.0 %), backend 438 (82.0 %), simulate 339 (63.5 %), validate 312 (58.4 %). Per domain and what stops the rest: [docs/COVERAGE_4.1.0.md](docs/COVERAGE_4.1.0.md).
<!-- coverage-4.1.0:end -->

Every published run, by date, with the trend: the [coverage site](https://openmodelica.github.io/OMLibraryTesting.jl/) (MSL 3.2.3; MSL 4.1.0 in [msl-4.1.0/](https://openmodelica.github.io/OMLibraryTesting.jl/msl-4.1.0/)).

## Directory Structure

Expand All @@ -41,14 +46,18 @@ OMLibraryTesting.jl/
│ ├── site.jl # Coverage site: one page per run, index, badges
│ └── testsuite.jl # Fetch + run the upstream OpenModelica testsuite (rtest / OM.jl modes)
├── models/
│ └── models.toml # Per-model overrides (atol, reltol, expected phase, signal mapping)
│ ├── models.toml # Per-model overrides (atol, reltol, expected phase, signal mapping), MSL 3.2.3
│ ├── models_4.1.0.toml # The same for MSL 4.1.0
│ └── experiments_Modelica_<version>.txt # The experiment models omc lists (record_experiments)
├── reference/
│ ├── csv/ # Dymola reference CSVs (MAP-LIB, MSL 3.2.3)
│ ├── signals/ # Per-model comparisonSignals.txt files
│ ├── 4.1.0/ # The same for MSL 4.1.0 (csv/, signals/)
│ ├── download_refs.sh # Script to fetch/refresh reference files from GitHub
│ └── generate_refs.mos # OpenModelica script to generate reference results
├── docs/
│ └── COVERAGE.md # The latest published coverage (scripts/publish_coverage.jl)
│ ├── COVERAGE.md # The latest published MSL 3.2.3 coverage (scripts/publish_coverage.jl)
│ └── COVERAGE_4.1.0.md # The same for MSL 4.1.0
├── results/ # Runtime output (ignored by git)
└── reports/ # Generated coverage reports (ignored by git)
```
Expand All @@ -70,6 +79,11 @@ results = OMLibraryTesting.run_coverage(domain = "Thermal", msl_version = "MSL:3
# Run a single model
results = OMLibraryTesting.run_coverage(model = "Modelica.Thermal.HeatTransfer.Examples.TwoMasses")

# MSL 4.1.0: its experiment list, registry (models/models_4.1.0.toml) and reference data
# (reference/4.1.0/); OMFrontend bundles 3.2.3 and 4.0.0, so 4.1.0 is the installed library
# (~/.openmodelica/libraries: Modelica, ModelicaServices and Complex 4.1.0)
results = OMLibraryTesting.run_coverage(msl_version = "MSL:4.1.0")

# Generate an HTML report
OMLibraryTesting.generate_report(results)

Expand All @@ -81,11 +95,13 @@ OMLibraryTesting.print_error_analysis(results)
```

The "Latest coverage" section above and docs/COVERAGE.md are published by the `MSL coverage` workflow
(.github/workflows/coverage.yml: weekly, or by hand from the Actions tab with `commit` set). It runs
`scripts/run_full_coverage.jl` with `OMJL_COVERAGE_NOSKIP=1` (every model through every phase) and passes the
serialized results to the publisher. It also adds the run to the coverage site, served by GitHub Pages from
the `gh-pages` branch (`scripts/publish_site.jl`): `runs/<date>/` with the analysis and the per-model report, an
index over all runs, and the badges (`badge.json`, models that validate; `badges/<phase>.json`). The same by hand:
(.github/workflows/coverage.yml: MSL 3.2.3 weekly, MSL 4.1.0 every third day, or by hand from the Actions
tab with `msl` and `commit` set). It runs `scripts/run_full_coverage.jl` with `OMJL_COVERAGE_NOSKIP=1` (every
model through every phase) and passes the serialized results to the publisher. It also adds the run to the
coverage site, served by GitHub Pages from the `gh-pages` branch (`scripts/publish_site.jl`): `runs/<date>/` with
the analysis and the per-model report, an index over all runs, and the badges (`badge.json`, models that
validate; `badges/<phase>.json`); MSL 4.1.0 has the same in `msl-4.1.0/`, docs/COVERAGE_4.1.0.md and its own
section above. The same by hand (`OMJL_COVERAGE_MSL`, default `MSL:3.2.3`, selects the version for all three):

```bash
OMJL_COVERAGE_NOSKIP=1 julia --project=. scripts/run_full_coverage.jl
Expand Down Expand Up @@ -201,7 +217,7 @@ Reference CSVs use Modelica dot notation (e.g., `rev.phi`). OM.jl uses underscor

## Reference Data

Reference trajectories come from the MAP-LIB Dymola reference results (MSL 3.2.3 branch). Run `reference/download_refs.sh` to fetch or refresh them.
Reference trajectories come from the MAP-LIB Dymola reference results (branch v3.2.3, and v4.1.0 for MSL 4.1.0). Run `reference/download_refs.sh` (MSL 3.2.3) or `reference/download_refs.sh --msl 4.1.0` to fetch or refresh them.

## License

Expand Down
Loading
Loading