Skip to content

Fixes open-metadata/openmetadata-collate#7380: Keep JAVA_TOOL_OPTIONS secrets out of the startup logs - #35402

Merged
mohityadav766 merged 3 commits into
mainfrom
fix/java-tool-options-startup-log
Oct 15, 2026
Merged

mohityadav766 merged 3 commits into
mainfrom
fix/java-tool-options-startup-log

Conversation

@mohityadav766

@mohityadav766 mohityadav766 commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Describe your changes:

Fixes open-metadata/openmetadata-collate#7380

When JAVA_TOOL_OPTIONS is set, for example to pass a truststore password from a Kubernetes secret, its value showed up in the server and migration logs:

Picked up JAVA_TOOL_OPTIONS: -Djavax.net.ssl.trustStore=... -Djavax.net.ssl.trustStorePassword=**** ...

Why it happened: our scripts never echo it. The JVM prints JAVA_TOOL_OPTIONS to stderr at startup whenever the variable is set, and no flag turns that off. So every launcher that starts a JVM with the variable in its environment leaked it: the server container, the Helm run-db-migrations init container and compose execute-migrate-all (both via openmetadata-ops.sh), and bare metal (logs/catalog.err).

What changed:

  • bin/openmetadata-server-start.sh, bootstrap/openmetadata-ops.sh and bin/openmetadata.sh now write JAVA_TOOL_OPTIONS to a -XX:VMOptionsFile and unset the variable.
    • The JVM parses that file exactly like JAVA_TOOL_OPTIONS, quotes and # included, and doesn't echo it.
    • The file is created in /dev/shm (tmpfs, still writable with a read-only root filesystem), falling back to $TMPDIR. It is opened on fd 3 and unlinked at once, so nothing stays on disk.
    • The JVM's command line, which every local user can read through ps or /proc/<pid>/cmdline, only shows -XX:VMOptionsFile=/dev/fd/3. That also holds on musl JDKs (Alpine), because the JVM reads the file itself after the launcher re-execs.
    • If neither directory is writable, the launchers print a one-line warning without the value, and leave the variable to the JVM as before.
    • The file goes ahead of the scripts' own options, so the scripts' options still win over the env ones, as before.
  • openmetadata-ops.sh builds its classpath first, so in dev mode the mvn call still sees JAVA_TOOL_OPTIONS.
  • The change is plain POSIX sh, so it runs under bash, dash and busybox sh alike.
  • scripts/docker_smoke_test.sh gets a new assertion. It starts the image with JAVA_TOOL_OPTIONS="-XX:+PrintCommandLineFlags -Dsmoke.secret=...#1", then checks four things:
    • the value is not in the container log;
    • the value is not on PID 1's command line;
    • no options file is left in /dev/shm or /tmp;
    • the flags line shows the options still reached the JVM.
  • How this got here: the first version put the options on the command line, which gitar flagged as visible in ps. The second used an @argfile, which can't carry # or quotes, so those values stayed in the environment and still printed. This version has neither problem.

The 2.0 backport is #35518. The Collate server launchers get the same change in open-metadata/openmetadata-collate#7411. The Collate image takes bootstrap/ from this repo, so its migration init container needs this PR.

Type of change:

  • Bug fix

High-level design:

N/A — small change.

Tests:

Use cases covered

  • With JAVA_TOOL_OPTIONS set, its options still reach the JVM. The value no longer appears in the server log, the migration log, logs/catalog.err, or the JVM's command line.
  • Values containing # or quotes arrive exactly as JAVA_TOOL_OPTIONS would deliver them.
  • The scripts' own options still take precedence over JAVA_TOOL_OPTIONS ones, as before.
  • Without JAVA_TOOL_OPTIONS, nothing changes. With no writable temp dir, the launcher prints one warning line and falls back to the old behavior.

Unit tests

Not applicable: shell scripts only. The new docker smoke test assertion is the regression test.

Backend integration tests

  • Not applicable (no backend API changes).

make java_affected selects the full IT suite, but only because no impact-map area owns bin/**. The ITs start the server in-process and never run these scripts, so I left them to the merge queue.

Ingestion integration tests

  • Not applicable (no ingestion changes).

Playwright (UI) tests

  • Not applicable (no UI changes).

Manual testing performed

  1. Reproduced the leak on docker.getcollate.io/openmetadata/server:2.0.5, with JAVA_TOOL_OPTIONS="-Dsecret=hunter2". The server log and ./bootstrap/openmetadata-ops.sh --help both print Picked up JAVA_TOOL_OPTIONS: -Dsecret=hunter2.

  2. Confirmed the review's premise in a container: a second uid can read a process's /proc/<pid>/cmdline, but gets "permission denied" on its /proc/<pid>/environ.

  3. Ran this branch's bin/openmetadata-server-start.sh and bootstrap/openmetadata-ops.sh with stub main classes. Each stub reports whether the property arrived, whether JAVA_TOOL_OPTIONS is still in its environment, whether its own command line contains the value, and its max heap.

    • Shells: macOS bash 3.2; dash on Temurin 21 / Ubuntu (glibc); busybox ash on the 2.0.5 Alpine image (musl).
    • Values: -Dsmoke.p=changeit -Xmx256m, -Dsmoke.p=pa#ss, and -Dsmoke.p='pa#s"s' -Dsmoke.q="a b".

    Results, in every shell:

    • nothing is printed;
    • every value arrives intact (changeit, pa#ss, pa#s"s, a b), the same as the env var gives;
    • JAVA_TOOL_OPTIONS is gone from the environment and from the command line, including on musl;
    • max heap is unchanged: 1024 MB for the server, where the script's -Xmx1G still wins, and 256 MB for the ops tool;
    • no options file is left behind.

    An unmatched quote still makes the JVM refuse to start, as it does with the env var. Its error names /dev/fd/3, not the value.
    On gcr.io/distroless/cc-debian13:debug-nonroot, as uid 65532 with a Temurin 21 JRE:

    • the default setup works;
    • with a read-only root filesystem, the launcher uses /dev/shm;
    • with /dev/shm read-only as well, it prints the one-line warning and falls back.
  4. bash -n, plus sh -n under dash and busybox, pass for every changed script. Shellcheck reports the same warnings as origin/main and nothing new.

  5. On this PR, the "Smoke test the server image" step runs the new assertion against the OSS image, since bin/** triggers that step.

UI screen recording:

Not applicable — launcher scripts only, no UI.

Checklist:

  • I have read the CONTRIBUTING document.

  • My PR title is Fixes <issue-number>: <short explanation>

  • My PR is linked to a GitHub issue via Fixes #<issue-number> above.

  • I have commented on my code, particularly in hard-to-understand areas.

  • I have added tests (unit / integration / Playwright as applicable) and listed them above.

  • Decision check: no record needed. The rule is stated in each launcher, and the smoke assertion enforces it.

  • I have added a test that covers the exact scenario we are fixing.

🤖 Generated with Claude Code

… secrets out of the startup logs

The JVM prints JAVA_TOOL_OPTIONS to stderr at startup ("Picked up
JAVA_TOOL_OPTIONS: ..."), so a truststore password passed that way ended up
in the server and migration logs. bin/openmetadata-server-start.sh,
bootstrap/openmetadata-ops.sh and bin/openmetadata.sh now pass the options to
java as arguments, ahead of their own so precedence is unchanged, and unset
the variable. A value with quotes is left to the JVM, which strips quotes
that shell word splitting would keep.

The docker smoke test starts the image with JAVA_TOOL_OPTIONS set and checks
that the value stays out of the log while the options still reach the JVM.
@github-actions

Copy link
Copy Markdown
Contributor

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

  • Linked issue open-metadata/openmetadata-collate#7380 does not exist or is not accessible.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

@github-actions github-actions Bot added backend safe to test Add this label to run secure Github workflows on PRs labels Oct 14, 2026
Comment thread bin/openmetadata-server-start.sh Outdated
@mohityadav766 mohityadav766 self-assigned this Oct 14, 2026
Passing the options as plain arguments traded the startup log for ps:
/proc/<pid>/cmdline is readable by every local user, while
/proc/<pid>/environ is readable only by the process owner. The launchers now
hand the options to the java launcher as an @argfile on fd 3, fed by a
heredoc, so the JVM's command line only shows @/dev/fd/3 and no file is left
behind. musl JDKs re-exec the launcher with the options expanded, so on
Alpine-based images ps still shows them. A value containing '#' now also
stays in the environment, because argfile syntax reads '#' as the start of a
comment.

openmetadata-ops.sh now builds its classpath before taking the variable, so
the dev-mode mvn call still sees JAVA_TOOL_OPTIONS. The docker smoke test also
checks that the value is not on PID 1's command line.
The argfile could not carry values containing '#' or quotes, so those stayed
in the environment and the JVM still printed them, with no warning.
Generated passwords often contain '#'. The launchers now write the options to
a -XX:VMOptionsFile instead, which the JVM parses exactly like
JAVA_TOOL_OPTIONS, quotes and '#' included, without echoing it. The file is
created in /dev/shm (tmpfs, writable even with a read-only root filesystem)
or $TMPDIR, opened on fd 3 and unlinked at once, so nothing stays on disk
and ps only shows -XX:VMOptionsFile=/dev/fd/3. The JVM reads the file itself
after the musl launcher re-exec, so this also holds on Alpine-based images.
If neither directory is writable, the launchers print a one-line warning
without the value and leave the variable to the JVM.

The docker smoke test now puts a '#' in its value and checks that no
options file is left behind.
@gitar-bot

gitar-bot Bot commented Oct 15, 2026

Copy link
Copy Markdown
Code Review ✅ Approved 1 closed / 1 findings

🟡 Medium risk · Changing JAVA_TOOL_OPTIONS delivery across server and migration launchers could alter JVM option handling or prevent those processes from starting.

Moves JAVA_TOOL_OPTIONS out of the environment and hands it to the java launcher via an @argfile on fd 3, so its values no longer print to server, migration, or catalog.err logs. The earlier finding that the secret would move to world-readable argv was resolved. No open issues found.

✅ 1 closed
✅ Security: Secret moves from env to world-readable argv instead of out of logs

📄 bin/openmetadata-server-start.sh:83-90 📄 bin/openmetadata-server-start.sh:142 📄 bin/openmetadata-server-start.sh:144 📄 bin/openmetadata.sh:156-163 📄 bin/openmetadata.sh:39 📄 bootstrap/openmetadata-ops.sh:37-44 📄 bootstrap/openmetadata-ops.sh:67
The block expands JAVA_TOOL_OPTIONS into java's command line, so a -Djavax.net.ssl.trustStorePassword=... that used to sit in the process environment now sits in argv. On Linux, /proc/<pid>/environ can only be read by the process owner. /proc/<pid>/cmdline can be read by any user, so ps -ef shows it. For bare-metal bin/openmetadata.sh, that means any local user can read the truststore password from ps for as long as the server runs. In containers, a shared PID namespace, sidecars and host process collectors (node agents, ps on the node) can see it too. The startup log leak is gone, but the secret is now readable through ps for the whole life of the process. A fix that avoids both: write $JAVA_TOOL_OPTIONS to a mode-600 argfile (umask 077; printf '%s ' "$JAVA_TOOL_OPTIONS" > "$f") and pass @"$f" to java. The launcher expands argfiles on JDK 9+, so the options never show up in argv or the log. The smoke test could also check /proc/1/cmdline for not-for-the-log.

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

For more verification details, comment gitar display:verbose.

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@mohityadav766
mohityadav766 added this pull request to the merge queue Oct 15, 2026
Merged via the queue into main with commit 3f215f8 Oct 15, 2026
72 of 74 checks passed
@mohityadav766
mohityadav766 deleted the fix/java-tool-options-startup-log branch October 15, 2026 23:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend safe to test Add this label to run secure Github workflows on PRs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants