Skip to content

fix(publish): support short-form port mapping with variable interpolation - #14198

Open
Hoomanghkhani wants to merge 1 commit into
docker:mainfrom
Hoomanghkhani:fix-publish-short-form-ports
Open

fix(publish): support short-form port mapping with variable interpolation#14198
Hoomanghkhani wants to merge 1 commit into
docker:mainfrom
Hoomanghkhani:fix-publish-short-form-ports

Conversation

@Hoomanghkhani

Copy link
Copy Markdown

What I did

Fixed an issue where docker compose publish failed with:
'services[...].ports[0]' expected a map or struct, got "string"
when a service defined short-form port syntax containing variable substitutions (e.g. ${PORT:-3000}:3000).

During preChecks, loadUnresolvedFile runs with options.SkipInterpolation = true to detect uncommitted secrets and suspicious literals. Because interpolation is skipped, types.ParsePortConfig cannot parse un-interpolated port expressions into numeric ports, leaving them as raw string slices. Mapstructure subsequently failed attempting to decode strings into types.ServicePortConfig structs.

To fix this:

  1. loadUnresolvedFile now loads the model via loader.LoadModelWithContext and deletes ports from services before calling loader.Transform. Neither collectEnvCheckFindings nor checkForSensitiveData inspects ports (only environment, env_files, extends, and configs are checked).
  2. composeFileAsByteReader now reads the compose file directly from disk via os.ReadFile, guaranteeing the secret scanner examines the exact file bytes to be published without loss of comments or unnecessary decoding.
  3. Added unit tests in publish_test.go verifying loadUnresolvedFile, checkForSensitiveData, and collectEnvCheckFindings with short-form port syntax containing variable substitutions.

Related issue
Fixes #13672

@Hoomanghkhani
Hoomanghkhani requested review from a team as code owners September 8, 2026 14:00
@Hoomanghkhani
Hoomanghkhani force-pushed the fix-publish-short-form-ports branch from a85b23a to f99ca7e Compare September 8, 2026 14:04

@glours glours left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for digging into this, the diagnosis is correct, but I see one blocker and a broader issue with the proposed fix.

Blocker: secret detection regression

Changing composeFileAsByteReader to use os.ReadFile bypasses the canonicalization performed by loadUnresolvedFile. The secret scanner doesn’t detect list-form environment entries such as:

environment:
  - MYSQL_ROOT_PASSWORD=secret

Canonicalizing them to map form is what makes the scanner detect the secret, which is why that round-trip was introduced in [#12620](#12620). Using the raw file would silently regress the docker compose publish secret check, so this needs to be addressed before merge.

ports is only one symptom

Removing ports before loader.Transform fixes this case, but other typed fields fail for the same reason when interpolation is skipped, including:

deploy:
  replicas: ${REPLICAS}
mem_limit: ${MEM}
healthcheck:
  retries: ${RETRIES}

The underlying problem is the strict conversion of an unresolved model into types.Project.

Suggested approach

Neither consumer needs that full conversion:

  • collectEnvCheckFindings can decode only the fields it uses.
  • composeFileAsByteReader can keep LoadModelWithContext, normalize list-form environment values with types.NewMappingWithEquals, and marshal the raw model directly to YAML. This preserves full-file scanning without strict-decoding unrelated fields.

We’ll also need to temporarily pin [compose-go#928](compose-spec/compose-go#928), as it fixes unresolved variables in build.ssh and ulimits during canonicalization:

replace github.com/compose-spec/compose-go/v2 => github.com/compose-spec/compose-go/v2 v2.15.1-0.20260910130034-0e17c6437ebb

Finally, could you add a test using mem_limit or deploy.replicas so the fix covers the whole class of issue rather than only ports?

@Hoomanghkhani
Hoomanghkhani force-pushed the fix-publish-short-form-ports branch from f99ca7e to 8f42e12 Compare September 10, 2026 16:20
@Hoomanghkhani

Copy link
Copy Markdown
Author

Thanks for the thorough review and guidance @glours!

I have refactored the fix following the suggested approach:

  1. Secret Detection & Canonicalization:

    • Reverted os.ReadFile in composeFileAsByteReader. It now retains loader.LoadModelWithContext, normalizes list-form environment variables to map form using types.NewMappingWithEquals, and marshals the raw dictionary directly to YAML.
    • Added Test_checkForSensitiveData_list_form_secret to ensure secrets declared in list-form environments are properly caught by the scanner.
  2. Selective Decoding for Pre-checks:

    • Rather than stripping ports or decoding into a full types.Project, loadUnresolvedFile now decodes strictly into a lightweight unresolvedFile struct containing only the fields inspected by collectEnvCheckFindings (environment, env_file, extends, and config content).
    • Unrelated typed fields (ports, deploy, mem_limit, healthcheck, etc.) are ignored during mapstructure decoding and will no longer fail on un-interpolated variable syntax.
  3. Pinned compose-go#928:

    • Added the temporary replace directive in go.mod to bring in compose-go#928.
  4. Extended Test Coverage:

    • Updated existing tests (loadUnresolvedFile, checkForSensitiveData, and collectEnvCheckFindings) to include mem_limit: ${MEM}, deploy.replicas: ${REPLICAS}, and healthcheck.retries: ${RETRIES} alongside short-form port syntax with variable substitutions.

The PR has been updated with these changes.

When publishing a Compose project containing unresolved variables in typed
fields (e.g. short-form port syntax `${PORT:-3000}:3000`, `mem_limit: ${MEM}`,
or `deploy.replicas: ${REPLICAS}`), publish previously failed with
type decode errors because `loadUnresolvedFile` strictly transformed the raw
model dictionary into a full `types.Project` with `SkipInterpolation = true`.

Neither consumer requires strict decoding of the entire `types.Project`:
1. `collectEnvCheckFindings` only inspects service environment, env_files,
   extends, and project configs. It now decodes only these fields into a
   dedicated lightweight struct via `loader.Transform`, ignoring unrelated
   typed fields like ports, mem_limit, and deploy.
2. `composeFileAsByteReader` keeps `loader.LoadModelWithContext`, normalizes
   any list-form environment declarations into map form using
   `types.NewMappingWithEquals`, and marshals the raw dictionary directly to
   YAML. This preserves full-file scanning and prevents secret detection
   regressions on list-form environment entries.
3. Temporarily pin compose-go#928 via `replace` in go.mod to fix unresolved
   variables in build.ssh and ulimits during canonicalization.
4. Added tests verifying `loadUnresolvedFile`, `checkForSensitiveData`, and
   `collectEnvCheckFindings` with unresolved variables in ports, mem_limit,
   and deploy.replicas, as well as secret detection in list-form environments.

Fixes docker#13672

Signed-off-by: Hooman <hooman.ghkhani@gmail.com>
@Hoomanghkhani
Hoomanghkhani force-pushed the fix-publish-short-form-ports branch from 8f42e12 to bcb81f0 Compare September 10, 2026 16:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] docker compose publish rejects short-form port mapping

2 participants