fix(publish): support short-form port mapping with variable interpolation - #14198
fix(publish): support short-form port mapping with variable interpolation#14198Hoomanghkhani wants to merge 1 commit into
Conversation
a85b23a to
f99ca7e
Compare
glours
left a comment
There was a problem hiding this comment.
Thanks for digging into this, the diagnosis is correct, but I see one blocker and a broader issue with the proposed fix.
Blocker: secret detection regression
Changing composeFileAsByteReader to use os.ReadFile bypasses the canonicalization performed by loadUnresolvedFile. The secret scanner doesn’t detect list-form environment entries such as:
environment:
- MYSQL_ROOT_PASSWORD=secretCanonicalizing them to map form is what makes the scanner detect the secret, which is why that round-trip was introduced in [#12620](#12620). Using the raw file would silently regress the docker compose publish secret check, so this needs to be addressed before merge.
ports is only one symptom
Removing ports before loader.Transform fixes this case, but other typed fields fail for the same reason when interpolation is skipped, including:
deploy:
replicas: ${REPLICAS}
mem_limit: ${MEM}
healthcheck:
retries: ${RETRIES}The underlying problem is the strict conversion of an unresolved model into types.Project.
Suggested approach
Neither consumer needs that full conversion:
collectEnvCheckFindingscan decode only the fields it uses.composeFileAsByteReadercan keepLoadModelWithContext, normalize list-formenvironmentvalues withtypes.NewMappingWithEquals, and marshal the raw model directly to YAML. This preserves full-file scanning without strict-decoding unrelated fields.
We’ll also need to temporarily pin [compose-go#928](compose-spec/compose-go#928), as it fixes unresolved variables in build.ssh and ulimits during canonicalization:
replace github.com/compose-spec/compose-go/v2 => github.com/compose-spec/compose-go/v2 v2.15.1-0.20260910130034-0e17c6437ebbFinally, could you add a test using mem_limit or deploy.replicas so the fix covers the whole class of issue rather than only ports?
f99ca7e to
8f42e12
Compare
|
Thanks for the thorough review and guidance @glours! I have refactored the fix following the suggested approach:
The PR has been updated with these changes. |
When publishing a Compose project containing unresolved variables in typed
fields (e.g. short-form port syntax `${PORT:-3000}:3000`, `mem_limit: ${MEM}`,
or `deploy.replicas: ${REPLICAS}`), publish previously failed with
type decode errors because `loadUnresolvedFile` strictly transformed the raw
model dictionary into a full `types.Project` with `SkipInterpolation = true`.
Neither consumer requires strict decoding of the entire `types.Project`:
1. `collectEnvCheckFindings` only inspects service environment, env_files,
extends, and project configs. It now decodes only these fields into a
dedicated lightweight struct via `loader.Transform`, ignoring unrelated
typed fields like ports, mem_limit, and deploy.
2. `composeFileAsByteReader` keeps `loader.LoadModelWithContext`, normalizes
any list-form environment declarations into map form using
`types.NewMappingWithEquals`, and marshals the raw dictionary directly to
YAML. This preserves full-file scanning and prevents secret detection
regressions on list-form environment entries.
3. Temporarily pin compose-go#928 via `replace` in go.mod to fix unresolved
variables in build.ssh and ulimits during canonicalization.
4. Added tests verifying `loadUnresolvedFile`, `checkForSensitiveData`, and
`collectEnvCheckFindings` with unresolved variables in ports, mem_limit,
and deploy.replicas, as well as secret detection in list-form environments.
Fixes docker#13672
Signed-off-by: Hooman <hooman.ghkhani@gmail.com>
8f42e12 to
bcb81f0
Compare
What I did
Fixed an issue where
docker compose publishfailed with:'services[...].ports[0]' expected a map or struct, got "string"when a service defined short-form port syntax containing variable substitutions (e.g.
${PORT:-3000}:3000).During
preChecks,loadUnresolvedFileruns withoptions.SkipInterpolation = trueto detect uncommitted secrets and suspicious literals. Because interpolation is skipped,types.ParsePortConfigcannot parse un-interpolated port expressions into numeric ports, leaving them as raw string slices. Mapstructure subsequently failed attempting to decode strings intotypes.ServicePortConfigstructs.To fix this:
loadUnresolvedFilenow loads the model vialoader.LoadModelWithContextand deletesportsfrom services before callingloader.Transform. NeithercollectEnvCheckFindingsnorcheckForSensitiveDatainspects ports (only environment, env_files, extends, and configs are checked).composeFileAsByteReadernow reads the compose file directly from disk viaos.ReadFile, guaranteeing the secret scanner examines the exact file bytes to be published without loss of comments or unnecessary decoding.publish_test.goverifyingloadUnresolvedFile,checkForSensitiveData, andcollectEnvCheckFindingswith short-form port syntax containing variable substitutions.Related issue
Fixes #13672