feat: sync secrets to GitLab CI environment scopes

[nimish-ks]

🔍 Overview

Closes #1043.

GitLab CI/CD variables can be limited to an environment with an environment scope, and the same key can exist once per scope. Until now, GitLab syncs always wrote to the default * scope. There was no way to sync e.g. Staging and Production secrets into the matching GitLab environments of one project, as GitHub Actions syncs can with GitHub environments.

Existing syncs also broke as soon as a synced key existed in more than one scope: updates failed with There are multiple variables with provided parameters. Please use 'filter[environment_scope]'.

💡 Proposed Changes

Environment scope on GitLab syncs

• The GitLab sync form has a new GitLab Environment Scope picker. New syncs default to All environments (*), and any scope GitLab accepts can be typed, including wildcards like review/*.
  • For projects, it lists the project's GitLab environments.
  • Groups have no environments, so for groups it suggests the scopes already used by the group's variables, as GitLab's own form does.
  • What's shown is what's used: a typed scope is kept when the picker closes without picking an option (Escape discards it). An invalid scope is kept too, shown as an error, and blocks creating the sync instead of falling back to the previous scope.
  • Picking another project or group resets the scope; a scope chosen before the first pick is kept.
• New gitlabEnvironments(credentialId, projectId) query. It is paginated and uses the same credential permission check as the sibling GitLab queries. Projects with Environments disabled return an empty list instead of an error.
• New gitlabGroupEnvironmentScopes(credentialId, groupPath) query, using GitLab's GraphQL group.environmentScopes. It only fetches scope names, never values, and returns an empty list on GitLab versions without that field.
• createGitlabCiSync takes an optional environmentScope, validated against GitLab's own rules (charset, 255 chars) and stored in the sync options.
• The sync card shows the scope, e.g. phase/backend (production).
• Pressing Enter in the GitLab sync form no longer goes back to step 1. The Back button had no type, so it was the form's default submit button.

Scope-aware sync

• A sync with a scope only creates, updates and deletes variables in that scope. Updates and deletes select the scope with filter[environment_scope], so keys that exist in several scopes no longer break the sync.
• Two syncs in an App can't write the same GitLab project or group and scope, since they would overwrite each other's variables. Different masked/protected settings no longer make such a sync count as distinct.
• This is checked when a sync is created, when a sync's credentials change, and when a credential's GitLab host changes. Using a new token for the same GitLab instance is never blocked. http:// and https:// URLs of a host count as the same instance. Errors only name syncs of Apps the member can access.

Existing syncs (verified with an upgrade test, see Testing)

• Syncs created before this change have no stored scope and keep their previous behaviour: they sync to all environments (*).
• As before, they also update and delete a variable that a user moved to another scope in GitLab, the workaround for missing scope support. This applies as long as the key has no * variable and the scope isn't managed by another sync to the same project or group.
• They never write one value into several scopes.
• They used to fail with GitLab's 409 when they touched a key with variables in several scopes, e.g. a per-environment override of a synced key: they compared the secret with the variable listed last, and GitLab refused to update (projects) or delete (projects and groups) the key. Group updates went to whichever variable GitLab found first, which overwrote overrides. Now:
  • A synced key with a * variable updates that variable and leaves overrides alone.
  • The sync never deletes a variable the old sync wouldn't have: wherever the old sync stopped, it still creates and updates, but deletes nothing, and fails naming the keys and what to do. Pipelines may rely on variables the old sync never got to delete.
  • Where it can't tell which variable to update (several copies, no *), it updates none, and reports it wherever the old sync failed or changed one of them.
• Syncs created before July 2024 only stored the project or group path. They are matched by path, and store the ID on their next run, so they're recognised after a rename too.
• A new scoped sync to a project or group that still has such a sync, in any App, is rejected with a message to recreate that sync with a scope first. Otherwise the old sync could copy a variable moved to the new sync's scope into *.

Safety

• GitLab tiers without scoped group variables (Free) silently drop environment_scope and create the variable for all environments.
  • Before writing anything, group syncs with a scope create a throwaway PHASE_SCOPE_CHECK_* variable holding a random value. If GitLab dropped the scope, they abort with a clear error, so no secret is ever written there.
  • Every create also verifies the scope GitLab returned and removes the variable if it landed elsewhere.
• GitLab's group PUT falls back to any variable with the same key when the scope filter matches nothing, e.g. when a concurrent run deleted the variable after it was listed, and GitLab ≤ 16.7 ignores the filter on group PUT entirely.
  • Group updates first check that the variable still exists (GET doesn't fall back) and create it again if not. Otherwise GitLab would update another variable with the key, e.g. a manual one in another scope.
  • Updates also send the scope in the body, so a fallback in the remaining window moves the other variable into the sync's scope instead of exposing the secret to its environments.
  • If GitLab still updated another scope's variable, it is restored from the listing and the sync fails.
• All GitLab API calls go through one helper that sets a timeout. Redirects are only followed to the same host and port, or from http:// to https:// on 443, keeping the method and body. The host is read with the parser requests connects with, redirects with credentials are refused, and in cloud each hop goes through the SSRF check again. In cloud, GitLab hosts that urllib.parse and requests read differently (e.g. with a backslash) are refused. Before, requests followed any redirect, which:
  • sent the Private-Token header (and, on 307/308, the secret payload) to wherever the GitLab host redirected, bypassing the cloud SSRF check on the configured host;
  • turned creates behind an http:// → https:// redirect into GETs: the sync reported success without ever creating a variable. GitLab also redirects the old path of a renamed project, so path-only syncs to renamed projects had the same problem.
• Tokens and hosts are trimmed. A token containing characters that are invalid in a header is no longer included in the error; previously requests echoed the header value into sync logs and GraphQL errors.

🖼️ Screenshots or Demo

Screenshots of the scope picker, the configured sync and the sync card are in the docs PR, under public/assets/images/platform-integrations/gitlab/: https://github.com/phasehq/docs/pull/261/files

📝 Release Notes

• New: GitLab CI syncs can target a GitLab environment scope (e.g. production, staging, review/*), so each Phase Environment can be synced to the matching GitLab environment of the same project or group.
• Fixed: GitLab syncs failed when a synced key existed in more than one environment scope in GitLab, e.g. with a per-environment override. Group syncs overwrote such an override with the default value.
• Fixed: GitLab syncs using an http:// host that redirects to https:// (or the old path of a renamed project) reported success without creating new variables.
• Fixed: pressing Enter in the GitLab sync form went back to the credentials step.
• Changed: existing GitLab syncs leave other syncs' scopes alone, and report keys with variables in several scopes clearly. To add environment-scoped syncs to the same project or group, recreate the existing sync with a scope first.
• Environment scopes for group variables require GitLab Premium or Ultimate. On other tiers the sync fails with an explanation instead of exposing scoped secrets to all environments.

❓ Open Questions

• Duplicate detection is per App, as for the other providers, so two Apps can still sync to the same GitLab project and scope. In QA, a second App's sync to edge/svc staging replaced all of the first App's staging variables, and the two syncs would keep overwriting each other. Should GitLab (or all providers) check across the organisation?

🧪 Testing

Automated

• backend/tests/utils/syncing/test_gitlab.py: sync behaviour against an in-memory fake of the GitLab variables API.
  • The fake models per-scope keys, the group PUT fallback (and that GET/DELETE don't fall back) and Free-tier scope dropping.
  • Covers scoped, unscoped and legacy syncs (including when the old sync used to stop, for projects and groups), pagination, the scope check, rollback and its failure, the group update check, redirects (same host followed; other hosts, ports, downgrades, credentials and backslash tricks refused), timeouts, environment and group scope listing, path-only syncs, resolving their ID and host normalisation.
• backend/tests/graphene/mutations/test_gitlab_sync_mutations.py: scope validation and conflict detection on create, credential switch and credential host change (legacy and path-only syncs, other Apps, other GitLab instances, token rotation, syncs without credentials, errors for Apps the member can't access).
• backend/tests/tasks/test_syncing.py: dispatch for scoped vs. legacy syncs, and storing the ID of path-only syncs.
• New cases in the existing query/mutation permission matrices.
• frontend/tests/utils/gitlabEnvironmentScope.test.ts: scope validation and picker options.

Upgrade test: syncs created with main, then run with this branch (self-hosted GitLab CE 19.4)

• Ten syncs in the format main creates (confirmed against main's own mutation), each on its own project or group, covering:
  • variables that already existed in GitLab (in and outside *, file type, raw: false, comments);
  • a variable moved to production by hand, and per-environment overrides on a project and a group;
  • a key not in Phase in two scopes;
  • masked + protected (with an unmaskable value), and a "masked and hidden" variable;
  • a path-only sync (pre-July 2024), a group sync, a paused sync, and a host that redirects to another path.
• In place: ran the scenario with main's backend, worker and frontend, switched the same database to this branch, and re-ran every sync with nothing changed. No sync that passed on main failed, nothing main kept was deleted, and the only changes were the fixes below. The path-only sync stored its ID.
• Side by side: ran the same five steps (first sync; moves and overrides in GitLab; edits, renames and deletes in Phase; deletes; no-op) from the same starting state on main and on this branch, and diffed every variable and sync result after each step. Everything is identical except:
  • project override: main failed with the 409 on every run once the override existed, so * stayed stale. Now * is updated and the override kept.
  • group override: main overwrote the manual production override with the default value and left * stale. Now * is updated and the override kept.
  • redirecting host: main reported "synchronized" on every run but never created a variable. Now they're created, and unchanged runs report "No changes needed".
  • key in two scopes: both fail and leave the variables as they are; the error now says which key and how to fix it.
  • No sync passes on main and fails here, at any step.
• After the upgrade: in the UI and the API, creating a scoped sync next to these syncs (including the path-only and the group sync, and from another App) is refused with the "recreate that sync" message. Their credentials can be switched to a new token, and their credential's host edited. Recreating one with * changes no variables, after which scoped syncs can be added.

Manual QA against self-hosted GitLab CE 19.4 and GitLab EE 19.4 on the Free and Ultimate tiers, with a GitLab Runner executing real pipelines.

Sync behaviour

• Synced Development → review/*, Staging → staging and Production → production, plus an existing unscoped sync, into one project. Checked every variable's value, description (comment), masked/protected/raw flags and scope through the API.
• Ran a pipeline with jobs for production, staging, development, review/feature-login and no environment. Each job received exactly the expected values: scoped values override *, review/* matches review/feature-login, and $ in values is not expanded.
• Edited, re-commented and deleted secrets in Phase; only the matching scope changed in GitLab. Re-syncs report "No changes needed".
• 150 variables in another scope, a stale key on page 2 and 122 environments: pagination works and other scopes are untouched.
• GitLab EE Free: a scoped group sync fails with the tier explanation and leaves no variable behind; scoped project syncs work.
• Duplicate syncs, invalid scopes, path-traversal project IDs and foreign credentials are rejected through the API.

Several syncs on one project and group with existing variables (scripted against real GitLab, checked through its API and 3 pipelines):

• Variables that already existed in GitLab before the syncs:
  • in a sync's scope: overwritten with the Phase value and the sync's masked/protected settings, or deleted when not in Phase;
  • in other scopes: untouched;
  • a file variable keeps its type;
  • a "masked and hidden" variable is updated by a masked sync.
• Three scoped syncs plus a path-filtered sync (/api → development) on one project, and * and staging group syncs on its group. Pipelines confirm that project variables beat group variables, and that review/feature-login beats review/*.
• Phase edits (value, comment, rename, delete) change only the sync's own scope. Drift made in GitLab (edited, deleted, re-flagged or extra variables) is reverted on the next sync.
• A * variable moved to production by hand: the * sync recreates it, and the production sync deletes the moved copy.
• Second syncs to the same scope are rejected, including from another Phase Environment, with another path, or without a scope (defaults to *). The UI shows the error and keeps the form.
• Pausing a sync stops updates; resuming applies them. Deleting a sync leaves its variables, and it can be recreated.
• A masked secret GitLab can't mask fails the sync with GitLab's error. Updates before it are applied, and fixing the secret in Phase completes the sync.
• Empty, multi-line and unicode values; group scope suggestions; no PHASE_SCOPE_CHECK_* variable left behind.

GitLab EE Ultimate

• Scoped group syncs (production masked and protected, staging, *, review/*) on a group, a scoped sync on a nested subgroup and a project sync below it, with pre-existing group variables in and outside the synced scopes.
• Pipelines confirm that the subgroup beats the parent group and the project beats both, and that production-only group secrets only reach production jobs.
• Group scope suggestions, for a top-level and a nested group, in the API and the UI. A group sync created through the UI with a typed scope.
• The group PUT race, reproduced against real GitLab by deleting the variable after the listing: the sync created it again, and the other scopes' variables (including a manual one) were untouched.
• Removing the license: existing scoped group variables keep their scopes, and * group syncs keep working. Scoped group syncs fail with the tier explanation without writing anything, and catch up once the license is back.

Form state (31 checks, Playwright, payload captured from the request):

• Switching between the project and group tabs after choosing a resource and scope, in either order; picking another project or group.
• Switching credentials, Back → Next, and closing and reopening the dialog.
• Typed, invalid (by click and by Enter), cleared, Escape-discarded and filtered-then-picked scopes; a scope chosen before the first project or group; Enter in each input.
• Environments from the second page; a double-click on Create sends one request.

Redirects: besides the unit tests, 18,468 Location values (userinfo and backslash tricks, percent-encoding, IDNA, IPv6 and zone IDs, control characters, scheme-relative and urljoin oddities) against 8 starting URLs: every redirect that is followed connects to the same host and port as checked, and a backslash proof of concept that reached another server before the fix is refused.

yarn build, tsc, ESLint and yarn test pass. The backend suite passes except test_file_read_permission_error, an unrelated test that relies on chmod 000 and so fails whenever tests run as root.

🎯 Reviewer Focus

1. backend/api/utils/syncing/gitlab/main.py: sync_gitlab_secrets, gitlab_sync_conflict, same_gitlab_resource, get_environment_scopes_of_other_syncs, gitlab_request and _connection_target.
2. backend/backend/graphene/mutations/syncing.py: CreateGitLabCISync, check_gitlab_host_change (used by UpdateSyncAuthentication and UpdateProviderCredentials).
3. backend/api/tasks/syncing.py: perform_gitlab_sync (storing the ID of path-only syncs).
4. frontend/components/syncing/GitLab/.

➕ Additional Context

Docs: phasehq/docs#261

Known limitations of this change:

• A sync created before this change can't tell a variable that was moved to another scope from a per-environment override. If a key's * variable is removed from Phase and the key is later added back, the remaining override is treated as moved and gets updated.
• When such a sync is recreated with a scope, variables that were moved to other scopes by hand are no longer updated. Their jobs keep the old value until a sync for that scope is created or the variables are deleted. The docs say so.
• A path-only sync (before July 2024) that hasn't run since this change, e.g. a paused one, is matched by path until it runs: if its project was renamed, a scoped sync can be created next to it.
• Two credentials for the same GitLab instance under different host names (or non-default ports) count as different instances.
• If a group variable is deleted in the moment between the check and the update, GitLab's fallback can still move another variable with that key into the sync's scope. Nothing is exposed.

Pre-existing issues noticed while working on this, not changed here:

• validate_url_is_safe resolves the host once and requests resolves it again (DNS rebinding). It also parses URLs differently from requests (e.g. a backslash before @), so a URL can pass it and connect elsewhere. GitLab now refuses such hosts itself; the OIDC issuer and AWS STS endpoint checks still use it as is.
• The GitLab project list uses min_access_level=30 (Developer), but managing CI/CD variables needs Maintainer, so some listed projects can't be synced.
• Variables created as "masked and hidden" in GitLab return no value, so they are rewritten on every sync. With an unmasked sync, every run fails: GitLab won't change the visibility of a hidden variable.
• A variable that existed in GitLab before the sync, with raw: false and the same value, keeps variable expansion enabled.
• When GitLab can't mask a value, the sync error is GitLab's {"message":{"value":["is invalid"]}}, which doesn't say why.
• The timeout on requests applies per socket read, not to the whole request, and the project/group listings have no page cap.
• Overlapping runs of the same sync are possible: cancel_sync_tasks removes queued jobs but doesn't stop running ones.
• Any member who can update integration credentials can repoint another App's sync credential to a host they control. UpdateSyncAuthentication also accepts a credential of another provider.
• The Back button of the other sync forms has the same missing type, so Enter in their text fields goes back to step 1.
• Sync cards don't show the GitLab host, so syncs to the same path on two instances look identical.

✨ How to Test the Changes Locally

1. Run GitLab locally, e.g. docker run -p 8929:8929 -e GITLAB_OMNIBUS_CONFIG="external_url 'http://localhost:8929'" gitlab/gitlab-ce. Create a project with a few environments (production, staging, review/feature-1) and a personal access token with the api scope.
2. In Phase, add GitLab credentials with host http://localhost:8929, enable SSE for an App and create a GitLab CI sync. Pick an environment scope in the new picker, or type review/*.
3. Check the project's Settings → CI/CD → Variables in GitLab: the variables carry the chosen environment scope, and variables in other scopes are untouched.

💚 Did You...

• Ensure linting passes (code style checks)?
• Update dependencies and lockfiles (if required)
• Update migrations (if required)
• Regenerate graphql schema and types (if required)
• Verify the app builds locally?
• Manually test the changes on different browsers/devices? (Chromium only)