Skip to content

CLI CI job passes without running training #1229

Description

@ale18V

Describe the bug

The test-cli GitHub Actions job is reported as successful even though the CLI exits during argument parsing and never starts training.

The workflow runs:

pnpm -F cli start -t cifar10 -u 3 -e 1 -r 1

However, testID is a required CLI argument and the workflow does not provide it. The CLI prints:

Required parameter 'testID' was not passed. Please provide a value by passing '--testID=passedValue' or '-i passedValue'

The argument parser exits with a successful status, so GitHub Actions marks the job as passing.

As a result, the current CI check does not:

  • Create any Disco clients.
  • Start any training.
  • Test the CIFAR-10 task.
  • Test decentralized communication.

Evidence

This can be seen in successful CI runs:

Each job prints the missing testID error and is nevertheless marked successful.

The relevant workflow command is:

https://github.com/epfml/disco/blob/main/.github/workflows/_test.yml#L70-L78

The required argument is defined here:

https://github.com/epfml/disco/blob/main/cli/src/args.ts#L93-L96

Additional server setup issue

After adding a test ID, the job will also need to start a Disco server.

CIFAR-10 uses the decentralized scheme and the CLI defaults to connecting to http://localhost:8080. The test-cli job currently does not start a server. The server started by build-server-docker belongs to a separate GitHub Actions runner and is not available to the test job.

Expected behavior

The CI job should:

  1. Start a Disco server on the same runner.
  2. Wait until the server is ready.
  3. Run the CLI with a test ID and three CIFAR-10 users.
  4. Fail if argument parsing, server connection, or training fails.
  5. Shut down the server after the test.

The CLI should also return a nonzero exit code when required arguments are missing or when its main operation rejects.

Possible fix

  • Pass a test ID, for example -i ci-cifar10.
  • Start the server as a background process or job service within test-cli.
  • Add a readiness check for port 8080.
  • Change the CLI entry-point error handler to set a failing exit code, for example:
main(args.provider, args.numberOfUsers).catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    scope-ciRelated to ci/cd pipelinescope-cliRelated to the CLI moduletype-bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions