Describe the bug
The test-cli GitHub Actions job is reported as successful even though the CLI exits during argument parsing and never starts training.
The workflow runs:
pnpm -F cli start -t cifar10 -u 3 -e 1 -r 1
However, testID is a required CLI argument and the workflow does not provide it. The CLI prints:
Required parameter 'testID' was not passed. Please provide a value by passing '--testID=passedValue' or '-i passedValue'
The argument parser exits with a successful status, so GitHub Actions marks the job as passing.
As a result, the current CI check does not:
- Create any Disco clients.
- Start any training.
- Test the CIFAR-10 task.
- Test decentralized communication.
Evidence
This can be seen in successful CI runs:
Each job prints the missing testID error and is nevertheless marked successful.
The relevant workflow command is:
https://github.com/epfml/disco/blob/main/.github/workflows/_test.yml#L70-L78
The required argument is defined here:
https://github.com/epfml/disco/blob/main/cli/src/args.ts#L93-L96
Additional server setup issue
After adding a test ID, the job will also need to start a Disco server.
CIFAR-10 uses the decentralized scheme and the CLI defaults to connecting to http://localhost:8080. The test-cli job currently does not start a server. The server started by build-server-docker belongs to a separate GitHub Actions runner and is not available to the test job.
Expected behavior
The CI job should:
- Start a Disco server on the same runner.
- Wait until the server is ready.
- Run the CLI with a test ID and three CIFAR-10 users.
- Fail if argument parsing, server connection, or training fails.
- Shut down the server after the test.
The CLI should also return a nonzero exit code when required arguments are missing or when its main operation rejects.
Possible fix
- Pass a test ID, for example
-i ci-cifar10.
- Start the server as a background process or job service within
test-cli.
- Add a readiness check for port 8080.
- Change the CLI entry-point error handler to set a failing exit code, for example:
main(args.provider, args.numberOfUsers).catch((error) => {
console.error(error);
process.exitCode = 1;
});
Describe the bug
The
test-cliGitHub Actions job is reported as successful even though the CLI exits during argument parsing and never starts training.The workflow runs:
However,
testIDis a required CLI argument and the workflow does not provide it. The CLI prints:The argument parser exits with a successful status, so GitHub Actions marks the job as passing.
As a result, the current CI check does not:
Evidence
This can be seen in successful CI runs:
Each job prints the missing
testIDerror and is nevertheless marked successful.The relevant workflow command is:
https://github.com/epfml/disco/blob/main/.github/workflows/_test.yml#L70-L78
The required argument is defined here:
https://github.com/epfml/disco/blob/main/cli/src/args.ts#L93-L96
Additional server setup issue
After adding a test ID, the job will also need to start a Disco server.
CIFAR-10 uses the decentralized scheme and the CLI defaults to connecting to
http://localhost:8080. Thetest-clijob currently does not start a server. The server started bybuild-server-dockerbelongs to a separate GitHub Actions runner and is not available to the test job.Expected behavior
The CI job should:
The CLI should also return a nonzero exit code when required arguments are missing or when its main operation rejects.
Possible fix
-i ci-cifar10.test-cli.