Skip to content

Split up Python API in light and full service #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
PGijsbers wants to merge 3 commits into
base: main
Choose a base branch
from
+33 −11
Open

Split up Python API in light and full service #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1b5afee
Split up Python API in light and full service
PGijsbers Sep 16, 2026
f029cbb
Revert accidental changes to the php-api service
PGijsbers Sep 16, 2026
14e116a
Add documentation on the `light` service
PGijsbers Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,15 @@ docker compose --profile frontend up -d # Frontend, rest-api, elasticsearch and
```
Use the same profile for your `down` command.

The Python REST API is also available in a 'light' mode that allows for database interactions
but does not allow for file uploads/downloads. It has to be started directly (as opposed to
through a profile):
```bash
docker compose up python-rest-api-light -d
docker compose down
```
While it is up, the API is available under `http://localhost:8082`, e.g., `http://localhost:8082/datasets/1`.

Comment on lines +62 to +70

@PGijsbers PGijsbers Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I cannot simply add a profile to the service as it is defined right now, since the python-rest-api service will also get that profile (due to extends) but the other dependencies of the python-rest-api (e.g., php-api) will not have that profile (and should not have the profile, since we don't want to start them).

I think this is perfectly fine, we are probably defining too many profiles in the first place. But if we do want to keep using profiles even for single services, then we could work around that with yaml anchoring, e.g.

 python-api-light:
      profiles: ["rest-api-light"]
  -   image: openml/rest-api:0.0.1-alpha
  +   image: &python-api-image openml/rest-api:0.0.1-alpha    # define anchor

and then

   python-api:
      profiles: ["all", "rest-api"]            # only its own profiles, not merged
  -   extends: python-api-light
  +   image: *python-api-image                 # copied from python-api-light


## Known issues
See the Github Issue list for the known issues.
Expand Down
35 changes: 24 additions & 11 deletions docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,22 +85,21 @@ services:
extra_hosts:
- "localhost=172.28.0.2"

# This is the new REST API, which will later replace the PHP REST API
python-api:
profiles: ["all", "rest-api"]
# The Python based REST API is set to replace the PHP based REST API
# The 'light' profile can be used to test most of the REST API
# (everything except file uploads/downloads), but is
# faster to start and requires fewer resources to run.
python-api-light:
# if no profile is set below, it starts on all `compose up` invocations,
# which leads to port conflict if `python-api` is also started.
profiles: ["rest-api-light"]
image: openml/rest-api:0.0.1-alpha
container_name: "openml-python-rest-api"
container_name: "openml-python-rest-api-light"
ports:
- "8082:8000" # or localhost:8000/py with nginx
depends_on:
database-setup:
condition: service_completed_successfully
php-api:
condition: service_healthy # used for serving arff files
minio:
condition: service_started # used for serving parquet files
nginx:
condition: service_started # used for routing to MinIO and PHP REST API
Comment on lines -102 to -103

@PGijsbers PGijsbers Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wasn't sure whether to keep the nginx around for the 'light' version. For now, I figured to keep it truly minimal. It's easy enough to specify docker compose up python-api-light nginx if you do want the routing to be available.

volumes:
- ./config/python-server/config.toml:/python-api/src/config.toml
healthcheck:
Expand All @@ -110,6 +109,20 @@ services:
timeout: 3s
interval: 1m

python-api:
profiles: ["all", "rest-api"]
container_name: "openml-python-rest-api"
extends: python-api-light
depends_on:
database-setup:
condition: service_completed_successfully
php-api:
condition: service_healthy # used for serving arff files
minio:
condition: service_started # used for serving parquet files
nginx:
condition: service_started # used for routing to MinIO and PHP REST API

email-server:
profiles: ["all", "frontend"]
image: foxcpp/maddy:latest
Expand Down Expand Up @@ -138,7 +151,7 @@ services:
condition: service_healthy

minio:
profiles: ["all", "minio", "evaluation-engine"]
profiles: ["all", "minio", "rest-api", "evaluation-engine"]
image: openml/test-minio:v0.1.20260204
container_name: "openml-minio"
ports:
Expand Down