Skip to content

Better service #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lzx1413 wants to merge 6 commits into
base: main
Choose a base branch
from
+3,555 −761
Open

Better service #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7be0140
refactor(service): centralize task and artifact foundations
lzx1413 Jul 8, 2026
4f74224
feat(service): configure local artifact lifecycle
lzx1413 Jul 8, 2026
e87f5c0
fix(service): harden runtime configuration boundaries
lzx1413 Jul 8, 2026
a39bbe4
refactor(service): split request and stream route profiles
lzx1413 Jul 8, 2026
efe9d9e
feat(service): finalize artifact metadata and docs
lzx1413 Jul 8, 2026
b59fefc
feat(service): support dynamic form task parameters
lzx1413 Jul 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 135 additions & 9 deletions docs/en/service.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,7 +150,7 @@ telefuser serve /path/to/pipeline --task i2v [OPTIONS]
| `--cache-dir` | `-c` | string | `work_dirs/server_cache` | Cache directory |
| `--parallelism` | `-g` | int | `1` | Number of parallel workers |
| `--num-replicas` | `-n` | int | `1` | Number of independent pipeline replicas (Pipeline Pool) |
| `--security-level` | | choice | `strict` | Security level: none/basic/strict/sandbox |
| `--security-level` | | choice | `strict` | Validation level: none/basic/strict/sandbox. `sandbox` is a best-effort restricted-load check, not runtime isolation. |
| `--skip-validation` | | flag | `False` | Skip security validation |
| `--validate-only` | | flag | `False` | Only validate without starting |

Expand Down Expand Up @@ -186,7 +186,7 @@ telefuser validate /path/to/pipeline.py [OPTIONS]
| Parameter | Default | Description |
|-----------|---------|-------------|
| `pipeline_file` | **Required** | Path to pipeline Python file |
| `--level` | `strict` | Security level: none/basic/strict/sandbox |
| `--level` | `strict` | Validation level: none/basic/strict/sandbox. `sandbox` is a best-effort restricted-load check, not runtime isolation. |
| `--json` | `False` | Output in JSON format |

```bash
Expand Down Expand Up @@ -258,7 +258,17 @@ telefuser serve --help
| `TELEFUSER_HOST` | Server host | `127.0.0.1` |
| `TELEFUSER_PORT` | Server port | `8000` |
| `TELEFUSER_RATE_LIMIT_ENABLED` | Enable rate limiting | `true` |
| `TELEFUSER_RATE_LIMIT_RPM` | Requests per minute limit | `60` |
| `TELEFUSER_RATE_LIMIT_REQUESTS_PER_MINUTE` | Requests per minute limit | `60` |
| `TELEFUSER_TRUST_FORWARDED_FOR` | Trust `X-Forwarded-For` for rate-limit identity. Enable only behind trusted proxies. | `false` |
| `TELEFUSER_ARTIFACT_STORAGE_BACKEND` | Artifact backend. Only `local` is implemented currently. | `local` |
| `TELEFUSER_ARTIFACT_LOCAL_ROOT` | Local artifact root. Defaults to `TELEFUSER_CACHE_DIR`. | unset |
| `TELEFUSER_ARTIFACT_PERSISTENCE_MODE` | Local artifact retention mode: `persistent` or `ephemeral`. | `persistent` |
| `TELEFUSER_ARTIFACT_RETENTION_SECONDS` | Retention time for terminal task artifacts. `0` disables TTL cleanup. | `604800` |
| `TELEFUSER_ARTIFACT_TMP_RETENTION_SECONDS` | Retention time for temporary `.part` files. `0` disables tmp cleanup. | `3600` |
| `TELEFUSER_ARTIFACT_CLEANUP_INTERVAL_SECONDS` | Background artifact cleanup interval. | `3600` |
| `TELEFUSER_ARTIFACT_MAX_TOTAL_BYTES` | Local artifact cache size limit. `0` disables capacity cleanup. | `0` |
| `TELEFUSER_ARTIFACT_MAX_TASK_BYTES` | Per-task artifact size limit for terminal tasks. `0` disables per-task cleanup. | `0` |
| `TELEFUSER_ARTIFACT_PRESERVE_FAILED_OUTPUTS` | Preserve failed task directories from automatic artifact cleanup. | `false` |

### Configuration File Example

Expand All @@ -269,9 +279,91 @@ TELEFUSER_SECURITY_LEVEL=STRICT
TELEFUSER_PORT=8080
TELEFUSER_HOST=0.0.0.0
TELEFUSER_RATE_LIMIT_ENABLED=true
TELEFUSER_RATE_LIMIT_RPM=100
TELEFUSER_RATE_LIMIT_REQUESTS_PER_MINUTE=100
```

### Runtime Boundaries

The request-response service is intentionally local and single-process by default:

- Task status, queue state, cancellation state, and rate-limit buckets are kept in process memory. They are not shared across multiple server instances and are not restored after restart.
- File inputs, generated outputs, and temporary `.part` files are managed by the local artifact store under the configured cache root.
- `artifact_storage_backend=s3` is declared as a future backend boundary, but it is not implemented yet. Setting it to `s3` fails at startup instead of silently falling back to local storage.
- `telefuser serve` exposes request-response routes only: `/v1/tasks/*`, `/v1/files/*`, `/v1/images/*`, `/v1/videos/*`, and `/v1/service/*`.
- `telefuser stream-serve` exposes stream routes only: `/v1/stream/*`, `/v1/stream/webrtc/*`, and `/v1/service/*`. It does not expose task, file-download, or OpenAI-compatible request-response routes.

### Artifact Storage and Cleanup

Artifacts are local files rooted at `TELEFUSER_ARTIFACT_LOCAL_ROOT` when set, otherwise `TELEFUSER_CACHE_DIR`.
Uploads and remote downloads are written through temporary `.part` files and atomically renamed after successful completion.

Outputs are task-scoped when a task id is available:

```text
<artifact_root>/tasks/<task_id>/inputs/
<artifact_root>/tasks/<task_id>/outputs/
<artifact_root>/tasks/<task_id>/tmp/
```

Only paths resolved inside the artifact root are accepted. Absolute output paths and `..` traversal are rejected.
Download routes accept validated output paths and local artifact ids. Local artifact ids use the
`local:tasks/<task_id>/outputs/<media_type>/<filename>` form and only refer to downloadable outputs in the local
artifact backend. They are not remote object-storage ids.

Cleanup is best-effort and never removes active tasks (`pending`, `processing`, or `streaming`). Terminal tasks
(`completed`, `failed`, or `cancelled`) may be removed by TTL cleanup, total-cache capacity cleanup, or per-task size cleanup.
Temporary `.part` files are cleaned independently by `TELEFUSER_ARTIFACT_TMP_RETENTION_SECONDS`.

`TELEFUSER_ARTIFACT_PERSISTENCE_MODE=persistent` keeps terminal task artifacts until TTL or capacity cleanup removes
them. `ephemeral` removes terminal task directories on the next cleanup pass. If
`TELEFUSER_ARTIFACT_PRESERVE_FAILED_OUTPUTS=true`, failed task directories are protected from automatic cleanup so
partial outputs can be inspected manually.

Recommended cleanup profiles:

```bash
# Development/debugging: keep outputs for one day and preserve failed task outputs.
export TELEFUSER_ARTIFACT_PERSISTENCE_MODE=persistent
export TELEFUSER_ARTIFACT_RETENTION_SECONDS=86400
export TELEFUSER_ARTIFACT_TMP_RETENTION_SECONDS=1800
export TELEFUSER_ARTIFACT_CLEANUP_INTERVAL_SECONDS=600
export TELEFUSER_ARTIFACT_PRESERVE_FAILED_OUTPUTS=true

# Temporary benchmarking: remove terminal task directories on the next cleanup pass.
export TELEFUSER_ARTIFACT_PERSISTENCE_MODE=ephemeral
export TELEFUSER_ARTIFACT_TMP_RETENTION_SECONDS=600
export TELEFUSER_ARTIFACT_CLEANUP_INTERVAL_SECONDS=300
export TELEFUSER_ARTIFACT_PRESERVE_FAILED_OUTPUTS=false

# Long-running service: keep artifacts for seven days with bounded local storage.
export TELEFUSER_ARTIFACT_PERSISTENCE_MODE=persistent
export TELEFUSER_ARTIFACT_RETENTION_SECONDS=604800
export TELEFUSER_ARTIFACT_TMP_RETENTION_SECONDS=3600
export TELEFUSER_ARTIFACT_CLEANUP_INTERVAL_SECONDS=3600
export TELEFUSER_ARTIFACT_MAX_TOTAL_BYTES=107374182400
export TELEFUSER_ARTIFACT_MAX_TASK_BYTES=10737418240
```

### Rate Limit and Proxy Headers

Rate limiting is in-memory and keyed by client identity. By default, the server uses the direct client address and does
not trust `X-Forwarded-For`, because arbitrary clients can spoof that header. Set
`TELEFUSER_TRUST_FORWARDED_FOR=true` only when TeleFuser is behind a trusted reverse proxy that sanitizes forwarded
headers.

The default limited paths cover expensive generation endpoints, artifact downloads, and stream negotiation:

```text
/v1/tasks/create
/v1/tasks/form
/v1/images
/v1/videos
/v1/files/download
/v1/stream
```

Liveness/readiness endpoints remain unthrottled for infrastructure probes.

### Pipeline Contract and Parameter Definitions

The service does not infer pipeline capabilities from Python signatures alone. Instead, it loads an explicit
Expand Down Expand Up @@ -463,9 +555,9 @@ When the server is running:
| POST | `/tasks/create` | Create a new generation task |
| POST | `/tasks/form` | Create task with file upload |
| GET | `/tasks/{task_id}/status` | Get task status |
| DELETE | `/tasks/{task_id}` | Cancel a task |
| DELETE | `/tasks/{task_id}` | Request cooperative task cancellation |
| GET | `/tasks/queue/status` | Get queue status |
| GET | `/files/download/{file_id}` | Download output files |
| GET | `/files/download/{file_id}` | Download an output file by validated path or local artifact id |
| GET | `/service/health` | Health check |
| GET | `/service/status` | Service status |
| GET | `/service/metadata` | Service metadata |
Expand Down Expand Up @@ -814,7 +906,17 @@ TeleFuser provides an **OpenAI-compatible** REST API that allows you to use stan
"data": [
{
"url": "http://localhost:8000/v1/images/task_xxx/content",
"revised_prompt": "a beautiful cat"
"revised_prompt": "a beautiful cat",
"file_path": "/cache/tasks/task_xxx/outputs/images/result.png",
"artifact_id": "local:tasks/task_xxx/outputs/images/result.png",
"artifact_metadata": {
"backend": "local",
"relative_path": "tasks/task_xxx/outputs/images/result.png",
"task_id": "task_xxx",
"media_type": "image",
"filename": "result.png",
"size_bytes": 123456
}
}
],
"peak_memory_mb": 4096.5,
Expand Down Expand Up @@ -875,7 +977,7 @@ response.data[0].save("sunset.png")
"size": "1024x576", // Optional: video size
"seed": 1024, // Optional: random seed
"negative_prompt": "blurry", // Optional: negative prompt
"output_path": "/path/to/output.mp4" // Optional: custom output path
"output_path": "custom-output.mp4" // Optional: relative custom output path
}
```

Expand All @@ -891,7 +993,31 @@ response.data[0].save("sunset.png")
"created_at": 1699000000,
"size": "1024x576",
"seconds": "5",
"url": null
"url": null,
"file_path": null,
"artifact_id": null,
"artifact_metadata": null
}
```

Completed video responses include a content URL and local artifact metadata:

```json
{
"id": "vid_xxx",
"status": "completed",
"progress": 100,
"url": "http://localhost:8000/v1/videos/vid_xxx/content",
"file_path": "/cache/tasks/vid_xxx/outputs/videos/result.mp4",
"artifact_id": "local:tasks/vid_xxx/outputs/videos/result.mp4",
"artifact_metadata": {
"backend": "local",
"relative_path": "tasks/vid_xxx/outputs/videos/result.mp4",
"task_id": "vid_xxx",
"media_type": "video",
"filename": "result.mp4",
"size_bytes": 987654
}
}
```

Expand Down
15 changes: 13 additions & 2 deletions docs/en/stream_server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ TeleFuser stream server supports two interaction modes, both available over WebR
| **Server Push** | WebRTC (RTP) | Server → Client | Real-time preview, text-to-video streaming |
| **Bidirectional** | WebRTC (RTP + DataChannel) | Client ↔ Server | Interactive generation, keyboard/camera control, speech-to-video |

`telefuser stream-serve` starts a stream-only app. It exposes `/v1/stream/*`, `/v1/stream/webrtc/*`, and
`/v1/service/*`; it does not expose request-response task routes, file-download routes, or OpenAI-compatible
`/v1/images` and `/v1/videos` routes. Use `telefuser serve` for batch task submission.

### Server Push (WebRTC)

```
Expand Down Expand Up @@ -105,7 +109,7 @@ telefuser stream-serve <pipeline_file> [OPTIONS]
|------|---------|-------------|
| `--port`, `-p` | `8088` | Server port |
| `--host` | `0.0.0.0` | Bind address |
| `--security-level` | `strict` | Pipeline validation level (`none`, `basic`, `strict`, `sandbox`) |
| `--security-level` | `strict` | Pipeline validation level (`none`, `basic`, `strict`, `sandbox`). `sandbox` is a best-effort restricted-load check, not runtime isolation. |
| `--skip-validation` | `false` | Skip pipeline file security checks |

### Examples
Expand Down Expand Up @@ -227,6 +231,9 @@ Server-push chunks should contain the following fields:
| `/v1/stream/sessions/{session_id}` | DELETE | Both | Close session (pipeline + WebRTC) |
| `/v1/stream/sessions/{session_id}/status` | GET | Both | Get session status |

The stream app also exposes service endpoints such as `/v1/service/health`, `/v1/service/ready`,
`/v1/service/metadata`, and `/v1/service/metrics/json`.

### WebRTC: SDP Offer

**POST** `/v1/stream/webrtc/offer`
Expand Down Expand Up @@ -290,6 +297,10 @@ Response (`200 OK`):
}
```

**DELETE** `/v1/stream/sessions/{session_id}` closes the pipeline session and WebRTC session together when both
owners are present. If one side has already disappeared but the other closes successfully, the endpoint still returns
`200 OK`; pipeline-side close failures are logged as warnings instead of being silently ignored.

### WebRTC: DataChannel Protocol (Bidirectional)

In bidirectional mode, the client-created `"telefuser"` DataChannel carries JSON messages in both directions.
Expand Down Expand Up @@ -757,7 +768,7 @@ Pipeline files are validated before loading. Use `--skip-validation` only for de
| `none` | Disable validation |
| `basic` | Static AST analysis |
| `strict` | Static AST analysis plus import restrictions |
| `sandbox` | Full sandbox execution |
| `sandbox` | Strict checks plus a best-effort restricted-load validation step. This is not runtime isolation. |

---

Expand Down
Loading
Loading