Server Warden

A self-hosted process supervisor and live monitoring dashboard for long-running services — game servers (FiveM, MTA, Minecraft, whatever you run), Discord bots, background workers, anything that's just a command you want to keep alive. It is intentionally process-agnostic: Server Warden doesn't know or care what you're running, only that it's a command that should stay up.

Why this exists

Running game server infrastructure and backend services for a living means the same small set of problems keeps coming back: a process dies at 3am and nothing restarts it, you have no idea what its memory footprint looked like right before it fell over, the logs that would explain the crash rotated out or were never captured, and the "backup script" is a forgotten cron entry on one box. Server Warden is a single, small, self-hosted tool that does those four things properly — process supervision with real crash-loop protection, resource monitoring, live log tailing, and scheduled backups — with a dashboard on top, and nothing else. No database, no multi-tenant auth, no cloud dependency. One JSON file, one Node process, one operator.

Architecture

┌─────────────────────────────────────────────────────────┐
│  Browser dashboard (public/) — vanilla HTML/CSS/JS       │
│  REST for CRUD + actions, WebSocket for live state/logs  │
└───────────────────────────┬───────────────────────────────┘
                             │
┌───────────────────────────▼───────────────────────────────┐
│  Express API (src/server.ts, src/api/routes.ts)            │
│  single API-key auth (src/auth.ts)                          │
├──────────────┬──────────────┬───────────────┬──────────────┤
│ProcessManager│ResourceMonitor│  LogManager   │ BackupManager│
│spawn/restart │ real CPU/mem  │ rotating logs │ zip + cron   │
│backoff+crash-│ per-PID, OS-  │ + WS broadcast│ scheduler +  │
│loop protect  │ specific      │               │ pruning      │
└──────────────┴──────────────┴───────────────┴──────────────┘
                             │
                    data/store.json (atomic
                    write-tmp-then-rename)

Everything is plain TypeScript, no framework on the frontend. State lives in one JSON file (data/store.json by default) written atomically — same pattern used elsewhere in this author's tooling: write to a temp file, then rename() over the real one, so a crash mid-write can't corrupt the store.

Screenshots: none included in this initial release — this was built and verified against the real running dashboard over the REST/WebSocket API in a terminal-only environment without browser/screenshot access. The HTML/CSS is real and was written and reviewed carefully, but nobody has visually confirmed it renders as intended in an actual browser yet. Treat the visual polish as unverified until someone opens http://localhost:4790 and looks.

Features

• Process supervision — register a service (name, command, args, cwd, env vars, restart policy). Start/stop/restart via child_process.spawn. Tracks PID, start time, uptime, and restart count.
• Auto-restart with backoff + crash-loop protection — on an unexpected exit, if the restart policy calls for it, the service restarts with exponential backoff (backoffBaseMs * 2^attempt, capped at backoffCapMs). If a service crashes more than maxRestartsInWindow times within crashWindowMs, Server Warden gives up and marks it crash-looped instead of retrying forever — this is what actually got tested against a script that crashes on a timer (see Verification below).
• Resource monitoring — real per-PID CPU% and memory, polled every 2s. Windows uses Get-CimInstance Win32_PerfFormattedData_PerfProc_Process (falls back to wmic if PowerShell is unavailable); Linux reads /proc/<pid>/stat and /proc/<pid>/status directly and computes CPU% from deltas. Both paths are real implementations, not stubs — see Verification.
• Live log tailing — stdout/stderr captured per service, written to a size/line-count-rotating log file (keeps the last N rotated files), and streamed to connected dashboard clients over WebSocket in real time.
• Scheduled backups — config-driven: back up a file or directory to a timestamped zip on a small built-in 5-field cron schedule (min hour dom month dow, including the standard OR semantics when both day-of-month and day-of-week are restricted), keep the last N backups, prune older ones automatically. Uses archiver for real zip creation (the one non-trivial runtime dependency in this project, justified because Node has no built-in zip container writer, only zlib's raw compression).
• REST API + API-key auth — CRUD for services and backup jobs, start/stop/restart/backup-now endpoints, historical log range endpoint. Single configured API key checked via x-api-key header (or Authorization: Bearer) — this is a tool for one operator running their own infrastructure, not a multi-tenant service, so that's all the auth it has.
• Dashboard — service cards (status, uptime, CPU/RAM, restart count, start/stop/restart), a live WebSocket log viewer with pause-on-scroll-up and a resume button, a backups panel (list + manual "run now" trigger), and a settings page for the API key/server URL.

What's deliberately out of scope

• One-click restore. The backups panel lists what exists and where; you unzip it yourself into the right place. Restoring over a live directory is destructive, and automating that safely (stop the service, confirm the target, handle partial overwrites) is a bigger feature than this release covers. Documented, not silently missing.
• Multi-user auth / roles. One API key, one operator. If you need multiple people with different permissions, put this behind your own reverse-proxy auth layer.
• Process trees deeper than one level on Windows get killed via taskkill /T /F (kills the tree), and on POSIX via killing the process group (detached: true + process.kill(-pid, ...)). This was verified for a direct child; a process that re-parents its own children away from the supervised process is outside what any of these mechanisms can catch, on any platform.

Setup

git clone https://github.com/kasapdev/server-warden.git
cd server-warden
npm install
cp .env.example .env
# edit .env and set WARDEN_API_KEY to a long random string
npm run build
npm start
# or for development with auto-reload:
npm run dev

Open http://localhost:4790 (or whatever PORT you set) and enter your API key when prompted — it's stored in the browser's localStorage, nowhere else.

If you don't set WARDEN_API_KEY, Server Warden generates a random one on startup and prints it to the console — fine for a quick local test, useless across restarts since it changes every time.

API reference

All routes below are under /api and require the x-api-key header (or Authorization: Bearer <key>) except /api/health.

Method	Path	Description
GET	/health	No auth. Liveness check.
GET	/services	List all services (config + runtime state).
GET	/services/:id	Get one service.
POST	/services	Create a service. Body: name, command, args[], cwd, env{}, restartPolicy (never|on-crash|always), backoffBaseMs, backoffCapMs, maxRestartsInWindow, crashWindowMs.
PUT	/services/:id	Update a service's config. Must be stopped first.
DELETE	/services/:id	Delete a service. Must not be running.
POST	/services/:id/start	Start the service.
POST	/services/:id/stop	Stop the service (kills the real process tree).
POST	/services/:id/restart	Stop then start.
GET	/services/:id/logs?lines=200	Historical log tail (from the on-disk log file).
GET	/backup-jobs	List backup jobs.
POST	/backup-jobs	Create a job. Body: name, sourcePath, schedule (cron), keepCount.
DELETE	/backup-jobs/:id	Delete a job (unschedules it).
POST	/backup-jobs/:id/run	Run a backup immediately; prunes old backups past keepCount afterward.
GET	/backups?jobId=	List backup records, optionally filtered by job.

WebSocket: connect to /ws?apiKey=<key>. Messages are {"type":"log","data":{serviceId,stream,line,timestamp}} for new log lines and {"type":"state","data":{id,state}} for service state changes (status/PID/uptime/CPU/memory/restart count), pushed on every resource poll and every status transition.

Verification

This is the part that actually matters, so here's exactly what was done, on this Windows 11 machine, against the real running server — not just unit tests:

1. Real process supervision. Registered test-service.js (included in this repo — ticks output every 500ms, exits with code 1 after 5 ticks to simulate a crash) as a service via the real REST API, started it, and confirmed via Get-Process -Id <pid> that a real Windows process existed at the PID the API reported.
2. Real resource monitoring. A CPU-bound dummy process (while(Date.now()-start<30000){Math.sqrt(Math.random())}) reported cpuPercent: 7.92 and memoryBytes: 14520320 from the live API — a genuinely non-zero, plausible value computed via Get-CimInstance Win32_PerfFormattedData_PerfProc_Process, not a stub. A near-idle process correctly reported cpuPercent: 0.
3. Real log capture. The service's stdout/stderr were captured to logs/<id>.log and streamed live over the WebSocket — a small test client (ws-test.mjs, not part of the shipped code) received 5 real-time alive tick N log messages pushed from the server as they were produced.
4. Real crash + auto-restart + backoff. With restartPolicy: "always", backoffBaseMs: 1000, backoffCapMs: 8000, the crash-tick service was observed restarting with delays of 1000ms, 2000ms, 4000ms, 8000ms, 8000ms (capped) — read directly out of the log file, matching computeBackoffMs exactly. Each restart produced a genuinely new PID (41916 → 15836 → 37932 → 18096 → 10380 → 2656).
5. Real crash-loop protection. With maxRestartsInWindow: 5 and crashWindowMs: 60000, the 6th crash within the window flipped the service to status: "crash-looped" and auto-restart stopped — confirmed via the API ("status":"crash-looped","restartCount":5) and the log line crash-loop detected (6 crashes within 60000ms) — giving up auto-restart.
6. Real process termination. POST /stop on a running service was followed by Get-Process -Id <pid> returning nothing — the real OS process was gone (via taskkill /T /F on Windows), and the status correctly settled at stopped rather than crashed, because the manual-stop flag suppresses the restart policy.
7. Real backups. Created a backup job pointed at a real directory with a real file in it, triggered backup-jobs/:id/run four times with keepCount: 2. Confirmed via the API and directly on disk (Get-ChildItem backups/) that only the newest 2 zip files remained — the older 2 were physically deleted. Extracted one of the surviving zips with Expand-Archive and confirmed the real file content came back intact.
8. Auth. Confirmed /api/services returns 401 with no key and with a wrong key, and 200 with the correct key; /api/health works with no key at all.

A separate bug was actually found and fixed during this process: a failed spawn() (e.g. a bad command) only emits Node's error event, never exit — the initial implementation left the service stuck reporting status: "running" forever with a dangling process reference. Fixed by funneling both error and exit through the same settle-once exit handler (src/process-manager.ts), so a spawn failure is now treated as an abnormal exit and goes through the same restart-policy/crash-loop logic as a real crash. Re-verified after the fix.

Automated unit tests

npm test runs 29 tests (Vitest) covering the pure logic that's cheapest and most valuable to pin down with unit tests: exponential backoff calculation, crash-loop window evaluation, restart-policy decisions, the cron parser (including step values, ranges, lists, and the day-of-month/day-of-week OR quirk), log rotation planning, and backup pruning. The process-supervision behavior itself (section above) is inherently about real OS interaction and was verified by actually running it, per the task's own reasoning — Windows process APIs and child_process behavior are exactly the kind of thing that looks right in a mock and breaks in reality (as the spawn-error bug above demonstrates).

Configuration reference (.env)

See .env.example. Key variables: PORT, WARDEN_API_KEY, WARDEN_DATA_DIR, WARDEN_LOG_DIR, WARDEN_BACKUP_DIR.

License

MIT — see LICENSE.