SD API Backend

A FastAPI-based backend service for Stable Diffusion image generation, inpainting, multi-model merging, and model management. Supports SD 1.5, SDXL, Illustrious XL, and Flux pipelines with asynchronous job processing, model caching, and optional LoRA adapter integration.

This is a rewrite effort of the SD-WebApp project, and has been brewing for a bit now. The idea is to decouple the UI from the actual inference backend. This repo will host the backend code.

Implemented Features

• Text-to-Image Generation - Submit async jobs for SD 1.5, SDXL, Illustrious XL, and Flux models, receive results via job status polling
• Inpainting - Image inpainting using specialized pipelines with base image and mask (base64) inputs, including Flux's strength-based re-noising inpaint
• Batch Generation - Submit multiple generation jobs in a single request
• Model Comparison - Generate images from multiple checkpoints side-by-side for comparison
• Multi-Model Merging - Merge two or more models using configurable methods (add, average, slerp), with weight coefficients and multi-step recipe execution (SD 1.5/SDXL/Illustrious XL only — see TODO.md for planned Flux merge support)
• Asynchronous Job Queue - Non-blocking API with worker-based async pipeline execution via asyncio.Queue
• Model Caching - LRU cache (max 2 pipelines) for expensive diffusion model loading, auto-eviction via access order
• LoRA Support - Load and fuse PEFT-based LoRA adapters into the base pipeline with configurable strength
• Custom VAE - Replace the default VAE with custom models loaded from disk
• Multi-Device Auto-Detection - Automatic fallback: MPS (Apple Silicon) > CUDA (NVIDIA) > CPU with dtype auto-selection (float16 for SD/SDXL/Illustrious XL, bfloat16 for Flux, float32 for CPU)
• Flux Memory Offload - Flux pipelines use enable_model_cpu_offload() instead of a full device move to fit its larger (~12B parameter transformer + T5-XXL encoder) footprint on consumer hardware
• Configurable Output - Optional disk persistence of generated images and metadata to outputs/images/ and outputs/json/ as PNG and JSON
• System Endpoints - Health checks, system info (GPU, cache stats, queue stats), and configuration exposure
• CORS Middleware - Configurable CORS support, enabled by default with wildcard origins

API Endpoints

All endpoints are prefixed with /api/v1.

Generation

Method	Route	Description
POST	/sd15/generate	Submit SD 1.5 text-to-image generation job
POST	/sd15/inpaint	Submit SD 1.5 inpainting job
POST	/sd15/generate/batch	Submit multiple SD 1.5 generation jobs
POST	/sd15/generate/compare	Generate from multiple SD 1.5 models for comparison
POST	/sdxl/generate	Submit SDXL text-to-image generation job
POST	/sdxl/inpaint	Submit SDXL inpainting job
POST	/sdxl/generate/batch	Submit multiple SDXL generation jobs
POST	/sdxl/generate/compare	Generate from multiple SDXL models for comparison
POST	/illustrious/generate	Submit Illustrious XL text-to-image generation job
POST	/illustrious/inpaint	Submit Illustrious XL inpainting job
POST	/illustrious/generate/batch	Submit multiple Illustrious XL generation jobs
POST	/illustrious/generate/compare	Generate from multiple Illustrious XL models for comparison
POST	/flux/generate	Submit Flux text-to-image generation job
POST	/flux/inpaint	Submit Flux inpainting job (strength-based re-noising)
POST	/flux/generate/batch	Submit multiple Flux generation jobs
POST	/flux/generate/compare	Generate from multiple Flux models for comparison

Model Management

Method	Route	Description
GET	/models/{model_type}/{resource_type}	List available models (model_type: sd15/sdxl/illustrious/flux, resource_type: checkpoints/loras/vaes)
GET	/models/{model_type}/{resource_type}/{model_name}/metadata	Get safetensors metadata for a specific model

Model Merging

Method	Route	Description
POST	/merge/merge	Merge two models into one new checkpoint with configurable method (add, average, slerp), alpha weight, and metadata preservation
POST	/merge/batch	Merge one base model against multiple target models simultaneously, saving each output into a specified subdirectory
POST	/merge/recipe	Execute a multi-step merge recipe where each step takes the current result and merges it with a new target model using its own method/alpha

Merging currently supports sd15, sdxl, and illustrious only. Flux's transformer-based state dict isn't compatible with the current key-by-key UNet tensor merge — see TODO.md.

Job Management

Method	Route	Description
GET	/jobs	List all jobs with current status
GET	/jobs/{job_id}	Get detailed status and result for a specific job
DELETE	/jobs/{job_id}	Cancel a pending or running job

System & Info

Method	Route	Description
GET	/system/health	Health check endpoint
GET	/system/info	System info (GPU, data type, cache stats, queue stats)
GET	/info/schedulers	List available noise schedulers
GET	/info/app_settings	Return current application configuration

Additional Swagger UI docs at /docs and ReDoc at /redoc (auto-generated by FastAPI).

Class Diagrams

Core Pipeline Architecture

┌─────────────────────────────────┐
│     BasePipelineGenerator       │  (ABC)
│  ─────────────────────────────  │
│ - _model_path: str              │
│ - _pipeline_type: PipelineType  │
│ - _device: str                  │
│ - _dtype: torch.dtype           │
│  ─────────────────────────────  │
│ + loadPipeline()                │
│ + loadInpaintPipeline()         │
│ + pipeToDevice()                │
│ + forward(prompt, params)       │  # text-to-image
│ + forward_inpaint(base, mask…)  │  # inpainting
│ + addLorasToPipeline()          │
│ + loadCustomVAE()               │
│ + getResolutions()              │
│ + SCHEDULERS (overridable)      │
└──────────┬──────────────────────┘
           │ inherits
    ┌───────┬───────┬───────┐
    │       │        │       │
┌────────┐┌────────┐┌─────────────┐┌────────┐
│ SD15   ││ SDXL   ││ Illustrious ││ Flux   │
│Pipeline││Pipeline││ Pipeline    ││Pipeline│
│Generator││Generator││ Generator  ││Generator│
└────────┘└────────┘└─────────────┘└────────┘
                                      overrides:
                                      _init_device (bf16),
                                      pipeToConfiguredDevice
                                      (enable_model_cpu_offload),
                                      forward/forward_inpaint
                                      (true_cfg_scale, strength)

Service Layer

┌──────────────────────┐     ┌─────────────────────┐     ┌──────────────────┐
│  GenerationService   │────>│   ModelCacheManager  │────>│ BasePipelineGen  │
│ ──────────────────── │     │ ───────────────────  │     │ ───────────────  │
│ + generate()         │     │ + get_pipeline()     │     │ - cache: dict    │
│ + inpaint()          │     │ + load_pipeline()    │     │ + forward()      │
│ + batch_generate()   │     │ + clear_cache()      │     │ + addLoras...()  │
│ + compare_models()   │     │ + get_stats()        │     └──────────────────┘
└──────────┬───────────┘     └─────────────────────┘
           │
           │ submits
           v
┌──────────────────────┐
│     JobQueue          │
│ ──────────────────── │
│ + submit_job()       │  # returns job_id
│ + get_job_status()   │
│ + cancel_job()       │
│ + list_jobs()        │
│ + stop_workers()     │
│ - _queue: Queue      │
│ - _jobs: dict        │
│ - _worker()          │  # async background loop
└──────────────────────┘

Request / Response Models

┌─────────────────┐   ┌────────────────┐   ┌──────────────────┐
│ Generation      │   │ InpaintRequest │   │ LoraConfig       │
│ Request         │   │ ────────────── │   │ ──────────────── │
│ ──────────────  │   │ - image_base64 │   │ - name: str      │
│ - prompt: str   │   │ - mask_base64  │   │ - strength: float│
│ - model: str    │   │ - image: str   │   └────────┬─────────┘
│ - params: dict  │   │ - mask: str    │            │
│ - loras: list[] │──>│ - resolution   │            │
│ - vae: str|None │   │ - negative_prompt
│ ──────────────  │   └────────────────┘
│ + to_dict()     │
└────────┬────────┘
         │ validates
         v
┌─────────────────────────────────────────┐
│        JobResponse / JobStatusResponse   │
│  ─────────────────────────────────────  │
│  - job_id: str (UUID4)                  │
│  - status: Job (QUEUED/RUNNING/...)     │
│  - result: JobResult|None               │
│    - image_base64: str                  │
│    - metadata: dict                     │
│    - seed: int                          │
└─────────────────────────────────────────┘

Configuration

┌──────────────────┐     ┌─────────────────┐
│ parameters.yaml  │────>│   AppConfig     │
│ ─────────────── │     │ ─────────────── │
│ checkpoints:     │     │ - _params: dict │
│   sd15: path     │     │ ─────────────── │
│   sdxl: path     │     │ + get_model_... │
│ loras:           │     │ + setup_paths() │
│ vaes:             │     │ + get_output_.. │
│ storage:         │     │ + as_dict()     │
│ api:             │     └─────────────────┘
│   queue:         │          ▲
│   cache:         │          │ lru_cache
│   cors:          │     ┌─────────────────┐
└──────────────────┘     │ get_app_config()│
                         └─────────────────┘

Overall Call Flow

Image Generation (Primary Workflow)

1. Client sends POST /api/v1/{sd15,sdxl}/generate with prompt, model, and generation parameters
2. Router resolves model path from AppConfig, converts LoRA configs to dicts, validates request
3. JobQueue generates a UUID4 job_id, creates a Job object, enqueues it, and returns immediately with JobResponse(status="queued")
4. Background workers (_worker loop) dequeue jobs and set status to RUNNING
5. GenerationService calls ModelCacheManager.get_pipeline() to get or load the diffusion pipeline
6. Pipeline loads in a thread pool executor (blocking diffusers call)
7. LoRA adapters are fused via peft and custom VAE is swapped if provided
8. Pipeline's forward() (text-to-image) or forward_inpaint() is invoked with generation params
9. Result is an image (base64-encoded PNG) plus metadata dict (GenerationMetadata container)
10. If persist_to_backend is enabled, image is saved to outputs/images/{uuid}.png and metadata to outputs/json/{uuid}.json
11. Job is marked COMPLETED and result stored for client polling

Model Merging Call Flow

1. Client sends POST /api/v1/merge/merge (or /merge/batch, /merge/recipe) with model specs, merge method, and weight parameters
2. Router validates model paths from AppConfig
3. JobQueue generates a job_id and returns immediately with JobResponse(status="queued")
4. Background workers dequeue jobs and invoke MergeService.merge_models()
5. MergeService loads model weights using diffusers and safetensors, applies the specified merge algorithm (add, average, or slerp), and saves the merged checkpoint to the configured output directory
6. Recipe merges loop through each step sequentially, feeding the result of the previous step as the base for the next
7. Job is marked COMPLETED with the path to the newly created merged model

Job Status Polling

Client: GET /api/v1/jobs/{job_id}
    → JobQueue.get_job_status()
    → Returns JobStatusResponse { job_id, job_type, status, result, error, timestamps }
    → If status=CALLED: extract result.image_base64 → decode → save PNG
    → If status=FAILED:  display result.error

Model Cache Flow

ModelCacheManager.get_pipeline(model_type, model_path, pipeline_type)
  1. Build cache key: "{model_type}:{model_name}:{pipeline_type}"
  2. Cache HIT → update access order → return cached pipeline
  3. Cache MISS → _load_pipeline() in thread pool
     → Instantiate pipeline generator → loadSDPipeline() → pipeToConfiguredDevice()
  4. If at capacity (>= max_models) → evict least-recently-used
  5. Insert new pipeline into cache

Startup / Shutdown Flow

Uvicorn starts → api.app:app (pre-created via create_app())

Lifespan startup:
  1. Load AppConfig from parameters.yaml
  2. Create FastAPI app with CORS middleware
  3. Mount routers (sd15, sdxl, models, merge, jobs, system, info)
  4. JobQueue singleton created → start_workers() spawns N async worker tasks
  5. Log "SD API Backend started"

Lifespan shutdown:
  1. JobQueue.stop_workers() → sets _running=False, drains queue, cancels workers
  2. Log "SD API Backend shutdown complete"

Usage Examples

Python (Async with httpx)

import asyncio
import httpx
import json

BASE_URL = "http://localhost:8000/api/v1"

async def generate_image():
    async with httpx.AsyncClient() as client:
        # Submit generation job
        response = await client.post(
            f"{BASE_URL}/sd15/generate",
            json={
                "prompt": "a beautiful sunset over mountains",
                "model": "v2-1024px-pruned.safetensors",
                "params": {
                    "width": 512,
                    "height": 512,
                    "num_inference_steps": 30,
                    "guidance_scale": 7.5,
                    "seed": 42
                }
            }
        )
        job = response.json()
        job_id = job["job_id"]
        print(f"Job submitted: {job_id}")

        # Poll for completion
        while True:
            status = await client.get(f"{BASE_URL}/jobs/{job_id}")
            data = status.json()
            if data["status"] == "completed":
                result = data["result"]
                # Decode and save image
                import base64
                with open(f"output_{job_id}.png", "wb") as f:
                    f.write(base64.b64decode(result["image_base64"]))
                print(f"Image saved. Seed: {result['seed']}")
                break
            elif data["status"] == "failed":
                print(f"Job failed: {data['error']}")
                break
            await asyncio.sleep(1)

asyncio.run(generate_image())

cURL

# Submit a generation job
curl -X POST http://localhost:8000/api/v1/sd15/generate \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "a cyberpunk city at night",
    "model": "v2-1024px-pruned.safetensors",
    "params": {
      "width": 512,
      "height": 512,
      "num_inference_steps": 30,
      "guidance_scale": 7.5,
      "seed": 42
    }
  }'

# Submit a Flux generation job
curl -X POST http://localhost:8000/api/v1/flux/generate \
  -H "Content-Type: application/json" \
  -d '{
    "positive_prompt": "a photo of an astronaut riding a horse on mars",
    "model_checkpoint": "flux1-dev.safetensors",
    "width": 1024,
    "height": 1024,
    "steps": 20,
    "cfg_scale": 3.5,
    "scheduler": "FlowMatch Euler",
    "seed": 42
  }'

# Check job status
curl http://localhost:8000/api/v1/jobs/{job_id}

# List all jobs
curl http://localhost:8000/api/v1/jobs

# Cancel a job
curl -X DELETE http://localhost:8000/api/v1/jobs/{job_id}

# List available checkpoints
curl http://localhost:8000/api/v1/models/sd15/checkpoints

# Merge two models
curl -X POST http://localhost:8000/api/v1/merge/merge \
  -H "Content-Type: application/json" \
  -d '{
    "model_a": "model_a.safetensors",
    "model_b": "model_b.safetensors",
    "alpha": 0.5,
    "merge_type": "average"
  }'

# Multi-step recipe merge
curl -X POST http://localhost:8000/api/v1/merge/recipe \
  -H "Content-Type: application/json" \
  -d '{
    "base_model": "base.safetensors",
    "steps": [
      {"target": "style1.safetensors", "alpha": 0.4, "merge_type": "add"},
      {"target": "style2.safetensors", "alpha": 0.3, "merge_type": "average"}
    ],
    "output_path": "merged/"
  }'

# Health check
curl http://localhost:8000/api/v1/system/health

With LoRA Adapters

response = await client.post(
    f"{BASE_URL}/sdxl/generate",
    json={
        "prompt": "a portrait in anime style",
        "model": "sd_xl_base_1.0.safetensors",
        "params": {
            "width": 1024,
            "height": 1024,
            "num_inference_steps": 25,
        },
        "loras": [
            {"name": "anime_style.safetensors", "strength": 0.8}
        ]
    }
)

Configuration

Copy parameters.yaml.example to parameters.yaml and adjust paths:

checkpoints:
  sd15:
    path: 'models/checkpoints/SD15'
  sdxl:
    path: 'models/checkpoints/SDXL'
  illustrious:
    path: 'models/checkpoints/Illustrious'
  flux:
    path: 'models/checkpoints/Flux'
loras:
  sd15:
    path: 'models/loras/SD15'
  sdxl:
    path: 'models/loras/SDXL'
  illustrious:
    path: 'models/loras/Illustrious'
  flux:
    path: 'models/loras/Flux'
vaes:
  sd15:
    path: 'models/vae/SD15'
  sdxl:
    path: 'models/vae/SDXL'
  illustrious:
    path: 'models/vae/Illustrious'
  flux:
    path: 'models/vae/Flux'
# Flux checkpoints are transformer-only single-file checkpoints; the shared
# text encoders (CLIP-L + T5-XXL), tokenizers, VAE, and scheduler are loaded
# once from a local diffusers-format directory and reused across checkpoints.
flux:
  components_path: 'models/flux_components'
storage:
  persist_to_backend: true
  output_images: 'outputs/images'
  output_json: 'outputs/json'
api:
  host: "0.0.0.0"
  port: 8000
  workers: 1
  job_queue:
    num_workers: 1
    result_ttl: 3600
  model_cache:
    max_models: 2
  cors:
    enabled: true
    allow_origins: ["*"]

Flux Setup

Flux checkpoints only ship the transformer; the shared text encoders, tokenizers, VAE, and scheduler must be populated once under flux.components_path in standard diffusers subfolder layout (text_encoder/, tokenizer/, text_encoder_2/, tokenizer_2/, vae/, scheduler/):

# FLUX.1-schnell is ungated and license-friction-free for testing;
# FLUX.1-dev requires accepting its gated license on Hugging Face first.
huggingface-cli download black-forest-labs/FLUX.1-dev \
  --exclude "transformer/*" --local-dir models/flux_components

Then place a single-file Flux transformer checkpoint (e.g. flux1-dev.safetensors) under checkpoints.flux.path.

Running

# Install dependencies
uv sync

# Start the server
uv run uvicorn api.app:app --host 0.0.0.0 --port 8000