Slemify

Generate, train, and validate small specialist models. One YAML, one command.

Small specialist models handle the high-volume, repetitive tasks in your AI workflows (classification, routing, extraction) so your LLMs can focus on what they're best at. Slemify automates the path to a validated, production-ready model: a CPU-trained encoder classifier for routing and labeling, or a stock generative SLM (GGUF) served on CPU and grounded by RAG for free-form reasoning. How you deploy that model is up to you.

slemify deploy --config expert.yaml

Slemify picks the right model family from project.task:

• task: generation — a causal LM served stock on CPU (GGUF/llama.cpp): Slemify downloads the base model, converts it to GGUF, and quantizes it. No fine-tuning; knowledge comes from RAG at serving time. For reasoning and free-form output.
• task: classification — a frozen encoder + a lightweight head, trained and served entirely on CPU in seconds. For routing, intent, and labeling.
• task: scoring — the same encoder-head family with a regression head, trained and served on CPU. Returns a single number in [0,1]. For risk/quality/confidence guardrails.
• task: extraction — a CPU-trained token tagger that pulls typed entity spans out of free-form text. Returns a list of {type, text} spans. For entity/field extraction from tickets, logs, and messages.
• task: embedding — a domain-tuned text embedding model, contrastively fine-tuned and served on CPU (ONNX). Returns a vector. For retrieval (RAG) over your own corpus.

When to Use an SLM

Not every task needs a frontier model. Most agentic AI systems have "hot spots". repetitive sub-tasks that run thousands of times a day with the same pattern. These are ideal for a specialized SLM:

Task Type	Example	Why SLM
Classification	Alert triage, intent routing, document categorization	Same pattern, different inputs. Fast, predictable output.
Scoring	Risk/quality/confidence guardrails on a config, answer, or request	One number in [0,1] decides auto-approve vs escalate. Cheap on every request.
Extraction	Pull structured fields from logs, invoices, clinical notes	Rigid output schema. Doesn't need world knowledge.
Routing	Pick which tool/API/agent handles a request	Binary or multi-class decision. Sub-100ms matters.
Validation	Safety checks, compliance gates, format verification	Rule-based logic baked into weights. Runs on every request.

The criteria: high repetition, low semantic variation, structured output. If the task looks the same every time with different inputs, an SLM can do it faster and cheaper than a general-purpose LLM. often with higher accuracy for that specific task.

SLMs + LLMs Together

Slemify doesn't replace LLMs. It adds a fast, cheap layer alongside them.

[Request] → [SLM Router] → high confidence → [SLM Result] → done (50ms, $0)
                          → low confidence  → [LLM Fallback] → done (3s, $0.01)

The inference endpoint exposes an OpenAI-compatible API (/v1/chat/completions). Any agent, orchestrator, or application can call it directly via HTTP. Set llm_endpoint in your config to any OpenAI-compatible API (vLLM, llama.cpp, Bedrock proxy) for LLM fallback. The SLM handles 70-90% of requests at fixed cost. The LLM handles the rest.

How It Works

expert.yaml → [DATA] → [TRAINING] → [SERVING + VALIDATION]
                 │          │                   │
            Ingest +    CPU train, or      Deploy model,
            Synthetic   convert+quantize   measure it live,
            via Bedrock (generation)       write the report

1. Data. Ingests your raw data from S3. For trained tasks, Bedrock generates synthetic training pairs from your source content and you verify them before training. Generation is served stock, so it skips synthetic data.
2. Training. Encoder tasks fit a head or contrastively tune an encoder on CPU in seconds to minutes. Generation has nothing to fine-tune, so this stage downloads the base model, converts it to GGUF, and quantizes it on CPU. Either way the output is uploaded to S3.
3. Serving + Validation. Deploys the model on a live endpoint and runs the report Job against it: held-out accuracy against a majority-class baseline for classifiers, stock-vs-tuned recall for embedding models, a serving profile (decode speed, time to first token, bandwidth ceiling) for generation models, plus endpoint latency and the on-demand rate of the node it landed on.

The output is a model in S3 and a report you can read in the terminal or as HTML. The serving deployment that Slemify creates is production-quality and serves as a reference for your own infrastructure. You can use it as-is, adapt it, or serve the GGUF with any compatible runtime (llama.cpp, vLLM, Ollama). See the Serving deep dive for deployment guidance and best practices.

Quick Start

Prerequisites

• EKS cluster with EKS Auto Mode or self-managed Karpenter
• S3 bucket for data and artifacts
• AWS credentials with Bedrock access
• kubectl configured for your cluster

1. Define your task

apiVersion: slemify/v1

project:
  name: k8s-autoscaling-triage
  task: classification
  domain: >
    Classify Kubernetes autoscaling support queries into a routing
    category. Each message is classified into exactly one category:
    karpenter_config, keda_config, hpa_config, pdb_disruption,
    spot_interruption, multi_resource, or noise for off-topic messages.
  labels:
    routing:
      - karpenter_config
      - keda_config
      - hpa_config
      - pdb_disruption
      - spot_interruption
      - multi_resource
      - noise

model:
  base: ""       # encoder model ID (a text encoder for classification)
  head: logistic # classifier head: logistic | linear | mlp

data:
  bucket: slemify-data
  path: k8s-autoscaling/data/
  sources:
    - path: queries/
      type: raw
  synthetic:
    pairs: 1200   # model: defaults to Slemify's Bedrock model (override: SLEMIFY_BEDROCK_MODEL)
  evaluation:
    pairs: 150    # model: optional, same default
    sources:
      - path: eval-queries/
        type: raw

training:
  spot: true

2. Upload your training data

aws s3 sync ./data/queries s3://slemify-data/k8s-autoscaling/data/queries/
aws s3 sync ./data/eval-queries s3://slemify-data/k8s-autoscaling/data/eval-queries/

3. Deploy

slemify deploy --config expert.yaml

Slemify handles data processing, synthetic pair generation, training, quantization, and validation. The resulting GGUF model is uploaded to S3. You then deploy it in your own infrastructure using the reference deployment as a starting point.

4. View the report

slemify report --config expert.yaml

Prints the report summary and opens the HTML version in your browser. What it contains depends on the task: accuracy against the majority-class baseline, real-vs-synthetic split, confusions, and calibration for classifiers; stock-vs-tuned recall@k for embedding models; a serving profile for generation models. Endpoint latency and the instance type's on-demand rate are in every report. See the Report deep dive.

How Much Data Do I Need?

Task Type	Training Examples	Notes
Classification (routing, triage)	200-500	Binary or multi-class. Clear categories.
Scoring (risk, quality, confidence)	500-1,200	Regression target in [0,1]. Spread examples across the full range.
Extraction (fields from text)	500-1,000	More examples = better edge case coverage.

These apply to the trained (encoder-family) tasks. task: generation is served stock and grounded by RAG, so it needs no training data. Quality matters more than quantity. 500 well-curated instruction-response pairs beat 10,000 noisy ones. Bedrock generates synthetic examples from your source data, so you don't need to write them all by hand.

Cost

Item	Cost
Model prep (CPU: encoder train, or download + convert + quantize for generation)	<~$1
Synthetic data for trained tasks (Bedrock)	~$10-50
Total to produce a model	~$10-50

Generation no longer uses a GPU: it is downloaded, converted to GGUF, and quantized on CPU, so it has no training cost and no synthetic-data cost. Inference cost depends on how you deploy. The reference deployment (llama.cpp on CPU Spot) runs at ~$117/mo per replica. Throughput scales linearly: 3 replicas = 3x throughput at 3x cost. No rate limits, no per-token charges. See the Serving deep dive for cost comparisons across CPU, GPU, and LLM API options.

This isn't only a $/token argument. Many production Kubernetes clusters report single-digit percent GPU utilization, because latency-insensitive and structurally simple work (routing, classification, validation, embedding) ends up parked on the same expensive pool as the generation workloads that actually need it. Moving that work to CPU isn't just cheaper per call, it frees up GPU capacity for the model that genuinely needs it. For the four encoder-family tasks (classification, scoring, extraction, embedding), there's effectively no volume threshold to clear: CPU training and serving are cheap at any scale, so the usual self-hosting break-even math (which only favors self-hosting past a real volume threshold, often millions of tokens/day) doesn't apply — that math is specific to serving a generative model, and Slemify sidesteps it by not fine-tuning generation in the first place.

Examples

• K8s Autoscaling Analyst. Tiered SLM system: a triage classifier routes queries, a 30B-A3B MoE analyst (~3B active params/token, served on CPU) produces structured reasoning about Karpenter/KEDA/HPA misconfigurations
• K8s Autoscaling Risk Scorer. A task: scoring encoder-head model that rates a config change's operational risk 0.0–1.0 on CPU — a cheap guardrail that auto-approves low-risk changes and escalates high-risk ones to the analyst
• K8s Autoscaling Retriever. A task: embedding model contrastively fine-tuned on in-domain (question, document) pairs — a domain-tuned RAG retriever that beats a stock encoder on recall, trained and served on CPU
• Support Ticket Extractor. A task: extraction token tagger that pulls service, error, version, and environment entities out of free-form support tickets on CPU — and a worked demonstration of when extraction earns ML over a regex baseline (open-vocabulary prose) and when it doesn't (structured configs)

Deep Dives

Technical docs covering the design decisions, best practices, and research behind each pipeline stage. Written for Platform Engineers.

• Getting Started. End-to-end tutorial: build an agentic K8s expert from scratch
• Data Stage. Raw data quality, synthetic generation, label taxonomy, verification
• Training Stage. Encoder-head and embedding training on CPU, the stock generation convert/quantize path, model sizing, quantization
• Serving Stage. Reference deployment, CPU inference, autoscaling guidance
• Report Stage. Held-out metrics against baselines, real-vs-synthetic split, endpoint latency, generation serving profile

Architecture

The pipeline runs on Kubernetes (EKS). The output is a GGUF model in S3.

• EKS Auto Mode or Karpenter. CPU nodes for training, conversion, and the reference deployment (no GPU in the pipeline). Slemify detects which one the cluster runs and creates a matching NodePool
• llama.cpp. GGUF conversion and quantization, plus CPU inference (used in the reference deployment and validation report)
• Pod Identity. IAM access to S3 and Bedrock, no static credentials
• Systems Manager. Remote container builds via SSM, no SSH keys or open ports required

The reference serving deployment (llama.cpp on CPU) is included for validation and as a starting point. You can serve the GGUF model with any compatible runtime: llama.cpp, vLLM, Ollama, or any tool that reads GGUF files.

Commands

Command	Description
slemify deploy	Run the full pipeline
slemify deploy --stage training --no-wait	Submit a stage and exit
slemify status my-project	Show pipeline progress
slemify status my-project -o json	Machine-readable status for agents
slemify validate	Validate config without deploying
slemify report	Print the report summary and open the HTML report in the browser
slemify report --output my-report.html	Save report to a custom path
slemify report --no-open	Download without opening the browser
slemify build	Build container images to ECR

FAQ

Q: When should I use an SLM vs just calling an LLM API? A: If the task is repetitive, structured, and runs more than ~1,000 times/day, or if data can't leave your VPC. Below that volume, an LLM API is simpler and fine. This threshold applies to the encoder-family tasks (classification, scoring, extraction, embedding) — training and serving them on CPU costs cents regardless of volume, so there's little downside to starting early. task: generation is a different calculation: you're comparing a self-hosted CPU (or GPU) deployment against an LLM API's per-token price, and that comparison only favors self-hosting past real volume (industry self-hosting break-even estimates for generative models commonly land in the millions of tokens/day). Below that, keep generation on the LLM API even if you've already adopted Slemify for routing/classification around it.

Q: Can a small model really match a frontier LLM? A: For general, open-ended tasks, no. For a scoped task with the right grounding, yes, and we measured it rather than assuming it. In the k8s-autoscaling example we ran a control experiment: swap the CPU-served SLM analyst for the frontier LLM the demo uses for escalation (via Bedrock), through the identical pipeline (same retrieved context, same faithfulness gate, same judge, same eval). The frontier LLM scored the same as the SLM. On the cases both missed, they missed the same way, which points at retrieval and eval quality as the bottleneck, not model capability. The demo keeps this experiment reproducible: set ANALYST=llm on the orchestrator and re-run the eval to compare any change against the LLM baseline yourself. External evidence agrees: Salesforce's xLAM-2-8B beat GPT-4o and Claude 3.5 at tool calling on the Berkeley Function-Calling Leaderboard. Specialization plus grounding beats size.

Q: If adding CPU replicas gives me the throughput, when would I still want a GPU? A: Replicas scale throughput linearly (3 replicas = 3x requests at 3x cost), but they never make a single request faster. That distinction decides it. A GPU earns its 3-10x hourly premium in three situations: (1) a single-request latency floor CPUs cannot meet, which in practice means cold prefill of long contexts (on the demo's MoE analyst, a 2,400-token RAG context takes 1.5-2 minutes of prompt processing cold, sub-second warm); (2) sustained aggregate demand high enough to keep a GPU busy around the clock, where its tokens-per-dollar beats a fleet of CPU replicas — this is a utilization crossover you should compute with your own traffic, not a rule; and (3) training and fine-tuning, which stay on GPUs. What a GPU does not buy on grounded, in-domain tasks is quality (see the previous question — measured at parity). Bursty or modest traffic, output-heavy tasks, and anything a warm prompt cache serves fast are the CPU fleet's home turf. See When you still want a GPU for the full breakdown.

Q: Does fine-tuning always improve quality? When doesn't it help? A: No, and Slemify is deliberate about this. Fine-tuning helps most when the model has to learn something it doesn't already know, and it backfires when the model already has the skill and only lacks the facts:

• Generation — not fine-tuned, on purpose. For a knowledge task the base model can already reason and write; what it lacks is your facts, and RAG supplies those at serving time better than training does. We tested fine-tuning the generative analyst in the k8s example and it made answers worse, so Slemify serves generation stock (download, convert to GGUF, quantize) and grounds it with RAG. (Adapting weights to a lower-precision quantization grid — QAT/QAD — is a different kind of fine-tuning and would be its own future task.)
• Classification / scoring — the head (a router taxonomy, a risk rubric) doesn't exist in any pretrained model, so it must be trained. These tasks always benefit; the question is just whether you have enough data.
• Embedding — domain-tuning a retriever measurably helps when your corpus uses vocabulary or relationships a general encoder hasn't specialized in (in our k8s example, recall@1 improved ~12 points over stock).
• Extraction — domain-dependent, and the example shows both sides. Pulling entities from open-vocabulary prose (support tickets) a trained tagger beats a regex/gazetteer baseline by a wide margin (F1 0.63 → 0.89, driven by open-vocab service/error names). But pulling fields from structured text (k8s YAML configs) a plain parser already wins, so there a trained model adds nothing. The extractor example documents both.
• Reranking — not a Slemify task, on purpose. A strong general-purpose cross-encoder is already excellent at judging (query, document) relevance, and fine-tuning it reliably needs curated hard negatives (human-labeled "looks relevant but isn't"). We tested it: synthesizing those over an overlapping technical corpus produces false negatives that degrade a good model (NDCG@5 0.85 → 0.58 on a fair eval). Since fine-tuning doesn't help, Slemify doesn't do it — running a stock cross-encoder reranker on CPU with no GPU is a serving pattern, shown in the k8s-autoscaling demo, not a model Slemify builds.

The reports always show the metric against an honest baseline (majority class for classification, stock-vs-tuned for embedding, predict-the-mean for scoring, a regex/memorization baseline for extraction) so you can see whether training actually helped on your data — not just trust that it did. Give the report human-labeled held-out records (data.evaluation.labeled) and it also scores real and synthetic inputs separately, which is where synthetic-data drift shows up first.

Q: How much context can the router/classifier handle? A: The default text encoder caps at ~512 tokens (roughly 350-400 words) and silently truncates beyond that — but that's usually fine, because a router only needs the decision-relevant slice, not the whole input. Feed it the question, the latest turn, or one retrieved chunk at a time, and scale by sending it less, more often. If the signal is genuinely buried in long text, trim or summarize to the relevant part first, or point model.base at a longer-context encoder. See the k8s-autoscaling routing example and the serving deep dive for the right-tool / wrong-tool guide.

Q: What about RAG? A: SLMs and RAG solve different problems, and Slemify now covers both sides. RAG retrieves relevant context for knowledge questions; the retrieval step itself is an embedding model, which you can domain-tune with task: embedding. The classification/scoring tasks handle routing and guardrails where you don't need retrieval, you need a fast decision. They work well together: an encoder classifier routes the query, a domain-tuned embedding model retrieves the knowledge, and a generative SLM writes the answer.

Q: Can I use a different base model? A: Yes. For task: generation, any HuggingFace causal LM that llama.cpp's GGUF converter supports. For encoder-head tasks (task: classification, task: scoring) and task: embedding, any sentence-transformers text encoder. task: extraction (v1) is the exception — its feature-based token tagger uses no encoder, so model.base is omitted. The auto-sizer adjusts infrastructure based on the task and model size.

Q: What happens during a Spot interruption? A: The encoder-family training jobs run in seconds to minutes on CPU, so an interrupted job simply re-runs. The generation convert job (download + GGUF + quantize) is a one-shot, bandwidth-heavy run, so Slemify pins it to on-demand capacity to avoid a mid-run reclaim forcing a full re-download. Serving runs on Spot and is replaced automatically.

Agent Skill

Slemify includes an agent skill compatible with Claude Code, OpenAI Codex, Gemini CLI, and Cursor. The skill teaches AI coding agents how to identify SLM opportunities in your system, design the agent's role, write the expert.yaml config, run the pipeline, and interpret results.

Install in Claude Code:

/plugin install slemify@<your-repo>

Or reference the skill directly:

"Use the Slemify skill to identify which of my LLM calls could be replaced with a specialized SLM."

The skill includes templates for two patterns:

• Router Agent (task: classification): a CPU encoder classifier for fast routing and intent decisions
• Analyst Agent (task: generation, dense 7-8B or small-MoE): structured reasoning grounded by RAG

References

• Small Language Models are the Future of Agentic AI (NVIDIA, 2025). Position paper arguing SLMs under 10B parameters can handle 60-80% of agentic AI tasks
• xLAM: Large Action Models (Salesforce). 8B model that beat GPT-4o at tool calling, proving specialization beats size
• Forbes: Don't Default to the Biggest AI Model. 40-70% of agentic AI invocations can use SLMs
• Hallucination Propensity in Small Models. Research on knowledge mismatch between fine-tuning data and base model knowledge
• llama.cpp. GGUF conversion, quantization, and CPU inference engine
• Model Context Protocol. How SLMs expose tools to AI assistants
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (Reimers & Gurevych, 2019). The sentence-embedding approach behind Slemify's encoder-head and embedding tasks
• EKS Best Practices: AI/ML CPU Inference. When CPU inference is appropriate and how to tune it