Quickstart

spn is the missing everything tool for writing and building C code:

An extremely fast build system with a built-in object cache that's designed for CI
TOML manifests for defining projects
A package manager that supports everything from "pin the world to exact commits" to "treat C like JS"
Sandboxed, hermetic builds that never run arbitrary native code

It ships as a single file static executable, or an embeddable library with a stable C ABI. It runs natively on Linux, macOS, and Windows and it supports most common toolchains (MSVC, GCC, Clang, MinGW, Zig) and can cross compile to any target your toolchain supports.

If any part of your build pipeline would benefit from fast, cached, reproducible compilation of C code backed by tools that are obsessed with making your life easier, then you're in the right place!

spn is in alpha. Join our Discord if you'd like to chat, need help, or want to scream into the void that I'm wasting my life.

Quickstart

Install

macOS & Linux

curl -fsSL https://spn.spader.zone/install | sh

Windows

irm https://spn.spader.zone/install.ps1 | iex

Create a project

Let's use the following program, which needs to link to Lua, as an example:

#include <stdio.h>
#include "lauxlib.h"
#include "lualib.h"

int main(void) {
  lua_State *L = luaL_newstate();
  luaL_openlibs(L);
  if (luaL_dofile(L, NULL) != LUA_OK) {
    fprintf(stderr, "%s\n", lua_tostring(L, -1));
    return 1;
  }
  lua_close(L);
  return 0;
}

Add a dependency

First, initialize your project. We'll pull Lua from the default package index for now; later, we'll talk about making your own packages.

spn init
spn add lua

Those commands create a project manifest, spn.toml, which defines a single [[bin]] executable.

[package]
name = "demo"
version = "0.1.0"

[[bin]]
name = "demo"
source = ["main.c"]

[deps.package]
lua = "5.4.6"

Build and run

Then, after copying the source code into main.c:

spn build
./build/debug/demo

Cross compile

Cross compiling works out of the box. The following commands produce build/x86_64-windows-gnu/debug/main.exe, build/aarch64-macos/debug/main, and a bare metal build/aarch64-freestanding-none/debug/main.elf with no libc or startup files:

spn build --target x86_64-windows-gnu
spn build --target aarch64-macos
spn build --target aarch64-freestanding-none

A kernel image needs its own memory layout. Give the target a linker script and any raw flags the link needs:

[[bin]]
name = "kernel"
source = ["kernel.c", "start.S"]
linker_script = ["kernel.ld"]
link_flags = ["-Wl,--build-id=none", "-Wl,-n"]

Congratulations! You now have a C program which:

Can be compiled to any OS, architecture, and ABI
Builds extremely quickly and incrementally

Editor setup

spn generates compile_commands.json out of the box, with no instrumentation needed.

Overview

Building C code should be this easy, always. This demo is just a fraction of the functionality spn offers. It can replace nearly every tool in what is today a patchwork ecosystem of developer tooling:

Build tools like CMake, Make, and Ninja
Package managers like Conan, vcpkg, pkg-config, or "whatever's on my distro"
Build caches like ccache or sccache
Binary tools like xxd, patchelf, and objcopy
Compilation database tooling like bear or compiledb
Toolchain managers like crosstool-NG, dockcross, CMake toolchain files

Let's take a look at everything it can do in more detail!

Packages

Packages are defined by TOML manifests.

Targets

A package can define as many executables, libraries, tests, and examples as it would like. These are called targets. They have fields like source, or include, or system_deps, and most fields can be configured at the package level for all targets or for an individual target.

[package]
include = ['common']

[[lib]]
source = ['source/lib.c']
include = ['source/lib']

[[example]]
source = ['example/foo/main.c']

[[test]]
source = ['test/main.c', 'test/foo.c']
define = ['SOMETHING']
system_deps = ['m']

A [[bin]] and a [[script]], for instance, are both executables, but their difference isn't strictly cosmetic. [[bin]] entries are taken to be exports of your package, able to be pulled in by consumers. [[script]] entries aren't. [[test]] entries are executed in spn test, and compiled to test/ instead of bin/.

Target fields

source

include and headers

define and flags

system_deps

deps

kinds

cxx

System dependencies

Conditional configuration

Any field in your package can be keyed on any fact of the build (target OS, architecture, ABI, build mode, optimization level, sanitizer settings). Clauses are structured data, not a DSL, and an entry with multiple clauses gets them ANDed together. Here's some common examples:

source = [
  { path = "source/backend/win32.c", when = { os = "windows" } },
  { path = "source/backend/posix.c", when = { os = { not = "windows" } } },
]
flags = [
  { value = "-mfpu=neon", when = { os = "linux", arch = "aarch64" } },
]
define = [
  { value = "USE_DEBUG_ALLOC", when = { mode = "debug", sanitize_address = false } },
]
system_deps = [
  { value = "pthread", when = { os = { not = "windows" } } },
]
deps = [
  { pkg = "tracy", when = { mode = "debug" } },
]

when clauses

Fact keys

Options as keys

Options declared in the manifest work as keys, too, so a feature flag can gate sources and dependencies together:

[options.freetype]
type = "bool"
default = true
define = "UI_FREETYPE"

[[lib]]
name = "ui"
source = [
  { path = "source/text_freetype.c", when = { freetype = true } },
]
deps = [
  { pkg = "freetype", when = { freetype = true } },
]

Options

Packages provide options. Options can be enumerations or booleans; enums are mutually exclusive, and an unresolvable conflict is a build error. Booleans are additive. For example:

[options.tls]
type = "enum"
values = ["schannel", "openssl", "off"]
default = [
  { when = { os = "windows" }, value = "schannel" },
  { when = { os = { not = "wasi" } }, value = "openssl" },
  { value = "off" },
]

[options.zstd]
type = "bool"
default = false

Consumers can then set these options

Platforms

macOS

[package.macos]
min_os = { major = 12 }

[lib.macos]
frameworks = ["Cocoa", "IOKit", "CoreVideo", "OpenGL"]

When linking, your binary's min_os is the max across everything in it. If a dependency needs macOS 12, your binary targets macOS 12.

Windows

[[bin]]
# ...
windows = { subsystem = "windows" }

Embedding files

spn can embed arbitrary files and bytes (from build scripts) in your binary by creating an object file and header that anything can link to trivially. For example, this configuration:

[[bin]]
name = "whatever"
source = ["main.c"]
embed = [
  "asset/fonts/inconsolata.ttf",
  { path = "asset/data.json", symbol = "data_json", data_type = "u8", size_type = "u64" },
  { path = "asset/shaders", dir = true, dest = "shaders" },
]

Produces this header:

extern const unsigned char asset_fonts_inconsolata_ttf [109384];
extern const unsigned long long asset_fonts_inconsolata_ttf_size;

extern const u8 data_json [2213];
extern const u64 data_json_size;

extern const unsigned char shaders_quad_frag [412];
extern const unsigned long long shaders_quad_frag_size;

extern const unsigned char shaders_quad_vert [287];
extern const unsigned long long shaders_quad_vert_size;

typedef struct {
  const char* path;
  const void* data;
  unsigned long long size;
} spn_embed_entry_t;
static const unsigned int spn_embed_count = 4;
static const spn_embed_entry_t spn_embed_manifest[] = {
  { "asset/fonts/inconsolata.ttf", asset_fonts_inconsolata_ttf, 109384 },
  { "asset/data.json", data_json, 2213 },
  { "shaders/quad.frag", shaders_quad_frag, 412 },
  { "shaders/quad.vert", shaders_quad_vert, 287 },
};

Dependencies

spn is a real package manager, built specifically for C and C++.

Local dependencies

The simplest dependency is a path to a directory with an spn.toml:

[deps.package]
foo = { path = "packages/foo" }

And this is what the package might look like:

[package]
name = "foo"
version = "6.9.0"

[[lib]]
name = "foo"
source = ["foo.c"] # Relative to foo's manifest

This is just an old fashioned vendored dependency. You could download LLVM, check in the entire source tree, and have an spn.toml that builds this local copy. If you then regained your sanity, you can point the exact same manifest at Git and spn will manage the checkout for you exactly as if it were an "official" package:

[package.upstream]
url = "https://github.com/tspader/foo.git"
commit = "6937fa02243da7b693c5692cea84a696950d4669"

Conditional and private dependencies

Configuration

Options

Packages can provide options, either as an enumeration or a boolean. Options for foo are set in [config.foo], like this:

[config.foo]
tls = "openssl"
zstd = true

Linkage

Libraries declare the linkages they support:

shared
static
source, which compiles the package's sources directly into your target as if they were your own files.

[config.foo]
kind = "shared"

Patching

What if you need to patch foo? You don't want to go back to the technology of the ancients, like submodules or vendoring. You want a specific commit with just a few changes. Do this in your spn.toml:

[deps.package]
foo = "6.9.0"

[patch.foo]
files = ["patches/foo.patch"]

That's it! Patches fold into the build system like everything else. If you change the contents of a patch, add or remove patches, you'll still get a correct, incremental build.

Your dependencies can't patch. If you ask for a given commit, you get exactly that commit.

The lockfile

Building

spn build compiles your project and all dependencies. Compilation targets a triple (e.g. x86_64-windows-gnu) and uses a profile (e.g. release, O2, statically linked).

Build output

By default, spn outputs to build/. If you explicitly pass a triple, it uses build/triple/profile. Otherwise, it uses build/profile. If you were to run this:

spn build
spn build --target x86_64-windows-gnu

It would produce this:

build/
├── debug/
└── x86_64-windows-gnu/
    └── debug/

Inside a given build, executables are placed at the top level for easy running. All other artifacts (headers, libraries, your dependencies' artifacts) are placed in store/ in the usual way. Finally, a JSONL file with a detailed trace of the build is in .spn/build.jsonl

build/debug/
├── store/
│   ├── bin/
│   │   ├── main.exe
│   ├── lib/
│   │   ├── libwhatever.a
│   ├── include/
│   │   ├── whatever.h
│   │   └── ...
└── .spn/
    └── build.jsonl

Selecting targets

Profiles

Packages are compiled against a profile, which contains the target triple, toolchain, build mode, optimization level, sanitizers, etc.

[profile.default]
toolchain = "zig"
linkage = "static"
standard = "c11"
mode = "debug"

Modes and optimization

Sanitizers

Cross compilation

Target triples

--os, --arch, --abi

CI

spn was built for CI from day one:

The build cache is designed to scale to your entire team
First class cross compilation destroys your insane matrix of runners
Out of the box, zero configuration integrations with GitHub Actions
The CLI can output a structured JSONL stream. You never have to scrape an error message
If it doesn't work how you want, write your own CLI that links to libspn instead
@spader Are these weak? People care about fast builds mostly but we got that elsewhere...

Caching

spn has a built in build cache which caches any intermediate artifact that your build creates, like ccache or sccache. Builds are incremental by default. That doesn't mean "incremental on my machine". That means incremental. Build foo.exe on one machine, and the cache is designed such that any subsequent machine building foo.exe can see a fully cached build.

That's because, at its core, spn is a content addressed DAG. If you've never seen the terms, let's take a ride!

Incremental builds

This means that everything in your build is turned into nodes in a graph. If something in the middle of the graph changes, we know exactly what needs to be rebuilt and in what order. This is like every other incremental build system that has ever existed.

Content addressing

If you use CMake, you've probably encountered this:

# Build once. After this, builds are incremental. You are happy.
make

# Ah, but you gotta check out another branch
git checkout whatever
git checkout main

# This is a full rebuild, because Git touched every file and made it
# appear to have been edited since the last build. You are sad.
make

That's because CMake, and in fact most build systems, lie to you. They tell you that they know when foo.c changed. But they have no god damn idea! None whatsoever! Now, they have proxies which in practice are...fine. Like, for example, a file's mtime. Last build was at 3:00, foo.c says 3:05, let's rebuild it.

But files get touched all the time! For no reason! Sometimes, they even go backwards, like when you decompress an archive, and then your build is wrong instead of just slow. This is the first problem: Traditional build systems have trouble knowing when something changed.

There's a beautiful solution to both of these things at once! If you're having trouble giving everything a unique identity, and you're having trouble figuring out when a given thing changed, make their identity be their content. When we build foo.o, we hash its bytes. Let's say that hash comes out to, miraculously, 0x69. Now, there's no such thing as foo.o; there's just a file called 0x69 in the cache.

That's great, but the next build still needs to be able to know that, ah, yes, we need the cache entry keyed at 0x69. To do this, we look at all of the inputs to foo.o:

foo.c, of course
Let's say that the compiler reported that stdint.h was used, too

We hash their content too, and get 0x420 and 0x5F3759DF. Then, all we do is write down a fact:

If the inputs to the compiler are 0x420 and 0x5F3759DF, then the output will be 0x69

Next time, when we're ready to build foo.o, we have all of its inputs ready to go. Take those inputs, and ask the fact machine if it knows the answer for that set of inputs. If foo.c was edited, the inputs are no longer (0x420, 0x5F3759DF). If spum.h got added to the build, then there are now three inputs rather than two. And, if the inputs are the same but 0x69 isn't in the cache, all you have to do is rebuild!

This is beautiful. Identity is content; content is identity. A file doesn't have a name. It simply is. There are, of course, many kinks to work out in such a system, but they're all tractable.

This is the exact principle behind Bazel, BuildXL, and Nix, and it's the fundamental reason why spn is so good at caching your builds across machines.

--force

Running tests

Output

compile_commands.json

JSON event stream

Build scripts

You need to run code in your build. Most people solve this by writing brittle scripts in Bash or Powershell, or, at best, a language like Python. spn solves this with WebAssembly¹. It embeds a WASM runtime, and will automatically compile arbitrary C programs to WASM modules that run in the build graph. You do not write code against a subset of C, or against a DSL. You write regular code, in the language you were using anyway.

Phases

configure

build

Writing a script

The sandbox

Script dependencies

API

If you're porting over an existing build, you can continue to use what you have.

Targets

Files and directories

Custom nodes

Embedding files

Logging

Example: code generation

Toolchains

Defaults

Zig

Probing

Bring your own

[[toolchain]]

Hosted artifacts

Target support

MSVC

Package indexes

When you run spn add lua, a version appears from the mist. Where does it come from? And what happens when I want to maintain my own versions of Lua, or some private package?

Directory indexes

A package index is just metadata about what packages exist, and what versions are available. The simplest possible index is just a directory of packages:

packages
├── flecs
│   └── spn.toml
├── fmt
│   └── spn.toml
├── imgui
│   └── spn.toml
├── sdl2
│   ├── spn.c
│   └── spn.toml
├── sdl2_mixer
│   └── spn.toml
└── tracy
    └── spn.toml

Point your build at it like this, or enable it globally in ~/.config/spn/spn.toml:

[[index]]
name = "randy"
path = "./packages"

If you only care about one version existing at a time, this is all you need. If you just want to build your personal libraries, or use some common packages in different projects, you're done.

Git indexes

Let's say you have a project that uses sqlite==3.51.0. You like it. But then, Richard Hipp returns from an ayahuasca retreat having had a vision: 4.0. You can prompt an LLM from inside a query, to say the least.

You want to start a new project and use it, but you still need 3.51.0. You've graduated to a second directory. This time, it's a directory of JSONL metadata:

index
└── sqlite.jsonl

Each JSONL entry is a release specifying the version, where the source code is, where the manifest is, etc.

{
  "name": "sqlite",
  "version": "4.0.0",
  "yanked": false,
  "source": {
    "url": "https://github.com/sqlite/sqllm.git",
    "rev": "a1de0694c693a5d741b2b424e5dfea45eaa30e69"
  },
  "manifest": {
    "url": "https://github.com/you/manifests.git",
    "rev": "c7faa527fd",
    "dir": "sqlite"
  },
  "paths": {
    "manifest": "spn.toml",
    "script": "spn.c"
  }
}

Note the manifest field. If sqllm decided that spn is the tool of the future, and decided to have a first party manifest that lives in their repo, this field would point to sqlite/sqllm.git. But since we're writing a package for a third party library that doesn't know we exist, we point it at a repository that we own. These entries are created with spn publish, not by hand.

Use the index the same way as the simpler one:

[[index]]
name = "randy"
url = "git@github.com:you/index.git"

Let's dig into that a little more, because it's important: Package indexes are just Git repositories. You're going to really like this:

Commits are transactional, reversible, and everything else Git does for your source code
It comes with provenance and auth that you already have set up
Every god damn developer on the planet knows how to use it

If you've ever used Homebrew and waited multiple tens of seconds for kegs to be sterilized, you may balk at this. "Spader," you might say, "Git doesn't scale for package indexes". And you'd be right! But you'd also be wrong:

spn isn't tied to Git. Index backends are just interfaces; there's a stubbed out HTTP backend in there already if you want to host it on Cloudflare or whatever
C is not JS. It's not a distro, it's not Rust. Nobody uses packages in C because there are no good package managers for all its special little quirks. We're talking orders (plural) of magnitude less scale.

Global configuration

Publishing

spn publish

Namespaces

The ecosystem

Here's some of the packages which have native, high quality spn manifests. The full list is in the package manifest repository. If a package is on this list, it has full support. That means you can cross compile it, pull in its optional dependencies, build its examples, and so on. Everything.

Embedding `spn`

There's a stable C ABI

It works with any event loop

It's so much better at hotloading than your crappy script

Workspaces

`spn` is a task runner

`spn` is a test runner

Why do I care?

"My CI isn't absurdly fast with zero configuration"

Look. I get it. Setting up the mess of tooling and integrations needed to cache an object file reliably is hard. It's so much easier to just sink into that warm abyss of twenty minute builds (twenty minutes is normal, right?) and tell yourself it doesn't matter.

Twenty minutes is not normal! An hour is not normal! 90% of your builds in CI should be so fast that you have to double check that it actually happened.

If this isn't the case for whatever you work on, send me an email right god damn now! Building native code is an extremely well understood problem. The problem's not that we don't know how, it's that we know how, and, well, it's pretty fucking hard unless you spend a lot of time on it. You, person who are presumably making and/or selling something other than an obscure meta-tool, don't have that much spare time².

takes quite a bit of rigor and care that most folks who are actually making and selling things can't spare.

"My company doesn't approve tools with dependencies (e.g. Python or JS)"

spn has zero dependencies. It is exactly one binary. Builds are hermetic and sandboxed by default, and it's as easy to pin the exact commits of your dependencies as it is to use packages from the index.

Can this replace...

...bear, compiledb?

Yes. spn generates compile_commands.json out of the box, with no instrumentation needed.

Development

Building

Testing

PRs

WebAssembly is a platform agnostic binary target; instead of compiling code for x86_64 or ARM64 machine code and running it with your CPU, you compile it to WASM bytecode and run it inside a regular program. ↩
And if you did, you'd probably spend it running your hands through the soft day-warmed grass of some shaded grove, those whom you love most by your side, feeling the beams of the sun against your face and basking in the simple pleasure of being alive. ↩

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
assets		assets
docs		docs
example		example
include		include
source		source
test		test
tools		tools
vendor		vendor
.clang-tidy		.clang-tidy
.gdbinit		.gdbinit
.gitattributes		.gitattributes
.gitignore		.gitignore
CMakeLists.txt		CMakeLists.txt
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
spn.lock		spn.lock
spn.toml		spn.toml

Latest commit

History

Folders and files

Repository files navigation

Quickstart

Install

macOS & Linux

Windows

Create a project

Add a dependency

Build and run

Cross compile

Editor setup

Overview

Packages

Targets

Target fields

source

include and headers

define and flags

system_deps

deps

kinds

cxx

System dependencies

Conditional configuration

when clauses

Fact keys

Options as keys

Options

Platforms

macOS

Windows

Embedding files

Dependencies

Local dependencies

Conditional and private dependencies

Configuration

Options

Linkage

Patching

The lockfile

Building

Build output

Selecting targets

Profiles

Modes and optimization

Sanitizers

Cross compilation

Target triples

--os, --arch, --abi

CI

Caching

Incremental builds

Content addressing

--force

Running tests

Output

compile_commands.json

JSON event stream

Build scripts

Phases

configure

build

Writing a script

The sandbox

Script dependencies

API

Targets

Files and directories

Custom nodes

Embedding files

Logging

Example: code generation

Toolchains

Defaults

Zig

Probing

Bring your own

[[toolchain]]

Embedding `spn`

`spn` is a task runner

`spn` is a test runner