From 8c2b20e4bcd05b1785bfd35510ebc2c57b7d7bb2 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Thu, 27 Aug 2026 21:39:46 +0800 Subject: [PATCH 1/5] Move the portable example's pins, and record what the plan got wrong MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The specification does not change. Two implementations do: openkal-macos provides the five interfaces 0.8 added and openkal-windows provides four of them, so the example that demonstrates one program above three implementations names the versions that have them. .agents/docs gains section 9 of the capability-completeness plan --- what was implemented, the four decisions section 8 left open, and the five things the plan itself got wrong. Two of the five are worth naming here: ⚠️ `posix_spawn' was never missing. openkal-musl replaced musl's source when the port was written, and `system' and `popen' have worked through it ever since. What was missing was a criterion. ⚠️⚠️ `timeout_ns = 0' does not mean "do not wait" --- it means NO BOUND, which timeout.h states and kal_task_wait established. The plan had it backwards, and passing a caller's zero straight through turns the one call that must not wait into the one that never returns. And one thing nothing had noticed: no continuous integration anywhere selected these interfaces. Every backend ran the conformance suite as `full', which expands to the HOSTED set, so every section for net, datagram, timeout, exec and space was compiled with its body removed and reported as not examined --- in the same release that added them. --- ...ility-completeness-across-the-ecosystem.md | 491 ++++++++++++++++++ examples/portable/mcpp.toml | 4 +- 2 files changed, 493 insertions(+), 2 deletions(-) create mode 100644 .agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md diff --git a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md new file mode 100644 index 0000000..f0484c9 --- /dev/null +++ b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md @@ -0,0 +1,491 @@ +# Capability completeness across the openkal ecosystem + +**Date**: 2026-08-27 +**Scope**: openkal 0.8 and every repository that implements or consumes it +**Status**: design, for review. Nothing here is implemented. + +--- + +## 0. Why this document exists + +Three reports from one user, on one workload — a seven-member C++23 modules +workspace that compiles and links statically through the native path: + +| report | subject | +| --- | --- | +| `mcpp-community/mcpp#514` | engine: dependency units missed the target C++-ABI include set; cache key omitted the target axis | +| `mcpplibs/openkal-musl#13` | the port publishes musl's internal `hidden` macro to consumers | +| `mcpplibs/openkal-linux#13` | ENOSYS for fork, pipe, socket, chmod, symlink, copy_file_range | + +`#514` is closed: mcpp 2026.8.27.1 carries both halves. The other two are open, +and neither is a single defect. They are two visible points on one line: **the +specification has grown atoms that the layers above it do not yet use, and the +layers below it do not yet all provide.** + +This document states what is measured today, what "complete" can mean given +what openkal is, and what each repository would have to do. It does not decide; +it makes the decision reviewable. + +--- + +## 1. The measured baseline + +### 1.1 What openkal 0.8 declares + +Fifteen interfaces, ninety names in `SURFACE.txt`: + +``` +abort 2 datagram 6 env 5 exec 4 fs 19 memory 2 net 11 +process 8 random 2 space 2 stream 7 task 7 terminal 4 time 5 timeout 6 +``` + +### 1.2 What each backend implements + +Measured by searching each backend's sources for definitions of the names +`SURFACE.txt` lists. Every module's `*_props` entry is a `const` object rather +than a function, so a module reported as *n−1 / n* is complete. + +| interface | linux | macos | windows | opensbi | uefi | +| --- | :-: | :-: | :-: | :-: | :-: | +| abort, memory, stream | ✅ | ✅ | ✅ | ✅ | ✅ | +| env, time | ✅ | ✅ | ✅ | ✅ | — | +| fs, process, task, random, terminal | ✅ | ✅ | ✅ | — | — | +| **net** | ✅ | **—** | **—** | — | — | +| **datagram** | ✅ | **—** | **—** | — | — | +| **timeout** | ✅ | **—** | **—** | — | — | +| **exec** | ✅ | **—** | **—** | — | — | +| **space** | ✅ | **—** | **—** | — | — | + +⭐ **THE FIVE INTERFACES 0.8 ADDED EXIST ONLY ON LINUX.** macos and windows +decline them in whole, which clause 3 permits and clause 6.1 makes honest — a +program that uses one fails at the link, naming the operation. But it means any +capability built upon them is a Linux capability until those two move. + +### 1.3 What the C library routes + +`openkal-musl` 0.4.0 handles **70** `SYS_` cases. Of the surfaces the reports +name: + +| surface | routed | note | +| --- | :-: | --- | +| `pipe`, `pipe2` | ✅ | `kal_process_channel`, weak reference tested before call | +| `copy_file_range` | ✅ | | +| `execve`, `wait4`, `readlink`, `getdents64`, `dup`, `dup3`, `kill` | ✅ | | +| `socket`, `bind`, `listen`, `accept`, `connect` | **0** | falls to `default:` → `-ENOSYS` | +| `clone`, `fork` | **0** | `musl/src/thread/clone.c` excluded | +| `chmod`, `fchmod`, `fchmodat` | **0** | | +| `symlink`, `symlinkat`, `link` | **0** | | +| `poll`, `ppoll`, `select`, `pselect6` | **0** | | + +`musl/src/network/*.c` is **not excluded**: those files compile and call +`__socketcall`, which reaches the port's default arm. The gap is in the port, +not in musl's sources. + +Excluded sources that bound the surface today: + +``` +env/__libc_start_main.c env/__init_tls.c thread/__set_thread_area.c +thread/clone.c process/posix_spawn.c mman/mmap.c +internal/syscall_ret.c unistd/getcwd.c ldso/dl_iterate_phdr.c +linux/cache.c linux/{timerfd,eventfd,signalfd,inotify,epoll}.c +``` + +### 1.4 What the C++ runtime withholds + +`openkal-llvm-runtime` 0.2.0 excludes, on the bare-metal row only, +`libcxx/src/filesystem/*.cpp` and `libcxx/src/random.cpp`, keeping +`_LIBCPP_HAS_RANDOM_DEVICE` at 0 there. The hosted rows build both. So the C++ +side is already near its ceiling for hosted targets; what limits it is what the +C library below can answer. + +--- + +## 2. A correction that changes the shape of the problem + +An earlier reading of these reports concluded that `fork` was a capability the +specification had deliberately declined, on the grounds that a copy-on-write +duplicate is not a minimal capability every kernel has. + +**That was wrong.** `openkal.space` states: + +> Starts a context in a copy of the calling address space. The result is a +> process … The copy is taken at this call. + +That is fork's semantics. The atom exists, and openkal-linux implements it. What +does not exist is the *route*: `musl/src/thread/clone.c` is excluded and no +`SYS_clone` case is present. + +⚠️ The distinction that survives is narrower and still matters. `fork()` returns +twice at the call site; `kal_space_start` begins the copy at an entry function. +Section 4.2 addresses what follows from that. + +--- + +## 3. What "complete" can mean + +openkal is not POSIX and is not trying to become it. Clause 3 admits an +interface only when it is a minimal capability every kernel has and cannot be +composed from what is already present. So completeness has to be defined +against the atoms, not against POSIX: + +> **A layer is complete when every capability the atoms below it can express is +> reachable through the interface above it, and every capability they cannot is +> refused in a way the caller can act on.** + +That gives three categories, and the whole plan is the act of sorting the +reports' items into them: + +- **(A) Composable now.** The atoms exist and the layer does not use them. Pure + adaptation. No specification change. +- **(B) Composable with fidelity loss.** The layer can emulate the POSIX shape, + but what it reports will not be what the environment holds. Requires a stated + decision, not silence. +- **(C) Not composable.** The atoms are absent. Either the specification grows, + or the operation is refused permanently and the refusal is documented. + +--- + +## 4. The sort + +### 4.1 (A) Composable now — the largest group + +**Sockets, stream and datagram.** Every BSD operation the reports need maps: + +| BSD | atom | +| --- | --- | +| `connect` | `kal_net_connect` | +| `bind` + `listen` | `kal_net_listen` | +| `accept` | `kal_net_accept`, or `kal_timeout_accept` for a bounded wait | +| `read`/`write`/`send`/`recv` | `kal_net_stream` → `kal_stream_*`, or `kal_timeout_*` | +| `getsockname` / `getpeername` | `kal_net_local` / `kal_net_peer` | +| `shutdown` | `kal_net_shutdown` | +| UDP, whole surface | `kal_datagram_open/send_to/recv_from/local/close` | + +⚠️ **BSD SEPARATES `socket()` FROM `connect()`/`bind()`; openkal DOES NOT.** +`kal_net_connect` produces a connection; there is no unbound socket. The port +must therefore hold a descriptor in a *pending* state carrying domain, type and +protocol, and perform the atom at `connect`, or at `listen` after `bind` has +recorded the local endpoint. This is a port-layer state machine, not a gap. + +`sockaddr_in`/`sockaddr_in6` ↔ `kal_endpoint` conversion belongs beside it; +`openkal.kit`'s `parse_v4`/`format_v4` already establish the byte order and the +`addr_len`-selects-the-family convention. + +**Non-blocking and readiness.** `O_NONBLOCK`, `SO_RCVTIMEO` and small +`poll`/`select` sets are expressible with `kal_timeout_*` at `timeout_ns = 0` +and at finite bounds. This is O(n) per call over the set, which is correct and +unhurried; `epoll` stays excluded, as it is a Linux facility rather than a +capability. + +**`posix_spawn`.** `kal_process_spawn_with` + `kal_process_channel` + +`kal_process_wait` are exactly its three parts. `musl/src/process/posix_spawn.c` +is excluded today; a port implementation restores `posix_spawn`, `system` and +`popen` for every consumer that uses them — which is the majority of the +"shell out to a subprocess" workload the report describes. + +### 4.2 (A) with a design decision — `fork()` itself + +The atom is `kal_space_start(entry, arg, stack_top, out)`. The obstacle is +return-twice semantics: the child must resume at the call site, not at `entry`. + +Two candidate designs, and the choice should be made deliberately: + +1. **Trampoline.** The parent records a resume point before the call; `entry` + transfers to it. The copied address space contains the parent's stack at the + same addresses, so the recorded context is valid in the child. Cost: the + mechanism is architecture-aware and must be written per architecture. +2. **Refuse `fork()`, provide `posix_spawn`.** `fork()` returns `-ENOSYS` + consistently; everything that spawns a subprocess goes through 4.1. Cost: + programs that call `fork()` directly do not run. + +⭐ Option 2 is the smaller step and covers the reported workload. Option 1 is +the complete one. They are not exclusive — 2 can ship first and 1 later, and +neither changes the specification. + +### 4.3 (B) Composable with fidelity loss — permission bits + +`kal_node_info` carries `{ size, modified_ns, kind, writable }` — **one boolean, +not a mode word** — and `kal_fs_open_file` takes `write` and `create` flags, not +a mode. + +So `chmod(path, 0600)` cannot be honoured. Three answers, in decreasing honesty: + +1. Refuse: `chmod` returns `-ENOSYS`, `stat` reports a mode synthesised from + `writable` (`0666`/`0444` masked by nothing). The report's "expected 0600, + got 0777" becomes "expected 0600, refused" — which a caller can act on. +2. Map the owner-write bit onto `writable` and ignore the rest. `chmod(0600)` + succeeds and `stat` reports `0600` only if the port remembers it; across + processes it does not. +3. Ask the specification for a permission atom (see 4.4). + +⚠️ Option 2 is the one that silently lies, and it is also the one that makes the +most tests pass. It should not be chosen without saying so in the port's own +source. + +### 4.4 (C) Not composable — symbolic links, and the permission question + +`SURFACE.txt` has **no operation that creates or reads a symbolic link.** What +it has is an acknowledgement that links exist: `kal_node_link` as a node kind +and `KAL_FS_PROP_LINKS` as a property word. An implementation can therefore +*report* a link it encounters and cannot *make* one. + +⭐ That asymmetry is itself a finding. A property word declaring support for a +thing the interface offers no operation upon is a promise with no way to keep +it — the shape this ecosystem has recorded before as a specification's silence +being invisible from inside the specification. + +Two candidates for 0.9, each to be judged against clause 3 (minimal, universal, +not composable) and clause 6.4 (an operation some resources can never satisfy +does not belong on the interface): + +- `kal_fs_symlink(base, name, len, target, target_len)` and + `kal_fs_readlink(...)`. Universal? Windows has symbolic links but creating one + requires a privilege by default; a FAT volume has none. Clause 6.2's property + word is the established answer to "some resources cannot" — `KAL_FS_PROP_LINKS` + already exists and would finally have a reader. +- A permission operation. Harder: the concept is not universal in the same way + (a FAT volume, a UEFI system partition and a Windows ACL do not share a model), + and clause 6.3 records mechanisms considered and not adopted for exactly this + reason. **The recommendation of this document is to refuse permissions at the + specification level and choose 4.3 option 1 in the port.** + +### 4.5 Independent of the above — the `hidden` macro + +`openkal-musl#13`. `port/include/features.h` defines `hidden`, `weak` and +undefines `weak_alias` for every consumer, so a program cannot use `hidden` as +an ordinary identifier. + +Measured: **`musl/include/` — the public headers — contain zero uses of +`hidden`.** The block's stated purpose is a different case, and its own comment +says so: a non-musl C source compiled *with the internal overlay on its command +line*, which is what a board building compiler-rt does. + +Since mcpp 2026.8.27.1 added `[build] private_include_dirs` and this package now +uses it, an ordinary consumer no longer has the overlay on its line at all. The +fix is therefore to neutralise **only when the overlay is present** — that is, +only when the macro is already defined after `#include_next ` — +which preserves the compiler-rt case and releases the name to consumers. + +--- + +## 5. Per-repository work + +### 5.1 `openkal-musl` — the largest share + +| item | category | depends on | +| --- | :-: | --- | +| `hidden`/`weak` scoping (#13) | — | nothing | +| socket family → `kal_net_*` | A | backend provides `openkal.net` | +| datagram family → `kal_datagram_*` | A | `openkal.datagram` | +| `O_NONBLOCK`, `poll`/`select` → `kal_timeout_*` | A | `openkal.timeout` | +| `posix_spawn` → `kal_process_spawn_with` | A | `openkal.process` | +| `fork` (4.2) | A + decision | `openkal.space` | +| `chmod` family (4.3) | B + decision | — | +| `symlink` family | C | 0.9 | + +New descriptor kinds beside the existing `OKM_STREAM / OKM_CHANNEL / OKM_FILE / +OKM_DIR`: a pending socket, a connection, a listener, a datagram endpoint. + +⚠️ **EVERY NEW ROUTE MUST TAKE A WEAK REFERENCE AND TEST IT BEFORE CALLING**, +as `kal_process_channel` already does (`if (!kal_process_channel) return +-ENOSYS;`). `openkal.net` is optional; a strong reference would turn "this +backend declines the interface" into "this program does not link", making an +optional interface mandatory — which clause 6.1's link-time absence exists to +avoid, and which this port has been bitten by before. + +### 5.2 `openkal-macos`, `openkal-windows` — the gating work + +They implement ten of fifteen interfaces. The five they decline are the five +0.8 added, and three of them are what section 4.1 is built on. + +| interface | macOS | Windows | +| --- | --- | --- | +| net | BSD sockets, directly | Winsock 2, `WSAStartup` at first use | +| datagram | as above | as above | +| timeout | `poll` with a deadline; `SO_RCVTIMEO` | `WSAPoll`, overlapped I/O | +| exec | `mmap(MAP_JIT)` + `pthread_jit_write_protect_np` | `VirtualAlloc` + `FlushInstructionCache` | +| space | ⚠️ macOS: `fork` exists but is unsafe after threads; Windows: no equivalent | see below | + +⭐ **`openkal.space` ON WINDOWS IS THE ONE THAT MAY HAVE TO STAY DECLINED**, and +declining it is a legitimate outcome rather than a failure: clause 3 says in +whole or not at all, and clause 6.1 makes the absence a link error naming +`kal_space_start`. The design should not invent a fake. + +Until macos and windows provide net/datagram/timeout, the socket work of 5.1 is +Linux-only in effect. That is acceptable and should be stated in the port's +documentation rather than discovered. + +### 5.3 `openkal-linux` — small + +Already provides all fifteen. Its share is verification: the conformance suite +and the portable program exercise the atoms, but nothing yet exercises them +*through musl*. A test that opens a listener, connects to it, and transfers +bytes — written against POSIX, not against openkal — is the criterion that the +route in 5.1 works. + +### 5.4 `openkal-opensbi`, `openkal-uefi` — none + +Freestanding. They decline fs and above, and nothing here changes that. The +work is only to confirm that the new routes in openkal-musl remain absent +rather than undefined on these targets, which the weak-reference rule of 5.1 +already guarantees. + +### 5.5 `openkal-llvm-runtime` — follows, does not lead + +`std::filesystem::create_symlink` and `permissions` become available exactly +when 4.3 and 4.4 are answered; `` is already answered. No change is +needed in this repository for section 4.1 — the C++ layer reaches sockets +through the C library, not directly. + +⚠️ One item does belong here: the bare-metal row excludes +`libcxx/src/filesystem/*.cpp` while the hosted rows build it. If the hosted +rows are to report a *complete* `std::filesystem`, the exclusions and the +`__config_site` switches must be read against each other once more, in the way +`_LIBCPP_HAS_TERMINAL` was in 0.2.0. + +### 5.6 `openkal` — the specification + +Only if 4.4 is accepted: `kal_fs_symlink` / `kal_fs_readlink` in 0.9, with +`KAL_FS_PROP_LINKS` gaining a reader, plus `SURFACE.txt`, `SPEC.md` clause 3's +inventory, the conformance suite, and the five backends each providing or +declining in whole. + +**Recommendation: do not add a permission operation.** Section 4.3 option 1 in +the port is the honest answer, and clause 6.3 is where the reasoning belongs. + +--- + +## 6. Order, and why this order + +1. **`hidden` scoping** — independent, small, and its evidence is already + measured. Unblocks any consumer that uses the name. +2. **`posix_spawn` + socket/datagram/timeout routes in openkal-musl** — the + largest capability gain per unit of work, and it needs no other repository + to move first, because openkal-linux already provides the atoms. +3. **openkal-macos and openkal-windows: net, datagram, timeout** — turns step 2 + from a Linux capability into an ecosystem one. +4. **`fork` decision (4.2), `chmod` decision (4.3)** — both are decisions before + they are code. +5. **0.9 symlink atoms, if accepted** — last, because it moves the specification + and therefore every backend. + +⚠️ Steps 2 and 3 are independent and can proceed in parallel. Step 3 is the +larger effort and the one that decides whether this ecosystem's story is "POSIX +programs run on Linux" or "POSIX programs run". + +--- + +## 7. Criteria + +Each step is judged by a reading, not by a green run: + +| step | criterion | +| --- | --- | +| `hidden` | a consumer TU declaring `static int hidden = 7;` compiles, **and** a compiler-rt-style build with the overlay on its line still compiles | +| sockets | a POSIX program — `socket`/`bind`/`listen`/`accept` + transfer — runs over openkal-musl, written against POSIX and naming no openkal symbol | +| `posix_spawn` | `system("…")` and `popen` return a status the caller can read; the child's output arrives through the channel | +| macos/windows net | the same POSIX program, unchanged, on those hosts | +| declines | a program using a declined interface fails at the **link**, naming the operation — never at runtime, never silently | +| every new route | with the backend's interface absent, the call returns `-ENOSYS` rather than jumping through a null pointer | + +⚠️ The last row is the one the report `openkal-linux#13` raised directly: a stub +that is a null pointer produces `PC=0` with an empty backtrace, which names +nothing. A guarded weak reference returning `-ENOSYS` is the difference between +a program that fails and a program that cannot say why. + +--- + +## 8. What this document does not settle + +- Whether `fork()` gets a trampoline (4.2) or a permanent refusal. +- Whether permissions are refused (recommended) or emulated. +- Whether symbolic links enter 0.9. +- Whether `openkal.space` on Windows is declined permanently. + +Each is a decision about what the specification is for, and none of them should +be made by whichever implementation reaches it first. + +--- + +## 9. What was implemented, and what the implementation corrected + +**Status**: sections 4.1 through 4.3, 4.5 and 5.1 through 5.5 are implemented. +Section 4.4's specification change is **not**: neither a permission operation +nor a symbolic-link operation was added, and the specification did not move. + +The four decisions of section 8, settled: + +| decision | outcome | +| --- | --- | +| `fork()` — trampoline or refusal | **trampoline**, and the specification asked for it. `space.h` states in terms that a library above reaches `fork` "by saving its own execution state before the call and restoring it in the started context". `okm_setjmp.S` already carried the per-architecture half. | +| permissions | **refused**, as recommended. `chmod` reports `ENOSYS`; `stat` reports a mode assembled from `writable`. | +| symbolic links in 0.9 | **not now.** The asymmetry (§4.4) stands recorded; nothing composes them, and the specification is unchanged. | +| `openkal.space` on Windows | **declined permanently.** `CreateProcessW` starts a *named program*, which is `openkal.process`. Constructing a copy of the calling address space out of it would be clause 3.1's simulation. | + +### 9.1 Five things this document got wrong + +⚠️ **`posix_spawn` was never missing.** §4.1 says +`musl/src/process/posix_spawn.c` is excluded and that a port implementation +would restore `posix_spawn`, `system` and `popen`. The exclusion is real and the +conclusion was not: `port/src/okm_spawn.c` has *replaced* that source since the +port was written, and `system` and `popen` work through it. Measured: +`system("exit 5")` returns an exit status of 5, `popen` carries a line back. +What was missing was a criterion, not a capability — `examples/subprocess` is it. + +⚠️⚠️ **`timeout_ns = 0` does not mean "do not wait".** §4.1 says `O_NONBLOCK` +and a zero `poll` timeout are "expressible with `kal_timeout_*` at +`timeout_ns = 0`". They are the *opposite*: `timeout.h` defines zero as **no +bound**, following `kal_task_wait`. Passing a caller's zero straight through +turns the one call that must not wait into the one that never returns — +measured, as a hang, four lines into the network probe. The smallest bound is +`1`, which the environment rounds up to its own granularity. + +⚠️ **`SYS_clone` is not the only number `fork` arrives through.** musl's `_Fork` +issues `SYS_fork` where the architecture has it and `SYS_clone` where it does +not, so a dispatcher implementing only the second is reached on aarch64 and +riscv64 and never on x86_64. + +⚠️ **The macOS row's `exec` is not the clause 6.5 case.** §5.2 gives +`mmap(MAP_JIT)` + `pthread_jit_write_protect_np`. That pair is what a program +needs for memory writable and executable *at the same time*, which +`openkal.exec` does not offer: a region is writable, then published, then +executable, and never both. The implementation is the ordinary `mmap` + +`mprotect`, and the conformance suite calls the published region, so the reading +is settled by the system rather than by the argument. + +⚠️ **`SO_REUSEADDR` is not the same option on all three systems.** On Linux and +macOS it permits a listener whose predecessor is lingering; on Windows it +permits two listeners on one address at once. openkal-windows therefore does not +set it, and setting it "for symmetry" would have made that implementation behave +differently while looking the same. + +### 9.2 One thing neither this document nor anything else had noticed + +⚠️⚠️ **No continuous integration anywhere selected the interfaces this document +is about.** Every backend ran the conformance suite as `full`, which expands to +`standard,abi,stability,cost` — and `standard` is the *hosted* set. The five +interfaces 0.8 added are in `optional`. So every one of their sections was +compiled with its body removed and reported as *not examined*, in the same +release that added them. + +Nothing failed. Nothing was checked either. openkal-linux now runs +`full,optional` (143 observations held, 0 failed, 1 not observed); +openkal-macos the same; openkal-windows enumerates the six it provides, because +`optional` names `space`. + +### 9.3 The criteria, as they now stand + +| criterion | where | +| --- | --- | +| a POSIX program using sockets, datagrams, `poll` and `select` — naming no openkal symbol | `openkal-musl/examples/net`, 35 observations | +| another program started three ways, and the refusal asserted as a refusal | `openkal-musl/examples/subprocess` | +| `hidden`, `weak`, `weak_alias` usable by a program, in C and in C++ | `openkal-musl/examples/identifiers`, `openkal-llvm-runtime/examples/cxx` | +| `std::filesystem` over the port, and `permissions`/`create_symlink` **refused** | `openkal-llvm-runtime/examples/cxx` | +| every new route references its interface **weakly** | `openkal-musl` CI, eleven names, with `kal_time_sleep` as the strong control | +| the five interfaces examined rather than skipped | every backend's conformance run | + +⚠️ The weak-reference check found two apparent failures on its first run, and +both were the check's fault: a search under `target/` reaches the *dependency's* +objects, where openkal-linux's `kal_timeout_accept` refers to its own +`kal_net_accept` strongly — which is correct for an implementation and says +nothing about the port. diff --git a/examples/portable/mcpp.toml b/examples/portable/mcpp.toml index 181dd21..c9ff10d 100644 --- a/examples/portable/mcpp.toml +++ b/examples/portable/mcpp.toml @@ -15,7 +15,7 @@ openkal = "0.8.0" openkal-linux = "0.6.0" [target.'cfg(os = "macos")'.dependencies] -openkal-macos = "0.4.0" +openkal-macos = "0.5.0" [target.'cfg(windows)'.dependencies] -openkal-windows = "0.2.0" +openkal-windows = "0.3.0" From 2f8c6bb0e1a0e1d6dc9f944f85218d07dae0ef11 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Thu, 27 Aug 2026 22:11:42 +0800 Subject: [PATCH 2/5] Give the closure against published packages a place in the repository Every workflow in this ecosystem substitutes its siblings' working trees for the versions its manifests name. That is deliberate --- these repositories change together --- and the consequence is that NO WORKFLOW ANYWHERE RESOLVES A PUBLISHED PACKAGE. `openkal-kit' 0.1.0 shipped naming the specification by a path, which is true inside its own tarball and false for any consumer that also names an implementation; eight packages published, nine workflows green and eight repositories green on their own main, and the first resolution of the published set failed. The script that asks that question existed only in a scratch directory. It is here now, generalised over the versions and extended to exercise the routes this release adds --- a listener, a connection, a transfer and a duplicated image, written through POSIX and naming no openkal symbol, so that a published C library whose sockets do not work cannot satisfy it. --- ...ility-completeness-across-the-ecosystem.md | 1 + tools/sandbox-closure.sh | 270 ++++++++++++++++++ 2 files changed, 271 insertions(+) create mode 100644 tools/sandbox-closure.sh diff --git a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md index f0484c9..932075e 100644 --- a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md +++ b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md @@ -483,6 +483,7 @@ openkal-macos the same; openkal-windows enumerates the six it provides, because | `std::filesystem` over the port, and `permissions`/`create_symlink` **refused** | `openkal-llvm-runtime/examples/cxx` | | every new route references its interface **weakly** | `openkal-musl` CI, eleven names, with `kal_time_sleep` as the strong control | | the five interfaces examined rather than skipped | every backend's conformance run | +| the **published** packages resolved, built and run | `openkal/tools/sandbox-closure.sh`, which is the only thing in this ecosystem that resolves a published package | ⚠️ The weak-reference check found two apparent failures on its first run, and both were the check's fault: a search under `target/` reaches the *dependency's* diff --git a/tools/sandbox-closure.sh b/tools/sandbox-closure.sh new file mode 100644 index 0000000..1bda42f --- /dev/null +++ b/tools/sandbox-closure.sh @@ -0,0 +1,270 @@ +#!/usr/bin/env bash +# +# THE ECOSYSTEM AS A CONSUMER RECEIVES IT. +# +# sandbox-closure.sh [NAME=VERSION]... +# +# ⚠️⚠️ EVERY CONTINUOUS-INTEGRATION WORKFLOW IN THIS ECOSYSTEM SUBSTITUTES ITS +# SIBLINGS' WORKING TREES FOR THE VERSIONS ITS MANIFESTS NAME. That is +# deliberate --- these repositories change together, and a run must assert what +# is written today rather than what agreed when it was published. The +# consequence is that NO WORKFLOW ANYWHERE RESOLVES A PUBLISHED PACKAGE, so a +# defect belonging to the published FORM is invisible until someone outside +# meets it. +# +# ⚠️ That is not hypothetical. `openkal-kit` 0.1.0 shipped naming the +# specification by a path, which is true inside its own tarball and false for +# any consumer that also names an implementation: the specification was then +# reached by two routes, and the engine refused. Eight packages published, nine +# workflows green, eight repositories green on their own `main` --- and the +# first resolution of the published set failed immediately. +# +# ⇒ This script is the only thing that asks the question, which is why it lives +# in the repository rather than in somebody's scratch directory. +# +# WHY A SANDBOX. A machine that has been developing these packages has every one +# of them installed, and would answer for its own state rather than for the +# index. `xlings subos --sandbox --cmd` gives a fresh environment, and a +# fresh `/tmp` with it --- so nothing may be staged outside. +# +# It asks two questions, because they fail independently: +# +# ① Does the engine SAY the right things --- the layers, with the versions +# that were PUBLISHED rather than any that resolve? +# ② Does a PROGRAM depending on every one of them build and RUN? A layer table +# can be right while the artefact is wrong, and the published-and-verified +# assets of this ecosystem have been reachable and still unusable through a +# supported path before. +set -euo pipefail + +subos="${1:?usage: sandbox-closure.sh [NAME=VERSION]...}" +shift + +# The versions under examination. Named here so that a reader sees them, and +# overridable one at a time so that a release moves one line rather than the +# script. +MCPP=2026.8.27.1 +KIT=0.1.1 +MUSL=0.5.0 +LINUX=0.6.0 +RUNTIME=0.3.0 + +for pair in "$@"; do + case "$pair" in + mcpp=*) MCPP="${pair#*=}" ;; + kit=*) KIT="${pair#*=}" ;; + musl=*) MUSL="${pair#*=}" ;; + linux=*) LINUX="${pair#*=}" ;; + runtime=*) RUNTIME="${pair#*=}" ;; + *) echo "unknown pair: $pair" >&2; exit 2 ;; + esac +done + +xlings subos list 2>/dev/null | grep -q " $subos " \ + || { echo "::error::the environment $subos does not exist"; exit 1; } + +cat < /dev/null 2>&1 || true +# ⚠️ THE INDEX IS NAMED. `mcpp@` alone is AMBIGUOUS wherever more than one +# index repository carries the name --- measured: local, scode and xim all +# answer, and xlings refuses rather than choosing. The engine is published in +# xim. +xlings install "xim:mcpp@__MCPP__" -y -g + +# ⚠️⚠️ INSTALLING IS NOT BECOMING WHAT RUNS. In an environment that already has +# another mcpp the install succeeds and `mcpp` keeps resolving to the previous +# one --- xlings says so plainly and carries on. Without the assertion below +# this check would build the whole ecosystem with the OLD engine and report the +# release as verified. +xlings use mcpp __MCPP__ || true +mcpp self config --mirror GLOBAL +got="$(mcpp --version | awk '{print $2}')" +[ "$got" = "__MCPP__" ] || { echo "::error::mcpp is $got, not __MCPP__"; exit 1; } +echo " mcpp is $got" + +say "a project that names only published versions" +rm -rf /tmp/closure && mkdir -p /tmp/closure/src && cd /tmp/closure +cat > mcpp.toml < src/main.cpp <<'CPP' +// ⚠️ THE C HEADERS ARE INCLUDED ON PURPOSE. They are what pull musl's headers +// in, which is what makes the include path under examination the one used. +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +import std; +import openkal.kit.endpoint; + +// ⭐ THE THREE NAMES A PROGRAM ABOVE THIS STACK MAY USE. musl's internal +// overlay defines them, and openkal-musl used to publish the directory that +// does (openkal-musl#13). If any is a macro again this file does not compile. +static int hidden = 7; +static int weak = 11; +struct weak_alias { int value; }; + +int main() { + std::vector v{4, 2, 7}; + std::ranges::sort(v); + std::print("sorted:"); + for (int x : v) std::print(" {}", x); + std::println(""); + std::println("names: {} {} {}", hidden, weak, weak_alias{3}.value); + + // openkal 0.8's terminal interface. Clause 6.1 makes absence a LINK-TIME + // absence, so an older specification cannot satisfy this quietly. + const kal_uintptr tp = kal_terminal_props(kal_stdout()); + std::println("openkal 0.8 terminal interface linked: {}", + static_cast(tp)); + + // The kit, used rather than merely named. + const auto ep = kal::kit::parse_v4("10.0.0.255:8080", 15); + const auto bad = kal::kit::parse_v4("300.1.1.1", 9); + std::println("kit parses {} and refuses {}: {} {}", + ep.ok, !bad.ok, static_cast(ep.ep.addr[3]), + static_cast(ep.ep.port)); + + // ⭐⭐ AND THE ROUTES THIS RELEASE ADDS, THROUGH POSIX AND NAMING NO OPENKAL + // SYMBOL. A published C library whose sockets do not work would satisfy + // every line above. + const int lis = ::socket(AF_INET, SOCK_STREAM, 0); + sockaddr_in a{}; + a.sin_family = AF_INET; + a.sin_addr.s_addr = htonl(0x7f000001u); + ::bind(lis, reinterpret_cast(&a), sizeof a); + ::listen(lis, 4); + socklen_t alen = sizeof a; + ::getsockname(lis, reinterpret_cast(&a), &alen); + + const int cli = ::socket(AF_INET, SOCK_STREAM, 0); + const bool connected = ::connect(cli, reinterpret_cast(&a), sizeof a) == 0; + pollfd pf{ lis, POLLIN, 0 }; + const bool readable = ::poll(&pf, 1, 2000) == 1; + const int srv = ::accept(lis, nullptr, nullptr); + ::write(cli, "ping", 4); + char in[8] = {}; + size_t have = 0; + while (have < 4) { const auto r = ::read(srv, in + have, 4 - have); if (r <= 0) break; have += r; } + std::println("sockets: port {} connected {} readable {} carried {}", + static_cast(ntohs(a.sin_port)), connected, readable, + have == 4 && memcmp(in, "ping", 4) == 0); + ::close(srv); ::close(cli); ::close(lis); + + const pid_t kid = ::fork(); + if (kid == 0) ::_exit(23); + int status = 0; + const bool reaped = kid > 0 && ::waitpid(kid, &status, 0) == kid + && WIFEXITED(status) && WEXITSTATUS(status) == 23; + std::println("the calling image is duplicated: {}", reaped); + return 0; +} +CPP + +say "the program builds and runs" +mcpp run 2>&1 | tee /tmp/out.log + +grep -q 'sorted: 2 4 7' /tmp/out.log +grep -q 'names: 7 11 3' /tmp/out.log +grep -q 'openkal 0.8 terminal interface linked:' /tmp/out.log +grep -q 'kit parses true and refuses true: 255 8080' /tmp/out.log +# ⚠️ THE PORT IS NOT ASSERTED AS A VALUE --- the environment chooses it --- but +# everything else on the line is, and a port of zero would mean `getsockname` +# reported nothing. +grep -qE 'sockets: port [1-9][0-9]* connected true readable true carried true' /tmp/out.log +grep -q 'the calling image is duplicated: true' /tmp/out.log + +say "what the engine believes each layer is" +# ⭐ THE VERSION IS THE FIELD THAT DISCRIMINATES. That the engine knows a layer +# called `c-abi` says nothing about which package supplies it, and an older +# install answers the layer question exactly as this one does. +mcpp why toolchain 2>&1 | tee /tmp/layers.log +grep -qE "openkal-musl@__MUSL__" /tmp/layers.log \ + || { echo "::error::the c-abi layer is not openkal-musl@__MUSL__"; exit 1; } +grep -qE "openkal-llvm-runtime@__RUNTIME__" /tmp/layers.log \ + || { echo "::error::the c++ layer is not openkal-llvm-runtime@__RUNTIME__"; exit 1; } +grep -qE "openkal-linux@__LINUX__" /tmp/layers.log \ + || { echo "::error::the kernel-abi layer is not openkal-linux@__LINUX__"; exit 1; } + +# ⭐⭐ private_include_dirs: THE CRITERION IS THE DIRECTORY, AND IT IS PER UNIT. +# +# ⚠️ A grep over the whole compile database can only ever fail: openkal-musl's +# OWN sources are built in this same graph and appear in the same file, and they +# MUST carry those directories --- that is what the overlay is for. The question +# is whether a unit that is NOT openkal-musl's sees them. +cdb=$(find . -name compile_commands.json | head -1) +[ -n "$cdb" ] || { echo "::error::no compile database was written"; exit 1; } +python3 - "$cdb" <<'PY' +import json, sys +d = json.load(open(sys.argv[1])) +PRIV = ("musl/src/include", "musl/src/internal", "musl-generated/internal") +OWN = "openkal-musl" +outside, leaked, own_with = 0, [], 0 +for e in d: + cmd = e.get("command") or " ".join(e.get("arguments", [])) + has = any(p in a for a in cmd.split() for p in PRIV) + if OWN in e["file"]: + own_with += has + else: + outside += 1 + if has: leaked.append(e["file"]) +# ⚠️ DENOMINATORS BOTH WAYS. Zero units outside the package makes the absence +# vacuous; zero inside it carrying the directories means the overlay was never +# in use and the comparison is empty. +print(f" {len(d)} entries: {outside} outside openkal-musl, {own_with} of its own carry the private directories") +if outside == 0 or own_with == 0: + print("::error::the comparison has no denominator"); sys.exit(1) +if leaked: + print(f"::error::{len(leaked)} unit(s) outside openkal-musl see its private directories") + for f in leaked[:5]: print(" ", f) + sys.exit(1) +print(" no translation unit outside openkal-musl sees musl/src/include") +PY + +echo +echo "the closure holds: published packages, resolved from the index, built and run" +INNER + +SCRIPT="${SCRIPT//__MCPP__/$MCPP}" +SCRIPT="${SCRIPT//__KIT__/$KIT}" +SCRIPT="${SCRIPT//__MUSL__/$MUSL}" +SCRIPT="${SCRIPT//__RUNTIME__/$RUNTIME}" +SCRIPT="${SCRIPT//__LINUX__/$LINUX}" + +xlings subos use "$subos" --sandbox --cmd "$SCRIPT" From b5661271115587a1e76f182b39c5da12896d84f3 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Thu, 27 Aug 2026 22:12:02 +0800 Subject: [PATCH 3/5] The plan's status is its outcome, and the plan is left as written --- ...026-08-27-capability-completeness-across-the-ecosystem.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md index 932075e..945eb06 100644 --- a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md +++ b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md @@ -2,7 +2,10 @@ **Date**: 2026-08-27 **Scope**: openkal 0.8 and every repository that implements or consumes it -**Status**: design, for review. Nothing here is implemented. +**Status**: implemented. Sections 0–8 are the plan as it was reviewed and are +left as they were written; **section 9 records what was built, the four +decisions section 8 left open, and the five things the plan got wrong.** A plan +edited to agree with its outcome is a plan nobody can learn from. --- From e253d5d01045fe4fd2ed7f35e795c94f8dbf7049 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Thu, 27 Aug 2026 22:37:57 +0800 Subject: [PATCH 4/5] The exec section compiles under all three compilers, not two MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ⚠️ THE FIRST RUN IN WHICH THIS SECTION WAS SELECTED AT ALL is the run that found it. openkal-windows now names `exec' in its feature set, and the MSVC row stopped at exec.cpp(78): error C3861: '__builtin___clear_cache': identifier not found exec.cpp(82): error C3861: '__builtin_memcpy': identifier not found The suite is built by three compilers and had been written against two. The cache maintenance is guarded on the COMPILER rather than on the architecture, because what varies is which compiler spells the operation that way; where it is absent, the architectures that compiler targets here keep the two paths coherent in hardware. The copy becomes a byte loop: this suite reaches no C library --- it is written to run against an implementation that may be the only supplier of one --- and a builtin is not a C library, but it is not a language feature either. --- conformance/src/sections/exec.cpp | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/conformance/src/sections/exec.cpp b/conformance/src/sections/exec.cpp index b229733..e2b3fd1 100644 --- a/conformance/src/sections/exec.cpp +++ b/conformance/src/sections/exec.cpp @@ -75,11 +75,33 @@ void run() { // through the data path are not yet visible to the fetch path. The // specification does not place this upon an implementation, because the // program is the party that knows which bytes it wrote. + // + // ⚠️ THE BUILTIN IS NOT UNIVERSAL, AND THIS SUITE IS BUILT BY THREE + // COMPILERS. Measured on the MSVC row of openkal-windows, the first run in + // which this section was selected at all: + // + // exec.cpp(78): error C3861: '__builtin___clear_cache': identifier not + // found + // + // ⭐ The guard is on the COMPILER and not on the architecture, because what + // varies is which compiler spells the operation this way. Where it is + // absent, the architectures that compiler targets here keep the two paths + // coherent in hardware and there is nothing to do. +#if defined(__GNUC__) || defined(__clang__) __builtin___clear_cache(static_cast(p), static_cast(p) + kReturns42Size); +#endif + // ⚠️ A BYTE COPY RATHER THAN `__builtin_memcpy', FOR THE SAME REASON AND + // FOUND IN THE SAME RUN. The suite reaches no C library --- it is written to + // run against an implementation that may be the only supplier of one --- and + // a builtin is not a C library, but it is not a language feature either. fn f = nullptr; - __builtin_memcpy(&f, &p, sizeof f); + { + auto* d = reinterpret_cast(&f); + const auto* b = reinterpret_cast(&p); + for (unsigned long i = 0; i < sizeof f; ++i) d[i] = b[i]; + } observe(kind::behaviour, f() == 42, "instructions written into the region and published are executed"); From a66636af6cb058637baad82d7f7f0d7b819eb984 Mon Sep 17 00:00:00 2001 From: speak-agent Date: Thu, 27 Aug 2026 22:40:52 +0800 Subject: [PATCH 5/5] Record the five defects continuous integration found and reading did not --- ...ility-completeness-across-the-ecosystem.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md index 945eb06..389b8dd 100644 --- a/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md +++ b/.agents/docs/2026-08-27-capability-completeness-across-the-ecosystem.md @@ -462,6 +462,47 @@ permits two listeners on one address at once. openkal-windows therefore does not set it, and setting it "for symmetry" would have made that implementation behave differently while looking the same. +### 9.1a Five more, found by continuous integration rather than by reading + +Every one of these was invisible on the machine the work was done on, and each +names a different kind of blindness. + +**⚠️⚠️ The symbol namespace is shared across the layer boundary.** Winsock +exports the *BSD names* — `bind`, `listen`, `accept`, `connect` — and so does +the C library above it: openkal-musl compiles musl's own `src/network/*.c`, +which define those names and route them through the port. Naming `-lws2_32` on +openkal-windows's link line put both definitions in one program. It is not an +ordering problem: an import library's member defines the thunk *and* the +`__imp_` pointer together. The remedy is to resolve the library at run time, so +that nothing of it enters the program's symbol table. + +**⚠️⚠️ `kal_task_current()` is not stable across a copy of the address space, +and nothing ever said it was.** openkal-musl keeps its per-context state in a +table keyed on that identity. openkal-linux caches `gettid` in a thread-local, +so the *copy of the cache* answers the parent's value and the lookup works; +openkal-macos asks the kernel each time, so the started context answers a value +the table has never seen. The second is the honest answer to the question the +interface asks. The assumption was the port's, and the started context now +rebinds its slot before anything reads per-context state. + +**⚠️ `__builtin___clear_cache` is a call, not an inline sequence.** On aarch64 +and riscv64 it becomes `___clear_cache` / `__riscv_flush_icache` in the +compiler's support library — a dependency a package asserting "no C runtime +symbol" may not acquire. §9.1's fourth item was written before this was +measured; openkal-macos added the builtin and its own independence check +reported it within the hour. + +**⚠️ A bounded wait must answer with the set the interface defines for it.** +openkal-windows returned an error belonging to the *resource* where +`openkal.timeout` defines one for the *wait*, and the conformance suite reported +it — the second time on one runner out of three, because Wine chooses a +different error for the same condition. An error of the resource is the +transfer's to report, and the transfer follows the wait. + +**⚠️ The conformance suite was written against two compilers of three.** +`__builtin___clear_cache` and `__builtin_memcpy` are not MSVC's, and the first +run in which anything selected the `exec` section is the run that found it. + ### 9.2 One thing neither this document nor anything else had noticed ⚠️⚠️ **No continuous integration anywhere selected the interfaces this document