Notes

Fault count is not latency

I spent two weeks optimising the wrong number. Lazy restore in CRIU pulls memory pages on demand, so the obvious thing to reduce is the number of page faults the restored process takes before it can answer a request. Sequential prefetch reduces that count enormously. On loopback it removed 85% of the faults, and made time-to-first-request 88% worse.

The reason is not subtle once you accept it. At loopback, a fault is nearly free; the transfer is the cost. Prefetching spends bandwidth and queue depth on pages the process may never touch, and the pages it does need queue behind that waste. The fault counter goes down because faults were never the thing being paid for.

Everything inverts once the link has real latency. Around 125 microseconds of RTT the two strategies cross, and by 1 ms prefetch is 37% faster; at 2 ms demand-only does not finish at all. So the honest statement is not "prefetch helps" or "prefetch hurts" but that the decision belongs to the network, and therefore has to be made at runtime rather than at build time.

The controller I ended up with watches duplicate pressure and queue depth per fault window and switches prefetch off when it sees waste accumulating. It recovers the loopback regression to within 5.5 ms of demand-only, which is statistically indistinguishable, and still cuts prefetch volume by half on memory-light workloads. It is a small piece of code. The expensive part was believing a graph that disagreed with the metric I had chosen.

A verifier you can afford in an interrupt path

Loading code into a running kernel is only acceptable if you can say something about the code before it runs. In AxiomOS that check is a path-sensitive abstract interpreter over eBPF, and the requirement is unusual: it has to be cheap enough that attaching a program on a robot is not itself a timing event.

The measurement on the host is 943 ns for a minimal program and 376 µs at a thousand instructions, which is close enough to linear that the cost model can be published as a per-instruction-class table. That table is the useful artifact, not the verifier itself, because it lets admission be a scheduling decision. Attaching a program commits CPU time against a fixed budget under EDF utilisation, and the kernel refuses attachments it cannot pay for.

This is where verification stops being a security feature and becomes a real-time one. A program that is memory-safe but takes an unbounded number of cycles in a timer interrupt is still a way to lose a machine. Refusing the attachment is the boring, correct answer, and it is much easier to explain to someone who owns the hardware than a probabilistic one.

What I have not done yet: the cost model is measured on the host, not on the Pi 5 under load, and the JIT is only exercised on aarch64. Both numbers on the target are provisional until I can capture them from a clean image.

The evidence gate that demoted my best number

Every result AxiomOS publishes has to name a clean commit, a hashed artifact, and a committed capture. If any of the three is missing the number is marked provisional and may not be cited, including by me. I wrote the gate expecting it to catch sloppiness in other people's contributions. Its first victim was my favourite benchmark.

It was a comparison against Linux, measured on an image I had been editing while measuring. The number was probably fine. "Probably fine" is precisely the category the gate exists to eliminate, so it got demoted, and the comparison is still not in the README.

The reason I keep the gate is that a personal project has no reviewers. There is no one to ask which kernel the capture came from, whether the build had debug assertions on, or how many iterations were discarded. The gate is a substitute for that pressure, and it is only useful if it is allowed to hurt.

Practically it means the front page of the project claims less than it could. I think that is the right trade. A reader who checks one number and finds it reproducible will believe the rest; a reader who checks one number and finds it stale has correctly learned to believe none of them.

Writing an RP1 PWM driver with nothing to trust

On the Pi 5 the peripherals moved behind RP1, and on a bare-metal kernel that means the friendly parts of the ecosystem are gone. No device tree you can lean on, no driver to read, and a datasheet that describes registers rather than intent. The first working PWM output took three days and a logic analyzer, and two of those days were spent proving that my clock assumption was wrong.

The method that eventually worked was to stop trusting software observation entirely. UART prints tell you what your kernel believes. A capture on the pin tells you what the hardware did. Where those two disagreed, the kernel was wrong every single time, and the discipline of writing down the expected waveform before running the test caught three mistakes I would otherwise have explained away.

The output of that week is not really a driver. It is a habit: every actuator path in AxiomOS now has a physical check somewhere in its test story, because a servo does not care about your logs.

Ten seconds, thirty, sixty

A drone that loses its link has to do something, and the worst available choice is to keep doing what it was doing. ResQTerra degrades in stages instead: ten seconds without a heartbeat raises a warning, thirty commits to return-to-home, sixty commits to an emergency landing. The thresholds are arbitrary in the sense that any three numbers would have needed defending, and deliberate in the sense that each one is tied to how long the aircraft can still be recovered.

The interesting engineering is not the timers, it is that the ladder has to hold when the link comes back halfway through. State is explicit — idle, armed, taking off, in mission, returning home, landing, emergency stop — and a reconnect does not silently cancel a decision already taken. An operator who watches an aircraft commit to return-to-home and then sees it resume the mission because a packet arrived has lost trust in the system, correctly.

Transport does the same thing one layer down: a persistent session over 5G, automatic fallback to a Bluetooth relay when the primary drops, and every command acknowledged and timeout-tracked by the dispatcher rather than assumed delivered. Search-and-rescue flights happen exactly where network reliability cannot be assumed, so the protocol treats loss as the normal case.

Limitation worth stating plainly: the ladder has been exercised in the field by pulling the link on purpose, not by losing it in the conditions the system is meant for. Those are not the same test.