Hi! I'm the author and the person who was on a road trip when I decided to do this experiment back in July. I'm super excited to finally get the blog post out, and even better, alongside a post describing the root cause process that went into the project. You can read about Tailscale's story here: https://tailscale.com/blog/sqlite-wal-reset-bug
I'm only sad that I didn't put SQLite under test earlier in the year, or I would have found this issue right away. If you look at the workload[1], you can see how simple it is. Exactly the same kind of workload we write every day to help our customers find bugs.
If you have any questions about our process or how debugging with Antithesis works, please let me know! Thanks for reading!
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
How difficult would it have been to isolate that problem if you didn't already know the SQLite subsystem it was in? This feels, to someone relatively ignorant of the SQLite / Tailscale / Antithesis architectures as a "hindsight is 20/20" kind of thing, but I'm open to learning more.
Great question! The general approach we take with transactional systems like this is to put reachability statements throughout the complex stateful machinery and then stress-test them in Antithesis.
The workload I put in place does exactly that[1]. It runs a write workload from multiple processes concurrently on the same SQLite database to cause writes to build up in the WAL, and runs checkpoints concurrently. This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Said differently, this is exactly the approach we take with all stateful transactional systems. I am only sad that I didn't do this experiment months earlier, as it would have saved Tailscale and the SQLite team a lot of time.
[1]: https://github.com/antithesishq/sqlite/blob/3.51.2-instrumen...
> This exercises the portion of the WAL code that, from a trivial read-through, is most likely to contain bugs (and turns out, did!).
Suuure. So why exactly haven't you found this bug already a long time ago? I mean it is trivially obvious that this is where bugs in SQLite would be, right?
SQLite is open source and is one of the most popular databases, surely testing it to find bugs would also be obvious?
Brutally honest answer: it only recently became easy to get random systems under test with Antithesis without a bunch of manual/expert effort. Expect us to be reporting a lot of bugs against open source projects in the near future.
I've been jealous of the work antithesis is doing and support you in spirit; the world needs deterministic, tested software.
If you want a challenging open source deterministic game to test, may I suggest Beyond All Reason, powered by the Spring / Recoil engine? It's an RTS game that's been in various incarnations of development for 20+ years. (Disclosure: I'm an occasional code contributor to the project).
Oh man, Beyond All Reason would be a killer demo. We are actually testing another RTS as we speak, but I won't spoil it for you :) Look forward to a really epic post coming soon.
We put many things to the test, but it costs money, and until very recently we haven't had a way to do this without a lot of human oversight. Now that we have https://github.com/antithesishq/antithesis-skills and LLMs are getting pretty reliably good at writing workloads, we are looking at ways to open up our technology to the OSS community. We have already started down that path with Etcd last year (https://etcd.io/blog/2025/autonomus_testing_with_antithesis/) and are working with various OSS groups to put more systems under test in a responsible way.
I'm not gonna lie, though, you're hitting a sore point. I'm absolutely kicking myself for not putting SQLite under test when I joined. The first few projects I put under test after joining were all projects that had a few properties: 1. The project had a path to a commercial deal 2. The project likely had interesting bugs to find
TBH, I didn't expect that one of the most widely used and well-tested pieces of software in existence would have a bug our fuzzer could find in 15 mins. (To be clear, this is not a hit on SQLite, just that our fuzzer is stupid good at finding race conditions. SQLite remains one of my favorite pieces of software ever.)
With this in mind, are there other similar projects that I'm not thinking of that I should put under test? Before you ask, I'm already scaling up my SQLite harness to cover more of its state space and run against master. Maybe I'll find something novel in SQLite after all! Never too late to check.
Other embedded databases like Dolt? DuckDB? git?
We've started testing git and DuckDB! Haven't played much with Dolt, though. We have some initial findings on Git that we are validating and will hopefully be making public soon!
Matrix, specifically synapse/rustMatrixSDK :)
Test out Fossil SCM, it depends on SQLite and is also written in c89.
The scheduler in the Linux kernel jumps out to me as a place where it would be cool to find something of this nature.
So you got information where the bug is, and now you brag that your AI friend found it quickly :joy:
Go find bugs that haven't been yet found, and then brag about them :joy:
Typo in the post: "SQL 3.51.2" should presumably be "SQLite 3.51.2".
Nice article, but could you get someone to fix the blog layout? Having the author card pinned to the window rather than being able to scroll past it is distracting.
you asserted that Antithesis would have found it right away, but it seems like from your write-up that you first set up instrumentation on the source, then told it to reproduce the WAL bug.
i don't feel like i gained a strong appreciation for what the actual Antithesis capabilities are or why you had confidence this was anything the Antithesis tools would have helped find? i want to know more about what kind of analysis or capabilities Antithesis has and how it could have helped find the issue. what's clear is that Antithesis could confirm an already found issue. and even there I don't feel like I gained any connection or knowledge or understanding of what the tooling was doing or how it would help. i'd really like more technical meat to this post to understand the value proposition.