csbartus 1 day ago

Today this is the second post I see where the 'traditional' testing methods (unit, e2e, integration, perf, pen, etc) are re-categorized, are put in new perspectives.

The other one is https://alperenkeles.com/posts/vocab-for-testing/

I truly understand our (only) job is to become the oracles of the machine: https://www.osequi.com/manifesto

  • cipehr 1 day ago

    Not sure I’m following, are you implying that deterministic simulation testing is a just a new name for a “traditional” test method? I’m not clear because 1. Deterministic simulation testing is not a new concept. (Foundation DB is one of the poster boys for it) 2. All the test methodologies mentioned are a name for a specific pattern and practice of testing, and DST is the same thing.

    • csbartus 1 hour ago

      You are right, I was focusing on the first table, on testing/categorization vs the DST itself

  • Mitchem 1 day ago

    I'm betting on something adjacent but flipped. As agents are writing the code, humans are stopping (and getting worse at) reviewing, so the oracle needs to be mechanical. My personal bet: https://github.com/NMitchem/charpente/blob/main/MANIFESTO.md

    • akoboldfrying 23 hours ago

      This sounds interesting. I agree with your thesis that the oracle will become the bottleneck to most regular AI software development, so focusing effort on that will bear fruit. I've thought for a long time that bringing more or even all business logic into stored procedures would actually be a net win if the tooling support was a lot better and the language less clunky -- what you're doing has a similar goal, though you're approaching it from the other end (the app absorbing the DB). I also agree that 1 process (does that also imply 1 thread?) on 1 machine is sufficient to handle 90% of real-world use cases, so that shouldn't be the big limiting factor people are afraid it will be.

      (I didn't love the Claudish -- maybe you could remove more egregious examples like "When one process can hold all of them, the boundaries stop paying rent and start charging it".)

    • csbartus 1 hour ago

      My take is: if it's something new (new feature, new app), the oracle can't be mechanical, it must be human. Otherwise (migration, clone), it can be mechanical.

skybrian 19 hours ago

I'd like to try it, but Antithesis is $3k/month. What's a good testing framework for the rest of us?

  • hasyimibhar 10 hours ago

    If you are writing green field software, just structure it in such a way that makes DST possible. That's what I'm doing with pg2iceberg [1]. The DST harness has caught a lot of edge cases that cause data loss. It's hard to imagine building it without DST.

    [1] https://github.com/polynya-dev/pg2iceberg

  • giaour 7 hours ago

    https://hegel.dev/ is a local PBT framework from Antithesis. I work at Antithesis, and we use Hegel and Bombadil (a local DST-for-web tool) a lot internally.

Klaster_1 13 hours ago

I've been using deterministic simulators paired with Playwright for many of my projects - web apps, firmware, video games - and I'm amazed how nobody else in teams I worked in ever did something like that. Unfortunately, this mostly stops at handrolled simulators of other parts of the system or running stuff like FW in Renode, I'd really love to push this further and have something like open source Antithesis (which I'm a huge fan of) to fold in more parts of the overall system and be more ambitious.

drooby 1 day ago

This is the future, I'm convinced.

I’ve been building a framework of my own and it’s already found dozens of important and critical bugs on my team at work.

The next problem to solve after this, though, is formalized intent.

A bug is often a bug if the app is not behaving as intended. What we intend is not always so clear, and it will remain an area of the engineer's and product designer's responsibility, because software is complex, and what we want cannot be computed so simply. Defining and understanding intent will remain a "human bottleneck" for many companies.

  • andai 19 hours ago

    My experience agrees with both your points. An LLM volunteered a "deterministic core" architecture during a refactor a few months ago.

    During a different stage of the refactor, it implemented a change completely backwards, and produced abundant passing tests for the backwards change.

    I laughed when I realized that it might as well have supplied me with a formal proof of the backwards change's "correctness" !

TacticalCoder 1 day ago

Deterministic testing is one thing and it's badly needed.

But I'm totally convinced we're moving to deterministic execution: n various implementations, in different languages, running on different stacks, and a minimal (maybe even hardware) voter taking care of the n-modular redundancy to pick the correct "answer".

0-day Postgres CVE? No problem: one of the implementation gives bogus answers and is detected as broken.

0-day Java? Same thing.

This shall make 0-day in anything but the voters a thing of the past (and the voters shall be minimal and have a tiny attack surface).

The answer can be an API query but we can go much lower: network packets, deterministically created, by n systems and all going to the voter(s). Whatever the "answer" is, it's going to be deterministic and verified from the lowest possible level.

The implementations shall be written from specs, formally verified, etc. But the one thing that'll guarantee any fuck up is caught is going to be n independant implementations of the same specs.

And as soon as one of the implementation/stack return even just one network packet that doesn't match what the other systems do produce: something is fucked up and has to be fixed.

This shall require throwing a huge lot of the non-deterministic bullshit away but we'll see systems doing just that: n-modular redundancy and voters, at the lowest level possible.

I'm 100% sure we're going there (and AI is going to write most of that for us).

  • kulukrok 1 day ago

    interesting idea. any references (long-form) on this?

  • nullstyle 1 day ago

    I agree 100%, and I’m definitely experimenting in that direction with my hobby projects.

  • Kinrany 22 hours ago

    Determinism is only possible within a process, so we still need to complement it with something like DST for when messages arrive in arbitrary order.

  • DenisM 21 hours ago

    This is insane. I like it a lot.