Thomas Eizinger

September 25, 2026

Finding bugs you didn’t think to test for

Coverage-guided fuzzing meets deterministic simulation testing

We've all been there. A support ticket comes in from a customer, stating that the product isn't working. Apparently, the connection hangs and they can't access the dashboard. You think: "That is odd, we have a test for exactly that and it doesn't happen for me."

You dive into the logs and ping emails back and forth with the customer. After two hours of debugging, it turns out that a seemingly unrelated configuration option on the customer's machine broke things in an unexpected way. You go and modify the existing test and, sure enough, it now fails. Despite knowing that this isn't going to scale in the long run, you add a nearly identical copy of the test and commit it with the fix.

More test coverage is good, but as you go along like that, your test suite becomes more of a burden than a tool that supports you in making changes to your code. The same APIs are called across hundreds of tests. Every test you add slows down CI a tiny bit, yet that exact test is what reproduced the customer's problem.

The tension is real. As developers, we want our test suite to be thorough but also fast. We want it to use high-level APIs to make sure we can express real scenarios, but we don't want the tests to be flaky.

The challenge is making those tests cheap enough to run on every pull request, reliable enough that failures are worth investigating, and varied enough to find the bugs you didn't think to test for.

At Firezone, we do this by combining sans-IO protocol design, deterministic simulation, and coverage-guided fuzzing. Our harness drives clients, gateways, and relays through generated scenarios and checks their behavior against a reference model.

From examples to properties

Most test frameworks provide you with some abstractions where all you need to do is put #[test] or @Test on a function and you are good to go. If you strip those away, however, an automated test is really just a program with its own fn main that directly interacts with a specific component of your software. What differentiates a test from a regular program is its structure. In an ideal world, a test has three steps: arrange, act, assert. A test is only as good as its assertion. Without assertions, the outcome (i.e., the exit code) of the test is always the same, and the test runner can't tell whether or not the observed behavior is actually the correct one.

How do you write a good assertion? What to assert on depends on the input you are feeding to the component. Let's say I am working on a new data structure library and it has a collection with set-like functionality. I could write a test like this:

#[test]
fn does_not_allow_duplicates() {
    let mut set = MySet::default();

    set.insert(1);
    set.insert(1);

    assert_eq!(set.items(), &[1]);
}

As we can see, the assertion depends on the input elements. I can only assert the presence of 1 because that is the element I inserted. The space of possible inputs is a lot larger, however. How do I know that it also works for, say, negative numbers? Or really large numbers? What if we add the same element three or four times?

To cover these additional scenarios, the tool most people reach for is a parameterized test. We wouldn't want to duplicate the entire test just because we want to test with different inputs, right?

#[test_case::test_case(1)]
#[test_case::test_case(1000)]
#[test_case::test_case(-1)]
fn does_not_allow_duplicates(number: i32) {
    let mut set = MySet::default();

    set.insert(number);
    set.insert(number);

    assert_eq!(set.items(), &[number]);
}

That's better. Let's have the test framework do the job of de-duplicating that test. It is easy to add more test cases now, but we may still miss a bug that only happens if our input is a really big prime number or something obscure like that.

Instead of defining the test cases ourselves, why not let the computer do it for us? This is where property-based testing comes in. Property-based testing asks you to rewrite your test to assert a property instead of a specific state. Assertions that are based on state need an example value, like 1, -1, or 1000. A property, on the other hand, is something you can test for without knowing the exact value. In our example, the property we want to assert is uniqueness of elements. A different property could be the sort order of numbers or that all numbers must be positive.

#[test_strategy::proptest()]
fn does_not_allow_duplicates(#[strategy(any::<Vec<i32>>())] numbers: Vec<i32>) {
    let mut set = MySet::default();

    for number in numbers {
        set.insert(number);
    }

    let all_items = set.items().collect::<Vec<_>>();
    let unique_items = set.items().unique().collect::<Vec<_>>();

    assert_eq!(all_items, unique_items);
}

Instead of specifying inputs ourselves, all we are specifying here is that we want a list of numbers. Whether our set behaves correctly is asserted by checking that after making all items unique, we still have the same items.

Property-based testing libraries like proptest use deterministic randomness to generate many different inputs, repeatedly execute our test, and persist the seed for replay every time they find a failure. This allows you to re-run the test in e.g. a debugger to step through and inspect what is happening.

Letting the machine find the interesting inputs

We've come a long way in improving our test and yet, we can still do better! Using randomness to generate inputs is good because it will likely test inputs you wouldn't have thought of. But depending on how complex the component you are testing is, it might take many iterations before it finds "interesting" inputs that trigger behavior that is specific to your implementation, like internal caches or inlining optimizations. As the authors of the code, we know about these and therefore, we may want to add specific test cases ourselves. If only the machine knew the code too and could figure those interesting cases out itself instead of brute-forcing it with randomness!

Coverage-guided fuzzing to the rescue. Fuzzing is usually associated with automated tests for parsers or other components that take unstructured input and are e.g. exposed to the network. Such first-in-line components need to be able to deal with whatever input is thrown at them and must not crash, otherwise you are exposing yourself to DoS attacks. Coverage-guided fuzzers therefore run an instrumented version of your code. Instead of just generating a coverage report, the fuzzer uses the coverage information to mutate the input stream to try and purposely hit new branches in your code.

For example, if you have a branch in your code that handles numbers between 512 and 1024 differently from others, the coverage feedback will tell the fuzzer that it is still missing a branch there and it will try to mutate its input stream to hit it. With pure randomness, you might also eventually hit it, but it is not guaranteed. For a trivial example like a number range, this might happen very quickly, but as the scope of your test grows, this gets less and less likely.

You'll find that liberal use of debug_assert!s in your code pairs very well with coverage-guided fuzzers. Assertions are just branches, and a coverage-guided fuzzer will try to hit them.

Using (debug) assertions is a good idea if you are worried that certain invariants in your code might accidentally get broken. You might be surprised, though, that grepping for debug_assert! in our codebase doesn't actually yield that many results:

thomas@fedora ~/s/g/f/f/rust > rg "debug_assert!" --count
libs/windows-security/src/pipe_dacl.rs:1
relay/proto/src/server.rs:5
libs/connlib/ip-packet/src/slices.rs:4
libs/connlib/ip-packet/src/lib.rs:4
libs/connlib/tunnel-proto/src/gateway.rs:1
libs/connlib/tunnel-proto/src/client.rs:1
libs/connlib/l3-udp-dns-client/lib.rs:1
tests/fuzz/src/arb/values.rs:1
libs/connlib/tunnel-proto/src/gateway/client_on_gateway.rs:1
tests/fuzz/src/buffered_transmits.rs:1
tests/fuzz/src/arb/topology.rs:3
tests/fuzz/src/sim_net.rs:1
tests/fuzz/src/sut.rs:3
tests/fuzz/src/reference.rs:3
libs/connlib/snownet/src/channel_data.rs:3
libs/connlib/snownet/src/allocation.rs:4
libs/connlib/snownet/src/node.rs:1
libs/connlib/phoenix-channel/src/lib.rs:1
libs/connlib/dns-over-tcp/src/server.rs:1
libs/connlib/flow-tracker/src/lib.rs:4
libs/connlib/dns-over-tcp/src/client.rs:2
libs/connlib/tun/src/ioctl.rs:1
libs/connlib/tunnel/src/lib.rs:1
libs/connlib/tunnel/src/io/device.rs:1
libs/connlib/tunnel/src/io/udp_gso_queue.rs:2
gui-client/src-tauri/src/controller/ran_before.rs:1

All the examples we've gone through so far are pretty trivial and related to data structures. Firezone, on the other hand, is a WireGuard-based VPN. If we aren't adding assertions to our code, how then are we using fuzzing to ensure Firezone works correctly? Let's dive into that.

Deterministic simulation testing

Deterministic simulation testing is a hot topic in and of itself. It got popularized by the developers of FoundationDB and these days, entire companies like Antithesis are built around it. In a nutshell, deterministic simulation testing is the idea that all of a system's inputs, including time, are externally controllable. That allows you to run the system inside a simulation, subject it to any scenario you like and replay it exactly when something goes wrong, similar to the regression seeds that a failing property-based test produces.

There is more than one way to get there. FoundationDB wrote the entire database in Flow. Flow is an actor language with a runtime that provides interfaces for network, disk and clock interactions. This is powerful but requires you to a) learn a new language and b) write literally all of your software in it, including all dependencies that have side effects. Antithesis takes a different approach and runs your existing, unmodified program on a deterministic hypervisor.

Our approach is sans-IO. In principle, it is similar to the idea behind Flow, but instead of writing a language (extension), you turn your system into a state machine. That gives you determinism for free because it is all free of side effects by definition. The downside is that, similar to Flow, all dependencies used in the state machine need to follow the same approach, and a lot of the time that results in not taking a dependency and writing the code yourself instead. Unlike Flow, however, sans-IO is just a style of writing code, meaning that if a library happens to adopt it, you can drop it into your code without issues. Once the system you want to test is in the right shape, building the test harness on top of it is actually not that difficult.

At its heart, Firezone's data plane combines four things:

Messages to and from the control plane, like new resources, routing table updates or revoked access
Commands from the user interface, like updated configuration options
UDP datagrams being sent to or received from other peers
IP packets being sent to or received from the local TUN device

All of these are essentially either input or output of the internal state machine. Messages and commands simply mutate the current state. Inbound UDP datagrams are passed in and get transformed into IP packets. Inbound IP packets are passed in and get transformed into outbound UDP datagrams. An event loop ties all of this together and passes the data between the various IO sources and the sans-IO state.

Just looking at one state in isolation is not particularly interesting. Where it gets more exciting is when we create multiple of these and wire them together. The actual application requires IO resources and a physical link to another device in order to send packets. In the simulation, sending a packet is just a matter of passing an argument from one state to the other.

Why static test cases don't scale

If we were to write static test cases in this model, we would very quickly drown in boilerplate and repeated code. Why? Because compared to our tests for MySet, actually setting up the state for two Firezone nodes to talk to one another requires more than just instantiating it.

For our simulation testing, it is important to understand that testing a simple happy-path packet exchange isn't just:

#[test]
fn exchanges_packet_via_happy_path() {
    let mut client = ClientState::default();
    let mut gateway = GatewayState::default();

    let sent_packet = make_packet();

    let datagram = client.handle_packet(&sent_packet);
    let received_packet = gateway.handle_datagram(&datagram);

    assert_eq!(sent_packet, received_packet);
}

Instead, the full setup also has to handle timers, messages to and from the control plane, exchanging packets with one or more relays and exchanging the datagrams for the WireGuard tunnel between the two states. In a way, that is the price you pay for writing sans-IO code. Even heavily abbreviated, a hand-written version of that test looks more like this:

#[test]
fn exchanges_packet_via_happy_path() {
    let mut now = Instant::now();

    let mut client = ClientState::new(client_key, now);
    let mut gateway = GatewayState::new(gateway_key, now);
    let mut relay = RelayState::new(relay_addr, relay_key);

    // Portal: tell both sides about the relay and give the client a resource.
    client.update_relays(iter::empty(), [&relay], now);
    gateway.update_relays(iter::empty(), [&relay], now);
    client.set_resources(vec![cidr_resource("10.0.0.0/24", gateway_id)]);

    // The first packet can't be sent yet; it gets buffered and the client asks
    // the portal for a connection instead.
    let sent_packet = make_udp_packet("10.0.0.1:53");
    client.handle_tun_input(sent_packet.clone(), now);
    let Some(ClientEvent::RequestAccess { resource_ids, .. }) = client.poll_event() else {
        panic!("client should request a connection");
    };

    // Portal: authorize the flow on the gateway and hand the answer back.
    gateway.create_authorization(client_id, client_key, client_ice, gateway_ice, resource_ids[0], now);
    client.handle_resource_access_authorized(resource_ids[0], gateway_id, gateway_key, gateway_ice, now);

    // Pump ICE and the WireGuard handshake between all three parties until
    // the buffered packet finally comes out on the other side.
    loop {
        for transmit in client.poll_transmit() {
            deliver(transmit, [&mut gateway, &mut relay], now);
        }
        for transmit in gateway.poll_transmit() {
            deliver(transmit, [&mut client, &mut relay], now);
        }
        for transmit in relay.poll_transmit() {
            deliver(transmit, [&mut client, &mut gateway], now);
        }

        if let Some(received_packet) = gateway.poll_packets() {
            assert_eq!(sent_packet, received_packet);
            return;
        }

        now = [client.poll_timeout(), gateway.poll_timeout(), relay.poll_timeout()]
            .into_iter()
            .flatten()
            .min()
            .expect("something should be scheduled");

        client.handle_timeout(now);
        gateway.handle_timeout(now);
        relay.handle_timeout(now);
    }
}

And even with that in place, we haven't started to think about introducing things like latency or simulating symmetric NAT, let alone covering scenarios like "what happens if we first send a packet, then the resource gets updated and then we try to send another one?".

The test is a state machine, too

The above example shows that sans-IO allows us to simulate the entire system in a deterministic way, but writing out different test cases would result in an enormous amount of boilerplate because we'd have to re-create all the plumbing that advances the individual state machines for each individual test case.

The nice thing about sans-IO and state machines in general is that they compose very well. Naturally, the solution to this problem is to apply the same hammer we've already been using and express our entire test harness as a state machine.

At this point, the model we are working with is literally that of an abstract state machine: state = state + transition.

We don't just have one state but two: a ReferenceState and the actual system under test. The ReferenceState is a re-implementation of how we want Firezone to work that doesn't concern itself with details such as WireGuard encryption or NAT traversal. Firezone is an overlay network where packets pushed in on one end need to pop out on the other side. The fact that we can gloss over all the implementation details but still assert that this is what is happening makes it possible to build this reference model without completely re-creating the implementation.

In our test harness, ReferenceState models:

a varying number of Clients and Gateways
a varying number of Relays
the routing table of a simulated network

The state is paired with a set of possible Transitions. These Transitions model inputs to our system. For example, EditResource describes an arbitrary edit to a resource in our control plane. Our customers make changes to their Firezone deployment all the time, and those changes are broadcast to their entire fleet of devices. On the other hand, we have transitions like RoamClient, which simulates that a given client roamed from e.g. the office network to their home WiFi and is now connecting through a different IP. We also have transitions like ConnectTcp, which creates a new TCP connection using a userspace TCP stack to a resource reachable through Firezone.

For each Transition, the ReferenceState models the effect that we expect to see in the system. For example, updating the name or description of a resource should be a non-disruptive change, whereas tightening the traffic filters to TCP/22 must block e.g. web traffic on port 80 from there onwards.

Putting all of this together results in something like this:

let mut generator = Generator::new(...);

// Sample an initial state.
let mut portal = generator.portal();
let mut reference = generator.reference_state(&portal);
let mut tunnel = TunnelTest::init_test(&reference, &mut portal);

for applied in 0..20 {
    // Sample a new transition
    let transition = generator.transition(&reference, &portal);

    tracing::debug!("Applying transition {applied}: {transition:?}");

    // Discard bookkeeping this transition makes stale
    reference.invalidate(&transition, &portal);
    tunnel.invalidate(&transition, &reference);

    // Advance the state
    portal.apply(&transition, &reference);
    reference = reference.apply(&transition, &portal);
    tunnel = tunnel.apply(transition, &reference, &mut portal);

    // Make sure everything checks out
    check_invariants(&reference, &tunnel, &portal);
}

Asserting invariants

At the beginning of the post, I claimed that a test is only as good as its assertions, and that is also true for our simulation. The above code snippet shows that after each transition, we check whether all our defined invariants hold. The list of invariants we check is quite long, too long to explain all of them. The one I'd like to pick out as an interesting case is DNS.

DNS is a fairly simple protocol: send a request, receive a response. Regardless of what is going on inside a Firezone Client, an invariant that must hold at any given point is that every DNS query produces a response. From a distance, this might look like something that is easy to uphold. Let's zoom in.

To support DNS-based routing, Firezone ships with a stub resolver that becomes active as soon as you are connected. The stub resolver looks at incoming DNS queries and matches them against the list of DNS resources. If none of them match, the query gets forwarded to the system-defined DNS resolver that was active before Firezone started. This is what allows us to implement split DNS. If you are browsing youtube.com and that isn't in your resource list, youtube.com will continue to resolve to the same IPs as if Firezone were not running.

In addition to resources, Firezone allows you to define a custom DNS server in the control plane that should be used for all DNS queries instead of the system-defined one. A common use case is to host a company-wide DNS server within a subnet that itself is only accessible via Firezone:

DNS server: 10.10.0.1
CIDR resource: 10.10.0.0/24

In such a setup, all DNS queries are forwarded to a Gateway in the site that hosts the DNS server, including the very first DNS query that triggers the setup of the tunnel. Before, querying youtube.com resulted in a DNS query to the system-defined DNS server. Most of the time, this is your home router. With an admin-enforced DNS server, the query for youtube.com needs to be sent through the tunnel to a Gateway, which then forwards it to the DNS server.

What this shows is that a seemingly simple invariant such as "every DNS query must have a response" can cover a lot of ground because it implicitly asserts that:

we correctly buffer DNS queries until the connection to the site has been established
connection setup itself works correctly
connection failures result in a SERVFAIL response

We could test all of these things in isolation if we built in seams that allow us to insert mocks or stubs. But why would I do that if I can also just test the system as a whole without the brittleness and performance issues of traditional integration tests?

From random inputs to coverage-guided ones

As we have seen, deterministic simulation testing is a very powerful technique to ensure your system behaves correctly in all kinds of scenarios. But as your system, and therefore your state machine, becomes more complex, some doubts might creep in: Are we really hitting all the code paths? Are we generating interesting and/or realistic transitions, or are we simply adding and removing the same resources 20 times in a row?

A coverage-guided feedback loop solves exactly that. Instead of just generating tens of thousands of different random seeds and hoping that you hit all the interesting code paths, a coverage-guided fuzzer will deliberately mutate its input stream to hit paths it hasn't hit yet. For this to be effective, it is important that your sampling code preserves mutation locality. Mutation locality means that a single bit flip in the input stream results in an output state that only changes at that particular location. For example, if a sequence of 4 bytes in the input stream maps to an IPv4 address, mutating those 4 bytes should only change the IP address and everything else stays the same. Preserving mutation locality is what allows the fuzzer to gradually explore deeper branches and more complex states of the system.

With coverage guidance in place and a generator that preserves mutation locality, generating interesting code paths is suddenly no longer a roll of the dice. Instead, each generated sample gives the fuzzer a better understanding of our code's behavior and increases the chances that we find a model mismatch. Not all model mismatches are bugs. As it turns out, correctly and exhaustively modeling the behavior of your system isn't actually that easy, and it often exposes imperfections in your implementation.

In order to continuously optimize our coverage, we have a nightly CI job that takes the existing corpus, replays it to learn the current coverage and then fuzzes for another 30 minutes to try and hit new branches. If new inputs have been found at the end, a bot account opens a pull request that adds them to the existing corpus.

The caveat

Building a harness like this has an upfront cost: designing the system as sans-IO state machines and building the simulation around them. We could do that because we own every major component of the stack, including NAT traversal, TURN servers and the WireGuard state machine. Building in a sans-IO way requires you to push more of the interesting code down the stack and move the side effects further up. Pursuing this consistently thins out the layer above your state machines to the point where it really just contains the side effects. Those can of course still contain bugs, which is why we additionally have Docker-based end-to-end tests and a dedicated QA environment.

Conclusion

Our simulation test harness describes operations such as editing a resource, opening a connection, roaming a client between networks or changing the DNS records of a domain. These operations and their observable effects are what users actually care about. Driving the test harness with a coverage-guided fuzzer means we don't have to write out all the different scenarios ourselves. Instead, the fuzzer will automatically explore them and build up an input corpus that maximizes code coverage. Replaying only around 500 scenarios is enough to reach a code coverage of more than 85%, and it takes less than two minutes. Fast, thorough, reliable. Pick three.