Pushing WireGuard on macOS to 5+ Gbit/s
A story of networking fundamentals and what we learned about the XNU kernel.
We are on a never-ending quest to offer the fastest WireGuard VPN possible, on every platform. Historically, performance improvements to VPNs have often been limited to Linux as it offers the most versatile and most optimized APIs for fast I/O. We'll get into the details as to how that matters later in this post.
Much to the dismay of many Linux enthusiasts, even 2026 isn't really the year of the Linux desktop. Especially in the enterprise world where VPNs are used extensively to protect company networks, most users connect from Windows or Apple devices. As such, when building a fast VPN you really ought to focus on Apple and Windows devices too, and in our humble opinion, your VPN shouldn't limit you in how fast you can transfer data. If you have a 5 Gbit/s Internet connection, you should be able to use that completely.
Unlike our Linux and Windows builds, Firezone on macOS and iOS wasn't performing as well as we would have liked, achieving 1 Gbit/s download speed at best and upload speeds often being crippled to a few hundred Mbit/s. The problem got worse with higher latency, which was a hint that packets might be getting lost somewhere.
In this post, we are going to talk through how we optimized the Firezone macOS Client from around 1 Gbit/s to over 5 Gbit/s. It is a mix of networking theory and hunting through the XNU source code to uncover how exactly TUN and UDP work on Darwin, the core of Apple's operating system that macOS and iOS build upon.
What even is the maximum throughput?
A network like the Internet is built from many intermediary devices and paths. Unsurprisingly, the maximum throughput you can get between two endpoints is limited by the slowest device or link along the path. What is perhaps surprising is how we even know what the maximum throughput is. Throughput isn't something that gets advertised and then both ends of the connection adhere to it. Instead, throughput is something you measure and adapt to dynamically.
The Internet Protocol (IP) was designed to tolerate packet loss. An IP packet is sent as fire-and-forget and it is the job of protocols built on top of it to deal with lost or untransmitted packets.
Protocols like TCP and QUIC therefore come with a congestion controller. It is the congestion controller's job to track what got sent, what the remote received (i.e. acknowledged), how long it took to send the data and from there calculate how fast it should send data. This continuous feedback loop is the reason why a speed test doesn't just immediately report your maximum speed and it first has to "warm up".
Essentially, the congestion controller starts with a default window size, sends the data, waits for it to be acknowledged by the remote, measures the round-trip time (RTT) and then starts to increase the window size. The bandwidth-delay product defines how these variables interact. At an RTT of 20 ms, a window size of 1 MB gives us an upper bound of 400 Mbit/s of throughput. Increasing the window size to 2 MB allows for a maximum throughput of 800 Mbit/s.
By varying the amount of data that is in flight (and thus unacknowledged), the congestion controller steers the speed of the transfer.
On modern Windows (10+), Linux and macOS, the default congestion controller for TCP is CUBIC. CUBIC's algorithm is loss-based, which means it will reduce the window size (and thus the send rate) as soon as it concludes from the remote's acknowledgments that packets got lost.
In TCP for example, this happens as part of the TCP header. TCP numbers every byte it sends and the receiver's acknowledgment says "I have everything before byte N". Based on this feedback, the sending side knows which packets are missing. If the gap doesn't close within a short reordering window, it retransmits them and the congestion controller treats the gap as loss.
This exchange happens between the sender and the receiver of the connection. Devices along the path do not explicitly negotiate a send rate with the sender. They can signal congestion by dropping packets or, with Explicit Congestion Notification (ECN), by marking them. In both cases, it is the sender that adjusts the rate based on feedback from the receiving end.
For physical network devices and links, this makes a lot of sense. An intermediary device has physical constraints in how many packets it can forward based on e.g. its clock speed. Packets arriving faster than they can be processed get dropped, which is how the sender learns that it has been sending packets too quickly.
Given some ramp-up time, this is how the congestion controller finds a send rate the path can sustain by varying the size of the congestion window.
When CUBIC detects packet loss, it reduces the congestion window by 30%. This sounds aggressive at first but does not directly translate into a 30% dip in the transfer rate. After all, data that is still in flight is still being drained and delivered. The congestion window only controls at what rate we are sending new data.
Most importantly, all of this works based on feedback from the receiver, which is why it takes longer to settle on the effective throughput as latency to the receiver increases.
Losing packets is expensive
What does that mean for an overlay network like a VPN? The congestion controller doesn't know that the interface it is sending packets into is a virtual one. It treats it like any other network device. It is therefore important that we play by the congestion controller's rules: If it controls throughput by measuring packet loss, we better not lose or reorder any packets in our software.
Why does the order matter? To the sender, a packet that arrives late initially looks just like a lost one: there is a gap in the acknowledgments. Networks can legitimately reorder packets, for example when they spread them across parallel paths. That is why TCP waits for a short reordering window before declaring a packet lost. But if a packet arrives later than that, the sender has already retransmitted it and likely reduced its send rate for nothing. So, to get the best performance, we really want to avoid reordering packets and we definitely must avoid dropping packets.
If we drop packets while we are below the possible bandwidth capacity of the link, the congestion controller will reduce the window size prematurely. Effectively, this results in lower throughput numbers than our processing speed would allow.
Backpressure beats packet loss
Similar to a physical network link, the code that makes up our virtual one can be thought of as a queue. Packets go in on one side and come out on the other.
Internally, a packet has to go through multiple stages. In Firezone, those stages run in different threads which again are connected by queues / channels.
All of these channels are bounded, which is important for two reasons:
- We need to make sure that our overall memory use is bounded and it is not possible to OOM a Firezone Client by forcing packets into it.
- Just like the slowest link in a network should slow down the others, the slowest processing stage should signal to the previous stage that it is at capacity.
Bounded channels achieve both of these: If a channel is full, the writing task will suspend and - in Rust terms - register a waker. Whenever the receiver takes out an element and finds a registered waker, it then uses it to notify the sender that there is now capacity for another element.
A processing task that takes elements from one channel and sends them into another will therefore automatically "pass on" the backpressure signal of a slower, later stage.
Applying this approach consistently throughout the code ensures that no packets get lost within Firezone even when a given stage starts to become the bottleneck.
You might wonder now: Why are we even bothering with this when we've just established that the most commonly used congestion controllers are loss-based?
The reason is latency. Loss-based feedback is delayed whereas local backpressure is essentially instant and has the same effect, assuming we can create a backpressure chain all the way back to the user's socket.
For example, say the user wants to upload a file through the browser to a website that gets accessed via Firezone. What happens underneath is that the browser opens a TCP stream to the server and starts writing to the provided file descriptor. To establish the connection, the kernel's network stack creates a TCP SYN packet and routes it to the interface, which, in this case, is Firezone's TUN device. Firezone then receives an IP packet and performs the necessary encryption and routing to send it out as a UDP packet containing a WireGuard data packet. Once the TCP handshake has finished, more IP packets with TCP segments start to stream in.
Within the kernel, the TCP socket has a send buffer that gets filled by the syscalls of the browser handling the upload.
Data only leaves that buffer once the server has acknowledged it, because TCP may have to retransmit it.
Crucially, the kernel rejects the syscall if the buffer is full.
On Linux for example, in the case of a non-blocking socket, this results in EWOULDBLOCK being returned as the error code.
The same idea applies to the UDP socket that carries the WireGuard datagrams.
Each IP packet passed to Firezone by the kernel maps directly to one UDP datagram carrying a WireGuard data packet.
Similar to TCP, a UDP socket also has a send buffer where the data passed to write initially lands.
From there, the kernel passes the datagrams to the corresponding NIC driver which ultimately sends them out via the network.
The main difference from TCP is that UDP doesn't define any flow or congestion control. In other words, it doesn't have a congestion controller that modulates the rate at which packets leave your device. There are no acknowledgments that we have to wait for. Instead, the packets essentially go out at line rate.
Backpressure, however, still applies. If the send buffer is full, the kernel will reject the syscall and, through that, link the backpressure signal from the UDP socket to the sending application.
Most likely, the rate at which UDP packets can leave your device is the limiting factor for throughput as your CPU can create new packets at a much faster rate than your network connection can carry.
The one caveat in this system is the TUN transmit queue, i.e. the queue that Firezone reads new IP packets from. That one does not have a backpressure signal based on queue size alone and therefore needs to have enough capacity to absorb the packets TCP (or QUIC) can still send after we stop reading. If it can't, we are back to loss as the congestion signal: the kernel drops packets at the full queue, and the congestion controller slows down only once it notices the missing acknowledgments. If it is large enough, TCP runs out of window before the queue fills up and holds off sending until more ACKs arrive.
In summary, what we are relying on here are the contracts of a non-blocking I/O system:
- When a buffer is full, a write gets rejected with "would block".
- After subscribing to "write-readiness", we receive a notification that the descriptor is writable again.
On Linux, (1) is implemented as the EWOULDBLOCK error code and (2) is implemented via the epoll syscall.
Reactors like mio are then built on top of this system to offer building blocks for non-blocking I/O.
As mentioned above, this design had been working great in our Windows and Linux builds. On macOS, however, upload performance was nowhere near those other platforms with throughput often crippled to a few hundred Mbit/s or even less as latency increased.
Diving into the XNU source code
XNU is to Darwin what Linux is to Fedora or Ubuntu: the kernel. What may come as a surprise to some people is that Apple actually publishes the XNU source code on GitHub. It is usually delayed by a few months from the release of a new macOS or iOS version but it does land there eventually.
XNU's networking stack has its roots in FreeBSD, and its socket API goes back as far as 4.2BSD in 1983.
Linus adopted the same API when he created Linux, which is why the two look alike on the surface with their syscalls of bind, connect, read and write.
Underneath, however, they work very differently and, as it turns out, XNU diverges quite heavily in its behavior and API when it comes to non-blocking I/O for network sockets.
UDP sockets do not use a send buffer
When debugging throughput problems, one of the first things to try is to increase buffer sizes. The reason is that larger buffers help in absorbing bursts and scheduling delays which would otherwise lead to packet loss. That didn't help in our case and looking at the source code revealed why.
XNU does allocate a send buffer for every UDP socket, but it effectively remains unused. Instead of being buffered, the packet is handed directly to the network interface's transmit queue and the buffer's fill level always remains 01.
So what did we configure when we tried to increase the buffer size via SO_SNDBUF?
It turns out that configuring the send buffer really only sets a high-water mark that enforces how big the datagram is allowed to be2.
Somewhat amusingly, this is also documented as part of the send buffer's default value3.
With the fill level always being at 0, the check for space succeeds no matter how full the transmit queue is4.
For other protocols like TCP, this is where XNU returns EWOULDBLOCK in case the buffer is full.
EWOULDBLOCK. UDP packets skip that buffer and go straight to the interface's transmit queue, where they are dropped once it is full while the write still succeeds.In other words: on macOS, UDP backpressure is broken in practically every async runtime.
Reactors and language runtimes rely on the behavior that, on a Unix system, a write to a non-blocking socket fails with EWOULDBLOCK when there isn't enough space for the message.
On XNU, that never happens for UDP: the socket happily accepts every message and reports no error, while the packets are dropped at the interface transmit queue.
Connected sockets are different
Going through this code path revealed something else. A connected UDP socket is treated differently from an unconnected one.
What is a connected socket? On a protocol level, UDP is stateless. Each packet merely describes source and destination attributes. UDP has no handshake, sequence numbers, or message types. By default, a single UDP socket can be used to send to and receive from many peers at the same time.
Calling connect on a UDP socket restricts the destination to the provided peer and only routes messages from that peer to the socket.
That is convenient if you need to manage several long-lived UDP "connections" without doing your own demultiplexing of inbound datagrams.
Instead of a single socket, you create one socket per destination peer, call connect on it, and handle the sockets in different tasks / threads.
On XNU, connecting a UDP socket has additional side effects.
Most importantly, using connected sockets allows us to obtain a backpressure signal, although it isn't what typical reactors and language runtimes expect.
Once the interface transmit queue is full, further sends fail with ENOBUFS instead of EWOULDBLOCK.
Runtimes treat ENOBUFS as an error, so using it for backpressure needs special handling.
Despite the different error code, registering for write-readiness after receiving ENOBUFS does produce a wake-up as soon as the queue has room again.
Restoring backpressure
This explains the slow upload speeds we had been seeing. On average, we were sending well below the capacity of the link, but the interface's transmit queue only holds a limited number of packets. Whenever draining it was briefly delayed, for example because the CPU was busy with other work, the queue filled up and XNU dropped every further datagram while still reporting the write as successful. To the TCP stream inside the tunnel, this looked like congestion, so CUBIC shrank its congestion window even though the link had capacity to spare: exactly the premature reduction we described earlier. And because the sender only learns about loss a round-trip later, the higher the latency, the longer it took to recover from each of these drops.
By moving to connected sockets and special-casing the ENOBUFS error, we were able to implement a dedicated code path that suspends our UDP transmit thread upon a full interface queue, restoring backpressure.
The details of this work are in PR 13725.
Now such a delay merely pauses our transmit thread until the queue has room again, and a longer stall propagates through our bounded channels back to the TUN device instead of turning into packet loss.
Amortization, the holy grail of throughput
So far, we've focused on not losing packets in order to not upset the congestion controller. That is a good foundation, but it doesn't make anything faster by itself. Ironically, measuring speed-ups while some part of the system still drops packets often produces worse results because you are still losing packets, now just faster.
To make packet processing faster, it helps to split the cost into two parts. There is the per-packet work: encrypting and decrypting, NATing towards a new IP, updating checksums, recording flow logs. Because it is per-packet, the CPU time we spend on this grows with the number of packets.
And there is per-call cost: syscalls, thread wakeups, allocations. It is paid once per operation, regardless of how many packets that operation carries.
That is what makes amortization powerful. Read 32 packets in one syscall and the syscall costs 1/32nd per packet. Hand 100 packets to another thread at once and the wakeup costs 1/100th per packet.
Thus, the answer for faster packet processing is to do it in batches, and XNU exposes several ways for us to do that.
sendmsg_x and recvmsg_x
These are the equivalent of Linux's sendmmsg and recvmmsg syscalls.
Instead of only passing a single packet / buffer, you can pass several in one call and have the kernel process them all at once.
Unlike on Linux, these syscalls work both on a network socket and the TUN file descriptor!
This allows us to pass packets in batches between the kernel and userspace, thus splitting the cost of the syscall across several packets. These APIs still require you to pass an individual buffer for each packet, meaning we mostly amortize the syscall overhead itself.
On Linux and Windows, we can make use of syscalls that not only amortize the syscall overhead but also the actual processing in the kernel. This is enabled by an API that accepts a single contiguous buffer instead of a list of buffers, together with a known segment length that specifies where the buffer must be split to obtain the individual datagrams.
Just like a userspace program, the kernel is also composed of many layers. Passing a single but large buffer all the way through to the NIC driver therefore amortizes that processing cost compared to passing each individual datagram.
Unfortunately, XNU doesn't offer this capability for BSD sockets. Nevertheless, amortizing at least the syscall overhead is better than nothing and offers a good performance improvement.
Combining sendmsg_x with connected sockets
A nice side effect of the connected sockets we've already built is that when used together with sendmsg_x, the kernel implements a fast path5 where it does some of the socket setup, like updating the socket's policies and checking for buffer space, once per batch instead of once per datagram.
And because the socket already knows its destination, the kernel also skips the per-datagram address and route lookup.
The idea is the same as using sendmsg_x to avoid the syscall overhead, now simply in the kernel.
Anything that we can do once per operation instead of once per packet frees up processing capacity for more packets.
Coalescing TCP segments
When configuring a TUN device, we not only need to set the IPv4 and IPv6 addresses but also the MTU (maximum transmission unit) of the link. The MTU tells the kernel how big we'd like the IP packets to be that it creates for outbound traffic that is sent through Firezone.
Firezone's TUN device is configured with an MTU of 1280 across all platforms. This matches the minimum MTU for IPv6 and leaves sufficient room for WireGuard's encapsulation overhead on paths with the common 1500-byte Ethernet MTU.
An interesting finding we've made in digging through XNU is that the kernel accepts packets larger than its MTU on the write path, i.e. when Firezone delivers packets back to the kernel. Internally, it stores each packet in a fixed-size slot6, and on the Macs we tested, the NetworkExtension sizes those slots at 4096 bytes, which is about three times the 1280-byte MTU.
This slot size can't be changed after the TUN device has been created, but it can be queried! That allows us to merge up to 3 consecutive MTU-sized segments of the same TCP flow into a single, larger segment before we write it to the TUN device. This only works for TCP because it is a byte stream: the receiving application never sees where one segment ends and the next one begins. UDP, on the other hand, is message-based. Every datagram reaches the application as a separate message, so merging two would deliver them as a single message which the receiver then cannot parse.
Coalescing TCP packets requires us to recalculate the checksum, which means a full pass over the entire payload. That cost is, however, outweighed by the savings in the kernel's per-segment processing, which now happens up to three times less often.
Results
The real world is messy.
We didn't make these discoveries in the "right" order: some of the packet processing optimizations landed before we had working backpressure.
In particular, Firezone had been using the batched send and receive syscalls sendmsg_x and recvmsg_x on its UDP sockets for a long time before we discovered that backpressure on XNU needs special handling.
As such, it is tricky to point to historical benchmark results.
That doesn't mean we can't measure the impact of these individual improvements, however.
The following results have been measured in our continuous benchmarking environment, which for the macOS Client consists of a pair of M1 Macs, directly connected via Thunderbolt at 40 Gbit/s. On these M1 Macs, Firezone achieves 5.87 Gbit/s upload and 5.76 Gbit/s download.
For the purpose of the following numbers, we selectively disabled the optimizations discussed in this post to show how much impact each one has.
In each run, we measure a single TCP stream three times per direction with iperf3.
The displayed number is the median of those three measurements.
What's next
Achieving 5+ Gbit/s throughput is great and should mean that for the majority of people's workloads, Firezone is not going to be the bottleneck. Yet we'd like Firezone to be faster.
The measured rate translates to ~600k packets per second at our MTU of 1280.
Profiling shows that the busiest thread is now our UDP thread, and it already spends most of its time inside the kernel.
The key thing that it reveals is that each UDP packet, despite the use of sendmsg_x, is handled individually as soon as it reaches the IP layer of XNU.
On Linux and Windows, segmentation offloads drastically reduce the per-packet cost here. Instead of passing an individual buffer per packet, we hand the kernel one large buffer together with a segment size, and it splits that buffer into individual datagrams as late as possible, ideally on the network card. This allows the cost of forming and routing an IP packet to be amortized across all packets in the buffer.
If Apple were to expose such a capability through its BSD socket API, the throughput numbers could very well be 2x or 3x higher and thus provide a great boost for any UDP-heavy workloads on macOS, userspace QUIC stacks included.
References
Footnotes
-
XNU
bsd/net/dlil.c, lines 3222 to 3241. ↩ -
XNU
bsd/kern/uipc_socket.c, lines 1943 to 1946. ↩ -
XNU
bsd/kern/uipc_socket.c, lines 1947 to 1965. ↩ -
XNU
bsd/kern/uipc_syscalls.c, lines 1906 to 1918. ↩ -
XNU
bsd/net/if_utun.c, lines 144 to 162. ↩