In my opinion, this is Tailscale's largest issue.
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
Yes. Just fucking stop doing userspace wireguard on linux <a href="https://github.com/tailscale/tailscale/issues/426" rel="nofollow">https://github.com/tailscale/tailscale/issues/426 - issue has been open for 6 years (and is locked now, lol), btw.
And if any tailscale employees are reading this - <a href="https://github.com/tailscale/tailscale/issues/15724" rel="nofollow">https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
You're correct, kernel isn't faster by default. With that said, the following is true:
1) the WireGuard kernel implementation, despite not even being zero-copy, exceeds the performance of the userspace implementation
2) implementations utilizing the userspace network stack have a maximum potential performance (context switch + memcpy is very slow, and that affects UDP disproportionately). It's the wrong approach for meaningful improvement.
What do you mean by “userspace network stack” when we’re talking about io_uring? That’s a contradiction. Unless you mean extra memcpy’s within the kernel network stack, but when talking about the buffer you supply I don’t believe that’s true - your data generally gets directly DMA’ed into the device because the buffer you supplied is pinned and can’t be released until the second CQE is delivered. It would be helpful if you clarified.
I do not understand why people continue parroting this performance nonsense.
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.
iscoelho · · focus · HN ↗
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
[1] <a href="https://tailscale.com/blog/more-throughput" rel="nofollow">https://tailscale.com/blog/more-throughput
boomer_joe · · focus · HN ↗
And if any tailscale employees are reading this - <a href="https://github.com/tailscale/tailscale/issues/15724" rel="nofollow">https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
lokar · · focus · HN ↗
iscoelho · · focus · HN ↗
1) the WireGuard kernel implementation, despite not even being zero-copy, exceeds the performance of the userspace implementation
2) implementations utilizing the userspace network stack have a maximum potential performance (context switch + memcpy is very slow, and that affects UDP disproportionately). It's the wrong approach for meaningful improvement.
vlovich123 · · focus · HN ↗
iscoelho · · focus · HN ↗
vlovich123 · · focus · HN ↗
Veserv · · focus · HN ↗
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.