In my opinion, this is Tailscale's largest issue.
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
Yes. Just fucking stop doing userspace wireguard on linux <a href="https://github.com/tailscale/tailscale/issues/426" rel="nofollow">https://github.com/tailscale/tailscale/issues/426 - issue has been open for 6 years (and is locked now, lol), btw.
And if any tailscale employees are reading this - <a href="https://github.com/tailscale/tailscale/issues/15724" rel="nofollow">https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
Tailscale's netstack is barely even WireGuard and they aren't compatible whatsoever. It's all marketing at this point.
So it's not that simple: it's impossible for Tailscale to use any existing kernel or accelerated WireGuard implementation. They could derive inspiration, but a kernel module for Linux won't fix Windows & Mac. With that said, I feel they have enough funding to maintain a few platforms (:
(Tailscale cofounder) That’s a good callout on DoH support, thanks.
That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)
You're correct, kernel isn't faster by default. With that said, the following is true:
1) the WireGuard kernel implementation, despite not even being zero-copy, exceeds the performance of the userspace implementation
2) implementations utilizing the userspace network stack have a maximum potential performance (context switch + memcpy is very slow, and that affects UDP disproportionately). It's the wrong approach for meaningful improvement.
What do you mean by “userspace network stack” when we’re talking about io_uring? That’s a contradiction. Unless you mean extra memcpy’s within the kernel network stack, but when talking about the buffer you supply I don’t believe that’s true - your data generally gets directly DMA’ed into the device because the buffer you supplied is pinned and can’t be released until the second CQE is delivered. It would be helpful if you clarified.
I do not understand why people continue parroting this performance nonsense.
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.
Remember that at least one LPE CVE associated to kernel IPSec implementation has been discovered (copy.fail), which means that whatever gains you get from this vpn tunneling, is lost by breaking the basic user security system guarantee.
You are better off not using a VPN at all rather than using kernel crypto
By that logic, we should avoid TCP as the Linux kernel implementation has had plenty of CVEs. Thankfully our expert critical thinking helps us acknowledge that as silly.
Not true. There are plenty of userspace TCP stacks in production today, just like there are also userspace IPsec stacks. UDP is also an option if TCP is too complex for your tastes.
If you believe that, you don't understand TCP, I recommend reading the actual RFC (RFC 793) and the IP RFC if necessary.
A port identifies a process, an IP address identifies a machine. The machine receives the packet, and then passes it to the process, since it's the machine that needs to receive the packet before routing it to the process, it's the kernel with ring 0 privileges that needs to process the packet and map the port to the process.
There's nothing stopping you from giving each stack its own IP, making the kernel only responsible for routing.
I don't think it's a good idea to put more of the stack than necessary into each individual program though, because then you're reliant on each program implementing everything correctly and you have to worry about bugs and security issues in every single program you're running rather than just in the kernel.
Considering that even something as simple as opening a socket (which involves calling one function to give you three numbers which you pass unchanged to a second function) is largely a clusterfuck in existing programs, I don't trust them to implement the whole TCP stack.
The performance hit for running an emulated CPU would be significant, to the point that it wouldn't really be the same argument.
I was thinking more along the lines of network namespaces -- although actually, I should have been thinking of TUN interfaces. Any packets sent into one of those are delivered to the attached program, which can do what it likes with them. Those are usually used by e.g. OpenVPN for L3 things, but nothing stops a program from handling the L4 headers.
You might want to read about exokernels. The kernel knows only enough about networking to pass the packet to the correct userspace process; it has no TCP machinery.
iscoelho · · focus · HN ↗
It is slow. It cannot achieve speeds of greater than 1Gbps on clients systems (Windows & Mac), where you'd normally see it being used. On Linux, it struggles to achieve 10Gbps even when using a synthetic large packet benchmark [1]. With an IMIX benchmark, it would not be competitive whatsoever.
This problem is fixable. WireGuard achieves higher performance (Kernel vs Userspace implementation) and IPsec implementations can achieve 100Gbps/400Gbps (DPDK/XDP). Zero-copy networking.
From this blog post, I can say Tailscale still seems to not have the appetite for that, which is a shame.
[1] <a href="https://tailscale.com/blog/more-throughput" rel="nofollow">https://tailscale.com/blog/more-throughput
boomer_joe · · focus · HN ↗
And if any tailscale employees are reading this - <a href="https://github.com/tailscale/tailscale/issues/15724" rel="nofollow">https://github.com/tailscale/tailscale/issues/15724 please fix this too. Regular users not using some sort of enterprise saas DNS (whatever their thing is?) deserve DNS privacy too.
iscoelho · · focus · HN ↗
So it's not that simple: it's impossible for Tailscale to use any existing kernel or accelerated WireGuard implementation. They could derive inspiration, but a kernel module for Linux won't fix Windows & Mac. With that said, I feel they have enough funding to maintain a few platforms (:
apenwarr · · focus · HN ↗
That said, note that if you run your own DNS server on your tailnet, the regular UDP DNS is automatically private because it’s carried over Tailscale. That’s the most common setup for non-SaaS DNS servers. DoH doesn’t really add anything in that arrangement. (And it’s more fiddly because you need to get and refresh a TLS cert.)
lokar · · focus · HN ↗
iscoelho · · focus · HN ↗
1) the WireGuard kernel implementation, despite not even being zero-copy, exceeds the performance of the userspace implementation
2) implementations utilizing the userspace network stack have a maximum potential performance (context switch + memcpy is very slow, and that affects UDP disproportionately). It's the wrong approach for meaningful improvement.
vlovich123 · · focus · HN ↗
iscoelho · · focus · HN ↗
vlovich123 · · focus · HN ↗
Veserv · · focus · HN ↗
Memory copying is on the order of 100 gigabytes per second. You can do 80 full payload copys and still out-pace a dog-slow 10 gigabit per second connection. If your bottleneck is memory copying you are either doing something very wrong and doing way too many copys or congratulations you have implemented one of the fastest network stacks.
Supervisor calls are also very fast, on the order of 100 ns up to maybe 1 us with all the Spectre mitigations. Even if you did something as stupid as one supervisor call per packet, you would still be getting on the order of 10 Gbps at the long end there and 100 Gbps at the short end. Which, again, means congratulations are in order because you have implemented one of the fastest network stacks. If you do batching and add just 10 us (us, not ms) of latency then that entire cost is so small as to be irrelevant. You are going to bottleneck on your memory copying first.
Network stacks are so slow almost entirely due to poor protocol design and poor protocol implementation. Usually both.
TZubiri · · focus · HN ↗
If they would have taken that advice, tailscale instances would have been pwned by copy.fail
TZubiri · · focus · HN ↗
>IPsec
Remember that at least one LPE CVE associated to kernel IPSec implementation has been discovered (copy.fail), which means that whatever gains you get from this vpn tunneling, is lost by breaking the basic user security system guarantee.
You are better off not using a VPN at all rather than using kernel crypto
iscoelho · · focus · HN ↗
TZubiri · · focus · HN ↗
iscoelho · · focus · HN ↗
TZubiri · · focus · HN ↗
A port identifies a process, an IP address identifies a machine. The machine receives the packet, and then passes it to the process, since it's the machine that needs to receive the packet before routing it to the process, it's the kernel with ring 0 privileges that needs to process the packet and map the port to the process.
Dagger2 · · focus · HN ↗
I don't think it's a good idea to put more of the stack than necessary into each individual program though, because then you're reliant on each program implementing everything correctly and you have to worry about bugs and security issues in every single program you're running rather than just in the kernel.
Considering that even something as simple as opening a socket (which involves calling one function to give you three numbers which you pass unchanged to a second function) is largely a clusterfuck in existing programs, I don't trust them to implement the whole TCP stack.
TZubiri · · focus · HN ↗
Dagger2 · · focus · HN ↗
I was thinking more along the lines of network namespaces -- although actually, I should have been thinking of TUN interfaces. Any packets sent into one of those are delivered to the attached program, which can do what it likes with them. Those are usually used by e.g. OpenVPN for L3 things, but nothing stops a program from handling the L4 headers.
yencabulator · · focus · HN ↗
<a href="https://pdos.csail.mit.edu/papers/exo:tocs.pdf" rel="nofollow">https://pdos.csail.mit.edu/papers/exo:tocs.pdf
adgjlsfhk1 · · focus · HN ↗