I run a similar setup. I didn't end up going for a full router-side hub inside the LAN, to avoid extra hops for things when they're inside the network.
Some of my notes:
* I have multiple servers outside the LAN that have WG connections back to specific interior systems for things like metrics/log-shipping, monitoring, and backups. All of those are "point-to-point" (the cloud-hosted systems have multiple Peers for different internal systems). This means that they can only talk to the things I expect them to, w/o me needing to also handle that at the firewall level.
* The Wireguard Mac and iOS apps are very solid. I use them heavily on multiple devices, and they can be trained for OnDemand activation based on SSIDs. This means that when my phone tries to load my NVR dashboard, it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub). The iOS app is reliable enough that I set it up once on my partner's phone so they could access the cameras and then never had to think about it again.
* Because the only things you need to set up a peer link are shared pubkeys and to agree on IPs, wiring up my endpoints via Puppet was super smooth and adding/adjusting has likewise been smooth. My systems generate a keypair during setup, publish their public key where the other nodes can find it, and all I have to do is update the map of "hey, this server get this WG-internal IP, and this is who should link to it".
> it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub)
Appropriate netmasks in the routing tables do this regardless of physical interfaces. For example, if your local network has netmask 255.255.255.128, the local link is prioritised over the tunnel link which could have netmask 255.255.255.0, but you have to decide which nets/subnets are useful to you.
My solution to this is a pair of udp ports relayed to each other on a VPS via socat. Then you can point a home WireGuard server (behind cgnat etc) at one, and clients at the other, with no need to trust the vps
I use defined.net (managed Nebula) for my homelab and side project overlay networks, and it works great. They have a very generous free tier and I automate enrollment with a set of systemd units that I wrote. Configuration is very easy and the Nebula android app does what I need so I can access everything over my phone. A few friends use wg-easy to manage their wireguard setups, and they sing its praise.
I still use wireguard for simple point to point tunnels into my datacenter rack but anything important I use Nebula.
An issue with his remote setup is that the remote VPS decrypts packets from the remote laptop, then re-encrypts them for the LAN — this means that the remote VPS can see the plaintext of all those packets. He’ll need to layer TLS or something similar, or run Wireguard over Wireguard.
As somebody with a very similar setup, all the things I'm making accessible over the Wireguard network are HTTPS, SSH, etc. The handful of things that couldn't do native TLS (or were irritating to get configured with automated TLS certs) I stuck behind nginx for TLS.
Short of setting up dns validation and using 3rd party dns service as many registrars don't support API for dns management, how is domain validation done for acquiring TLS certs when serving only via wireguard ?
There's the private CA route but its a pain to setup the certs on all (mobile) devices and Android makes it very scary and hard.
I use https://github.com/go-acme/lego and DNS validation. I'm not sure what DNS provider you're using, but a ton of them have robust APIs. In my case, I'm using Route53 which is notable in that I can generate API credentials that can only update the specific record needed for a particular name's ACME validation record.
What options are available to use a remote VPS to facilitate connecting Wireguard directly through the CGNAT? It seems most "client" devices are going to be behind at least some kind of NAT as well.
I did the p2p between any sensible nodes in my network then added routing layer via BGP so any node can "see" any other node regardless of underlying mesh.
The access limits are better served by firewalls than limiting tunelling
In my opinion Tailscale is the realistic option for most people. The author is familiar with Tailscale having worked with it previously, but my interpretation is that he wanted to get more familiar with the underlying Wireguard technology.
It's almost as if these people are offended someone would not use tailscale. Do you also find stories about people hiking and comment "I just look up scenic locations on google"? Why would you think your +1 to the easy, commercial route is at all interesting to a forum literally having Hacker in the name?
It is quite different here as the OP stated headscale which is own hosted and I understand they use just the tailscale clients. That is mine setup too after using tailscale first and then wanted something own hosted without my traffic going through someone else's network infrastructure (internet excluded of course)
“Each Tailscale agent in your distributed network streams its logs to a central log server (at log.tailscale.com). This includes real-time events for open and close events for every inter-machine connection (TCP or UDP) on your network.”
That reminds me of a complaint that I had with a visualization library that kept pushing the open source project to be tied into their hosting racket. I brought up how lots of security organizations at the time wouldn't understand how much of a massive problem it was for the end user to decide to make the data immediately available to the entire Internet. It feels wrong that such implications could be missed from a set of environment variables just being missing. Usually you want things to fail safely ...
Agreed it's a bit annoying how non-discoverable this is. Wish it was in some sort of onboarding prompt. But I can see Tailscale's argument for making it the way it is.
Tailscale is really targeting the business market, especially since their product is basically free for personal use. In a corporate environment, I imagine that the client logs are actually hugely valuable to the corporate customers themselves. It lets them see who is accessing what and is super critical when doing a post-mortem after a hack. (also no actual traffic content is logged)
But I still keep client logging disabled for my personal use.
Or you can setup Yggdrasil by installing it on each machine and adding a few public volunteer peers to the configuration. My own summary of WireGuard->Nebula->Yggdrasil journey: https://news.ycombinator.com/item?id=43967082
I've looked at Yggdrasil multiple times now, and I always end up backing away because it's security critical software intentionally facing the public internet and talking to untrusted peers, but it's a smaller project with no security audit. In contrast, wireguard has a lot of eyes on it, simpler code that does less, only exposes a listen-only invisible UDP port to untrusted parties, and uses formal verification to ensure that its smaller codebase is secure. Perhaps I'm paranoid, but it's a really hard sell for me to use anything but wireguard:\
The refusal to use a control plane tells me the author would probably enjoy openvpn much more than vanilla wireguard.
It’s so nice to edit openvpn configuration, maybe some ccd file, restart the openvpn server and see clients reconnect one by one and pick up new options (eg: nee routes etc)
As a random aside: in a past life, I was helping somebody with a project where we needed a VPN connecting two locations. We were working with OpenVPN, and the nodes we had to handle the connection were beefy with large uplinks but we were hitting the single-core limitation so it didn't matter. We ended up building a proof of concept that launched one OpenVPN instance per core and then bonded them together, which let us get way closer to line rate.
Thankfully we never had to try them for real to see how horribly that would have gone under real load.
FWIW I found using one wireguard interface/instance too restrictive. The problem is that wireguard is an L3 interface that has its own internal IP routing table, so only one node can serve as a default (horizon) route for the instance. So for my setup, on each node I have a separate wg interface for every peer node. Every packet that goes into a given interface is destined to come out to the one peer on the other side.
Having been travelling just now with Wireguard as my primary VPN the main thing I noticed is it's completely unusable on airport wifi.
So the big feature I need is any sort of gateway which works over port 80 or 443 natively - ideally at a subpath so it's easy put behind a reverse proxy.
I run a similar setup. I didn't end up going for a full router-side hub inside the LAN, to avoid extra hops for things when they're inside the network.
Some of my notes:
* I have multiple servers outside the LAN that have WG connections back to specific interior systems for things like metrics/log-shipping, monitoring, and backups. All of those are "point-to-point" (the cloud-hosted systems have multiple Peers for different internal systems). This means that they can only talk to the things I expect them to, w/o me needing to also handle that at the firewall level.
* The Wireguard Mac and iOS apps are very solid. I use them heavily on multiple devices, and they can be trained for OnDemand activation based on SSIDs. This means that when my phone tries to load my NVR dashboard, it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub). The iOS app is reliable enough that I set it up once on my partner's phone so they could access the cameras and then never had to think about it again.
* Because the only things you need to set up a peer link are shared pubkeys and to agree on IPs, wiring up my endpoints via Puppet was super smooth and adding/adjusting has likewise been smooth. My systems generate a keypair during setup, publish their public key where the other nodes can find it, and all I have to do is update the map of "hey, this server get this WG-internal IP, and this is who should link to it".
> it uses Wireguard if I'm mobile and doesn't if I'm at home (which saves me a hop through the cloud hub)
Appropriate netmasks in the routing tables do this regardless of physical interfaces. For example, if your local network has netmask 255.255.255.128, the local link is prioritised over the tunnel link which could have netmask 255.255.255.0, but you have to decide which nets/subnets are useful to you.
My solution to this is a pair of udp ports relayed to each other on a VPS via socat. Then you can point a home WireGuard server (behind cgnat etc) at one, and clients at the other, with no need to trust the vps
I use defined.net (managed Nebula) for my homelab and side project overlay networks, and it works great. They have a very generous free tier and I automate enrollment with a set of systemd units that I wrote. Configuration is very easy and the Nebula android app does what I need so I can access everything over my phone. A few friends use wg-easy to manage their wireguard setups, and they sing its praise.
I still use wireguard for simple point to point tunnels into my datacenter rack but anything important I use Nebula.
https://github.com/quickvm/defined-systemd-units
An issue with his remote setup is that the remote VPS decrypts packets from the remote laptop, then re-encrypts them for the LAN — this means that the remote VPS can see the plaintext of all those packets. He’ll need to layer TLS or something similar, or run Wireguard over Wireguard.
As somebody with a very similar setup, all the things I'm making accessible over the Wireguard network are HTTPS, SSH, etc. The handful of things that couldn't do native TLS (or were irritating to get configured with automated TLS certs) I stuck behind nginx for TLS.
Short of setting up dns validation and using 3rd party dns service as many registrars don't support API for dns management, how is domain validation done for acquiring TLS certs when serving only via wireguard ?
There's the private CA route but its a pain to setup the certs on all (mobile) devices and Android makes it very scary and hard.
I use https://github.com/go-acme/lego and DNS validation. I'm not sure what DNS provider you're using, but a ton of them have robust APIs. In my case, I'm using Route53 which is notable in that I can generate API credentials that can only update the specific record needed for a particular name's ACME validation record.
Here's a list of supported providers: https://go-acme.github.io/lego/dns/
And in case you're curious, the API perms dance to do specific-record updates: https://github.com/armorfret/terraform-aws-r53-certbot/blob/...
Author here! Indeed, it is mostly HTTPS terminated by Caddy in the server at home. Otherwise, it is SSH.
What options are available to use a remote VPS to facilitate connecting Wireguard directly through the CGNAT? It seems most "client" devices are going to be behind at least some kind of NAT as well.
I did the p2p between any sensible nodes in my network then added routing layer via BGP so any node can "see" any other node regardless of underlying mesh.
The access limits are better served by firewalls than limiting tunelling
This sounds interesting. Could you say more? Do you run BGP on the hosts themselves over the Wireguard tunnels? Do you self-manage a Wireguard mesh?
I just use Tailscale (headscale)... It's Wireguard in essence but much easier to deploy and maintain.
In my opinion Tailscale is the realistic option for most people. The author is familiar with Tailscale having worked with it previously, but my interpretation is that he wanted to get more familiar with the underlying Wireguard technology.
Without fail, in every post on HN mentioning Wireguard:
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
It's almost as if these people are offended someone would not use tailscale. Do you also find stories about people hiking and comment "I just look up scenic locations on google"? Why would you think your +1 to the easy, commercial route is at all interesting to a forum literally having Hacker in the name?
It is quite different here as the OP stated headscale which is own hosted and I understand they use just the tailscale clients. That is mine setup too after using tailscale first and then wanted something own hosted without my traffic going through someone else's network infrastructure (internet excluded of course)
Important to know the security tradeoffs of the Tailscale commercial product. At the very least people will learn about all the pieces.
The privacy tradeoffs are much worse :/ https://tailscale.com/kb/1011/log-mesh-traffic
“Each Tailscale agent in your distributed network streams its logs to a central log server (at log.tailscale.com). This includes real-time events for open and close events for every inter-machine connection (TCP or UDP) on your network.”
RE: https://kieranhealy.org/blog/archives/2013/06/09/using-metad...
I believe setting the "TS_NO_LOGS_NO_SUPPORT=1" env disables this logging.
Yep, despite the extremely FUD-y variable name, though there's still no way to opt out on iOS or Android: https://github.com/tailscale/tailscale/issues/13174
There's an unmerged PR for the Android client: https://github.com/tailscale/tailscale-android/pull/695
That reminds me of a complaint that I had with a visualization library that kept pushing the open source project to be tied into their hosting racket. I brought up how lots of security organizations at the time wouldn't understand how much of a massive problem it was for the end user to decide to make the data immediately available to the entire Internet. It feels wrong that such implications could be missed from a set of environment variables just being missing. Usually you want things to fail safely ...
Agreed it's a bit annoying how non-discoverable this is. Wish it was in some sort of onboarding prompt. But I can see Tailscale's argument for making it the way it is.
Tailscale is really targeting the business market, especially since their product is basically free for personal use. In a corporate environment, I imagine that the client logs are actually hugely valuable to the corporate customers themselves. It lets them see who is accessing what and is super critical when doing a post-mortem after a hack. (also no actual traffic content is logged)
But I still keep client logging disabled for my personal use.
Headscale isn't commercial
Or you can setup Yggdrasil by installing it on each machine and adding a few public volunteer peers to the configuration. My own summary of WireGuard->Nebula->Yggdrasil journey: https://news.ycombinator.com/item?id=43967082
I've looked at Yggdrasil multiple times now, and I always end up backing away because it's security critical software intentionally facing the public internet and talking to untrusted peers, but it's a smaller project with no security audit. In contrast, wireguard has a lot of eyes on it, simpler code that does less, only exposes a listen-only invisible UDP port to untrusted parties, and uses formal verification to ensure that its smaller codebase is secure. Perhaps I'm paranoid, but it's a really hard sell for me to use anything but wireguard:\
I use Tailscale with headscale and it is very simple. I used to have manual WG but this is way easier.
The refusal to use a control plane tells me the author would probably enjoy openvpn much more than vanilla wireguard.
It’s so nice to edit openvpn configuration, maybe some ccd file, restart the openvpn server and see clients reconnect one by one and pick up new options (eg: nee routes etc)
Only if you’re happy with your CPU being pegged at 100% whenever you go beyond 40 mbit/s.
I’d rather use wireguard and get 800 mbit/s.
Hasn't openvpn had hardware acceleration for some time now? Those days are long gone.
Nope it’s still busted and worse of all it’s single core bound!
As a random aside: in a past life, I was helping somebody with a project where we needed a VPN connecting two locations. We were working with OpenVPN, and the nodes we had to handle the connection were beefy with large uplinks but we were hitting the single-core limitation so it didn't matter. We ended up building a proof of concept that launched one OpenVPN instance per core and then bonded them together, which let us get way closer to line rate.
Thankfully we never had to try them for real to see how horribly that would have gone under real load.
It’s surprising to me how bad the scaling is too. A 200megabits/s i what I was able to get on a fairly modern 10900k!
This with the default cipher? From what I can see, AES on a modern CPU should be orders of magnitude faster than that.
Are OpenVPN and Wireguard really comparable?
Wireguard is creating a series of point-to-point links with crypto that's basically impossible to mess up.
OpenVPN sets up a hub and spokes and has far more baggage carried around with it.
tinc is good for setup like that as it automatically meshes the nodes
FWIW I found using one wireguard interface/instance too restrictive. The problem is that wireguard is an L3 interface that has its own internal IP routing table, so only one node can serve as a default (horizon) route for the instance. So for my setup, on each node I have a separate wg interface for every peer node. Every packet that goes into a given interface is destined to come out to the one peer on the other side.
Having been travelling just now with Wireguard as my primary VPN the main thing I noticed is it's completely unusable on airport wifi.
So the big feature I need is any sort of gateway which works over port 80 or 443 natively - ideally at a subpath so it's easy put behind a reverse proxy.
Are you saying the protocol is being blocked? I have used Wireguard over 443 at airports with no issues.
UDP can be blocked. Doesn't affect browsing DNS is local anyway.
However it's arranged generally rovust VPN requires being able to send TCP over a well known port - which Wireguard by default doesn't do.
So I'm currently looking for an Android VPN app which can multiplex via a long live lived TLS connection.