Open SSH failed to restart on Linode instance

So last night my Linode instance rebooted after performing an unattended upgrade. When it rebooted my sshd service failed to restart because I have the service bound to an IP address that is running on a VPN (zerotier, for anyone interested). The sshd daemon must have been started before the VPN because it failed to start due to the IP not being available.

I restarted the sshd service and it’s working again, but it’s a bit of a nuisance to have to log into my Linode account to reset this service. I’ve been using this cloud instance for a little over a year and this is the first time this has ever happened, but I’m looking for a way to prevent this from happening again.

My first thought is to edit /etc/systemd/system/sshd.service and add the line
Requires=zerotier-one.service
or maybe
After=zerotier-one.service

I’m not too fond of messing with the systemd service files, if there is a better way to solve the problem. I often miss a much better/easier way to solve an issue, so I wonder if anyone would recommend a better fix. Or is this the best solution?

Systemd should be the way to go, in all honesty. It should be both Requires and After. If you specify “Requires” only, then it is possible the SSH server will be started at the same time as zerotier-one, before the vpn connects, leading to the same problem. If you specify only “After,” then the vpn might not connect and the sshd will be started after the vpn service was launched.

I believe the syntax for hard requirement in systemd is “Wants.” I don’t use systemd, so I just added an if in my service file, for ntpd to wait until the host gets an ip address. Loop the if, if the ip is detected, quit the loop, otherwise sleep a second. Once the specific IP is detected in a grep, launch the service.

while ( [ $(ip a | grep '192.168.0.2' -c) -lt 1 ] ) do sleep 1 ; done

For a VPN setup, or anything that waits for a network to start up and give an IP, this is an easy trick. Systemd is technically superior when it comes to other dependency resolution, like the wants option, which basically kills any service if its dependency is not running. I’m no systemd guru though, just read some man pages once. I don’t know how Wants or BindsTo (another option) work, so I’d say use both Requires and After to ensure the sshd only starts after the vpn is connected and you got an ip.

Also of note, systemd might be starting things a bit early in the bootup process, which means things like “while” and “ip” and “grep” might not be available, due to a lack of shell, so it is possible this option might not work for you, even if you change the start script of sshd in the systemd service file. Just use after and requires, it’s fine.

Out of curiosity, have you thought of using Linode’s firewall service to restrict access to sshd, rather than using your current method? I haven’t had a chance to test wireguard yet, so maybe my thought process isn’t compatible. But you could restrict sshd traffic to a private and restricted network within Linode (using their VLAN feature, for example) and then have wireguard bridge your local network and the remote VLAN, making sure Wireguard is the only way of accessing the private VLAN from the outside. I would think that should make it such that you wouldn’t have to adjust anything at all when things restart.

Of course, I realize that wasn’t what you were asking, but I figured I’d mention that in case it’s fruit for your thoughts.

1 Like

Nope. I haven’t thought about that. But that’s mostly because I don’t understand at all how that would work. The way I understood (which is to say that I don’t) VLAN’s on cloud services like Linode would set up a virtual LAN within multiple instances hosted on the VPS. Since I only have the one VM running on Linode I was under the impression that the VLAN would not be of much use to me. Are you suggesting I could set up a VLAN through Linode that is connected to my VPN so when network services are started it would instantly connect to my VPN?

And oddly enough I added all “Requires,” “After,” and “Wants” for zerotier-one.service and the system still tried (and failed) to start sshd before zerotier had configured an IP. At least that’s what I assumed since the log I was getting is

Error: Bind to port 22 on 10.10.10.10 failed: Cannot assign requested address

I did a little more google-fu and found this article with the solution of adding

Wants=sys-devices-virtual-net-tun0.device
After=sys-devices-virtual-net-tun0.device

where you replace the tun0 with the name of the network device your VPN creates. This is effectively the same as your while loop, but in systemd.

I’ve only tested a couple of reboots but so far this is working for me.

2 Likes

I’m using wireguard at home on my router. I restrict everything in iptables incoming on WAN and allow some incoming, but not all traffic from the wg tunnel. It’s my own 2 sites, so I trust the networks at both ends of the tunnel, but if I didn’t, I would restrict more.

I also don’t allow traffic incoming from LAN to go outbound on WAN, I only allow traffic outbound on the wireguard interface. It makes it so all my devices go out on the internet through the other site.

Bonus points for doing it in iptables via port names: you can set them up even when the interfaces don’t exist. I have BLOCK IN on wlan0 (wan), BLOCK OUT from eth0 to wlan0, ALLOW OUT from eth0 to wg0 and a few more rules based on ports (like 22). Whenever the system runs, iptables loads this. If the tunnel is down, my internal hosts don’t reach the internet, because of the explicit block on WAN, they only go out through wg0.