12 mo. ago • 100%

Remote Server won't come back online after "sudo reboot"

I know I probably fucked something up, but still want some advice.

I have two houses, halfway across the world from each other. Whenever I am on holidays in my second home, I would like to still access my home network and vice versa. I have a Tailscale VPN setup for this and I regularly SSH into my server from other devices to configure it, rather than use the physical device. I tend to only access it whenever I need to turn it on or off.

TIFU by trying to reboot it. I was configuring some network stuff for my brand-new project with installing PiHole, and after debugging a little issue, having changed many configurations and being unsure about how to restart everything needed for the configs to take effect, the answer that I was following suggested rebooting to sort all issues in one go. Having tried `sudo reboot` on my local VM earlier today, I thought it couldn't hurt, ran the command on my remote, and it hasn't come back online yet.

It should be automatically connecting to tailscale on startup, it has worked like that in the past, but it hasn't this time. It has been an hour since tailscale last connected with the device.

What did I do wrong, and how do y'all handle rebooting your bare metal when you don't have access to the physical server atm?

5 comments

how do y'all handle rebooting your bare metal when you don't have access to the physical server atm?

I once forgot I was SSH'd into a pi at home and ran sudo shutdown on my laptop. I was in Japan at the time.

I asked a friend to unplug and replug it for me.

3
I always go for low hanging fruit when troubleshooting. You initially said you rebooted a VM, then said your bare metal wasn’t up. I’m assuming you can’t ping your server. Maybe the VPN service didn’t start like you expected. If you have no mechanism to SSH, or otherwise connect to your server you’re not going to have a lot of options.

2
You should put a smart plug on your remote server with the bios setting to always power on after power loss.

This won’t help you if you’ve fucked up network settings, but it’s a life saver if your computer locks up or doesn’t come back.

Turn off the smart switch, wait a few seconds and turn it back on again, your computer should start back up.

2
I dont think there is anything to do now, except manually interact with your server.

For next time, I suggest buying a smart plug, smart outlet, whatever you call it, and configure your server bios to always come back on after power comes back to it.

Though, if you fucked the network settings of your machine, that wouldn't fix anything about it.
What I have is a Raspberry Pi with a wireguard vpn on it, so in the worst case, I can ssh into it, then SSH into a server in the same local network.

But again, if the network settings are so fucked that it doesn't even connect to the local network anymore, there is nothing much to do except going to the server and fixing it there.
Unless you have a PiKVM (Its a raspberry pi with an HDMI In/Out that can interact with your computer even if its off), an IMPI/ILO/iDRAC (its a chip on server motherboards that can power on/power off/show screen of the server even if it's off).

Good luck for your adventures

1
It can have many reasons as to why the server won't connect again. Most likely one of the configurations changes caused it to loose internet connectivity. Don't worry, it happens to everyone at some point.

I can't tell you what exactly you did wrong, but I can give you tips on how to prevent it in the future:

Firstly, take the time to research about whichever service you want to change. Typically, there is a way to reboot any service, without rebooting the entire server. Once you figured that out, you can write a little script that backd up your original config, applies the new one and then reverts the old one back unless you manually cancel within 30 or more seconds.

This ensures that, if your server does loose internet connectivity, your changes are reverted automatically. If everything works as exoected, you just CTRL + C to cancel the rest of the script. Also make sure to spawn the script in an independant tool like tmux, so it doesn't end with your session.

Only once you are sure that all your configs 100% work, you may or may not fully restart the server.

1