Why is starting with Self-hosting so daunting?

Part of working with tech is knowing what to search for und using the right keywords. If you could give an example what guide / information you are unable to find, someone could give you an example how to search for it. I personally know a few junior devs and junior devops which use llama2 / chatgpt since they dont know how to search for it or read the docs.

You don't have to do it, 99.9999%^([citation needed]) of the population don't. If it doesn't interest or excite you then it's not worth it.

You're kidding?

I'm an avid home-labber trying to get into software engineering and I'm finding the software dev work to be super daunting lol

I can setup a reverse proxy in my sleep - but write a JS function without double checking google? forget it!

I can barely find info on HOW to set up this things. Most blogs and articles I find are about what they are which I already know. And the few that actually explain how to set it up are just throwing so many more concepts at me that I can’t keep up.

This is big part of why I, even as an IT professional, don't do much homelabbing or selfhosting. So often I'll be scrolling through these subs and see something someone has done and I'll decide I want to do that too, but so often I end up abandoning whatever it is because doing x requires googling for y and z because without y and z, x doesn't work. And after a day of dealing with issues and tinkering on work stuff, do I want to spend more time troubleshooting? Often not.

Maybe you should read the documentation for the programs you try to install

Because most people who develop these things are, frankly, terrible at good documentation, or understanding the end-user perspective.

There's also a downward spiral effect when you start getting into these things, because lots of them require dependencies, or ask you to do things but don't explain why, and you're just left wondering why you added that line to a config file somewhere, but if you don't put it there, nothing works.

A vertical slice of the amount of knowledge you need passes through so many different disciplines, operating systems, GUIs, and programming languages that it would look like a Milhojas cake.

I've been a technical writer in the software industry for 17 years. The number one challenge in my work is extracting all of the information I need to write good documentation from the experts elsewhere in my company.

Each one of these things is a separate topic/project. There is of course, overlap, but concentrate on just one at a time. If you try to do everything, you'll go mad. Application Administrator, Network Engineer, Sysadmin, Network Security, are often separate full time jobs. Just concentrate on your own needs, one at a time, then focus on how simple you can pare down that one thing you need/want to self host. What is the minimum set of capabilities you must have for this one thing? Then, expect to have as much research for this one thing as taking a 1 credit college course, at least. Maybe more, maybe less, depending what it is. That's a more realistic set of expectations.

I've done a lot of web design and web hosting, and messing with Linux server stacks for around 30 years, now. You absorb a LOT over time, but it is all always changing. When I set up my first server, Cloudflare didn't exist yet. VPNs weren't a thing yet. The only people I knew with a network in their home were Computer Science professors. Wifi was not a thing yet. It isn't you being inadequate, it's you trying to do multiple jobs that each require constant professional development.

Back in the day you had to find a book if you didn't know people

Most blogs and articles I find are about what they are which I already know.

Exactly.. I neither want "what z is" nor all the history behind z, the Z inventors life, his grand parents lives etc. I just want a solution to the problem im having..

I've studied automotive and never really graduate, but I love reading manuals since I was a kid and always believe that even I don't know what am I doing as long I can read It's manual, guide, documentation and read support I could get a grasp of it even if it takes more time.

A part that made it easier for me to understand self-hosting is the raspberry pi. It let me thinker self-hosting without worrying about power bill, how can I easily reinstall its OS even if I make bunch of mistake, make different copies of my setup on different sd card without worries of hdd or ssd.

Then from that just upgraded stuff from what I've learned.

Invest some time in understanding Docker containers and then install something like Portainer (for example on top of Proxmox + VM, or ProxMOX + LXC, or some Linux or bare metal) and then installing new apps will become very trivial once you understand how to "create" the containers and how to map the volumes and ports.

Depends on what you're trying to do.. Plex/Jellyfin server on a Pi with Zerotier for remote access? Super easy and you can get running in 20 minutes with zero previous knowledge.

Self hosted email and reverse proxy with auto SSL and best security practices? Well that's a rabbit hole...

Use a decent VPS instead of trying to host off a residential internet connection behind 47 layers of NAT and you don't have to worry about 90% of that stuff

i.e. use the right tool for the job

I think the reason is that there's a preferred method for doing things for each person.

Like take reverse proxies for instance.

You can use nginx proxy manager if you're comfortable with docker but uncomfortable with CLI or text file configs.

Or you can just use straight nginx.

Or if you want you can use Caddy, which is what I personally do, because I dig how well Caddy does automatic management of letsencrypt certs. But that means being comfortable grokking how to write a Caddyfile (the config), and how to launch a daemon on some sort of Linux.

So if you want to take my route you've got even more choices, you can run it on a Linux VM, a Linux container, or let docker manage your Linux container. Or you can use someone's Caddy docker container and config that for yourself.

Personally I hand configured a Linux LXC container on proxmox and am using that for all the reverse proxying on my proxmox NUC server.

It's just so many variables now.

I remember back in the day you just installed a LAMP server and ran everything with PHP.

I'm not sure which is better.

But I've got to say, the sheer amount of abstractions and ways to orchestrate a system are mind numbing.

Like you can do proxmox or truenas or truenas scale or Synology or rancher or bare metal. There are so many things to decide on now.

And I've been running Linux for the past 18 years or so.

The biggest thing for me is that you should decide on the outset if you want a simple setup, or if you want to play with bleeding edge enterprise stuff.

One is relatively plug and play and somewhat tends to be less secure.

The other can be fun for some or job training for others. I find the people who find it fun to be a bit weird, like they are basically LARPing being a system administrator. But it might be just dressing up for the job they want instead of the job they've got, or they just like it and that's fine for them.

It's not really for me. I look for simplicity, but also security. Luckily I'm a fan of minimal installs and am happy with operating on a CLI. OpenBSD is a pretty solid base to build on. It makes you think about everything you add and how it's a possible security hole.

Anyway, enjoy reading manuals. That's most of what we do here. Join a forum or two (a real one). Find a supportive community there, ask good questions, contribute good answers. It's computer karma and by helping others you're broadening and solidifying your own comfort levels.

Don't give opinions as fact. They are opinions.

This comment is clearly riddled with opinions. YMMV. Good luck with your goals.

This is true for any thing in life. Especially technical stuff. It helps that you have some understanding to start with, but you should learn how to start small and organically grow. You can't setup all and everything in one go.

I would do in this order to make self hosting bite size,

1. Get docker working, with basic docker stuff.
2. Host one service and use it over local lan.
3. More services, but still strictly internal lan only. Get comfortable.
4. Get VPN and try accessing services through VPN
5. Now start research on how to expose, tools to use etc etc.

Maybe my path can help you. I don’t have any educational / professional background in tech, but it is my hobby. I started simple, wanting a NAS for my home. I downloaded TrueNAS and reading the docs and using a lot of YouTube videos (from multiple creators) I managed to get it up and running. Then I tried sharing but using permissions. It took me weeks to understand ACL and NFS shares. It took me a week to figure out that I need to share both parent and child dataset to access a child via NFS. I had to goggle a lot, but it was such a stupid mistake that there was no information about, so I resort to try and error. After I had it setup I thought it would cool to install pinhole. Then I had to learn VM on trueNas, Linux server, package managers and so on. I had zero experience in Linux. I managed to install docker and portainer via a tutorial but then I learned docker compose files and it made everything easier. I destroyed and created that vm multiple times. Tried Debian, Ubuntu server, fedora server … The pattern goes on, home lab will be about reading a lot and not afraid to try and break stuff. It is better to try when you don’t have a lot of services. Always backup your vm, if something breaks, nuke it and start from the backup. Keep it a piece each time. Don’t try to setup all at once. Have an idea about the end state but solve a small problem at time. They say the best engineers are the ones that are able to break complex problems in really small ones. Doing that I have a vm on true nas running more than 20 containers, a redundant pinhole on a separated server, everything accessible via VPN.

Now I’m learning about podman and kubernetes, probably I will spin a second vm, try to migrate what I have a nuke docker. It never ends while you are curious.

so many more things get thrown around so much in this sub and other resources, yet I can barely find info on HOW to set up this things.

no offense but maybe you just aren't using the right search terms.

Yeah I agree it can be quite daunting, I recommend starting out small and trying to understand and getting one piece working at a time.

That is the approach I've taken while migrating my workloads from manually deployed services on old PC hardware into a fully declarative Kubernetes setup on a more energy efficient mini PC. Sounds scary but it's been a lot of fun actually!

I tried writing down some help on getting started, as well as short explanations on most of the infrastructure here in my homelab git repo: https://github.com/FruitieX/homelab/blob/main/docs/configuration.md

Hopefully that's helpful if you decide to try setting up something similar! 😎

just chip away at it, I'm slowly learning SWAG/reverse proxy. Its complicated for me but its rewarding and eventually becomes easy. Such is the way with linux from my experience

Well I mean there's a reason why these skills are in such high demand.

People are overcomplicating stuff because they fear for security.

Move fast and break stuff, and just start somewhere.

You will not die if you run your setup without reverse proxy for a week.

Just go step for step till everything is nice and secure.

Also the thing about understanding everything is a habit that you get in school. But in bigger software projects it's not feasible at least in my experience.

If you don't want to deal with these things and it's just you or maybe a few others, here's the easy way:

• Define the services for each host (i.e. physical + virtual machine) in a single Docker Compose file. You can even put all of these docker-compose files in a single Git repo
• Install Tailscale on all your machines - this is really easy and will let you securely access everything without having to forward ports or worry about the attack surface of the public web
• Set up something like Heimdall or https://github.com/tailscale/golink to have easy to use shortcuts for your services

This gets you out of SSO signin, certificates, reverse proxy setups, all the things that are (understandably!) complex and annoying to set up

For me and I'm not sure if it's been mentioned here already or not, but I don't really have any programming skills or super geek out on this stuff. But I like the tools, right? So teaching myself some of this is hard, especially when there is an attitude with most of this stuff that you should know everything about docker containers and weird networking rules, etc. Sometimes people are helpful, more often than not, it's a super short answer you have to interpret and decipher to figure out.

Because security isn't easy. Know you know why system admins have a problem with programmers that just expect things to work.

I self host for the same reason I play Factorio, it's a constant stream of solving problems and adding new ways to do things. There is a lot to do, but you don't need to do everything right away. Like a lot of people said, just break it up. I use Vikunja to create a list of things I want to and then break those things down into hour or weekend projects.

There is a challenge in the "bootstrap" process. But once you get something going then its easier to keep adding. Stuff like "Reverse proxy, VPN, Cloudfare bullshit" is not super necessary at first. Start with just getting an app going at an ip address and port that's only accessible on you LAN, then add the other stuff.

Networking is extremely dense and extremely hard.

Its also a space filled with professionals who do it for a job. Many on this sub are current or ex networking or security professionals who understand this shit like the back of their hand.

Im no idiot with IT - I’ve had pc’s my whole life and work in close proxiimity to IT. But networking, security and the clusterfuck of everything associated with it is honestly about as hard to get into as actuarial science.

It requires such a heavy base of fundamental concepts that if you dont start at the bottom and work up, you have no clue wtf is going on.

This repo is worth checking out.

I can’t get anything actually running!

Most of my time in my sysadmin work is sitting there staring at the screen reading error logs about why some new product isn't working.

make sure you ACTUALLY have a solid grasp of networking. dig into the osi model or the 3 layer dod model. Go further than "yup. encapsulation. got it". Look at the actual headers in layer 2 and layer 3. Figure out how packets get routed and how frames get forwarded. Look at how NAT actually mangles packets.

Understanding hos the traffic flows will make troubleshooting much easier.

If you've never gone and built your own system, or started a network from scratch without a container based system, and you've relied on stacking your own environment for a long time, you might necognize it as you would any skill.

I'm sure I could build a shed, for E.g. I know to get the wood, the panels, the cement, and all the things I'll have all documented in front of me.

Then I will start, realize I forgot I also needed a permit, which I then found out I might also need to run power to it, so now shed has become something that doesn't feel impossible, it feels daunting

Why? Well, I don't build sheds, I don't keep building things that would allow me to have all the framework, project of getting my shed built. So, as with a homelab, take it one step at a time, and if rhe shed isn't the house, you're allowed to skip a few things and stick with what works for you.

I use homelab and self hosting as a passion, but I also have 25 years of networking and systems administration experience so docker and containers just made my normal, tedious job of building up complex network infrastructure, feel like a job, and my homelab and self host happens to be a hobby

Treat your homelab as a passion or simple hobby, or use it as a skill building tool for work

Overall it's just time, experience and expectations that either drive you forward or set you back

Decide what you want to self-host. Keep it simple at first then work your way up as need be. You could first self-host a simple web site. You could use Tailscale free vpn with 100.x.x.x ip's to test your setup from another computer anywhere.

my love here :D

That's not bad

The greatest advice my teachers in university told me was to tackle everything in life with the foundation/philosophy of K.I.S.S. = Keep It Simple Stupid.

It's more of a code to live by; to which you should ask yourself the question first "why do I need this thing?" What problem is it solving, and is there a simpler method?

Usually, most labbers don't even need 10% of what we think we need. We introduce complexity for the sake of complexity.

Just be comfortable with an ecosystem or make your own that aligns with your values/principles.

Hone your senses by questioning the conceptual integrity of others (everyone has an opinion, don't take their opinion unless they're a trusted person that you respect). If you want to emulate someone based on their own system that matches yours, then do exactly to the letter what they have done and mimic their approach. Most times, you find that once you've concluded your journey you often feel it wasn't worth the journey in the end and you take those hard lessons and sometimes failures and it bridges your understanding further to the truth of a matter.

Because it's a complicated topic, you're just feeling the actual responsibilities of doing it right that many are ignoring now.

I got hooked by Youtube videos from TechnoTim Christian Lempa and others. Videos include screen capture what they are doing so it bridges some things that may be missing from text guide because it may be 'obvious' if you are doing it all the time.

I keep to principal that I have two 'sections' in my lab 'production' and 'testing' Testing I am mucking about testing stuff it works and production has some stuff that I rely on.

On a certain level, most of those things don't matter.

Yes, reverse proxying and being able to point a domain is nice

A VPN more secure

Cloudflare (either tunnels, DNS, or DDoS protection) is also good

You can always do better with more "best practices" but if you want to stream a movie to some friends? Throwing up a server, opening a port and providing an IP just works as well. The better you get the more fancy you will be.

Its very much a tinkering hobby. Find a reason you want to tinker, and if you enjoy the tinkering you will keep up with it. My current dashboard points to a direct IP. Most of my services are reverse proxied but for some reason I can't get CORS to work well with dashy. At some point I'll get around to fixing it, but it gets the job done now.

As a computer science student you should know: just break it down into small chunks.

1. Get service to run locally (IP / port)
2. port forwarding, access remotely
3. Set up a free DuckDNS domain, Lets Encrypt, use for a bit
4. Buy a cloudflare domain, set up dynamic dns, Lets Encrypt (again), swap away from DuckDNS
5. Set up Nginx reverse proxy, port forward to that instead, proxy to service’s internal IP + port
6. Run more things!

That’s generally what I did.. I’ve been self hosting for about a year now and I’ve got 6 services going now.. once you get going, it’s quite easy!

Senior DevOps Engineer here with 15+ years of experience in system engineering now, even with my skillset and experience I am improving my setup piece by piece.

When I look at my old docker-compose versions and general setup, I sometimes laugh how I could've been so dumb.

You don't have to and won't learn and set everything up at once, ever.

Well, I can somehow understand you, but I didn't have so many problems until now. Especially reverse proxy is in my opinion pretty well documented. I got everything running what I wanted, except LDAP. I can't seem to understand this shit and there aren't any good guides either :D.

@mooglestilzkin has already posted the url for Christian Lempa‘s YT channel. CL has just started a series „made for you“ about Docker: https://youtu.be/Nm1tfmZDqo8?si=g_L8bmm9qcFx9wve

For VPN home labbing you should check out wireguard, it's stupid simple and very powerful. Reverse proxy stuff I handle through nginx, mostly because I'm using it for web hosting anyways and I'm comfortable with the workflow. I don't bother with cloudflare much because I host a jellyfin instance and I believe that's against their TOS, but just take it one step at a time and you'll figure it out in no time.

Check YouTube. I haven’t found a lot of written out guides but someone has made a video on it, I bet. Christian Lempa, Techno Tim, Db Tech, NetworkChuck come to mind for self hosting tutorials on containers and new applications. I’m sure there’s a lot more as well.

at a theoretical level

Theory + experience => skill

I say this because I am not sure how much of what you find "rough" comes from you not being familiar with it and how much comes from real hurdle (not that getting familiar with a subject is not a hurdle in itself).

Most blogs and articles I find...

Those are nice to get an idea but come on, read the doc. Do spend that time. Unless the service was extremely well designed and small in scope, getting a serviceable mental map of how it works can take days (or weeks if you really have no clue). There is no real shortcut to this. If the doc is not good enough, either walk away or engage with the dev/community to get things right.

It might be the first time in your life where you are confronted to having to get things perfectly right for a service to work at all (I say this neutrally, without meaning to offend) but this is how it is: this is what "work" means.

I implement everything with the help of chatgpt without problems. Of course, as the service on github has a lot of issues I skip it.

Because that means nothing?

"Why are computers hard"

"Why is car hard"

Maybe figure out what you're trying to do? "I can't figure out VPNs" is a pretty weird starting point considering you can just google VPN and click literally anything

As with everything in life you need to crawl before you run. And as someone mentioned before skills issue. You can assume most people who self host are Linux engineers / Devops engineers. So they do these kind of setups daily. This is why these channels exist. So we can assist or guide. So you can learn as well.

Self-hosting is like any other topic in computing. You have to read, read and read. If you want to self-host a reverse proxy, go to the official documentation of the reverse proxy. What you are doing is searching for magic recipes in blogs and YouTube, that is why you feel it so "daunting".

The responses you get here are from people that have already perfected their setup so take them with a grain of salt.

I mean I have my setup running with Traefik, Crowdsec, Authentik and that's before the request even reaches the application.

2 months ago that was only Traefik.

A year before that I was using Nginx Proxy Manager instead of Traefik because it was easier to manage and understand.

Half the fun is evolving your homelab. Trying to start out with the full stack of things someone suggests is daunting and nearly impossible.

Take things one step at a time. And honestly if you don't understand what the documentation is talking about, YouTube videos are great. I've had to use it lots to understand how Authentik works but now I understand the docs

If you want security and ease of use for remote access, just use Tailscale. Twingate if you share access with others.

As a software engineer of 17 years, you’re only 2 years in, you ain’t hot shit yet and have lots to learn. It can be difficult. Also, you’re probably just not great at searching/finding answers yet. Sometimes you need to look further down the Google results, or comb through closed issues/PRs in GitHub, or read a weird issue of StackOverflow that has a comment that links to another issue with the exact problem.

Researching / finding answers / problem solving / learning new things will be the #1 skill that will get you somewhere.

I am right there with you. I am a Mechanical Engineer and if we tried putting out a product with the limited documentation that these come out with, we would be taken off the project. ME projects have documentation meant to guide a 10 year old through the process and details. The documentation would fill a 3" 3ring binder.

Yet these network products are released with 1 page of upper level description of the product. Yeah I already know what the product already does, that's why I got it, how about the actual documentation?