Starting off let me be clear, traditional distros aren’t going anywhere. They have their place and people already know and love them. Businesses rely on them. Yet people aren’t using them and we’ve been talking about “Year of the Linux Desktop” for longer than I have been alive. Immutable distros (or “Atomic” as they now like to be called) have been around for a while as well, but are relatively new in the Linux Desktop space and allow end users to be able a Linux desktop like an appliance. They can turn it on and know it will boot properly without having to worry about things users had to worry about in the past like drivers, kernel mods, or new packages breaking things – the ghosts of Linux Desktop Past. If you want to see what can be possible, look at the Steam Deck.

Alright, you get it by now – atomic distros are cool. What makes them even cooler is how you can make your own atomic distro. Enter Bootc! Bootc allows you to make an OS the same way you make an application, using containers! There is no denying that any applications recently developed are shipping today as a Docker container and most developers have some experience with them. Bootc takes this existing developer experience and directly translates it to your OS. Lets look at an example to see how easy it can be:

1
2
3
FROM quay.io/fedora/fedora-bootc:41
COPY nginx.container /usr/share/containers/systemd
COPY nginx.conf /etc/nginx

Simple, right? This copies a file to run a Nginx container as a quadlet and a config into the /etc folder. Anytime I make a change or on a schedule I set it creates a new image and pushes it to my registry essentially bringing GitOps to my OS. With this whenever my machine starts, whether for the first time or millionth time, its going to be configured to work exactly as expected with no extra work or additional configuration. Day 1/2 configuration with other tools like Ansible or Chef are no longer necessary. Everything together in the same repo.

But that was a simple example for an Nginx server, what about something complex like a Linux desktop with all the software I want like VSCode, OpenRazr, and Nvidia drivers? Look no further than Universal Blue. This project is the epitome of “I want everything to work out of the box with no extra work”. In fact The Verge recently published an article saying that one of the Universal Blue projects, Bazzite, was a better experience than SteamOS. High praise! They put a lot of work into streamlining the user experience so anything you could want is just a simple command away and it always “Just Works” which is exactly the kind of experience normal non-technical users want. Something I can give to my parents and know they’ll have access to all the applications they need and won’t be able to break. Bootc is the first step to be able to achieve that.

Bootc allows OS configuration to shift left and be testable and reproducible using the same tools developers already use for application development. Bootc isn’t the only piece to the puzzle and I’m just scratching the surface with what Bootc can do. There are more pieces like ComposeFS & Systemd-sysext but Bootc is where I think people should start. If you want to see more examples feel free to check out my repo on GitLab.

Edited 2025-03-25 with some minor changes around feedback from HackerNews post

Last weekend I was able to attend my first Ubuntu Summit in Den Haag, Netherlands! I was kindly invited to represent the Late Night Linux family of podcasts as one of the hosts of the Hybrid Cloud Show. I was very excited as this means I could finally get to meet one of my co-hosts in person! Canonical paid for my flight and hotel and I was off to meet old friends as well as new ones. This was my first time really participating with the Ubuntu community and it was not quite what I expected.

Canonical has been talking about going public for a while now, at least since 2018 according to my memory. I was positive that as time marches on and we get closer to this seemingly inevitable conclusion that their developer conference would share the same tones. However that was not my experience at all. There were no sponsored talks by partners, the schedule was tight and focused on interesting and innovative subjects, and many of the booths were for the various communities around Ubuntu rather than just companies looking to make a sale.

The booths actually were my favorite activity because it was just time to sit and talk about the various things going on in the community: Linux gaming (!), making better documentation not just for Ubuntu but for all, developing with upstream, and the intricacies of Object Oriented Programming in Rust – just to name a few. To be clear, there were a few companies out there selling things like System76 and Framework but that wasn’t the whole reason they were there. System76 was giving us a sneak peek of their latest alpha for Cosmic and discussing the complexity of making a desktop environment (almost) from scratch.

My favorite thing I saw was how a company called DeepComputing wanted to make a RISC-V developer kit. This isn’t as straightforward as many people might think. Qualcomm found out the hard way with its Windows ARM developer kit. Instead of trying to eat a whole elephant they partnered with Framework to make a mainboard that fit their 13-inch model. Framework laptops are famous for their easily repairable and swappable components. Thanks to Framework’s ethos for open sourcing everything, including its hardware specs, it was a perfect match. Now any developer can start trying to develop for RISC-V by just swapping out a piece that the Framework CEO showed us takes under 5 minutes to do.

While all that was impressive to me, what really blew me was the people I met. Everyone was extraordinary. No, really, I mean it. Every single person I interacted with outside getting coffee, at the booths, or during the talks was sincerely excited to be there and wanted to share their perspective on things. The Ubuntu community has some very intelligent people, not just the Canonical employees, and they’re all working towards making things better for everyone. A very sincere “Thank you” to the whole community for being very welcoming and open to all my questions about the past, present, and future of Ubuntu.

Immich has been a game changer for my family and has fully replaced Google Photos for over a year. The project has been going strong for a while with no signs of stopping and continues to push out updates full of new features! Something happened though on the update to v1.91, Immich was able to remove a dependency on a service called Typesense which they used for indexing and searching through content by using a new extension called vectors inside of their existing PostgresQL database. You can read more about the reasoning here.

The result is that people who were using their own Postgres instance that wasn’t part of the Docker Compose or Helm Chart setup were out of luck with this upgrade and would have to figure out how to move forward. They recommended people switch to a new container image tensorchord/pgvecto-rs as it is a drop in replacement on top of the normal Postgres image but with the extensions needed included. It’s impressive to see a project this big be able to switch something like their database backend and not cause their users any fuss. However I was using the Zalando Postgres Operator which has its own Postgres image called Spilo, so switching away was not so trivial for me.

Zalando is an enterprise-type business and as is customary, can move slowly sometimes. Someone created a feature request to get the vector extension into Spilo back in July 2023. Then in January 2024 someone brought it up at the operator level and someone responded that support has already been baked into Spilo and that the default image for the operator will be updated in the next release. So it seemed like all the pieces were in place to use the vector extension it was just a matter of time. I also couldn’t switch to tensorchord/pgvecto-rs as there are some basic issues preventing high availability like stream replication not working.

I saw someone else mention in the comments of the release that they were also looking for ways forward with their Postgres instances and one person even went so far as to build his own custom container image which can be found here. This was way too much work for me to try and take on at the time so my Immich instance sat on v1.90.2 for a long time, hoping for someone to come along and provide a path forward. Finally I found some time to tackle the issue and I think my solution is worth sharing.

The Solution

This was actually very simple, if the vector extension was already supported in the newest Spilo image ghcr.io/zalando/spilo-15:3.1-p1 then all I need to do is switch the one deployment over to the latest version. Like so!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
    apiVersion: acid.zalan.do/v1
    kind: postgresql
    metadata:
      labels:
        team: acid
      name: acid-immich
      namespace: immich
    spec:
      dockerImage: ghcr.io/zalando/spilo-15:3.1-p1
      allowedSourceRanges: []
      databases:
        immich: immich
      numberOfInstances: 2
      postgresql:
        version: '15'

Once I had ensured that my database was running as normal on the new image, I found that in v1.95.0 they introduced compatibility for the vector extension as opposed to vectors that is bundled with pgvecto-rs by adding this environment variables:

1
    DB_VECTOR_EXTENSION=pgvector

Then I just had to run the following commands against the database according to the migration documentation:

1
    CREATE EXTENSION vector;

That’s it! My terabyte of photos were migrated in about 2 hours and I’m back to the latest version of Immich. Shout out to the team for this amazing project!

Setting Up Separated Ingress Controllers for Internal & External Traffic

First, update your existing chart/manifests to distinguish that your current ingress controller is going to be used for external traffic only. So change its ingressClassName, containerName, and anything else you feel necessary. Secondly, go to your existing ingresses and make sure they’re updated to have the correct ingressClassName or kubernetes.io/ingress.class (which is deprecated at time of writing) so that they will all still work correctly.

Setting Up Internal Traffic inside Kubernetes

Now we have to create another instance of an ingress controller for internal services that I want to run on my cluster. Very simple, just duplicate your helm chart or manifests, do the same changes as the previous step and change the values like ingressClass, controller.name, etc … to correctly route for the new internal ingress controller, then add the new LoadBalancer IP to your local DNS. I suggest making a subdomain for all your internal traffic. For example all my internal services are on local.thrailkill.cloud instead of the base domain. Totally up to you.

Setting Up Internal Traffic outside Kubernetes

Finally – the reason for the article! The secret here is a Kubernetes resource called Endpoints and how they relate to a service. A service does not link straight to pods. When Kubernetes sees a service, and if the service selector matches a pod label, Kubernetes will automatically create an Endpoints object with the same name as the service, which stores the pod’s IP address and port. We can create our own endpoints in comboination with a headless service to forward the traffic outside our cluster. So for example:

1
2
3
4
5
6
7
8
9
10
    apiVersion: v1
    kind: Endpoints
    metadata:
      name: my-service
      namespace: my-namespace
    subsets:
      - addresses:
          - ip: 10.0.0.100
        ports:
          - port: 8080

Then, to create a headless service you need to set clusterIP to None.

NOTE: The endpoint name and the service name must be the same.

1
2
3
4
5
6
7
8
9
10
11
    apiVersion: v1
    kind: Service
    metadata:
      name: my-service
      namespace: my-namespace
    spec:
      clusterIP: None
      ports:
      - port: 8080
        protocol: TCP
        targetPort: 8080

That should be it! Update your local DNS to point a URL to your internal ingress controller and it should work now!

That was too easy…

Yeah, unfortunatly nothing is ever easy. Somethings like Proxmox or GitLab want to handle SSL termination on their end instead of serving HTTP traffic and letting the SSL termination be handled by a reverse proxy. In that case, you will have to allow SSL Passthrough and that should get you where you want to be.

Also, Endpoints are in the process of being replaced by EndpointSlices. I havn’t figured out quite yet how to use this resource but when I do I’ll update this guide.

Let me know if you have any questions or want to share your setup!

Desktop Environment vs Operating System

So when looking at running Linux there are 2 big things you want to look at: the Desktop Environment and the Operating System. The Desktop Environment is how your user interface will look and feel – things like your file browser, application menu, settings, etc. This is usually the most important choice. For a full list of possible Desktop Environments, see this list.

The other big choice is the operating system. This is all about the underpinnings of your computer. How often does it updates? Does it update to the latest and bleeding edge or does it prefer stable packages? If you are a casual user, this shouldn’t mean too much to you however if you are an advanced Linux user, this matters a whole lot. The “which OS (or usually called a Distro) is right for me?” question gets asked almost everyday and everyone has a different opinion. I’m not kidding.

In my trials I found KDE Plasma wasn’t quite ready for a touch centered experience. However I have seen some work to that end recently. Hopefully as more Linux devices like the Steam Deck come out that use Plasma we will see some more development time towards that effort.

Wayland vs X11

While Gnome worked better I never found a setup that had it all. X11 worked great for some features like auto-rotation and on screen keyboard but lacked features like gesture support on screen and trackpad. Wayland on the other hand supported the gestures and ran smoother but couldn’t auto-rotate and I had to manually open the keyboard when I wanted to type. And if you’re using any kind of software KVM like barrier or synergy or something then forget about support coming for Wayland anytime soon.

So How Did It Go?

So for my touch screen laptop, I knew I wanted a Desktop Environment that had good touch support, large icons, and an Operating System that was stable since my laptop isn’t used constantly. For this I tried GNOME Desktop Environment versions of Ubuntu 21.10, OpenSUSE Tumbleweed, Fedora 35, and Pop OS 21.10. Here’s how each went:

• Ubuntu 21.10 - Overall was fantastic. Stable, had immediate support for touch screen, partial scaling for resolutions, and I’m already very familiar with Debian based systems.
• openSUSE Tumbleweed - Honestly this is more of a bleeding edge distro but I’m becoming more of a fan of Suse and wanted to give it a try.
• Fedora 35 - Good initial support, had to run some commands to get things working and never could get fingerprint reader to work. Good option.
• Pop OS 20.10 - Worked just as well as Ubuntu (no suprise since it’s based on Ubuntu) and had some extra goodies like Flatpack support right off the bat.

Due to Pop OS working just as well as Ubuntu, the extra goodies it had, and the recent trouble with Ubuntu I decided to go with it for now. So far besides an issue with Sleep mode, it has worked really well. Battery life is about 8 hours, all my applications run fast and smooth, and it’s easy to administrate. Let’s hope it stays that way!

What is Longhorn?

So if you have a Kubernetes cluster, chances are you want to use the local storage on your nodes in a pool to run your cluster with. In my opinion, the only real option for this kind of setup is Longhorn. I’ve been using Longhorn for about a year and when I first installed it, I did it quickly and easily though the Rancher interface. Between the reliability of the storage and the seamless integration with Rancher, Longhorn has been absolutely amazing.

Then what is the problem?

Rancher has had 2 different interfaces for a while – Cluster Manager (the older one) and Cluster Explorer (the newer one). Cluster Manager was a great start but Rancher realized that its inherent design left it limited for areas like multi-tenancy and so it began working on a new interface that was more suited for bigger and better things. In both UIs, Rancher tries to make things easier for its users and offers “Apps”. Apps are just helm charts but it fills in a lot for you and runs it locally on the cluster so you never have to even drop to terminal.

So a lot of us installed these apps in the older interface and things were great. Then Rancher decided its new interface was ready for use and swapped the default interface to Cluster Explorer. While a welcome change in some ways, it was unwelcome in other ways. Apps that had been installed in the old interface no longer appeared nor were manageable in Cluster Explorer. That means you can’t upgrade, change, or even remove the application unless you go into terminal and run some commands/scripts/crds/etc. This was not ideal for many of us.

How do we fix this?

Long story short, you don’t. The old interface using Helm 2 and the new interface uses Helm 3. There was a migration project called Helm 2to3 but for whatever reason, the way Cluster Manager installed it didn’t play nice with the conversion. There is no way to easily upgrade, but there is a hard way.

My recommendation is to backup everything to S3 or NFS storage, follow the uninstall instructions, and then reinstall using Cluster Explorer or just using Helm on your own. After which, you can restore your volumes and proceed as normal. While not ideal, I would much rather play it safe with my Kubernetes storage than something that might break down the line because of some hacks I did previously.

P.S. Make sure to follow the uninstall instructions carefully and give the operations time to finish, otherwise you might end up in a bad situation where you’ll have your backups delete themselves. Shoutout to @JenTing for helping me through that.

Bonus!

After reaching out to the Longhorn team about the incompatability, I received this explanation from Vincent:

Technically it was still Helm before, but Tiller in Helm 2 had no concept of multi-tenancy and everyone ran it as admin and let anyone with access deploy whatever they wanted through it. So we added an “Apps” middleman that ran Tiller for you, only on-demand, as the user requesting the operation instead of giving away arbitrary admin. Helm 3 does the same on its own (while eliminating the daemon  entirely) so our abstraction is no longer needed or useful.  And they have their own problems upgrading from 2->3 even without involving us. You can manage existing old Apps, under Legacy in the left nav when a single Project is selected in the filter at the top of the UI.  They are not “gone”.

So the other day I had an application not doing its daily jobs correctly. Unfortunately, because of my docker setup on this particular node, I didn’t have the logs to go back and see what had happened. This left me to wait until the next day to watch the console in real time to diagnose the failure. In an enterprise setting this just isn’t something I have ever had to worry about because logging is so important that it’s almost a reflex anytime I write code. So why would I treat my private cloud any differently?

First off, this is not a comparison of the various logging products out there. There are a lot of other well-written articles out there concerning that. I particularly thought The Chief I/O article to be insightful and helpful. To me it came down to a solution that would be able to show me a unified interface to search my logs across Kubernetes as well as my other machines like my NAS or Raspberry Pis. The kube-prometheus-stack helm chart also seemed to give me a bunch of Out Of Memory exceptions that I couldn’t solve. So I needed to find something else.

For me, it came down to Graylog or Loki. I was already familiar with the Grafana, Prometheus, Alert Manager and the like. Graylog seemed like a really good product that has a lot of backing and adoption however I could not find an official way to bring in Kubernetes logs into Graylog. There are third-party solutions out there for what it’s worth.

The Work

I decided to run this on my Kubernetes cluster as a helm chart. Here are the various ways to run it in case that isn’t for you.

1. Create a new Loki namespace (always separate workloads by namespace)
2. Create your chart values file. Here is what mine looks like:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
    loki:
      enabled: true
      persistence:
        enabled: true
        size: 5Gi
       
    promtail:
      enabled: true
       
    grafana:
      enabled: true
      sidecar:
        datasources:
          enabled: true
      image:
        tag: 7.5.0
       
    prometheus:
      enabled: true
      alertmanager:
        persistentVolume:
          enabled: true
      server:
        persistentVolume:
          enabled: false
   

3. Run the helm install command:

   1
   helm install loki grafana/loki-stack --values values.yaml -n <YOUR NAMESPACE>
   

4. That’s it! If everything worked correctly you should have logging via Loki and monitoring through Prometheus. To see grafana you can get the admin user account password by running: kubectl get secret --namespace loki loki-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

5. To see Grafana either do:

• kubectl edit svc grafana-loki -n <YOUR NAMSPACE> and change the service from ClusterIp to HostPort or LoadBalancer then access from the IP address.
• Or you can temporaily connect by forwarding the port to your localhost by doing: kubectl port-forward --namespace loki service/loki-grafana 3000:80

Congrats! Now make sure you familiarize yourself with the LogQL that Loki uses for queries. It’s a bit of a learning curve but I’ve found it to be intuitive. I might write up a review if I feel strongly one way or another.

For those of us who value our privacy and security of our data, self hosting our apps and services is really the only way to go. In almost no other manner can you guarantee that your data isn’t being spied on, metadata isn’t being collected, or network traffic being watched to name some examples. Sure there are ways to combat that like encrypting your data before sending it, but its easier and safer in our little controlled piece of the internet.

“With great power comes great responsibility” - Ben Parker

With complete control of our network, data, and security it also means that we who self host are also 100% responsible for managing the infrastructure, securing it, and making it available to all users. Some of this can be easy with the right tools. At home I use Terraform to maintain my Kubernetes cluster which then all my services run on Kubernetes. My hosts are all easily managed and updated using Ansible. I can spin up and spin down my whole stack in minutes since I have built them with good tools. However even with these good tools that doesn’t mean that my stuff is secure. For example, I have to set a lot of rules on my router and switches to make sure that other devices can’t snoop or see the traffic and vice versa. Even with these amazing tools, it’s not going to matter if my internet goes out. Like most problems, if you throw enough money at it you can probably solve it. I could use multiple ISPs, or have a 5G failover connection. What about if I lose power? I have to buy a generator? Man, things add up quick and suddenly you have to ask yourself, are all these things worth it?

Honestly, unless you’re self hosting something you probably shouldn’t be, the answer most likely is no. Self-hosting usually means your hosting your stuff. Stuff that you and maybe your family and friends use. They’ll understand if things go down for a bit. They can live without Plex or Nextcloud for a day. But for the things that matter, the things like your APIs or your website where you sell your handmade organic tree based writing surfaces, those might cost your money while they’re down. For these, its time to use a cloud service. Someone else needs to deal with failover connections, backup generators, data replication and backups, all the overhead that comes with hosting important apps and services.

WAIT HOLD ON DON’T CLICK AWAY YET

I know, I know, I want to self-host everything too. However, being a good engineer is knowing when to use the right tool. Unless you have a really good reason, you shouldn’t be reinventing the wheel and adding backup generators to your house… I mean…unless you live in an area with hurricanes or something and already have one…

Option A - Private Cloud

SO. If you shouldn’t self host, what are you going to do? Well it depends, if you’re looking at doing things long-term, then I believe it’s best to use a private cloud whether its a Virtual Private Cloud or Virtual Private Server. I wouldn’t go full buy-and-install-your-own-hardware-in-a-datacenter Private Cloud unless you are a larger business and have special needs like regulatory compliance. If you’re already self hosting then the added burdens of managing a virtual private cloud/server should be no problem to you. Doing so gives you the most control you can possibly get without installing hardware in a co-op or something similar. It also allows you to offload some of the management or configuration to the hosting company for when your company takes off and you haven’t hired some additional help yet. Remember that a major reason we’re running a private cloud is because we want as much control over our infrastructure as possible, and we’re most likely going to be paying a price for that compared to a public cloud. According to a study done by VMware running a private cloud usually is less expensive** in the long run** compared to a public cloud. Win-Win.

Option B - Public Cloud

If you are looking at doing something short-term then things can get complicated. It might be worth it to use the public cloud and be defensive about it by using tools to protect your privacy. For networking, technologies like Wireguard or Nebula make it easy to quickly add a public cloud instance to appear on your network so it can be treated as part of your private cloud. For storage, encryption is the name of the game. Almost all public cloud providers offer the ability to encrypt your data at rest. The trick is here to not give the provider the encryption key as well. Why would you give a burglar a key to get into your data? Usually it’s called something like “Customer-managed encryption”. If you HAVE to let the provider manage the keys then there are still options you can do like encrypting your data with a tool like Cryptomator before it lands on their hard drives. This is a lot of work to protect your data and privacy but man, public clouds are literallygivingawayfreeresources just to get you hooked – make it work for you.

Option C - Hybrid Cloud

¿Por que no los dos?

For those of you who don’t speak Spanish, that means “Why not both?”. I know that “Hybrid Cloud” is a huge buzzword but I’m going to spell it out so it’s very clear what I am saying here: Self host + Private Cloud. That’s what I mean. It can get complicated between configuring load balancers and duplicating data across local nodes and cloud nodes but it’s a valid option for making sure that something is always available for your users. If you’re going this route I recommend using an orchestrator of some sort. Like if you are using VMware at home, then setting up HCX and VMware Cloud so that way its very easy to vMotion your stuff over if something happens on either location. Kubernetes can also do this natively. If a node disappears because the power goes out at home, it will try to auto heal and move workloads to new nodes within the rules you have set in your cluster.

Final Thoughts

Like most things in Software Engineering, there are a lot of ways to go about things. A difference between a good engineer and a great engineer is knowing when to use which tools and how to best configure those tools. Just because I haven’t mentioned it here doesn’t mean its not a viable solution. If you find something that works for you, please comment or message me and let me know!

Why?

If you are just starting to get into Kubernetes and containerization, Rancher can be a great tool for you. It strikes the incredible balance of making a lot of things easy and streamlined for you without hiding the advanced configuration options and details. This makes learning and using Kubernetes really easy with Rancher. It can be a little intimidating to deploy something when you just barely stood up your cluster so using Rancher can help you get stood up quickly. Rancher also doesn’t use and proprietary APIs or libraries when it performs actions. Its all the same tools you would use if you were doing this with CLI. On top of all this, Rancher adds a lot of visualization to what is happening inside your clusters and their health. Rancher even has its own repository of charts that come preconfigured for use with Rancher for things like reverse proxies, monitoring, alerting, distributed storage, and more.

Why Not?

Rancher is a fantastic tool to get started like I said, however it makes things so easy that it might become hard in the future to do anything without it. This can come back to hurt you when you look for a guide or are trying to do something in an environment that doesn’t have Rancher. Learning kubectl and the different parts of Kubernetes is absolutely critical even if you use Rancher. **Rancher is meant to add value to your existing tool set. Not replace it. **So if you understand this and want to get started with Rancher, see below!

Steps:

We’re going to get you started by getting you a certificate to use with Rancher (SSL is always a good idea), then actually install Rancher.

1. Make sure you have helm installed
2. Install cert manager:

   1
   2
   3
   4
   5
   6
   7
    https://github.com/jetstack/cert-manager/releases/download/v1.3.1/cert-manager.crds.yaml
    kubectl create namespace cert-manager
    helm repo add jetstack https://charts.jetstack.io
    helm repo update
    helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --version v1.3.1
   

3. Install Rancher and expose the deployment so you can get to it. Make sure to set the hostname to whatever you’re going to use to connect to Rancher. For example, rancher.local or if you’re going to expose this on your domain (which I wouldn’t advise) then something like rancher.my.domain.

   1
   2
   3
   4
   5
    helm install rancher rancher-latest/rancher \
      --namespace cattle-system \
      --set hostname=rancher.my.domain
    kubectl -n cattle-system rollout status deploy/rancher
    kubectl expose deployment rancher -n cattle-system --type=LoadBalancer --name=rancher-lb --port=443
   

That’s it! It is really that simple. Now when you go to the IP of the loadbalancer that was created, you’ll be greeted with a screen to set up a master account. After you set that up, you’ll be able to see you “local” cluster which is your k3s kubernetes cluster! Congrats!

The next step is to install a reverse proxy and then add port forwarding on your router to the static IP for your proxy and start making your deployments accessible to the internet. We’ll cover that next time.

So for my self hosted setup, my workloads run on Kubernetes using k3s across 3 machines. For the sake of keeping it stupid simple, I’m going to provide some light context and documentation for where to go if you want to really understand whats happening.

If you want high availability (also commonly referred to as HA) then you have 2 options. You can either run an embedded database or an external database. The embedded DB uses etcd and the external DB can use any major DB (see the list here). With an external DB you have a minimum of 2 machines and with the embedded you need a minimum of 3 but also you have to have an odd number of machines. See the full list of requirements for external and embedded. For this guide, I used the embedded since I have exactly 3 machines and I didn’t want to have to maintain another DB. Don’t worry, even if you use the embedded there’s still a way to backup your DB (stay tuned for another guide on how to setup that).

One thing that I am going to change here for my installation of k3s is their service load-balancer. Its Klipper and while I’m sure it can work, I’d rather use MetalLB which I’m more comfortable with. Since I’m not using AWS or GCP for this deployment and therefore unable to use the load-balancers they offer, MetalLB will allow me to create and use a bare-metal load-balancer which will be helpful to help make my workloads HA across my different machines. More on that later.

So to keep this as stupid simple as possible, this is what you’re going to get by following this guide:

1. A load-balancer for your machines (so if one machine goes down, the other machines can still talk to each other)
2. High Availbility Kubernetes Cluster (the dream)
3. A load-balancer for your workloads (so your workloads can use static IPs which makes configuring workloads WAY easier)
4. Sanity

Steps:

1. Make sure your machines have static IPs. Kubernetes REALLY doesn’t like it when machines start moving around, even with a load-balancer in front of them.

2. Open up the correct ports on your machines. See here for a list of what each is for.

The TCP ports are:

• 80
• 179
• 443
• 2376
• 2379
• 2380
• 6443
• 9099
• 9796
• 10250
• 10254

The UDP ports are:

• 22
• 80
• 443
• 2376
• 4789
• 6783
• 6784
• 7946
• 8472

1. Set up your machine load-balancer. You need to set up a load-balancer that will act as the single point of reference for your cluster. Then that load-balancer will distribute the requests to the machines as it sees fit. I used NGINX. Simple, light, and powerful. Feel free to use whatever you want. This can be run on docker or bare-metal. Literally thousands of ways and guides on how. Here is the config for NGINX:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   #uncomment this next line if you are NOT running nginx in docker
   #load_module /usr/lib/nginx/modules/ngx_stream_module.so;
       
   events {}
       
   stream {
     upstream k3s_servers {
       server server0:6443;
       server server1:6443;
       server server2:6443;
     }
       
     server {
       listen 6443;
       proxy_pass k3s_servers;
       ssl_verify_client optional;
     }
   }
   

2. Make sure your machines can talk to each other. a simple nc -vz server:port will tell you if your machines can talk to each other over these ports.
3. Choose which computer is going to be the master machine, which are going to be servers, and which are going to be worker nodes. If you don’t know what this means, don’t worry. Click here to understand. Got it? Great. We’re going to install a small tool called k3sup to help us easily install and manage k3s. Run the installer script like so: curl -sLS https://get.k3sup.dev | sh sudo install k3sup /usr/local/bin/
4. Now that you got it installed, here is a quick little bash script I made to make this super easy for you. All you got to do is add your hostnames or IPs and the location of the SSH key for these machines:

   1
   2
   3
   4
   5
   6
   7
   8
   9
   10
   11
   12
   13
   14
   15
   16
   17
   18
   19
   20
   21
   22
   23
   24
   25
   26
   27
   28
   29
   30
   31
   32
   33
   34
   35
   36
   37
   38
   39
   40
    #!/bin/bash
    main_server=server0
    servers='server1 server2'
    agents='agent0 agent1 agent2'
    ssh_key='~/.ssh/ssh.pem'
       
    echo 'Installing K3s on main server'
    k3sup install \
      --ip $main_server \
      --user sean \
      --cluster \
      --k3s-extra-args '--no-deploy traefik --no-deploy servicelb' \
      --ssh-key $ssh_key
    echo ' '
    echo 'Joining other servers'
    for s in $servers; do
    echo ' '
    echo 'Joining ' $s
    k3sup join \
      --ip $s \
      --user sean \
      --server-user sean \
      --server-ip $main_server \
      --server \
      --ssh-key $ssh_key
    done
    echo ' '
    echo 'Joining other agents'
    for s in $agents; do
    echo ' '
    echo 'Joining ' $s
    k3sup join \
      --ip $s \
      --user sean \
      --server-user sean \
      --server-ip $main_server \
      --ssh-key $ssh_key
    done
       
    echo 'K3s Install Completed'
   

5. Last step, you need to be able to control this cluster. So take the contents of kubeconfig that was generated for you and put that in the ~/.kube/config folder on whatever machine youre going to use to run kubectl on.
6. Now on said machine, test to see if everything is up and running by doing: kubectl get nodes and make sure you see all your machines (from now on, they’re called “nodes”).

If you did everything right, you should see all your nodes when running

1
kubectl get nodes

What Now?

Now you’ve got the dream! Congratulations! Hold up though, before you start applying kubectl commands all over the place, we wanted to use MetalLB, remember? So lets do that real quick.

1. Open up a terminal on your master node. Were going to run some kubectl commands.
2. Run kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
3. Run kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
4. Run kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"

So now we have MetalLB installed but not configured. To do that we need to specify what range of IP addresses MetalLB can assign from. So create a file like this:

1
2
3
4
5
6
7
8
9
10
11
12
    apiVersion: v1
    kind: ConfigMap
    metadata:
      namespace: metallb-system
      name: config
    data:
      config: |
        address-pools:
        - name: default
          protocol: layer2
          addresses:
          - ip.address.range.start-ip.address.range.end

Hopefully it’s obvious, but replace ip.address.range.start and ip.address.range.end with the starting and ending IP address of the pool MetalLB can assign from. Something like 192.168.0.2-192.168.0.100.

1. Final Step, apply that file you just created: kubectl apply -f fileName.yaml

Congratulations!

You’re done. For now. Next step is to deploy some workloads or something. Next post will be about getting Rancher up and running to make deploying workloads way easier for someone not familiar or learning Kubernetes. Peace.

Edited 06/15/21 with updated commands to install K3s using k3sup