Portainer and Me - The End of the Road

For the last few years, Docker has been an integral part of Grumpy Networks here at home. It all started out with just a single container running on the home server, but as the power and flexibility that containers offer became more apparent, the need to run more of them kept growing.

Unfortunately, with multiple containers comes great responsibility many configurations. I began to worry that managing all of these configurations via CLI would become painful¹. I really wanted to expand my usage of Docker though, so what was I going to do?

As a long-time user of openmediavault, I discovered a Docker tab on the OMV-Extras screen, and inside there was an option to install something called… Portainer.

What Is This Vision of Loveliness I See Before Me?⌗

Look at it! Menus! Proper separation of concepts! You can just look at it and know where to go to find what you’re looking for. It seemed to have it all, and upon closer examination, it did! I was hooked.

I could, at a glance, see if there were issues, whether health-checks were passing or not, and generally have an idea of what was happening on my server. But the real beauty seemed to come from the use of stacks.

What is a stack? This page gives a good overview, but TL;DR, it lets you orchestrate containers across multiple machines. It seemed to be built in out of the box in Portainer, and I didn’t fancy learning Kubernetes at the time, so using Stacks seemed like an easy win.

But hang on a second, I’m running these containers just on a single box. I didn’t need to orchestrate multiple machines. What was the big draw here? This is where it gets a bit convoluted.

Let’s Take A Step Back For A Second…⌗

Most of the containers that I use come from linuxserver.io, who I’ve generally found to produce good reliable images for a bunch of stuff that I need to use. (I like their stuff so much I sponsor them monthly for a small amount too, hopefully it helps!) On every one of their image instruction pages, they list a bunch of important things describing what the image does, how to set it up, important configuration options, and a bunch of other stuff - all really helpful and easy to follow.

When it comes to configuring the container, there’s normally something like this in the instructions:

When you click on the link for more info, you go here, which gives you some background on how Docker compose works, and some examples of Docker compose YAML files. All pretty straightforward. There’s even a helpful “Common Gotchas” section.

This is where my brain takes its first wrong turn. At some stage, I read something on the internet². Something about Docker compose files needing to be fired up each time you reboot your server. I couldn’t tell you where I read that, but I’m sure I did. Now it turns out that I have a near limitless capacity for laziness³. What I really wanted was something that would do the restarting of containers for me on reboot. Portainer seemed to be that thing.

Building The Relationship⌗

So, I setup every image as a stack, with a nice bit of Docker compose YAML defined for each one. I setup volumes using the nice handy dialog boxes, and marked those volumes as “external” in the compose files. It all worked just perfectly. 14 stacks later, I was in Docker heaven, with a myriad of different services running happily together on one machine - MariaDB, Syncthing, NetData, Ubooquity, CalibreWeb, Paperless, Plex.

Yes, Plex. It was there from the beginning, and would be the end of my relationship with Portainer.

Relationship Worries?⌗

After 2 or more years of using Portainer to manage my containers, one day, Plex refused to start. It happened around the time that I upgraded the Grumpy PiHole to a faster RPi, and also tried using unbound, so assumed that something I’d done with the network setup was stopping Plex from starting. The error logs had a bunch of errors that seemed related to some form of resource access on the internet too, so this seemed to be the culprit.

Plex is not a mainstay in our daily routines at Grumpy Manor. It mostly comes up when we want to stream some obscure black metal to our Sonos kit that isn’t available on <insert streaming service here>. With Plex down, we couldn’t easily scratch that itch when it popped up. But that was fine, we could always just shrug it off and play something else instead. It was something that I’d get around to fixing shortly.

While I’m a Lazy Grumpy Metal Guy, I’m also a Stubborn Grumpy Metal Guy⁴. I refused to give in. I would fix this thing properly by myself, with no outside aid other than Kagi and a bunch of reddit links.

Every week or so, I’d go back and have another go at fixing it. I’d wipe out the config folder, and restart, change my PiHole settings, adjust the unbound config to ensure that Plex could just get to the resources it needed. None of it worked. Nada. Nothing.

Eventually, after 3 or 4 months, I gave in. I needed help.

Outside Counselling⌗

Help in this case came from the LinuxServer.io Discord page. I did my best to describe what was going on. The LinuxServer.io folks responded 7 minutes later, with some general advice. Their response also included the following nugget:

as an additional note, we do not support or recommend portainer for deploying containers due to the incredible number of issues it causes

Whoa. Surely not? I mean, it’s being working so well for me. It can’t be Portainer, it must be me! Asking for more clarification, I got the following response:

Portainer has many issues which make it hard for us to support, such as:
- Incorrect order of source and target of mounts
- Inconsistent case-sensitivity
- No automatically created custom networks for inter-container communication
- Inconsistent compose implementations on different architectures
- Pulls every tag on update when you don't set a specific tag
- Capabilities are hidden and some don't work at all on ARM platforms

even if it runs fine for you doesn't change the fact that we do not support it or recommend it

Mind. Officially. Blown. Could this have been it? Plex was working up until recently! None of those points seemed to apply to my case either, particularly the reference to ARM platforms, as the server is running on Intel hardware. It’s worth noting that at no point did I not believe the LinuxServer folks. It just didn’t seem that this could possibly be the problem.

But, after all, it’s easy to check, right? Shut down the stack in Portainer, and cobble together a quick command line Docker call to imitate the Docker compose file that I was using. And yep, of course, it worked straight away.

It wasn’t Plex that was the problem. It was Portainer.

It’s Just Not Working Out For Us⌗

So, with Portainer identified as the cause of my Docker weirdness, the solution was obvious. The thing that gave me great Grumps was that there was absolutely no mention of this on the LinuxServer.io “Common Gotchas” section. There was mention of Docker compose, but nothing to say “DO NOT USE PORTAINER”. If this had been more clearly stated somewhere, it would have saved me a lot of effort. Having to wait for someone to post something in response to a question on Discord felt a bit much.

Obviously, having realised all this, I knew that I had to ditch Portainer and replace it with something else. But what could I find that would restart all my containers after each reboot and be easy to manage?

Here’s where it comes to light that I’ve been doing it wrong for a very long time. It turns out that at some stage (before I started using Portainer naturally!), a restart mode was added for Docker containers called restart: unless-stopped. This will ensure that containers will always be restarted if they go down, unless they’ve been manually stopped. This includes restarting them after a server reboot. So in other words, I never needed to worry about this in the first place. I was using restart: unless-stopped in my compose files within Portainer all along, so it wasn’t Portainer making the magic happen, it was Docker all along.

Sigh.

I’m Moving On⌗

So now, as per the LinuxServer.io guidelines, I’ve got all of my Docker containers running with Docker compose YAML files, and all of them use restart: unless-stopped. They quite happily start after each reboot too⁵. As for GUI goodness to view running containers, I’m using yacht, which is… fine?

Compared to Portainer’s sweet sweet UI goodness, it’s a bit bare-bones. But it at least works⁶. And at the end of the day, the containers all run happily anyway, so the GUI is just a nice-to-have.

So there we have it. A tale of love found, love lost and sanity recovered. Not too Grumpy after all as it turns out…