GitLab Runners

As was hinted at in my last post, having gotten GitLab up and running with a nice fancy self-hosted domain name, I wanted to get CI/CD running in order to build some self-hosted websites. The last line of the post started with “I’m expecting this to be a bit tougher than getting it up and running”. Oh, the naïveté! There’s a reason this site isn’t called www.optimism.pays.off.com! Of course it was tougher, and once again, finding good resources on the web to help out was harder than resisting another turn of Civilisation II at 3am.

Running Away Bravely⌗

CI/CD in GitLab is managed by GitLab Runners. These are separate processes from GitLab itself that exist to run builds and automated tests, then deploy the build results somewhere. You register runners as either shared, project-level, or group-level, and tag them with various useful terms (python, web, etc). For each of your projects, you setup a CI/CD build file called .gitlab-ci.yml, and configure your build steps in it. Each step is given a tag, and the actual CI build will be assigned to a runner that matches that tag. This allows you to have custom runners for specific build types.

This is all fairly sensible, although I completely overlooked the tags when I was getting everything up and running, so make sure you pay attention and RTFM, don’t just RTFPOTS1.

Composition Is Key⌗

I initially debated whether to run the GitLab Runner in the same docker compose file as the main GitLab instance or not. I figured it probably didn’t make much difference either way, so I popped it all together in the same file, following the instructions here. Although there is a section there for self-signed SSL certs, that doesn’t apply here, as GitLab is running happily withouth them, letting Caddy do all the hard work. I added this to the existing compose file:

  runner:
    image: gitlab/gitlab-runner:latest
    container_name: gitlab_runner
    hostname: 'gitlab.example.com'
    restart: unless-stopped
    volumes:
      - '$GITLAB_HOME/runner:/etc/gitlab-runner'

I saved, I restarted, and I followed the instructions on how to register a new runner. I wasn’t really sure what to do for the executor type, so picked (somewhat randomly) shell, as we can do everything on the shell. (I obviously hadn’t read this!) I restarted the GitLab containers, then excitedly created my first .gitlab-ci.yml file. I copied an example config from somewhere on the web, ending up with something that looked like this:

before_script:
  - python --version
  - pip install -r requirements.txt


build_job:
  stage: build
  script:
      mkdocs build

Beautiful, this will work like a dream. Hitting the commit button, I tabbed over to my GitLab dashboard to see how the build was going. The status was “Stuck”. Not quite what I’d hoped for. This was due to the aforementioned lack of tags in any of my config. I could update the tag easily enough in the CI file for my project, but not knowing how to update a runner config, I thought it was easier to create a new one. This time, I selected “python” and “linux” as tags on the runner, then restarted the containers again. At this point, stuff started getting weird.

Is Anyone Listening?⌗ 

gitlab_runner | WARNING: Checking for jobs... failed                runner=YpsmmxXrz status=couldn't execute POST against https://gitlab.mydomaingoeshere.com/api/v4/jobs/request: Post "https://gitlab.mydomaingoeshere.com/api/v4/jobs/request": dial tcp 192.168.64.3:443: connect: connection refused

What the…? How can a connection between two services in the same container be refused? Maybe it wasn’t the runner itself, but the config? Looking at the config.toml file used by the runner to store the configs, I had:

[[runners]]
  name = "python_runner"
  url = "https://gitlab.mydomaingoeshere.com"
  id = 3

Maybe if I change that to not be https?

gitlab_runner | WARNING: Checking for jobs... failed                runner=YpsmmxXrz status=couldn't execute POST against http://gitlab.mydomaingoeshere.com/api/v4/jobs/request: Post "http://gitlab.mydomaingoeshere.com/api/v4/jobs/request": dial tcp 192.168.80.3:80: connect: connection refused

No dice. Cracking my knuckles and grabbing a pale ale, I knew I was in for a lot of Kagi-ing.

Trawling The Archives⌗

As usual with anything “odd” with GitLab, it seems that a lot of internet trawling is needed to find helpful information. It also seems that there’s no single source of info that helps. Instead, you have to act like a Source Repository Archaeologist, exploring dusty troves of knowledge in the hopes of finding enough small clues to piece together a full picture of what the issue is.

One of the posts that I stumbled across mentioned that having the hostname defined in a combined GitLab/GitLab Runner compose file could potentially mess things up. I tried without it, which seemed to solve one problem (no more problems with refused connections), but brought up another one:

 Couldn't execute POST against https://hostname.tld/api/v4/jobs/request: Post https://hostname.tld/api/v4/jobs/request: x509: certificate signed by unknown authority

So close and yet so far. But this one felt a bit more understandable. With the SSL setup that I had, GitLab was having trouble recognising the certificate. This made finding an answer online a bit quicker. This post did the trick perfectly - modifying the Caddy configuration for GitLab to use the full certificate chain got rid of the error.

One facepalm moment during all of this was when I saw this alarming message appearing in the logs when starting up the container:

gitlab_runner | WARNING: Checking for jobs... failed                runner=YpsmmxXrz status=502 Bad Gateway

My initial thought was that I’d messed something up again, sigh. But if you think about this a bit, the GitLab instance is just starting up, so the Runner can’t actually check for jobs yet. Exercising a bit of patience and ignoring this until after everything had started up, the “problem” vanished. So don’t panic right away if you see this, it should all be fine. Maybe.

One other thing that I tried that may have helped (a bit hard to tell though!) was setting up a custom docker network called git for use with the main services and the docker executors (by adding network_mode = "my_docker_network_name" to the executor config in config.toml). I may try removing this one day to see if it actually makes any difference or not, but for now, it’s still there!

Lastly, after a bit more reading, it was clear that the shell executor was definitely not the right way to run Python builds. The GitLab website has this page with a good example of how it could all work using a Docker executor. I manually triggered a new pipeline build for the test project, and a minute or so later, I had my first successful build.

Aren’t We Forgetting Something?⌗

The whole point of setting up the CI/CD function was to build home-hosted internal web sites for our own documentation. With the build working, it was now time to sort out the CD part of CI/CD. Out output of GitLab CI processes are known as artifacts. Reading up on how to deploy this, it seemed that GitLab only allowed artifacts to be downloaded from a web browser. This wasn’t what I was expecting, although it’s probably reasonable for a normally publicly-available service like GitLab or GitHub. So how could I get the websites deployed?

Turns out that old faithful is the simplest way. A simple file copy of the build output to a target directory should take care of things nicely. But given the build is running inside Docker, how do I configure the output paths? config.toml to the rescue! Each runner config has a volumes entry that allows you to supply mappings that the executor can see. I initially got a bit recursive and thought that the mapping needed to be from the GitLab Runner container to the executor container - in other words, a double-mapping. It turns out though that the executor mappings are for the machine that everything is running on, so adding this to the config:

volumes = ["/srv/sites:/sites"]

then adding the following step to my CI build script:

build_job:
  tags:
    - python
  stage: build
  script:
    - mkdocs build
    - cp -r /builds/my_user/my_project/site/* /sites/my_project

And hey presto, Caddy can now serve this up to end users. Looks like we’re done!

TL;CBA⌗

  • Don’t use hostname if you’re sharing a single docker compose file for GitLab and GitLab Runner, seems to mess things up a bit.
  • Possibly use a dedicated network, although I reserve the right to be wrong on this.
  • Ensure that your reverse proxy is using a full certificate chain.
  • Research the best executor up front.
  • Use tags!

Fire And Forget⌗

So at this stage, my GitLab setup is complete and doing what I want it to. There’s a vague chance I’ll do more with it for some other projects, probably Rust based, but that would be a way off if I did. In the meantime, Grumpy Metal Girl and I can continue on our documentation journey and let the server do the heavy lifting for us, just the way Beelzebub intended.

  1. Read The Fragging Post On This Site. ↩︎