Save Time, Save Money

I’ve recently been following the instructions here to build a simple CI/CD pipeline. I mean really simple. It contains my older static content, which I’ve decided to keep in GitLab and sync (from a merge to master) to Amazon S3 where it’s served using their simple static site hosting feature.

What’s nice about this is that it’s so very serverless that I have been able to terminate by old VPS without losing anything of any importance. Good times.

GitLab provides a lot of decent features out of the box and seems intent on making a unified integrated environment in a way that GitHub achieves with an ecosystem of tools, and your private CI/CD mechanisms might achieve with the use of Jenkins, a suite of plugins, and a variety of additional off-the-shelf installations or SaaS packages.

But I digress.

One thing to watch out for when building these sorts of pipelines, especially when sharing runners/slaves, is how long the pipeline takes to run. This is for a couple of reasons:

  • Lowering the latency from development to tested and deployed, even from 30 seconds to 15 seconds, is an enormous win
  • In the future, the big issue with cloud-based CI/CD, when everyone’s doing it, will be how must power/metal is being squandered by inefficient pipelines.

GitLab CI/CD

GitLab uses a .yml file to define build pipelines, tools and steps. Here’s a snippet:

  image: python:latest
  - pip install awscli

The file describes the docker image used as the base of the build, for the build servers are dockerised, which is an enormously awesome thing! and then describes the steps to perform on top of that docker image.

As my deployment wants to use aws s3 sync as its deployment tool, the above was easy to get working. However, it took over a minute to execute. Why?


Docker themselves recommend excluding anything unnecessary from a docker build, and a build of Python, though not a huge ecosystem, seems to be at least one instance of Python too large for a script that just needs to run the AWS CLI.

Worse than that, the image doesn’t even contain the AWS CLI, the installation of which is the first thing that the deployment script needs to do before it can proceed.

Though not enormously problematic, the initial build was taking around a minute with the surplus and missing dependencies.

Enter the Hero

Michael Irwin I salute you. In a world where if you can imagine it and google for it, someone’s probably done it, this chap has produced a docker image with the AWS CLI in it and little else… to be fair, he’s cheekily added jq but that’s genuinely tiny, so it’s no big deal.

Replacing the Python docker image with his aws-cli-docker image brought my build down to a few seconds, rather than a load.

Save The Planet

Since computers are generally very fast with plenty of resources, it’s easy to get complacent about the waste that to you seems to happen only once every so often when you run a flabby build. This waste multiplies horribly though. If you don’t keep an eye on it and refine things to get a balance of simple to maintain and efficient to run, then everyone’s double or treble-sized builds (or worse) will add up to a huge unnecessary investment in resource in our cloud platforms. Worse than that, the natural trend towards growth in development will mean that your builds will grow nonlinearly slower, slowed down further by other clumsily written builds on the same platform from colleagues/other users.

Throw away stuff you don’t need! Bring builds down to single figure times. Save the world!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s