Server upgrades are a mandatory aspect of keeping your IT infra healthy, but it becomes a bit tricky if you version your application stacks. You do that of course because you are smart 😉.

I recently did a production OS upgrade for a customer. To keep full control and a predictable end result it was done through a set of DTAP environments. The important difference from regular OS upgrades is that this one was done while maintaining the software application on top at the same version.

The original procedure was published as a blog post to help out companies, well their engineers, to maintain service availability during platform upgrades to keep customers happy. You can find that here: How to upgrade Ubuntu but preserve your docker versioned application stack.

Ok, on to software versioning and pinning.

Versioning your service

Wait, versioning your “service”? Don’t you mean “software”? Yes, I do but we’ll get to that in a minute. Versioning of software is a good practice when you have a bigger, more complex application platform. This is especially useful when you have dependencies to services of third parties. You being either a consumer of another service or a provider of a service to others.

Giving your software a version makes it easier for your company as a whole to reduce overhead and keep clarity on what’s happening. A simple example for instance is giving support to customers or troubleshooting issues. When you can refer to a version you make sure that all involved parties (customers, developers, managers) are looking at the same thing when discussing it.

If you’re not versioning your application you might just end up with an endless first version. So, about versioning a “service” as opposed to software.

Software dependancies and services

Software engines, e.g. like the Python development language, use a mechanism to easily install a specific version of your application again and again. As an example I refer to Python’s ‘requirements.txt’ components list. This is a source for the Pythong package manager to install the exact versioned set of dependencies for your application. Assume an application has version “v2.7.1”. This could be built with these underlying software components:

Component A=11.3.1
Component B=4.14.4

So, if you want to recreate that exact same application instance for a second customer you know that the end result is exactly the same as a previously deployed setup.

Remember I said “versioning your service” in the beginning? You can apply this same principle to your platform if you abstract the above versioning mechanism one layer upwards. You then end up setting a version for your platform. Assume you have a platform composed of 3 different microservices:

Front-end web service v1.2

Application v2.7.1
Helper application v1.1.14

Backend data service v3.2

Analytics application v6.2.2
Postgres+gis v13.3-3

Backend computing service v1.4

Broker application v5.2.4
External (3rd party) api v3.5.1

Abstracting this structure upward one level, thus versioning your service as a whole, you could define it like this:

Super cool platform v3.7

Front-end customer service v1.2
Backend data service v3.2
Backend (3rd party) computing service v1.4

The main takeaway of this is that you always know that a certain instance of your platform (e.g. your acceptance server) will behave the same as on your production system. So, when upgrading to a new version and deploying it to production, the predicted outcome is as certain as possible.

You should of course scope what you register as versioned because it’s not always useful to version things out of your control. The main goal is to keep control of the quality and processes you are responsible for. While on the other hand, you should describe the full stack that is within your reach. These are:

• The Operation System. Although we can do that for you if you like (get in touch!)
• Low level building blocks. Think of postgres, nginx, docker or other software you use to service your customers
• Your software engine. Thus your language of choice, like php, python, node.js, etcetera
• You application version. The actual software you wrote to build your application.
• And if you’re aiming for scaling. The docker container versions, because you can tag (version) and optionally pre-build them.

NB: For proper (semantic) versioning as used in the example I refer you to: https://semver.org/spec/v2.0.0.html

Pinning software and why not to long

When, and only when, you version your software you can use pinning. Pinning means that you fixate a certain service or underlying software component on a specific version while the rest can be upgraded. Pinning of a software version is usually a temporary fix because in a newer version of a certain software component there could be “breaking changes” that needs to be addressed first.

If you version your software as explained above you already do this, because your upgrades are controlled by the versions you set. Having said that, pinning is usually done for packages living in an environment where everything is upgraded automatically. A good example of this is OS upgrades. In that case the versioning of all the tools and OS software is done by the Operation System’s vendor (or community incase of several Linux distributions).

The problem here starts when you add foreign repositories, because different policies can be in place between the linux distribution itself (as you can see in our blog post about updating Ubuntu) and the extra packages you install. 

I’ve seen breaking changes appear between Ubuntu and an upgrade of the way influxDB handles connections for inbound data streams from telegraf agents. The problem was that Influxdb package upgraded from 2.0 to 3.0 where SSL client certificate validation was mandatory all of a sudden. Since this was not configured as such, the next morning after the nightly updates, there was no data coming in because the clients couldn’t connect anymore. The data was back-pressured on the clients so nothing was lost, but as you can understand this was pretty disruptive.

The simple fix was downgrading influxdb back from 3.0 to 2.0, which made everything operational again. Thereafter temporarily pinning InfluxDB on version 2.0 made sure everything stayed operational during future nightly OS updates. So by pinning influxdb it gave us time to implement the new SSL client configurations while maintaining operational. Afterwards it was only a matter of removing ‘the pin’ from the InfluxDB package and upgrading it back to version 3.0 on the next nightly update.

To summarize. Pinning is a temporary but very valuable trick to keep everything working as it should without unforeseen outages.

Struggling to keep a healthy operational way of working: Get in touch and let’s see if there’s a way I can help you.