The Testing Manifold

When we have CI/CD that progresses between environments, it’s sensible to have some sort of automated test gate after deployment to the lower environment which determines whether promotion can occur to a higher environment.

Something like this:

This works great for monolithic repos where the build is building ALL services. A successful “test deployment” action can approve the build artifact to go to the higher environment. This too may have a test pack on the back of it to check if it can be promoted upwards, etc.

Note: there’s a test before the deploy. This is where many lower-level tests should be executed. This is at least unit testing, but it may also run myriad other pre-deployment checks, perhaps even standing up bits of application in Docker to see whether things seem to have a chance of working when deployed.

But What If It’s Not Monolithic?

There’s an odd dilemma when you make microservices:

  • Keep them independent – but then you have a test-permutation problem
  • Merge them into a mono repo – but then low level build things can take an age

I’ve tried two solutions to this:

  • Have a higher-level GitOps repo which brings together the smaller services for deployment and testing as a cohort
  • Share a test pack

Let’s look at this latter one.

Shared Test Pack

While each microservice may have its own contract, there may be cross-service dependencies which need to be smoke tested before final deployment. Perhaps, for cost reasons, we’re sharing a central database server (even if the schemata are per-service). Perhaps we want to check our front end is still compatible with the back end.

Any increment in any service might break the whole. But maybe we still want per service CI/CD lifecycle. We’ll do our best to make sure there are no breaking changes… but how do we prove that. (Note: you can’t prove anything, but you can come close.)

In this diagram, three independent pipelines each end by executing the same test pack. The evidence required for success is that the test pack works, no matter which of the services has just bumped its version.

This leads to some problems which may need resolving:

  • Can the test pack run concurrently if more than one service deploys?
  • If the test pack goes wrong when multiple deploys were running, who’s to blame?
  • How do we manage feature flags if there will be breaking changes?
  • How do we manage feature flags between environments?

All of these are implementation details, though I may offer some thoughts in them in future.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s