There’s been a bit of shitposting on LinkedIn recently about evolutionary design. Things like this:
Similar, I think, to the people declaring that TDD is a myth, or only works in books, or is just dogma and that “real developers know better”, arguments like this seem to rely rather heavily on a straw man definition of the thing that’s being criticised.
One of the biggest elements of the straw man is exemplified by this comment on the above post:
Let me be clear. Emergent design and evolutionary design do not rely on guesswork.
Emergent design and evolutionary design is practiced by people who are good at producing high quality smaller scale implementations which can later be easily refactored for future requirements.
It doesn’t follow that you cannot set a north star vision before you start building. I think it’s a good idea to establish either some key rules of the system, or some approximate guess of where it’s going to end.
However, the magic with evolutionary design is that we don’t build huge jigsaw pieces according to a grand vision. We instead, build small usable increments compatible with our best guess at the grand vision. We get early feedback and we iterate in the next best direction on each increment. We ditch bad approaches much sooner, and try to avoid the churn of regularly modifying the same component.
It does not mean that we hack together some rubbish and hope that it will randomly get better. That’s a technique, but it’s not the intent or correct application of evolutionary design.
Grand Designs vs Build to a Vision
So, let me illustrate this from real world experience.
About 20 months ago, I drew a diagram which we called “The Leviathan”. It predicted how a 2+ year project would solve the problem in hand. It was wrong. But it wasn’t too wrong. Much of what that diagram says is still visible in the software we’ve actually built, but not quite in the way it was predicted.
About 2 projects ago, I drew a diagram of the “big crazy everything solution” that I thought we might end up with. I was trying to show how big the system might get, and I was being quite idealistic with the architecture.
In both cases, we started by building something way way simpler, in the hope of using a small prototype to help us iterate in the right sort of direction, but having also establish the playbook of tools and techniques that might help us.
I believe the “big crazy everything solution” might be a decent prediction of the system it became.
However, the key to having built the right thing right was to produce earlier simpler versions on which theories could be tested, and against which we could fail fast.
In both cases, building the heavy middle bit of the whole system would have taken a long time, yielded minimal learning and visible progress, and set us up for failure.
The System That Became like the Vision
In my current project, I predicted that a particular legacy service that we intended to wrap would need a mock built so we could simulate its behaviour and also simulate failure modes that the real service might encounter, but which we couldn’t force the real one to do.
Let’s say the service was the “Zoo”. I drew on diagrams that there would be:
- Mock Zoo
- A control plane to make the Mock Zoo go wrong at test time for specific tests
- A stateful store of Mock Records which we could put into specific states
- The use of a Redis cluster to store the Mock Records so that we wouldn’t need to tidy up this test service – test data would just expire
I said it would be a serverless service and would use the same REST contract as the real Zoo.
So, where do you start? You can see what it appears to need. It needs a database (in redis), it needs all the contract behaviour of the original service. It needs a control plane. It needs to be a really good simulation of the real Zoo.
So, do we build all that on day one?
My goodness no.
On day one, we built one of its possible calls against a static record, held in a file in the codebase. We deployed this stateless service as a serverless lambda behind an API, and we developed it as minimally as possible so that it could satisfy the small number of use cases we were using.
We didn’t have any of the main features I’ve described. We weren’t sure how we’d secure the control plane. We didn’t know how we’d implement a redis cluster specifically (in general, sure). We didn’t know which APIs we want to ultimately simulate. Because we didn’t need them yet.
The “time to market” for version one was still longer than hoped for. Various of the basics got in our way, but because the version was so simple, they were at their easiest to resolve and get into shape.
It was months before the redis cluster first showed up. Several more weeks before stateful records (rather than static ones), and we’re still yet to simulate some other of the features of Zoo.
In the grand designs approach, we’d have put a team onto simulating the whole of Zoo, with the original design I wrote, and then have delayed some of the other more stimulating activities we did in other parts of the project.
We deliberately underbuilt, even though the vision told us the general trajectory this tool might have. We found that the features we expected to add were simpler when we came to add them.
As it stands, I think the Mock Zoo meets my original design intent pretty well, but I’m enormously glad we built it incrementally and worked out the finer details as we went along. Had we done the opposite, we’d have invested time we didn’t need to invest on features we’re still not using, and we’d have been constantly saying “not just yet” to a nervous product manager asking us when this supporting technology that he didn’t specificially care about would be ready.
TL;DR
Have good ideas, then underbuild them, then evolve them using “intelligent design” not “random chance” in the next best direction, constantly updating your perception of what that is.
The Coding Craftsman 