I thought I’d put together a catalogue of testing smells (having discussed making tests difficult in the past). There may be others, please feel free to comment on them. While I may focus on Java (and possibly occasional JS) examples, these should be pretty universal.
Here are some smells I’ve found:
- Missing parameterised test – when you did it the long way round because you didn’t bring in parameterisation
- Test body is somewhere else – when the test method calls another method entirely with no other implementation in the test method – often a sign of missing parameterised test
- Two for the price of one – sometimes a sign of missing parameterised tests – where a test is testing two use cases with the same set up.
- Over refactoring of tests – where you can’t read them because they’ve been DRYed out to death
- Boilerplate hell – where you can’t read the test because there’s so much code, perhaps a case of missing test data factory
- Missing test data factory – where every test has its own way of making the same test example data
- Unworldly test data – where the test data is in a different style to real-world data e.g. time processing based on epoch milliseconds near 0, rather than on sensible timestamps that would be used in the real world
- Second guess the calculation – where rather than using concrete test data, we use something that needs us to calculate the correct answer ahead of assertion
- Over exertion assertion – where the implementation of an assertion is heavy and in the body of the test, rather than in an assertion library
- Assertion diversion – where the wrong sort of assert is used, thus making a test failure harder to understand
- Conditional assertions – potentially a case of over exertion or diversion – the choice of assertion in a test appears to be a runtime choice, leading to tests whose objectives are harder to understand/guarantee.
- The True Believer – just enough tests to convince the author that the code must surely be right, not that it most likely isn’t wrong
- Assert the world – where the assertions prove everything, even uninteresting stuff.
- Blinkered assertions – where the assertions are blind to the fact that the whole answer is wrong, because they’re focusing on a subset of the detail.
- Integration test, masquerading as unit test – where there are too many layers involved in making a unit test, so it runs too long
- Overmocking – where tests are testing situations that are guaranteed to pass as they’re whitebox tested against perfect mocks that do not indicate anything to do with reality
- Mock madness – where even near-primitive values like POJOs are being mocked, just because.
- Making a mockery of design – where pure functions have to be dependency injected so they can be mocked.
- Wheel of fortune – where random values in the test can lead to error – see also It Passed Yesterday
- Is There Anybody There? – the flickering test that occasionally breaks a build – bad test or bad code?
- It was like that when I got here – ignoring the preparation of pre and post-test state, leading to all manner of shenanigans.
- Badly reimplementing a bit of test framework – related to over exertion asserts, where there’s an ad-hoc bit of what should be a test library, this also includes home-made shallow implementations for deep problems like managing resources such as database or file.
- I wrote it like this – testing the known implementation rather than the outcome of that implementation.
- Contortionist testing – this is really a design smell. You’re probably adding tests after the code was written and are required to bend over backwards to construct those tests owing to poorly designed code. This especially involves NEEDing to use mocking of static functions or types.
Please feel free to complain about your own testing smells in the comments below. I plan to flesh each of the above out into their own short article in due course.
Other Test Smells resources: