The True Believer Test


I’m quite interested in logical fallacies. These are arguments which seem convincing, but have enormous misleading holes in them.

For example: only a god could create a world which containds a cappuccino that tastes as nice as this one; my cappuccino is nice, therefore a god must exist. If you say it quickly enough, people might be convinced, but it’s a circular and misleading as a piece of logic.

While this is not a religious blog, I think we can agree that the logic of computer programming should be entirely logical. Belief should not come into it, and the behaviour of belief is not suitable for logical situations.

How does a believer operate in a world of hard evidence. They have two modes of operation. When I say they I mean humans as a whole. We all have cognitive biases. In particular:

  • Evidence for something, whether big or small, makes you believe in that thing all the more if you already believed in it
  • Evidence against something you believe in is something you genuinely don’t want to accept as valid, and will work hard to reject

If you want evidence of the above things happening, talk to someone whose political views you disagree with.

Surely as software developers and testers we are not that susceptible to this. I present to you The True Believer Test.

This is best illustrated with a simple example:

// perfectly reasonable test
public void findsValue() {
   // filtering list is a new class which wraps
   // a list of strings with some filtering methods
   FilteringList list = new FilteringList("arrow");

// but what if the code under test was this
String getFirstThatBeginsWith(String prefix) {
   // return the first item (oops - forgot to filter)
   return items.get(0);

The above test finds a shred of evidence for the assertion that the filter function works. On its own, it’s not wrong. If it’s the only test, then the test suite is not a test suite, it’s a statement of dedication to the unproven existence of an algorithm. In general you avoid this in testing with the following techniques:

  • A simple shred of evidence test in a happy days scenario
  • A second, third, fourth example, to allow for triangulation – it can’t be accidentally right for too many diverse use cases
  • A test for what happens when there’s no data
  • A test proving that the output is the opposite of what’s desired – e.g. when the data shouldn’t be found, it’s not, etc

The magician who sticks a sword into a box with their assistant in doesn’t convince the audience. They stick several swords into the box in order to leave the audience feeling like there’s a huge amount of evidence that the trick has really been performed against all odds. Keep that in mind when you write your tests.

It’s easy to find evidence for something you want to be true. If you do TDD, then you can use those initial easy tests to demonstrate the fact that you’ve developed nothing – they should go red. This proves they are meaningful. As you build incrementally, so you should be testing incrementally with increasingly more difficult tests to pass (increasingly awkwardly positioned swords if you will). Those incremental tests should fail; if they don’t you’re not testing properly. Then they should only pass when the algorithm is well rounded.

In software development, belief is the enemy of truth.

One comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s