Fuzzy Assertions

As discussed in other Test Smells, like Trail by Competitive Calculation, and It Passed Yesterday, it’s very easy to create obscure assertions that are either unreliable or don’t prove a huge amount.

While there are often good “bad reasons” for why we cannot write straightforward assertions, recognising when you are not writing one, and doing the best job possible to move back towards the norm is a very good idea.

Let’s look at a test in a fictional language:

// Note - this is is no specific language
test('adding user to database results in new user', {

    user = createUser('John', 'Smith')

    id = system.addUser(user)

    readUser = system.retrieveUser(id)

    assert(user.firstName).is('John')
    assert(user.lastName).is('Smith')
});

The above has many properties of a straightforward test:

  • The test data we’re using ‘John’ and ‘Smith’ is simple and right there in front of us
  • The system under test as an API which lends itself to testing
  • We’re asserting with precise discrete values that we were able to predict before the test
  • Any auto-generated things like id and maybe a userCreationDate (not shown) are not affecting our test

But What About…?

In the above example, there is the hint that maybe users are provided with an id and also a creation timestamp.

There might have been a temptation to try to write an assertion like this:

    readUser = system.retrieveUser(id)

    assert(user).is(expectedUser)

Where expectedUser has somehow been set up to contain an exact match of the user that the system would generate.

General tip – where there’s a constructed object in a test called expected, there’s a high risk of trial by competitive calculation.

To achieve the ability to predict system generated things, we end up having to plug in our own key generator and mock clock etc into a system. That might be valuable, or it might be a load of test garbage.

Let’s Fuzzy Match

An alternative for the whole object match is to create something like this:

// still a fictional language
    readUser = system.retrieveUser(id)

    assert(user).matches([
        { obj.firstName == 'John' },
        { obj.lastName == 'Smith' },
        { obj.id instanceof UUID },
        { obj.creationDate - now() < inSeconds(5) }
    ])

In this made up test, there are some concrete assertions and then some more fuzzy assertions.

Fuzzy Matching is Cumbersome

The above solution shows how you can make relatively meaningful assertions on object type, approximate object value, maybe you could even do regex matches on the contents of fields… it allows you to assert for a value you can’t predict…

But…

The reason the above assertion got big is that we were doing a whole object match on something that can’t be fully matched.

This may be the best option available.

Alternatives

  • Do fuzzy matches in separate tests, one field at a time – avoid the whole object fuzzy match
  • Filter out fields that cannot be matched from comparing data – either by blanking, by copying from actual to expected, or by exclusion rules in the comparison
  • Rig the generators to produce predictable values
  • Write lower-level tests that can predict the values so you don’t have to rely on fuzzy matching at a higher level

Conclusion

Having fuzzy matching in our assertions is a good trick to be able to pull, but it must be a last resort when nothing easier is available.

Asserting on whole payloads generally causes us to face fuzzy matching issues.

More precise lower-level field matching can remove the need for fuzziness.

7 comments

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s