Exceptional Happenstances

Happenstance testing is a test smell where you are asserting something more specifically than relevant to what you’re testing. For example, trying to assert the order of items in a set can lead to disappointment when your runtime changes its algorithm for arranging sets.

Similarly, asserting the exact test of an error message, unless it’s the construction of the message that’s the thing under test, can lead to brittle test failures when the message is improved in a way that the test shouldn’t care about.

Here’s a sanitised example of a recent test failure:

StringProcessorTest.subStringExtractionMustBeInRange:407 expected:<[begin -1, end 3, length 5]> but was:<[String index out of range: -1]> 

The original assertion was checking that some sort of error was thrown, but the assertion had been loaded with the specific text of the error from the string library in Java. One Java upgrade later, and this failed. Clearly, the error message wasn’t the point.

In general, make sure only to assert stuff you care about.

One comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s