When you want to disable a slow running test

A quick one. The code for this is here on GitHub.

When testing your software with automated unit tests, you want to be thorough to get code coverage and genuinely cover system behaviour, yet some of the things you want to test can take a fair bit of time to run through tests.

This poses a dilemma:

  • Make the test feedback process as quick as possible
  • Make the test process as thorough as possible

The best answer is to organise your code so that it can be thoroughly tested fast. If that’s not possible you may need a multi-tier automated test. The quick things can be tested straight away with a first-line of feedback, ideally within 5 minutes. Slower things should follow, with the whole end-to-end feedback in under 20 minutes if possible.

This leads to a dilemma – how and when do we disable long-running tests? And, given that we’re often using a combination of maven and within-IDE test runners, how can we disable a test in all contexts EXCEPT the one where it should be run. In my view moving a long-running test into the integration tests, just because you don’t like it, isn’t a great solution. Worse than that, you can jump through hoops trying to break things into categories, only to find that your IDE test runner is running them anyway.

A solution that worked for us recently was to disable certain JUnit tests based on a system property. If the property is not set, the test just doesn’t run. Simple as that. This can be done easily in JUnit by an assumption failure. More simple than that there’s the use of a JUnit rule to take the whole test class and make all of its tests abort if the system property isn’t set.

The code on GitHub is the JUnit rule I created. It is illustrated by this unit test:

/**
 * Quick example of how to use this using a simpler test than that used to verify
 * that the mechanism works
 *
 * The test within this class will only work if the system property "run.test" is set to "true"
 * 
 * To demo this running try setting the JVM option -Drun.test=true, which you can do in the maven
 * command line, or via your IDE's invocation of the junit runner
 */
public class ExampleTest {
	@Rule
	public RequireSystemProperty property = new RequireSystemProperty("run.test", "true");
	
	@Test
	public void systemPropertyDependentTest() {
		// do some sort of test we only want to do when a system property is set...
		assertTrue(true);
	}
}

As you can see – the presence of this rule effectively stops that test running unless you specifically configure the JVM that’s running it with the system property that would activate it. We use this on our CI server in the integration test build where we enable everything and the build takes maybe 12-15 minutes. On the front-line CI build, we want the build time to be about 5 minutes, so we don’t run integration tests, or code coverage tools, or slow-running things.

Hope this is useful.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s