In general, my heart sinks when I think I may have to write some unit tests for threaded code. This can be fiendishly difficult, and the secret with it is not to test things too absolutely, as timings and orders of events can be impossible to predict. Here are a few dos and don’ts.
Don’t
- Over assert everything – thing about the grand outcome you’re looking for and assert that
- Expect timings to be consistent between tests, even within margins
- Allow poor design in your code to lead to really hard to read tests – often concurrent things can make tight coupling in their code even harder to unit test – introduce interfaces to help with the testing
- Assume any particular execution order within your code
- Rely on Thread.sleep to keep things in sync – it can, but it won’t guarantee much.
Do
- Test on multiple machines, especially your CI server – you’ll find concurrency bugs that way
- Use sensible synchronization between your test code and any worker threads to keep control of things
- Keep it simple
- Use interfaces which can be mocked to see what your threads are doing to their outside world
- Keep it simple
- Keep it simple
and finally, here are two things you really should know when you’re doing concurrent testing
Tests Can Timeout
JUnit allows for a whole test to timeout. When you specify a timeout for the test, the test itself is run in a worker thread while JUnit watches to see if it takes too long, killing it with an exception if it overruns. This allows you to do things like join a thread that’s supposed to finish, in the hope that the whole test will end soon enough.
@Test(timeout=1000) public void testSomething() { Thread thread = new Thread(new Runnable() { public void run() { waitForSomethingToHappen(); // then we end } }); // the worker is now started and is waiting for something to happen thread.start(); // set the wheels in motion so that thing happens someService.triggerStartOfSomething(); // now we wait for the worker thread to spot it thread.join(); // if we get here without timeout then the system works. }
Mockito’s Verify Can Wait Until Something Has Happened
When you have done the thing which should have provoked your concurrent code to get the result, you really need to wait for that result to propagate back your end point. If it’s a mockito mock, then use the timeout within the verify call to wait for everything to come back.
interface SomeInterface { // this is where the answer would be written by a worker void storeResultFromWorker(Result result); } @Test public void workersCanConcurrentlyGetAnswer() { // the mock that will receive the answer SomeInterface someInterface = mock(SomeInterface.class); // create 100 worker threads to work on their answers for(int i=0; i<100; i++) { // let's assume the worker will do its thing and then tell someInterface about it createAndStartWorkerThread(i, someInterface); } // wait to see if all 100 threads got an answer verify(someInterface, timeout(1000).times(100)).storeResultFromWorker(any(Result.class)); }
In the above code, we’re testing that the workers all get an answer and tell the mock within the timeout period. This could well be enough testing. It saves us having to try to guess when they’ll all have run.
@Ashley – The problem I hit is when testing / asserting from multiple worker threads. Check out a library I wrote that helps with concurrent unit testing in general: https://github.com/jhalterman/concurrentunit
Thanks for the link. Interesting stuff. I think the secret to unit testing is to keep the tests simple and test from the outside. The more you need assertions in worker threads, the more it seems that you’re testing/asserting from the inside.
There’s a balance to strike and sometimes the black-box testing approach is harder.
On the whole my solution to this problem would be to have the operation of the workers target some external tracker, and watch that to see what happens.