So you want to write BDD tests?

Before you can write BDD tests, you probably need to know what they are. Short answer – BDD tests try to black-box test something according to its behaviour. One of the most common languages is Gherkin. In principle, Gherkin is a natural language based technique for describing feature and scenarios in terms of how they appear to an observer. You’ll know it’s Gherkin if you see a lot of:

  • Feature – to describe a capability
  • Scenario – to describe an example of a feature
  • Given – setup
  • When – execution
  • Then – expectations

Other BDD specification languages also exist. RSpec style test frameworks tend to use:

  • Describe – to describe a capability in terms of its behaviour
  • It – to describe a scenario
  • Expect – to verify things during testing that scenario

RSpec is Ruby based. It has a JavaScript counterpart – Jasmine. Jasmine has proved brilliant, especially when used in conjunction with Karma for testing Angular and other bits of JavaScript.

This post is concerned with Java and JVM. There are a cavalcade of possible tools you can use for BDD testing on the JVM. Here’s a quick round-up of tools I’ve heard of:

  • Cucumber JVM – one of the leading products – based on Gherkin – uses plaintext feature files and wires up test code using reflection and regular expressions
  • JBehave – also based on Gherkin – similar feature set to Cucumber JVM
  • Spock – this relies on Groovy and feels like a blend of code and script – it has Gherkin-like when and then labels.
  • Oleaster – this uses RSpec/Jasmine like syntax and requires Java 8. It includes a port of the “expect” framework you’d find in Jasmine.
  • Spectrum – this is intended to be a Polyglot and Principle of Least Surprise framework. It uses RSpec/Jasmine syntax and also supports Gherkin syntax, all expressed in Java 8. No expectations framework is supplied as you can take your pick of JUnit, AssertJ, Hamcrest‘s ones as you prefer.
  • Ginkgo4j – a port of Ginkgo to Java, including direct support for Spring. Ginkgo seems to be very similar to RSpec in its syntax.
  • JGiven – this takes a completely different approach, encouraging you to create your own DSL for using in the tests.
  • ColaTests – a Gherkin based JUnit runner where you write the steps and tests directly in the test class in Java annotated with Gherkin syntax.
  • Specsy – intended for Scala, this can also be used a Java 8 lambda-based test framework, with examples and support for Groovy and Scala to boot. It’s very lightweight and hugely supports parallel testing and control of sharing state between tests.

The above is intended to be a rundown of tools that are out there – please comment with any omissions or errors and I will try to update the list.

Full disclosure – I’ve contributed to the Spectrum framework.

Posted in bdd, Java, tdd

How to make your Unit Tests harder

This is written about JUnit in Java, but much of this applies to other test frameworks. I’m going to tell you a bunch of ways to screw up your tests. You can probably guess how to write better ones – do the exact opposite.

Unit testing should be easy. It should be resistant to unimportant change and it should be sensitive to important change. Unit test frameworks like JUnit and Mockito make it easy to write test cases, assertions and mocks. So how can you make all the testing mistakes to make you question why you bothered writing tests in the first place?

Here’s how.

Ignore the entry and exit state of a test

If you totally avoid worrying about who has to put the objects or resources into the right state before a test, and what state those objects or resources will be in afterwards, then you can guarantee that your tests will only run successfully if nobody changes the order of anything or shares those same resources for future tests.

Example – one test unzips a file into a temp directory and another one uses that same file since it’s probably there already.

Example – we use something like Spring to build our context and then do some stateful things with the beans, assuming no other test minds about the state change.

Manage Temporary Files Ourselves

Why use things like JUnit’s TemporaryFolder rule when you can just write to local file system with your own ad-hoc techniques for writing temporary files. Even better, why not use the src folder and its descendents for keeping these files – what could possibly go wrong? Don’t worry we can stop these temp files from checking in with a suitable gitignore file, so really? What’s the harm? Apart from the fact that every developer’s machine will be telling them that their temp files are actually fixed resources that are part of every workspace always…

Mock a POJO

Yeah. Mock simple objects. The simpler they are, the more you can really mock them. Sure, your map may have a putter and a getter, but just mock the calls to the getter. For goodness sake, don’t just instantiate an object with the right values in it and use it.

Make a function into an object and mock it

Spring lets you turn everything into a bean. This means that discrete functions can be inside beans, with interfaces, and then mocked. While there may be reasonable points where the ecosystem is so chaotic that this is actually a good thing to lock down for a test, why not do it always? Then you can have unrealistic data examples flowing through complex chains of instantiation and dependency injection, rather than have a nice static function whose behaviour you can easily predict and whose presence will allow you to focus entirely on the input data to the test, rather than all the bits of micro implementation you have to mock to make the test run.

Make your tests a mirror of the implementaton

You may even need to paste bits of the implementation code into the test to be able to successfully predict every last value that flows through every microscopic node of your code…

Only ever whitebox test based on implementation

Ignore the basics of “what’s the behaviour to the outsider?” and make every test a deep dive with god-like knowledge of how the whole implementation works, so that any small change in that implementation requires test rewriting.

Never change your implementation to make it more testable

If that class does so much that it’s hard to test, then work your backside off to make the test that’s hard to do, rather than find an easier test boundary by moving a few responsibilities around.

In Conclusion

Make testing harder so you can keep yourself busier and less successful more slowly!

Posted in Uncategorized

How to Learn a Language

I received some marketing email relating to a Java programming course from The email contained this (probably made up) horror story.

so I’ve been learning java for over 3 years. and I’ve given up due to feelings over sadness because of the sheer size of the language as well as every time I learn, I always feel like the resource I’m using isn’t the best and I end up switching (this has been going on for a long time). That has led me to not gaining any actual knowledge in Java, all I know is basic syntax and a little about classes and methods. I am currently using Head First Java and John Purcell’s Cave of Programming courses to learn while also building my own projects, but even now I feel like I’m not using the best resource available and I just want to give up. I am so confused and I feel like I won’t ever get this language down to a solid level of understanding. I’m very lost.

The conclusion, unsurprisingly, was that the course on offer would help you get beyond this sort of issue. However, the answer is a lot simpler than “go on yet another course”. The biggest clue for this person’s problem is in this part of the quote:

I end up switching (this has been going on for a long time). That has led me to not gaining any actual knowledge in Java

If you are going to make any progress in any technology you need to do one thing. So this is my…

One sure-fire trick to get you to understand any programming language

Build something non-trivial using one language with one set of frameworks/libraries. Start with a pre-cooked example, turn that into your full blown application and finish the thing.

You’ll have to pick up the skills you need to do this along the way, and having set a technology choice in stone, you’ll build a body of knowledge, rather than thrash within a swirling void of possibilities.

It’s that easy.

It’s also that hard.

Posted in Uncategorized

A SureFire Classpath Fail!

Sadly, even the best plugins can work against you. We’ve been having a problem recently with Spring, Maven and SureFire. Specifically, it seemed that some of our unit tests ran absolutely fine with Spring and properties files when run via Eclipse, but got different behaviour when run via Maven. Even more specifically, the workaround seemed to be to duplicate values from .properties files that were in one jar file by putting them into a .properties file in the module whose tests were failing under Maven – usually by adding more properties to files in src/test/resources.

What this seemed to imply was that the classpath when running via eclipse, was set in such a way that the full module dependency structure was harnessed when looking for properties to evaluate in the @Value(“${someproperty}”) annotations, but was not being utilised at test time by Maven. Notably, these properties were used from the correct places at runtime in production.

Aaagh. What undocumented hell is this?

The solution is quite simple. You must not use the system class path in surefire if you want this to work. Here’s a snippet from our surefire configuration in the Maven pom.xml file:


As you can see, we’ve set this key to false. Now everything works. That’s it. You don’t want a default classloader: you want the one with all your dependencies in. That allows the Spring class path based dependencies to work correctly.

There was nothing about this on StackOverflow and it took months before we found the answer. Here it is now for posterity.

Posted in Uncategorized

When you want to disable a slow running test

A quick one. The code for this is here on GitHub.

When testing your software with automated unit tests, you want to be thorough to get code coverage and genuinely cover system behaviour, yet some of the things you want to test can take a fair bit of time to run through tests.

This poses a dilemma:

  • Make the test feedback process as quick as possible
  • Make the test process as thorough as possible

The best answer is to organise your code so that it can be thoroughly tested fast. If that’s not possible you may need a multi-tier automated test. The quick things can be tested straight away with a first-line of feedback, ideally within 5 minutes. Slower things should follow, with the whole end-to-end feedback in under 20 minutes if possible.

This leads to a dilemma – how and when do we disable long-running tests? And, given that we’re often using a combination of maven and within-IDE test runners, how can we disable a test in all contexts EXCEPT the one where it should be run. In my view moving a long-running test into the integration tests, just because you don’t like it, isn’t a great solution. Worse than that, you can jump through hoops trying to break things into categories, only to find that your IDE test runner is running them anyway.

A solution that worked for us recently was to disable certain JUnit tests based on a system property. If the property is not set, the test just doesn’t run. Simple as that. This can be done easily in JUnit by an assumption failure. More simple than that there’s the use of a JUnit rule to take the whole test class and make all of its tests abort if the system property isn’t set.

The code on GitHub is the JUnit rule I created. It is illustrated by this unit test:

 * Quick example of how to use this using a simpler test than that used to verify
 * that the mechanism works
 * The test within this class will only work if the system property "run.test" is set to "true"
 * To demo this running try setting the JVM option -Drun.test=true, which you can do in the maven
 * command line, or via your IDE's invocation of the junit runner
public class ExampleTest {
	public RequireSystemProperty property = new RequireSystemProperty("run.test", "true");
	public void systemPropertyDependentTest() {
		// do some sort of test we only want to do when a system property is set...

As you can see – the presence of this rule effectively stops that test running unless you specifically configure the JVM that’s running it with the system property that would activate it. We use this on our CI server in the integration test build where we enable everything and the build takes maybe 12-15 minutes. On the front-line CI build, we want the build time to be about 5 minutes, so we don’t run integration tests, or code coverage tools, or slow-running things.

Hope this is useful.

Tagged with: , ,
Posted in Uncategorized

Multi Module Maven Builds, Code Coverage and Sonar

I love SonarQube for its potential. The reality of it is somewhat different, you need a byzantine series of configurations to make it play nicely with Jenkins, Maven and Jacoco. It gets more difficult when you have a multi module build – i.e. a parent pom with modules set up in it.

I’m not going to claim any ownership for the decent how-to on this. The amazing David Valeri in has nailed this issue. Here is a quick summary of what you need to look out for:

  • You probably run your integration tests using the failsafe runner rather than surefire
  • If multiple projects have different bits of integration test, you’ll need to configure the jacoco plugin so that it shares a single jacoco.exec file across all modules
  • If one of your projects is an integration test project (our case) you just need to know where jacoco will write its output
  • In the parent pom.xml you will create a property for sonar to tell it where to find the integration test coverage report
  • Paths, especially once jenkins gets involved, can be complex

If you get it right, it works a treat.

See the linked post for details. Kudos to David!


Posted in Uncategorized

Handy Tips and Tricks For Finding Linux Disk Usage

I’ve no idea how I know that df -h is how you find disk free space… if you want to do more directory based stuff, then this blog is for you:

I’m glad they wrote it so I don’t have to.

Tagged with:
Posted in Uncategorized