Two weeks ago I posted that the OU Moodle development team were going to spend a fortnight writing Behat scripts.
I promised to write up how we got on, so this post fulfills that promise, I hope.
Our aims for the fortnight were
- to generate automated Behat tests for a significant proportion of OU plugins that we can use in future;
- to create the know-how within the team of how to create and run those tests; and
- to establish writing tests like this as part of what we do as part of any development
Of the 15 developers, 4 had written Behat scripts before and acted as mentors for the team. By the end of the fortnight another 8 developers had got Behat setup and 7 had successfully written scripts.
The OU has over 200 plug-ins in our Moodle code base. Before the sprint only a couple had any Behat coverage. Now about 16% do.
We wrote over 3,000 lines of Behat-related code in the past fortnight, about 10% of which are for custom steps.
Our handover checklist now says that all new features should come with Behat scripts, unless there’s a really good reason not to.
People generally found the week useful both in terms of learning the same thing together so there were people able to help each other out and that Behat itself is beneficial to our work. Some found bugs and fixed them as they went along, so there has been a (small) positive impact on system quality. All agreed that it is enjoyable to see the scripts working and that this sort of automation is what computers are for. Hopefully we’ll use this approach when we need to learn something new again.
Obviously, that’s not the end of the story. We had a retrospective and came up with some tips & tricks to share with each other and a few questions. And we’ll carry on doing Behat and getting better coverage for our more of our plug-ins.
Tips and tricks:
- The Site admin tree takes too long to expand sometimes so times out before the next script step which then fails. Either restart the PC, or if that fails, write an “I expand” step then “I navigate to”.
- Sometimes when filling in forms, care is required over the enabling of other fields or buttons before moving to next step. You may need to alter the order of the steps, or inject extra steps like check the button exists.
- Cron only runs 1 per minute so if your script runs cron repeatedly with the “I trigger cron” step, you may need a step to reset cron so that it will work. This bug has been reported to HQ. In any case if you have a cron task which doesn’t fire on every cron run, will need to set this up specifically in your script so that it your conditions are met.
- It is possible to use PHPUnit generators as a way of doing some background setup, e.g. if you want to set up a number of activities with specific properties. It is quicker to setup the course in this way than writing Behat steps to setup through the UI.
- Lots of our plug-ins are interdependent, with if x is installed PHP so we can still share them individually. If you want to test integration between plugins you can write a custom step to check if the plug-in installed and throw a SkipException if not installed.
- Use names visible in the UI to identify screen elements rather than css ids. This is a) more readable by humans, b) if your feature is built this way it is likely to be more accessible to screen-readers.
- If you have to use a complex xpath to identify a screen element, add a comment to explain to human reader which bit of the page you’re trying to test.
- If you including test files it is ok to put them in the fixtures folder but because they’re in the webroot they must only include publicly available materials.
- If you are repeatedly writing the same collection of steps, you should convert these to a custom step.
- Don’t forget to update the steps list occasionally to see what’s available. Take care when relying on a custom step stored another plugin – can you be sure the plugin will be there (e.g. for contributed plugins)? You may need to copy the custom step to your plug-in.
- When writing custom steps, consider if they are Moodle generic and if so they should be given to core e.g. date picker.
- Write scripts where you set up a feature as one user and then switch log in as an account with lower permissions to test actual output.
Area of concern:
The latest FF upgrade to 32 causes behat to fail. In summary, don’t upgrade or use Chrome for a while. This has also been reported today in the GDF forum.
There was some discussion over the difference between test scripts and Behat scripts – they should not be identical. Test scripts should test more/different things to Behat or test the bits that cannot be Behat scripted easily.
Perhaps you have a view on some of these, or some better tips and tricks? If so, please share them in the comments.