Recently we embarked on adding automated UI tests to various existing projects and all our future projects. It’s pretty exciting as we’ve been trying for years to get something set up but we’ve always been held back, mostly because until now the tools really haven’t been that mature. We want to avoid brittle tests that break when UI elements are moved around or renamed in order to avoid too much maintenance which has always been a problem with UI tests. Some of this is unavoidable but there’s definitely ways to reduce this.
Pretty much all our projects are ASP.NET MVC based and we’ve decided to go with Seleno which is a great library that offers Page Objects and Page Components and reading/writing web page data using strongly typed view models. This means that a great deal of common UI changes will be easily updated in tests. We use NUnit and a test runner to run the tests. The tests run with Selenium and the default WebDriver opens FireFox to allow us to see our tests in action. This is great when you’re writing your tests but it can be very slow to run a lot of tests. Luckily we can easily swap out WebDrivers and use GhostDriver instead which uses PhantomJS (a headless browser) to run much quicker. Michael Whelan has a good post explaining how to use this combination in your UI tests.
So we got the technical part sorted and were really happy with the results but the more important part was deciding how much we were going to test, what environments/data we would use and what we were going to do with the results (should it stop builds deploying or do a quick rollback after the fact?). We only ever planned to supplement our automated unit tests to help catch things that slipped past these tests and provide a bit more confidence with our changes. It gets tricky when you want to run UI tests in production given you could be creating actual records, calling real APIs, sending out emails, etc. You need a way to either prevent some of these actions, roll them back or possibly have another environment that is a copy of all the data perhaps? You also need to handle authentication, roles and test users for some things that aren’t publicly accessible. We’re still looking into these possibilities.
What we did decide is to limit ourselves to the “happy” scenarios. Pretty much the best case acceptance criteria where you perform a task without errors and all the various conditions you could set a more complicated form up with. I feel that most of the granular logic testing should still be in the unit tests and the UI tests can just help to supplement that. The other thing to consider is that some of the UI tests will need the data to be in a particular state. Perhaps certain records need to exist before some tests are able to run. This means that a lot of the tests will need a part to set up and tear down which we felt should be done in the UI also. This can make the tests take longer to run but it ensures any particular test can run in isolation and is not dependent on the previous one or the order of a batch of tests (not good).
Because of the time taken to run UI tests we also are trying to figure out the best place to run them. Overnight tests are good because they ensure we know when systems aren’t behaving as they should but really it’s too late usually. You need to know before deploying to production or pretty soon afterwards! Either way the running of the tests needs to be automated too.
I’ll no doubt update this post as we travel down the UI testing path but I thought I’d link to some good articles at least…