Nov 4, 2011

Excuse #4 - Running the tests takes forever

So, after a long break (been focusing my time on reading about new technologies rather than writing - started a new job @ JoyTunes...) I'm back with the next excuse in the "Testing, Why Bother?" series.

"Running the tests takes too much time, 
so we never run them anyhow."


Well, this excuse is less common than the other ones on the list, and to tell the truth - if that's your case, you're in a pretty good spot - because, at least you have tests to run!

However, if you have tests and you're not running them, they obviously have little merit, so the effort of writing them goes to waste, which is definitely a shame.

So here are a few tips on how you can solve this tricky situation.



You don't have to run the tests yourself!


The word "automated" in "automated testing" is the key. It means the whole process must be automatic, not only the tests themselves.
If you don't have a Continuous Integration (CI) server that runs the tests yet, such as Jenkins, you should hurry up and setup one, because tests are just not the same without it.

If you haven't heard of continuous integration, I highly recommend you read the book:


In a nutshell, it means that for each new revision pushed into the source control management system (such as Subversion, git, ...), the CI server wakes up and starts a "build" for the project. Builds can automate very complicated things that are often done manually in the release process, and that's probably something I can write a whole set of blog posts about, but what is often done at the beginning of the CI build, is running all the automated tests in the repository, and sending e-mails to everyone involved in case the tests failed.

This way, tests that are running for too long are a much lesser pain, as you don't really have to run them manually and wait for them to finish in order to know if they passed or not.

"But I want to be able to run the tests manually as well"

That's a very good point. Tests are not only to be run automatically by a CI server after they are already in the repository. You often want to run them while you're working, from your IDE, in order to know that the latest line of code you changed didn't break any logic that was working previously.

Well, here are a few tip regarding how to speed up your automated tests:

Make sure you don't set up any unnecessary environment

Unit tests should be testing a single component separated from its context. So - if your tests have to set up 5 DB connections, open some sockets, read a bunch of XMLs and instantiate 100 objects just to run a simple functionality test on the tested component - you're doing it wrong!

If you design your code in a way that you don't have to use real environment and you can easily substitute external interfaces with mock objects (e.g. avoid evil DbConnectionManager singletons), it should be easy to save a lot of setup time for your tests.

If however, you feel you MUST use real environment for a specific test - make sure you only set it up once, in the global setup of your Test Suite (e.g. @BeforeClass in JUnit), and not before every test function...

Mock irrelevant operations that take to much time


This is pretty much similar to the previous tip, but this time I'm talking about mocking calls to some kind of 3rd party API or external interface.

Simply replace calls to the external interfaces with a mock that returns a value, without having to wait for the small 3rd party calls to finish before moving on with your test.

One interesting way of achieving this, is by taking an Integration Test that's already testing your component from a higher level, recording all the calls to external interfaces and APIs, and then replaying all the results in a unit-test of your component, that takes much less time.
There are some frameworks that may help you with this, like CaptureMock for python.

Separate the Integration Test Suite from the Unit Test Suite
Unit Tests in line for some CPU time...

This may come as trivial for a lot of you, but I've seen many violations of this principal.
As said, the Unit Test Suite should be no more than a few seconds to run, in order for us to be able to quickly run them and make sure we didn't break anything in our code.

If you have integration tests that take a lot of time - you don't need to run each time. Have a separate TestSuite for them, and make it run only in the CI server - and probably not after each build, but only before release candidates.





In conclusion
Having slow tests is much better than having no tests at all, but is still a real pain - but a very treatable one. If you have any more ideas of how to speed up your tests, don't hesitate and share it in the comments!

Stay tuned for more blog posts in the series, and make sure you read all the previous ones as well. I will try keeping 'em coming. If you haven't already, you should Follow me on twitter.

1 comment:

  1. Yes yes all UTs should run in no-time even on my old Pentium 4 machine!

    Also, a nice side effect is that it encourages designing the code to be more testable; e.g. a cleaner, faster, single-responsibility functions rather than an ugly mix of logic and a bunch of slow, bloated external dependencies that would have to be mocked later.

    ReplyDelete