On Unit Tests

Unit tests are the second of the three pillars of automated testing. A lot has been written about them so I’m not going to repeat it, just will point some highlights. Unit tests are integral part of the development and are written by developers.

The behefits

• Having done a second look at the production code you’ll find some bug you missed the first time that you wrote it. Think about how you re-read your school essays, you did not present your first draft, right?

• The outcome is better code structure at method level. If your code is to hard to unit test, then something is wrong and it needs refactoring. Maybe the method is too big, does too much stuff. Maybe you have wrong abstractions, or cannot easily substitute a behavior. Listen you your tests.

• You immediately know, when you introduce a change, does the new code breaks the existing behavior. You run your unit test on every commit. You can move faster and worry less about regressions.

• As a bonus, you get a set of unit tests. Note that I put this on the last place, because the goal of writing the unit tests in not the unit tests themselves. They just happen to be useful byproduct.

An Example

The developers writing the unit tests should be in contact with the person writing the high level tests so that they do not repeat the same tests, but at a higher cost. For example, let’s say you have a web page that accepts credit card to make a payment. Some of the possible test cases are:

• Invalid credit card number (has less or more digits, contains anything other than digits, Mod 10 check fails, unknown issuer)

• Invalid expiry month/year (set in the past, contains characters other than digits, too much/ less digits)

• Invalid CVV (not exactly three digits, contains characters other than digits)

Those checks should be performed on your backend (for security reasons, since an attacker can completely bypass any frontend checks) as well as on your frontend (for faster user feedback). Now assume that you need to automate all those test cases only with Selenium. This is the top of the testing pyramid, more brittle and very slow tests.

It may take more than 5 minutes to complete all the test through the UI. Each test case may create brand new user, interact with web service, database, login to your application, navigate to purchase a service, then navigate to the payment section and then proceed to enter test case data and validate the result. Why is all of this extra effort needed? Your frontend will call the backend. The backend will call three verification methods to do all the heavy lifting. Then why don’t you tests those three methods directly. It is where the decision is being made. No need to go through all that Selenium trouble to test something that more easily and reliable tested with unit test. It’s like killing a fly with bazooka.

The correct course of actions is to cover all the test cases with unit tests, then have no more than two acceptance high level automated tests. The one will test the happy path where all credit card details are correct, and the other will test what happens if any of the credit card details is not correct.

Organizational Anti Patterns

Unfortunately, most often I see two common patterns in companies that invert the testing pyramid (a.k.a the ice-cream cone). They have more automated blackbox tests that unit tests.

The first pattern is that developers do not write unit tests. This is because they think they code is bug free, think that unit tests are a waste of time, the code is such mess that it need refactoring before being able to be put under test harness, or developers just don’t know how to write unit tests. Then all the automated testing is shifted towards the QA team. The solution is for developers to start appreciating the value that the unit tests bring as well as to start writing them. This is pretty big topic by itself.

The second pattern is that Dev and QA are separate departments, they rarely talk to each other, except to throw something over the wall. As a result they either write too much overlapping automated tests for given functionality — unit and acceptance. Or do not cover it with sufficient tests, thinking the other team is covering it. The solution is this case is the three amigos, co-located team, sitting by each other, the whole agile shebang.

Guidelines

Except for the most straightforward cases (e.g. your acceptance tests, blackbox tests) it may be impossible to test for boundary conditions, exceptions, DB corruption with your high level tests that treat your system as blackbox. However this is trivial to do with unit tests.

Unit tests should be written whenever you change the code base. It does not matter if it’s for the development of a regular new feature or for a bugfix. Especially if you fix a bug, write unit test to verify the desired behavior and now you know that this exact bug will not appear again anymore. Unit tests and production code go together.

The tests are easy and cheap to write, and will pinpoint to the exact location of the problem (where as the blackbox tests will usualy guestimate where the problem is) when they fail.

Unit tests should also should not fail randomly. You should be able to run them in any order, only one test or all of them. Every unit test should setup and destroy everything that it needs.

Write unit tests for bugs you encounter in production. Tests should be closer to the code. Write automated test on the lowest level possible. Where is the decision beig taken? Test at that point.

Inexperience developers usually separate the development tasks in two and give different estimates, one for writing the code and another for writing the tests. With time and experience under their belt, there should be one estimate only as unit tests would become integral part of any development activity. And I’m not even talking about TDD here. In my experience one achieves mastery (knowing the test runner tool, the legacy code writing test for, how to refactor first in order to be able to write unit test at all) writing unit tests after he has written at least 2,000 tests cases.

Code Coverage

If you’ve worked with non-technical manager, he might have set code coverage threshold e.g.:

“The build should fail if the code coverage drops bellow 90%”.

This is bullshit and you should oppose this. When you measure a team against a metric, the team will do anything to meet this metric (for example to get the yearly bonuses). Developers are creative and they will find a way to game the system. They will most likely write tests with little or no assertions. Also remember the Pareto 20/80 rule — do 20% of the effort to get 80% of the results. Sometimes even with the best intentions it’s not realistic to achieve code coverage of more than 80-85%, it becomes exponentially difficult to hit the last 15-20%.

A final note on the code coverage obsession. I’ve seen a bug, the cost of which was $50,000 in direct losses in code that was 100% covered by tests. How did this happened? Simple: the tests we’re not covering some unlikely (at least to the developer) scenarios. The method was expecting the input array to contain two keys with specific values and did not have any extra checks what will happen if the input array did not contain these keys. Or for that matter if only one of the key was present but the other missing. It just was assumed that they will always be there. So there were no unit tests written for these missing extra checks. When dealing with code that handles money, you really need to program defensively. Are you making assumptions that the input data will always be in the same format? What will happen if an exception is thrown? What will happen if write operation in the database fails?

Speed

Unit tests should be fast and deterministic. This means that you should control all the external factors. This is achieved by mocks and stubs, but we’re not going into much more details about those for now. Your unit tests should not talk to the database or over the network. As a practical measure, you can try and run them on a machine with no network connectivity, just ti make sure that they still pass, and someone form your team forgot for example to disable a constructor that has side effect of opening connection to the database.

The tests should pass in milliseconds. As you can see from the screenshot above, more than 2,000 tests pass for 10 seconds.