The Three Pillars of Automated Testing

In order to have fast and quality release you need to implement and balance between the three automation pillars, not leaning heavily towards one or the other. They are interconnected and act as a different layers to defend agains various problems.

More posts will follow to describe in details the three pillars, but the highlights are listed bellow.

Static code analysis tools evaluate all the code according to some rules - syntactic, lexical, potential bugs. This step does not execute any code. For example, a PHP static code analysis tool chain might include: linter, hhvm in analysis mode, phpmd, php codesniffer, custom php code checks. A Java static code analysis toolchain might include: compiler (no way around this), findbugs, pmd, aspirator, checkstyle. These are the cheapest tests as they are configured once and require little or no maintenance.

Unit tests are written by developers, execute some of the code (more on code coverage fallacies in another blog posts). They should be fast, reliable and deterministic, hence they should not talk to the database or the network. All the external dependencies should be under developer’s control using stubs/mocks. All the execution happens only in memory.

Blackbox tests can be API, UI, Acceptance, Integration, System. Everyone understands what those names mean differently. What is common is that in order for them to pass, the application needs to be in a running state, the databases are hooked up, the filesystem and the networks are operational. They threat the application as black box, not knowing the external state. They use only the external syste interfaces - http, web services, database. These are the slowest and the most unreliable tests, and you should take care not to write more of those than needed. Comparing to the unit tests, you don’t have control over the external dependencies. Another common trait related to those tests is that they are usually written by the same group of people — either developers or automation test QA engineers. The gist of it is that there is a lot of code reuse between say API and UI tests, for setting them up, tearing down, use proper domain model, so it makes sense to share the common parts.

If you are familiar with the testing pyramid, the addition would be the static code analysis at the bottom. Also I classify the tests above the unit tests (API, UI) to be of the same type because they require the application to be in full running condition. Whereas the unit tests still execute the code only in memory, however are limited, because they use test doubles.

The pillars are ordered in the same order as they should be run. Upon commit, you automatically run the fastest check first (your first line of defense), then gradually proceed to the slowest. If any of the checks fail, then you fail the whole build, no need to continue, as it’s most likely that the next check will fail too. The checks also have another property, the fast and small they are, they pinpoint the exact location of the problem more easily

Depending on your CI/CD setup, you may not want to deploy to test/integration/staging or automatically install if the build fails. Also there is no need for manual tests to begin until the build is broken. No need to waste precious human time to see that the PHP developer has committed a line that does not end in semicolon. The linter from the first pillar (static code analysis) will automatically catch this error on commit. Of course, when using a compiled language, a syntax error like this would fail to compile the application, but in interpreted languages you don’t have this luxury.

When the build breaks it’s all hands on desk until the build is green again. Usually the response is for the developer who broke the build to fix it (this is common courtesy to your colleagues). So wait for the feedback before you leave work. Should be less than 5 minutes.

All the check should run automatically after every commit and everything should finish within 5 minutes — to give the developer fast feedback about the quality of the new code. For this is the average attention span of the developer. More than 5 minutes, and the developer moves to other things - Social networks, surfing Internet, porn, bathroom (maybe not in this particular order). The 5 minute rule has the added benefit to point at a problem while the code is still fresh in developers mind. The code push should be small (a few lines) so it’s easy to be fixed in case of a problem, easy to pinpoint where the bug is.

Do yourself a favor and push code at least twice a day!