melp.nl

I guess the guys at Drupal.org didn't really understand the concept of unit testing. In comparison to the Zend Framework 1.13 and Symfony 1.4 test suites, of which some tests did not pass mainly due to some configuration issues (98.6% and 99.9% respectively), Drupal had some, let's say, surprising results. Though it reported 100% of the tests to pass, 2 fatal errors occured. That might indicate expected fatal errors, but I doubt that. But the real surprising bit was that the tests took nearly 2 hours (!!) to finish.

In comparison

The timings of running the tests, all run at PHP 5.3.5 with the exact same configuration. Of course, they're indication, and I am not really proving anything statistically here.

Symfony 1.4

#!shell
$ time ./symfony symfony:test > /dev/null

real    0m56.659s
user    0m40.624s
sys     0m8.956s

This particular distribution of Symfony has 204,588 NCLOC ¹, only counting .php suffixes.

Zend Framework 1.13

#!shell
$ time ./runtests.sh > /dev/null
++ phpunit --verbose AllTests

real    4m20.609s
user    1m41.473s
sys     0m4.420s

Zend Framework, in this distribution, has 879,610 NCLOC, again, only counting .php suffixes.

Drupal

#!shell
$ time php ./scripts/run-tests.sh --url http://drupal7.dev/ --all > /dev/null

real    115m33.344s
user    53m45.673s
sys     3m1.591s

Drupal uses different suffixes for PHP files, so counting the NCLOC with the suffixes engine, profile, install, test, module, php and inc, the NCLOC come to a total of 164,395.

In perspective

Let's assume, for argument's sake, that the percentage of code actually covered by these tests is more or less proportionate in all frameworks. As a means of getting indication of the complexity of the test, we can calculate the time spent on the tests in proportion to the lines of code. That will give us a fair indication of how complex the tests are in respect to each other, without inspecting the code further.

Whoa!! An average of 24 lines of code tested per second. That is pretty steep.

Symfony and Zend Framework are more or less equivalent in test performance. Knowing for a fact that their tests are in good shape in terms of separation of concern and therefore actually being unit tests in most of the occasions, Drupal really doesn't come off very well, to say the least.

So, what is wrong with this picture?

Testing procedural code is very hard in PHP. Since there is, with standard PHP, no way to stub or mock functions that influence the code that is under test, it is pretty much impossible to test code without having to bootstrap the entire framework. And that is exactly what the Drupal tests do. In fact, they do not run the tests on isolated portions of code (units), but for each test the entire bootstrap process is executed, causing enormous performance overhead.

However, the performance overhead isn't the main concern. The main issue is that none of the tests can actually be called Unit tests. By Wikipedia's definition, a unit is the smallest testable part of an application². It doesn't take a lot of arguing to agree that, if bootstrapping the entire framework is necessary for testing, apparently the entire framework is the smallest testable part of the application, and therefore the unit under test.

So, the test author for Drupal is confronted with a serious issue. The Drupal code base imposes a design on the test author that is clearly not that well testable in isolation. And non-testable code typically is code that isn't that well designed. Or, at least, it doesn't apply Separation of Concerns and Design by Contract very well.

What to do?

There are two ways to go from here.

1. Either, Drupal's way of testing code is maintained, and the design of the code base isn't altered. Only once in a while, someone that has some time or CPU processing power left, the entire code base is tested against the load of functional tests, and the maintenance of the core codebase will remain cumbersome, as it is already.
2. Or, the current tests are used to devise new ones, aiding the Drupal maintainers to refactor the entire code base such, that it employs good design patterns and a well thought-out structure, with testable units of code.

Of course, number two has the advantage. Long term maintenance of the codebase will be much easier, and refactoring along the way will benefit any developer ever having to maintain any part of the Drupal core, let alone the thousands of modules out there.

How to get there

1. Define the current API, that is, all functions, hooks, variables, etc that are in the core
2. Devise tests for all parts of the API (firing hooks, theming data, installing / uninstalling modules, loading nodes, altering nodes, etc, etc.)
3. Do this while getting a good indication of code coverage (without fooling yourself).
4. As soon as you have 100% (yes, 100%) code coverage of the entire API, start refactoring, without touching the tests, while defining a new API.
5. Let the old API forward to the new API.
6. After refactoring, devise tests for the refactored API, without referring to the old API
7. Deprecate the entire old API in favour of the new one.
8. Roll out a new version of the software.

Of course, this can be done iteratively for all of the core modules.

But, if the current API isn't testable, how can you start with step 1?

Because writing unit tests for the Drupal code is close to impossible, but not actually impossible. Fortunately, there are some very useful PHP extensions out there:

• runkit, currently maintained by some fellow named Dmitry Zenovich.
• php-test-helpers, a similar extension.

With one of these extensions installed and enabled, you can rename and replace functions that live in the global scope, effectively making it possible to stub and mock anything that is part of Drupal's core. Anything, but the static variables. There is, to my knowledge, currently no way to replace or reset static function variables that aren't reset by the function itself.

That is the only remaining hurdle: first refactoring all static variables to a registry that is mutable (e.g. $GLOBALS or any other registry container), and we're good to go.

Who's going to do that?

That's the only question that remains unanswered at this point. You? Me? I'm not sure, but I think a code overhaul of Drupal will eventually benefit the entire PHP community, and ultimately anyone directly or indirectly affected by Drupal.