Say what now?

Here’s part of the problem: PHP developers usually don’t really know the difference between what’s static and what’s not. Simply put: there is only one variable. The request. Anything else should be as static as possible. If I would introduce a configuration file with loads of variables, a typical C programmer would argue: “Those are build parameters. They don’t change unless your environment changes, so they are static and should be handled by a preprocessor and macros. You don’t need continuous evaluation of static data”. A typical Java programmer would probably put stuff like that in a static final constant property somewhere, knowing (or assuming) that the compiler will optimize and inline usage of these settings ¹.

PHP doesn’t do either. We have no preprocessor, and we have no compiler optimizations on that level. Mainly, because PHP originally was a platform that was supposed to do some simple processing and spew out some dynamic content, as fast as possible. But the PHP community is growing up. We want real OO programming, we use design patterns, we think about dependencies, maintainability, scalability. Stuff that grown-ups do.

But then, what the hell is wrong with this picture?!

I was having performance trouble with one of my Symfony projects at work. So I decided to do some performance testing. Please note that I absolutely love Symfony. There is no framework out there that will actually help you design your application better, and it brought me and many of my colleagues more joy in life. Really. Please understand that this isn’t really about Symfony, because I am pretty sure most of it goes for any other current-generation framework.

I set up some benchmarking comparisons. This isn’t scientific evidence, to be frank. This is experimental science. I was having a hunch, and this indicates that my hunch was correct. So here goes.

The example controller in the Symfony standard edition

Whenever you start a Symfony project, you would typically bootstrap from the standard edition, remove the demo code and add your own bundle. The example controller does something pretty simple. It responds to a URL /hello/..., where the dots may contain anything but a forward slash. Here’s the code:

<?php
 
class DefaultController extends Controller
{
    /**
     * @Route("/hello/{name}")
     * @Template()
     */
    public function indexAction($name)
    {
        return array('name' => $name);
    }
}

The idea is simple. /hello/{name} routes to this controller, the controller returns an array of named variables, and the @Template annotation makes sure the template corresponding to the controller is rendered. This is usually a twig template, and in this case, the template looks like this:

Hello {{ name }}!

Not just to be a pain in the ass…

How would you have done this 15 years ago? Probably something like this. Create a PHP file in hello/index.php containing:

Hello <?= htmlspecialchars($_GET['name']) ?>!

and a .htaccess in the web root containing:

RewriteEngine On

RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule hello/([^/]+) hello/index.php?name=$1 [L]

Right? It doesn’t really do anything more than that. And if I’m absolutely honest about it, useless as the example may be, it shouldn’t be done any other way.

So, just for the sake of curiosity, lets see what a difference in performance this is. I set up two identically configured hosts on my local machine in Apache, but with different webroots, just to be as close as possible to a regular development environment I normally use. The first web root is the regular web/ directory from Symfony (and I renamed it to web-1). The second is another directory containing the setup as described above.

With ‘ab’ (Apache Benchmark), you can get a pretty good idea of performance increases if you’re optimizing your website, one way or another. You can basically pass in an URL and have multiple concurrent users hammer that URL for a specified number of times. So that’s what I did. I usually start with 1000 requests, with 20 concurrent users. This usually gives a good comparable impression of speed.

First off, the dev, debugging version of the symfony app. Of course, you wouldn’t normally test performance in a development or debugging environment, but it’s here for the sake of comparison. Here’s the most relevant² results of ‘ab’.

Concurrency Level:      20                    
Time taken for tests:   25.058 seconds        
Complete requests:      1000                  
Requests per second:    39.91 [#/sec] (mean)

Second, the prod, non-debugging version of the symfony app.

Concurrency Level:      20
Time taken for tests:   5.302 seconds
Complete requests:      1000
Requests per second:    188.61 [#/sec] (mean)

That seems pretty decent. Just under 200 requests per second were managed. Let’s increase concurrency to see what happens:

Concurrency Level:      250
Time taken for tests:   7.153 seconds
Complete requests:      1000
Requests per second:    139.80 [#/sec] (mean)

A slight drop in performance was to be expected, but it is at least pretty steady and stable.

Now let see how the 1998 version of the same “application” would perform.

Concurrency Level:      20
Time taken for tests:   0.178 seconds
Complete requests:      1000
Requests per second:    5614.76 [#/sec] (mean)

Wait. What? Over 5500 requests handled each second? That means a performance increase of 30. Not 30%, a FACTOR of 30. Nearly 30 times faster. The last time I saw numbers like these were when I ran benchmarks on a Varnish cache….

This would mean that if I were to run the same test with 30 times as much requests, the 1998 version should manage equally:

Concurrency Level:      20
Time taken for tests:   4.482 seconds
Complete requests:      30000
Requests per second:    6693.93 [#/sec] (mean)

At these numbers, the measurements apparently show even better results! And if we would increase concurrency, it would at least have the same relative performance as the other one:

Concurrency Level:      250
Time taken for tests:   6.366 seconds
Complete requests:      30000
Requests per second:    4712.83 [#/sec] (mean)

Let’s look at this for a few seconds. Just to be sure we understand what this means. This means the stripped version of the functionality runs roughly a factor 30 faster than the one implemented using the framework, both with 20 and 250 concurrent users.

But if these numbers are possible, why then, why do we think all the framework’s benefits outweigh these mind-boggling disadvantages?

The counter-arguments

You have no request listeners and therefore no security, no logging, etc, etc…

I know. And you are right. I don’t. But none of those are needed in this example. And that’s exactly what’s wrong with our mindset. We load tons of utilities and tons of logic to requests of which in 80% of the cases, only 20% actually need, and we keep on saying that the ease of use and the well-thought design are good arguments to have such a horrendous performance impact.

Be honest. If you would buy a new server, would you rather have it installed bare bone without any software on it so you can carefully pick your stuff, or would you have the entire Ubuntu Software center installed, so you can use it whenever you can?

Sure, there is a middle road, but lets find a middle road that is the best, not the one that is the most convenient just for us developers.

But you can cache it!

There is only one right answer to such a suggestion. Caching sucks³. Yes, I know, caching is cool, because it can make stuff insanely fast (try Varnish one time. Unsavory goo will drip from your lips, I promise you). But that’s not the point, really. Caching complicates stuff. Complications are just like expensive toys. 1) You don’t really need them, 2) they make you worry about breaking things that shouldn’t really matter and 3) you already have enough of them.

So we should install php3 again and get rid of the frameworks?

Balls, no! This is something the people behind the frameworks should be aware of. But at the very least, the people using them should be aware of it. Don’t tell your project manager (or worse, your client) that the performance is okay, and that you just need a few days R&D to implement caching, just tell them the framework is relatively slow but that you think the extra cost in hardware and complexity of caching mechanisms outweigh the design and maintainability benefits. And be very careful to really mean it and know what you’re talking about.

Code generation is paramount to performance!

I’m not the type of guy to point out problems and not think of solutions beforehand. I have been thinking about this for quite some time, and thought of building something myself. But the fact of the matter is: I don’t have the time nor the persuasive character to actually get it done.

What would be needed is a build system that “folds out” into smaller pieces of somewhat repetitive PHP. We write the development code just like we did before, but when deploying to a non-development environment, all the bits and pieces should be as static as possible. Anything that is dynamic but could, in theory, be static, must be factored out. This means, in case of Symfony, that

• It is a bad practice (or even unsupported) to have dynamics inside your Bundle classes. Use the service container for this. This way, the bundles needn’t be loaded nor initialized on each request;
• The service container and accompanying files should be loaded lazily. This can be achieved using require_once calls whenever necessary. The compiled code should do the same.
• Service container and compiled DI code (we have this already, but could even be more effective) compiles into separate files in stead of a big class that gets loaded everytime.
• Configuration must be compiled statically
• Templating must be optimized such that it generates the most efficient PHP code possible. Using magic like the detection of ‘getters’ and such should be removed. May be even consider it a bad practice to use objects in templates at all.
• Routing must be done statically. Dynamic routing is runtime logic and can be done inside the controllers.
• Request listener code is compiled into the controller files, so templating listeners, error handling and security are as efficient as possible.
• Resolution of bundle aliases is done at compile time
• Handling sub requests is done by simple includes

And on goes the list.

And now, for some action

I have built a little “proof of concept” that shows you the idea. I wrote the resulting code by hand, but with most of the logic already in place, a compiler with some optimization passes in the resulting AST could do anything that i did by hand automatically.

Here’s the performance impact difference:

The ‘hello {name}‘ example:

An example using doctrine

In the second example, (which is configured at route db/{name}), the entire service container is loaded, by including the appProdProjectContainer from the cache. In the last example, routed through db-optim/{name}, the doctrine service and it’s dependencies are loaded by including files with their service definition. This adds another ~10% increase in performance, which is still worth considering, imho.

Note that the doctrine services only use per-request caching (ArrayCache), in this example.

What’s the catch?

The catch is, most of the overhead from Symfony’s kernel is from the request listeners. There are a whopping 15 (fifteen) services that listen to every request in Symfony. All of these listeners can be optimized into pieces of code which are included in the controller files, where the “listening logic” (i.e., checking if the listener should do anything) can be done by a static line of code. In the examples above, you can see this happening for the templating listener, which is only checking if the return value of the controller is an array. The fact that it’s in place right there, can be controlled by the @Template annotation. The same would fly for @Secure and @Route annotations. Some listeners contain global static logic (such as the locale listener) or are generic, but nonetheless static (such as exception listeners.

With the Kernel structure as it is now, there is no way to have these listeners come into play only when needed. They are initialized every request, and therefore all dependencies that these listeners may have, are initialized. There is only one way to get around this. Compile all initialization into the controller files, but guarded within the conditions these listeners should provide. Just like the is_array() example for the logic of the Templating listener, this can be done for any other listener.

The sharp reader must have noticed that I disabled all listener logic in my example. This is the reason.

To conclude

I have posted before about a paradigm shift towards declarative programming. I am also arguing that this shift should include a better sense of balance between performance and maintainability. PHP’s road to success was paved by performance. If we lose that, there is not much reason left not to choose other platforms.

A final disclaimer: I am not bashing. I hope that this will lead to more good in the PHP community. At the very least, I dropped my thoughts and can leave it simmering for now. If you read the post entirely, thanks for your patience.

The UNIX principle applied?

Something that is currently overlooked in a large part of the PHP community is that not all design patterns are necessarily implemented in PHP code. Your webserver can be considered the front controller of your application too. Routes can be defined using your filesystem and webserver configuration. Not everything has to be Object Oriented to be well organised and well structured. To gain perspective, here are some ideas to think outside the box, and not have OO patterns drive you beyond reason.

Some perfectly good solutions to, for example, caching and distributed processing, are commonly overlooked because of the simple presumption that everything should be done inside PHP or should at least be controlled by PHP. In my opinion, this might be a symptom of the Not Invented Here syndrome, and is thus by definition something to be questioned. It makes modular reimplementation in a different platform or language virtually impossible, which troubles long term development of web applications. You can not and must not feel confident that anything you want can or should be done in PHP. The language is irrelevant, the platform is irrelevant, the choice for PHP should be one based on those aspects that make PHP a better choice than another platform or language. If the outcome is PHP, use it. If it is not, it should be of least possible impact on the design choices you make for the rest of your application.

Why is the question important?

We need to be more and more prepared for distributed and parallel processing, which means we need to slice applications down to small and simple bits to be processed on any CPU or filesystem. That means that, while employing nice OO implementations of design patterns inside frameworks like Zend Framework, Symfony and the likes¹, we are in fact building monolithical applications, that can be called and routed only through a single front controller file (usually index.php), which doesn’t make the design of the application internals monolithic per se, but does make the application an sich monolithic.

Continuing with this type of development is in huge conflict with various philosophies, such as the UNIX principle. We build one application that does a lot of things very well². This UNIX principle is not something to be overlooked lightly, because it is historically the single greatest success factor of free software and UNIX based development such as Linux, BSD, GNU, etc…, not to mention PHP. Note that free software means “Free to choose what software you want to use” also; not only free in terms of money or legalities, but also free in terms of dependency and freedom of choice.

Moreover, note that the UNIX philosophy holds strong resemblance to the Principle of Least Knowledge and Separation of Concern, which, in my opinion, are by far the two most vital and important fundementals of software design, when keeping maintainability, testability and long term development in mind. Parts of the application that need updating because either the code is irrelevant or some better component has arisen, should be easily cut out and replaced by another. Think of, for example, replacing Apache with NginX, or even replacing the current PHP payment module with a Java one.

Any component in an application should be responsible for one thing in the application, and one thing only. Vice versa, the component that does the job, should be the component that is the very best at that job. You wouldn’t write your own SQL implementation, if you know there’s perfectly good relational databases out there. So why build your own page caching mechanism if the webserver can cache pages for you? Why build your own routing and dispatching system if the webserver has perfectly good routing and dispatching systems in place?

What with those quote MVC unquote frameworks?

PHP does one thing very well: it is a very fast interpreter of a reasonably simple programming language, and is best suited for processing variables (typically CGI requests) and responding with text output (typically HTML formatted content), while communicating with any backend such as MySQL and filesystems.

MVC frameworks are deviating from this premiss. They hold the PHP application responsible for far more than just handling an incoming request and serving the response. This could be a problem. It complicates the application design very much that all those responsibilities come onto the shoulders of a single application. The danger lies in the fact that inside this application, there might be more trouble identifying the responsibilities and concerns of the different components. If you would separate those responsibilities from the main application, you would find that there is a lot that we’re dealing with inside our application, which is very irrelevant to the application’s own concern.

The best example of this is caching. Either caching stuff that is coming from the database (which a properly configured MySQL already does for you), or caching entire pages (which can be done by the web server): this does not necessarily belong in your PHP code. It belongs in your application, yes, but your application consists of more than just PHP. It is the entire stack of software that all follow the UNIX principle: Linux, Apache, PHP and MySQL. Of course, substitute any of the software with an equivalent alternative, if you have reason.

Another example is routing and dispatching. One of the most tedious and tiresome jobs inside your application is making sure that every link is SEO friendly; i.e. that it contains words that are relevant to the page’s content, and that it’s hierarchical position is relevant to where the page belongs in the site, while still having an idea of what the canonical url of that page should be to avoid duplicated indexing by search engines. I have had not a single project where this would cost not a few hours – at best – to take into account. Why? Because we think to hard about it, and we fear the simple solution might impact performance.

Where complexity is, is where the problem lies

Whenever you find you’re adding complexity to your application because of non-functional requirements such as performance reasons or cost reduction, you need to take a step back and re-evaluate the problem at hand.

A very simple example of this is a YAML file read by a set of PHP classes, rendering a PHP array. This array can be cached in a compiled form using var_export. Having the application itself responsible for this compilation will add complexity to the set of responsibilities of the application. It would be much easier to have the application depend on a PHP configuration file, and let some build process make sure the YML file is compiled into PHP.

Another example is compilation of template files. If you have a set of template files in another language than PHP, for example Twig or Smarty, you can have the compilation of these files be part of a build process. This doesn’t have to be on-the-fly. PHP programmers are used to having stuff work “on the fly”, but it is a convenience that can bite you in the ass. It’s far easier to have a build script that makes sure all cachable data is flushed, than have your application responsible for checking of freshness of these caches. To be even more exact: your application doesn’t even need to know that the stuff it’s reading is actually a cache. It only needs some files, and they’re there.

My final example is SEO friendly URL’s. There is no reason the application itself should worry about this. We have lots of available tools to rewrite outbound and inbound url’s to whatever we need to rewrite. The only thing this tool needs to know is a very simple “from” => “to” mapping of all these URL’s, and that’s it. The program can translate friendly URL’s into internal URL’s, and vice versa. If the program does that, and does that very well, we are out of the pickle.

Data versus layout

Formatting the data of your application in to whatever format (usually HTML), is the responsibility of some sort of view renderer. Action Controllers have the responsibility in translating a request into some sort of response, and usually dispatch to a view renderer internally to have the response formatted as HTML. Now here’s something fishy going on. The action controllers gain three responsibilities in this way; two explicit (handling the request with the appropriate action and returning it’s result) and one implicit: rendering the result. Since the result of any action always is data, the action controller should not be responsible for the formatting of the data. It is in fact irrelevant at that point in time. You’ll just want to know what the result of that action is; either some state (success or failure) or some data. It makes much more sense to have an intermediate format for this (be it a PHP array of data, XML, JSON, or some other form), and have the view on this data or result be detached entirely from the action’s operation. This way, the action controller has only one responsibility: returning the request’s result. Note that the request can be defined as simply as a few method parameters; it needn’t be an HTTP request as such.

The front controller of the application can be responsible for dispatching one front request to one or more action requests, that all have their result. Whether or not the front controller renders this response as HTML is moot.

Why not utilize PHP as a build result?

I think we all might gain more in using PHP as a result of our project’s build. We might simply program in a higher level programming language, and have some compilation or build process build all kinds of php files, that may contain a meriad of repetitive code, just because that is what ultimately the PHP interpreter is best at. Simply preprocessing the hypertext, in stead of doing it the other way around and having PHP implement byte code caches and high level programming features. Maybe PHP was the right language right before it got cocky…

• Tetris¹
• Source at github

But wait, there’s still hope! I thought I could write a post about me thinking of awesome stuff, getting the basics working and then never finishing it. So I started typing this post, but halfway through, I decided that the basics are good enough. And that the basics are worth posting too.

Sorry if I disappointed you. Have fun reading the code The gameplay is not that interesting.

However, none of that is the real core of Symfony2. The most interesting part lies in two main components of the framework. First, the HttpFoundation component and second, the Service container.

The HTTP Kernel

Ordinarily, with any controller framework out there, you’ll have a front controller, a router, a dispatcher and some action controllers that work together to generate output for any URL that is requested within your app. In essence, this is no different in Symfony2. There is something extra though. The HttpKernel, which is actually the core of your application is responsible for speaking HTTP. The developers of Symfony2 have put extra effort into nitpicking every aspect of the protocol, such as caching, so that any application written with the HTTP Component would have all the tools at hand to harness the power of HTTP.

The request loop

So, what actually happens when a typical Symfony2 app handles a request, is that a Request object is forged from all PHP’s relevant variables, it is dispatched into the Kernel as an event. The kernel holds a reference to the service container too. Each of the services registered as an event listener for this particular event is notified, so they can build a response. Then the response triggers a new event so all listeners can influence the response again, and ultimately the Response object is sent to the client.

Both the Request and Response objects reflect HTTP messages. As you probably know, any HTTP message is made up from a status line, a set of headers and an optional body. The headers contain protocol information and the body contains content. Usually, when you visit a page, a GET message is issued to the server, containing some headers on how your browser is configured and what host we want to get the URL from. The webserver responds with an answer, containing a status line which indicates the nature of the rest of the response, headers about the size of the response and some other useful information such as how long the resource may be cached by proxies and browsers, and, finally, a body, usually containing the resource data of the requested object if the request succeeded.

Parlez-vous HTTP?

In ye olden days, it was typical for a PHP script to always execute the exact same way, no matter how the user configured their browser or what information on caching was available. Moreover, it was common to have PHP scripts always send, what we affectionately called the “no cache headers”, which instructed the browser to absolutely never cache the page and please always come back to refresh the data, so all data was up to date for all PHP pages and all visitors, always. This had also to do with a particular browser that tended to cache more than was actually instructed by the web server, and customers who wanted to see changes rather sooner than later. Well, the conclusion was, that we, as a profession, never learned to parler HTTP.

Nowadays, with applications getting bigger and application speed, performance and stability are makers or breakers of the application’s success, you’re soon too late to dive in. There was a few years where we could get away with caching only in the backend by optimizing MySQL’s query cache, utilizing simple file caching mechanisms or firing up APC, Memcached and all the likes. All very useful, but only half the story.

Get up `ahhhh! Like a cache-machine

Basically, there are two ways of caching stuff. You can tell the client to keep a resource a maximum amount of time, or you can tell them when the resource was last modified ¹. These two models are called Expiration and Validation, respectively.

In the first case, a client may keep the data as long as is specified. In the second case, the client must check back with the server to see if their cached version of the resource is still valid. If not, the response proceeds as usual; if so, the response is set to a 304 Not Modified status and is returned without a body. Expiration and validation can be combined so the client is told how old a resource is and with what frequency they should check back to validate the resource. That’s the short version. Read the longer version here.

The service container

The service container is a registry. What’s a registry, you say? It is a container object that contains immutable objects based on unique keys (much like a “hash” or a “map” structure), which is application wide. Martin Fowler describes it as “A well-known object that other objects can use to find common objects and services”². The service container itself, therefore, is not that big a deal. THe idea behind it, though, is that all the services that are available in the service container can be loaded on demand. This phenomenon is called lazy loading or bootstrapping. That means that if you have a gazillion services defined, and not one of the lot is used, none of them will be instantiated. This means that if you have a PHP script cropping an image on-the-fly, that does not need the database connection will not open the database connection, even if the script uses the same front controller logic. Now, that sounds like it could be useful.

Depend this!

The clever thing about Symfony2′s service container is that it ships with a load of logic to resolve and inject dependencies. What that means is that if you have, say, a model class which depends on a database connection, you only have to explain to the container how the dependencies can be instantiated and how they reference each other, and the service container will know how to resolve the dependencies. This sounds like a lot of overhead, but as this dependency resolution is done at compile time, and PHP code is generated that actually has all such depencies resolved already, there really is next to none.

And then?

Get on the Symfony2-wave. It is rolling like crazy and Github is exploding with bundles (symfony2′s modules) and contributors. If you know the Kernel and the Dependency Injection components, you should head over to the Form component. Then you know about everything you need to know to build a real Symfony Bundle. The rest will come as you go.

One last tip, don’t use the Bundle Generator if you don’t know what it generates and whether you need it or not. Just create your bundle by hand, you can always start generating bundles if you actually know what it does. My next post on this topic will be a simple guide to help set things up by hand in stead of all the command line wizardry. Why? Because then you actually know what the wizard conjures.

cd /etc/nginx/
mkdir ssl
cd ssl
mv /path/to/pfx/file.pfx .
chmod 400 file.fpx

First extract the public certificate. You might be asked for a password.

openssl pkcs12 -in ./file.pfx -clcerts -nokeys -out public.crt

And extract the private key:

openssl pkcs12 -in ./file.pfx -nocerts -nodes -out private.rsa

Now you can test the server on an arbitrary port, using openssl:

openssl s_server -www -accept 443 -cert ./public.crt -key ./private.rsa

Make sure no one can read the files other than you:

chmod 400 /etc/nginx/ssl/*

With NginX it is now easy to fire up the server. I used a proxy for this, because from an architecture perspective, this is the easiest:

server {
    server_name example.org;

    listen 443 ssl;
    ssl_certificate /etc/nginx/ssl/public.crt;
    ssl_certificate_key /etc/nginx/ssl/private.key;

    location / {
        proxy_pass http://example.org/;
        proxy_set_header Host $host;
        proxy_set_header X-Ssl on;
    }
}

I pass an additional X-Ssl header to the backend so they know we’re publicly serving through the SSL proxy (e.g. for building absolute URL’s). Once you actually know how to do it, it is easy as pie.

With thanks to Yadab Das and Berk D. Demir

One simple example of proving closures’ power is in the concept of PHPUnit’s dataProviders. A data provider provides data to a test, in which a test consumes the data and asserts that the code is ok. As data can also be closures, you can also alter the execution of the tests appropriate to the data you want to test, without actually altering the test. I’ll show you an example ¹.

Consider the following class:

class FooBuilder {
    function add($property, $value) {
        $this->properties[$property][] = $value;
        return $this;
    }
 
 
    function remove() {
        if (!empty($this->properties[$property])) {
            array_pop($this->properties[$property]);
            if (count($this->properties[$property]) == 0) {
                unset($this->properties[$property]);
            }
        }
        return $this;
    }
 
 
    function build() {
        return $this->properties;
    }
}

Though there is no obvious use case for this class, the pattern is simple: we have some builder with methods to create something, and the builder provides a fluent interface to do so. This is also very common for query builders, including jQuery. With such a pattern, there is a meriad of possibilities to use this code, with at least as much edge conditions. The order in which the methods are executed might influence the way it is implemented, what would happen if properties are added, removed and then removed again, etc etc.

These edge cases won’t be displayed by simple use of code coverage, and introducing separate tests for each of these conditions would be cumbersome, having to introduce separate test names for each of the cases.

So, here’s where closures come in handy. We’ll simply call the test testBuildingWithFluentInterfaceWillResultInExpectedArray:

class FooBuilderTest extends \PHPUnit_Framework_TestCase {
    /**
     * @dataProvider builderCases
     */
    function testBuildingWithFluentInterfaceWillResultInExpectedArray($expected, $builderImpl, $description) {
        $builder = new FooBuilder();
        $this->assertEquals($expectedArray, $builderImpl($builder)->build(), "$description failed");
    }
 
 
    function builderCases() {
        return array(
            array(
                array(),
                function (FooBuilder $builder) {
                    return $builder;
                },
                "Initial construction"
            ),
            /* .... */
            array(
                array('a' => array('b', 'c'), 'x' => array('y', 'z')),
                function (FooBuilder $builder) {
                    return $builder
                        ->add('a', 'b')
                        ->add('x', 'y')
                        ->add('x', 'z')
                        ->add('a', 'c')
                    ;
                },
                "Adding different properties multiple times in a different order"
            ),
            /* .... */
            array(
                array(),
                function (FooBuilder $builder) {
                    return $builder
                        ->add('x', 'y')
                        ->add('x', 'z')
                        ->remove('x')
                        ->remove('x')
                    ;
                },
                "Adding different properties multiple times in a different order"
            ),
        );
 
}

This way, you´ll have a clear view of all the test cases and expected output, without having to only provide data through your data provider, but logic too. Clear separation of use case and consumer logic makes the test very clear and easily extendable.

Hope this offers more food for creativity in your tests

The “old” paradigm: Procedural

Procedural programming is a form of imperative programming in which you define “procedures” or “subroutines”¹ that encapsulate a certain set of operations, and accept parameters or “arguments” to that set of operations. A very simple example of this, is a procedure that will return the sum of two arguments. In C:

int sum(int a, int b) {
    return a + b;
}

We declare a procedure that accepts two integer arguments a and b and return an integer sum of the arguments. Simple, efficient, clear. If we want the sum to be present in a third variable, we simply execute the procedure:

#include <stdio.h>
 
int sum(int a, int b) {
    return a + b;
}
 
int main(int argc, char *argv[]) {
    int c;
    c = sum(10, 20);
    printf("c now contains %d\n", c); /* Will output: "c now contains 30" */
    return 0;
}

Procedural is imperative

The procedural style of programming is imperative. Imperative means that you program the execution flow. You decide what pieces of code gets executed by using control structures, loops, procedures, etcetera, and the state of the variables (the “context” or “environment”) ultimately define the execution flow. This is very intuitive, since you can read most imperative languages as “if something happens, you do this and that”. More or less like an instruction video for driving. If someone comes from the right, yield.

The “newer” paradigm: Object Oriented

Another, still imperative, paradigm is Object Oriented programming. Though it is not strictly different from procedural programming, it introduces a new concept: binding data and methods to a single object, that represent one single previously initialized state of the object. This is called instantiation and means that multiple objects of one type can exist in memory simultaneously. Each of these objects have their own methods that act only on the state of that particular object.

There are two concepts that are new in OO style programming, first of which is encapsulation, where the outside world does not need to know what is going on within some object’s internals. Second is polymorphism, which means that an object can explicitly express that it is capable of some sort of functionality and may implement it in a very different way than another object which exposes the same behaviour. Huh? Bear with me. Here’s an example.

class AreaCalculator {
    public float total = 0;
 
    public void add(HasArea obj) {
        total += obj.getArea();
    }
 
    public void subtract(HasArea obj) {
        total -= obj.getArea();
    }
}
 
 
interface HasArea {
    public double getArea();
}
 
class Rectangle implements HasArea {
    private double _side1;
    private double _side2;
 
    public Rectangle(double side1, double side2) {
        _side1 = side1;
        _side2 = side2;
    }
 
 
    public double getArea() {
        return _side1 * _side2;
    }
}
 
class Circle implements HasArea {
    private double _radius;
 
    public Circle(double radius) {
        _radius = radius;
    }
 
    public double getArea() {
        return Math.PI * Math.PI * _radius;
    }
}
 
class Example {
    public static void main(String[] args) {
        AreaCalculator calc = new AreaCalculator();
        // our yard is 10 by 4.2 meters
        calc.add(new Rectangle(10.0, 4.2));
 
        // But it has a pool, 3 meter in diameter
        calc.subtract(new Circle(1.5));
 
        System.out.println("The square metrage we need to buy grass seed for is " + calc.total);
    } 
}

The result:

The square metrage we need to buy grass seed for is 27.195593

The example shows that two different objects have some common characteristic. In this case, they both know how they should produce their own area. The implementation is bound to the meaning of the class itself, since the area of a circle is calculated differently from the area of a rectangle. So, they both provide their own implementation to do so. Since the two different implementations are tightly related, it does make sense that we defined a common interface for it, so the actual consumer of the objects (the calculator) does not need to know they are totally different kinds of objects. All it needs to know is that the object is capable of producing it’s own area. That’s it. Well, if you know OO you know this already, but you might not have known that this phenomenon is called polymorphism. The fact that we hide the properties _side1, _side2 and _radius from the outside world is encapsulation. Especially polymorphism is something that is hard to achieve with a purely procedural language.

It strikes to see that OO versus procedural, however still imperative, is much more descriptive. It describes natural things in a natural way. A pear is a fruit, an apple is a fruit, they both have some specific characteristics, but they share a lot of common properties and probably operations, such as feed() or ferment(). The programming style still is imperative as it follows the execution flow, but much of the organisation of the code revolves around typing and classifying, so it is more descriptive even though not yet strictly declarative.

It would also appear that OO moves focus from processing and decision-making to behaviour. Describing behaviour is typically part of the object model, because behaviour defines how objects interact and how they depend on each other. This is curious: the data no longer plays the main role in the design of the program, the behaviour does. Let’s stash that thought.

Declarative programming

The main difference between declarative and imperative programming, is that declarative means that you express what something is, rather than what something should do. A common example of declarative programming is build scripts. I’ll use a Makefile for this example.

As we define a task within a Makefile, we can specify what tasks it is dependent on. As this simply declares what the task does, without actually doing any decision-making within the program this is typically a declarative approach. The tasks get declared as what set of executions the represent. The decisions that are needed to make for execution of the tasks is made by the platform:

# building the app depends on mysum.c (the example above)
# if mysum alters, the app should alter too
mysum: mysum.c
    gcc mysum.c -o mysum

The approach is declarative. We do no checks, we just tell what needs to be done to generate the app. See the entire Makefile for the above examples on the github repo

Another relatively simple example is Haskell. Without explaining too much about functional programming, the main idea of Haskell is that you declare anything you need. Declaring things does not mean executing them. For example:

l = repeat 5

What happens here is that we declare a list that is made up of an infinite list of 5s. So, how come nothing happens? If the list is infinite, how can it exist? It exists because it is declared. It only gets meaning as soon as it is used in context, which means that the meaning is defined by evaluating the list in it’s context. When (or even if) this happens. The problem is, imperative programming leads to the assumption that everything that is actually declared, will be evaluated as soon as execution flow enters. This is different in declarative languages, simply because they do not have execution flow. Or, at least, the execution flow is irrelevant. Evaluation is delayed until the point it is required. That is called lazy evaluation. And we like lazy, because lazy means CPU cycles and memory is only consumed as needed.

The best of both worlds

Languages now tend to shift to a mixture of (imperative) Object Oriented and (declarative) functional. We see that in Python, Ruby, Erlang and maybe even the best example of them all: Scala. What’s even more surprising is that in PHP world, which isn’t the fastest cat of the bunch, the seed is actually planted. Symfony2 introduces declarative paradigms with an inversion of control service container. That is declarative. You simply declare what service depends on another. PHP code is generated to have these dependencies resolve gracefully and lazily and we no longer need to think about execution flow for this. The services and their dependencies are only declared, and the (imperative) code for initializing the services is generated under water. Consequentally, requesting the services from the container will have the services be initialized lazily.

Another fine example is NodeJS. Since the entire model is built around one simple event loop, all behaviour is essentially defined declaratively.

So, what’s next?

Learning imperative languages will become obsolete, just as learning assembly has. You will need to learn how to define behaviour and dependencies, but no longer what needs to be done to get the entire thing rolling. As languages get more and more declarative capabilities and functional programming paradigm features, the imperative idea will only be a resort where the awesomest go.

Or…. will it? Compiling code introduces overhead. There is no compiler that is able to optimize all and every situation as well as an advanced programmer could. This holds true for, say, a C to Assembly compiler, and would most certainly be the case for compiling declarative approaches. Because ultimately, the little pieces of hardware that get the work done, are not that professed in languages. They only do ones and zeros, and have no concept of meaning. This means that bringing the code from the keyboard to the processor’s registry will become further and further apart. And if that’s the case, there’s only two ways to go. Either software will become even less efficient, or the hardware needs to get smarter. Pick one.

NB: The code in this post is on github

I do not intend to convince you to use a content repository. I think the matters speak for themselves. If you know all the limitations classical setups using relational databases have, you’ll probably see a future in NoSQL databases, as they are affectionately called. MongoDB and CouchDB never convinced me the way the JCR specification did. Google the specs and read them yourself and know what I mean.

1. Get a CR backend running

I chose Apache Jackrabbit, a Java Content Repository (JCR), to do this. As pointed out on the Jackalope website, you’ll need a patched version of Jackrabbit to get things compatible with PHPCR. Actually, it is just version 2.2 with a backport patch of the 2.3 branch applied to it ¹, so don’t fear using it. I’m told it is pretty stable.

1. Get the patched Jackrabbit here.

2. Run the the server

   java -jar jackrabbit-standalone-*.jar

This will run jackrabbit at port 8080. Visit your jackrabbit front end at http://localhost:8080/. You’ll be asked a username and password, but you can leave them blank. By default, there is no access control. If you see a page about jackrabbit, it works.

2. Install Symfony

I’m assuming you know how to do this. If you don’t, follow the instructions at symfony.com. The short version is:

1. Download Symfony 2.0 latest version
2. Extract the archive in your document root, say /var/www/
3. Visit /Symfony/web/config.php and follow the instructions
4. Visit /Symfony/web/app_dev.php to determine if the installation worked

3. Install PHPCR, Jackalope, DoctrinePHPCR ODM and the DoctrinePHPCRBundle

Go to your vendor dir and clone the PHPCR libraries from github:

cd /var/www/Symfony/vendor
git clone git://github.com/phpcr/phpcr.git
git clone git://github.com/jackalope/jackalope.git
git clone git://github.com/doctrine/phpcd-odm.git
cd bundles/Symfony/Bundle
git clone git://github.com/symfony-cmf/DoctrinePHPCRBundle.git

4. Configure the autoloader

Open app/autoload.php and configure the namespaces for the downloaded libraries:

<?php
/* ... */
$loader->registerNamespaces(array(
    /* ... */
    'Jackalope'             => __DIR__.'/../vendor/jackalope/src',
    'PHPCR'                 => __DIR__.'/../vendor/phpcr/src',
    'Doctrine\\ODM\\PHPCR'  => __DIR__.'/../vendor/phpcr-odm/lib/',
    /* ... */
));

5. Add the DoctrinePHPCRBundle to your AppKernel

Open app/AppKernel.php and add the bundle to the registerBundles() call:

<?php
/* ... */
public function registerBundles()
{
    $bundles = array(
        /* ... */
        new Symfony\Bundle\DoctrinePHPCRBundle\DoctrinePHPCRBundle()
    );
    /* ... */
}

6. Configure the ODM default service

Add the following section to your app/config/config.yml file:

doctrine_phpcr:
    session:
        backend:
            url: http://localhost:8080/server/
        workspace: default
        username: ''
        password: ''
    odm:
        auto_mapping: true

7. Set up the Doctrine system types

To have Jackrabbit understand some type internals Doctrine uses for document mapping, we’ll need to make the content repository aware of these types. There is a console command for that, to make things easy:

app/console doctrine:phpcr:register-system-node-types

If you skip this step, persisting documents with a Doctrine DocumentManager will fail with a Jackrabbit error message complaining about phpcr being an unknown namespace.

8. Set up a PagesBundle

You should, of course, change the class and namespace names to whatever you wish. Here’s what I used:

In src/Melp, I created a PagesBundle. This bundle will hold my routes for editing, creating, deleting and showing pages from the content repository. It will also hold the Document model classes I will be using for mapping documents to objects:

cd /var/www/Symfony/src
mkdir -p Melp/PagesBundle/{Controller,Document,Resources{,/views}}

This creates the bundle’s directory structure. There is also a console command for it, which can scaffold the directory structure and some classes for you:

cd /var/www/Symfony
app/console generate:bundle

The MelpPagesBundle class will be in /var/www/Symfony/src/Melp/PagesBundle/MelpPagesBundle.php, and contains the following code:

<?php
namespace Melp\PagesBundle;
 
use Symfony\Component\HttpKernel\Bundle\Bundle;
 
class MelpPagesBundle extends Bundle {}

Add the bundle instance to the AppKernel class’s bundle initialization routine:

<?php
/* ... */
public function registerBundles()
{
    $bundles = array(
        /* ... */
        new Melp\Bundle\PagesBundle\MelpPagesBundle()
    );
    /* ... */
}

9. Set up the controller, routes and view scripts

I’ll use Annotations for the controller set up. I was a bit reluctant at first to use annotations for such configuration, but I find it much easier and clear to use annotations than the other configuration options. The only thing we’ll need is to let the router configuration know the controller exists. We’ll do that in app/config/routing.yml:

_pages:
    resource:  "@MelpPagesBundle/Controller/PageController.php"
    type: annotation

I will also use Twig view scripts. You can of course use PHP view scripts or whatever you prefer, but Twig has had my heart for some time already. The @Template annotation will tell Symfony to render the view resources using the twig template engine as is configured in the default Symfony2 distribution you downloaded. ²

The controller class will then look like this:

<?php
namespace Melp\PagesBundle\Controller;
 
use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\Routing\Annotation\Route;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Template;
 
class PageController extends Controller
{
    /**
     * @Route("/")
     */
    function indexAction()
    {
        return $this->redirect($this->generateUrl('view', array('path' => 'home')));
    }
 
 
    /**
     * @Route("/{path}", requirements={"path" = "(?!(edit|delete|create)).+"}, name="view")
     * @Template
     */
    function viewAction($path)
    {
    }
 
 
    /**
     * @Route("/edit/{path}", requirements={"path" = ".*"}, name="edit")
     * @Template
     */
    function editAction($path)
    {
    }
 
    /**
     * @Route("/create/{path}", requirements={"path" = ".*"}, name="create")
     */
    function createAction($path)
    {
    }
 
 
    /**
     * @Route("/delete/{path}", requirements={"path" = ".*"}, name="delete")
     */
    function deleteAction($path)
    {
    }
}

Since I want to have the routes reflect the path in the content repository, I use a wildcard pattern for the paths, so they may contain slashes. You can use PCRE in your routes, which I used to exclude the /edit, /create and /delete routes from the default viewAction() to avoid conflicts.

These actions will implement CRUD actions for nodes inside the content repository.

10. Create the document model class

To have Doctrine manage my model class, I will need to attach Doctrine annotations to it, so Doctrine will know how to read and write model data from and to the content repository backend. I will use a simple Page class, with title and content properties, wich we will later implement the CRUD for, as layed out in the Controller above.

In src/Melp/PagesBundle/Document, create the model reflecting the ‘Page’ data structure we’ll be using to edit and show pages ³.

<?php
namespace Melp\PagesBundle\Document;
 
use Doctrine\ODM\PHPCR\Mapping\Annotations as PHPCRODM;
 
/**
 * @PHPCRODM\Document(alias="page")
 */
class Page
{
    /** @PHPCRODM\Id */
    public $path;
 
    /** @PHPCRODM\String */
    public $title;
 
    /** @PHPCRODM\String */
    public $content;
}

11. Create a Pages facade and register it as a service

I created a facade for controlling how pages are persisted and added some logic to that for placing pages in a configured root:

services:
    pages:
        class: Melp\PagesBundle\Service\PagesFacade
        arguments:
            - @doctrine_phpcr.odm.default_document_manager
            - 'Melp\PagesBundle\Document\Page'
            - "/pages/"
            - "home"

The facade exposes the following methods:

public function __construct(\Doctrine\ODM\PHPCR\DocumentManager $dm, $documentClass, $root, $default)
{
}


function createDefault($title = "Homepage")
{
}


function removePath($path)
{
}


function findPath($path)
{
}


function remove($node)
{
}


function persist($node)
{
}

This saves some clutter in the controller code. The Facade dispatches to the document manager and the document repository for finding and persisting pages.

12. Create a PageType form class

Following a best practice not to initialize your forms inside your controller, I created a PageType class in src/Melp/PagesBundle/Form/Type/PageType.php:

<?php
 
namespace Melp\PagesBundle\Form\Type;
 
use \Symfony\Component\Form\AbstractType;
use \Symfony\Component\Form\FormBuilder;
 
class PageType extends AbstractType
{
    function buildForm(FormBuilder $builder, array $options)
    {
        $builder
                ->add('title', 'text', array('required' => true))
                ->add('content', 'textarea')
        ;
    }
}

This class will be used inside the controller to build the forms for creating new pages and editing existing ones.

13. Implement the CRUD Controller

I chose to implement a very, very basic CRUD for the document class. The code speaks for itself. I used twig templates to render the output of the edit, create and view actions, and rely heavy on flash messages and redirects for feedback to the user.

Summarizing

Much, much easier than an RDBMS based hierarchical system of pages, much more convenient and easier to control, and crazily scalable. I think there is much future in this kind of set up. So get comfortable with it while I’ll do the same, and I hope to get even deeper into the subject some next blog.

I hope this post helped you some further if you got stuck, and is of some inspiration if you want to get started. All tips and comments are appreciated.

set -e

Quoting the man page:

-e errexit

Exit immediately if a simple command exits with a non-zero, unless the command that fails is part of an until or loop, part of an if statement, part of a && or || list, if the command’s return is being inverted using !.

If you want to reset to normal behaviour, reverse it using:

set +e

Very useful, it’ll save you lot of tedious if’s and error checking.

I already knew of the nice program, with which you can alter the CPU scheduling prioritization of processes. nice usually prefixes a command to start it up with altered ‘niceness’:

$ nice -n 19 rsync -a ~ /net/share/backup

renice is used to alter an already running process’s niceness:

$ renice -n 19 -p $PID # where, of course $PID is de pid of the process you're altering

Pretty useful, and also pretty common knowledge, as far as I know. But then it dawned on me that CPU priority isn’t at all what’s useful here. You want I/O priority. So, a little googling lead me to ionice – how inventive a name – with which you can alter process I/O prioritization. Awesome!

$ ionice -c 2 -n 4 -p 1234 # Give process 1234 "best effort" priority 4

You should read the man pages of nice and ionice to find out what the parameters mean (just because it’s a good habit to read man pages, and not blogs ), but the basic usage is as follows:

# start a process 'some command with arguments' with a niceness of 19. 
# 19 means lowest priority, -19 means highest
nice -n 19 some command with arguments 
 
# The process with pid 1234 gets a new niceness of 10, provided you are either root,
# or you are the owner of the process
renice -n 10 -p 1234
 
# The process with pid 1234 gets "Idle" class priority, which means it only gets I/O access
# when no other process needs it.
ionice -p 1234 -c 3

Use top and iotop to monitor the usage of CPU and I/O usage respectively. From now on, I’ll be nice to the kernel and not let him figure out all the prioritization. Wow that’s cheesy