melp.nl

< Return to main page

Composer: create a local package repository to improve speed

When you're developing a lot of symfony projects with a relatively large set of dependencies, sooner or later you'll get annoyed by performance issues. By default, composer uses packagist.org to get package metadata (what versions are available and where to get them from). All packages are part of packagist.org by default, but 9 times out of 10, you'll only need a fraction of that.

So, to gain performance, I decided to generate a local package repository which only contains the package info for the libraries I regulary use.

1. Gather package names

In my homedir, I searched for composer.lock files, and extracted all github.com repository names with the following script, which, of course, started out as a oneliner.

#!bash
# command you can use to extract github.com package names from composer.lock files.
find ~ -maxdepth 3 -name "composer.lock" -exec cat '{}' \;  \
    | grep github.com                                       \
    | grep -v 'api.github.com'                              \
    | egrep '.git",$'                                       \
    | awk '{print $NF}'                                     \
    | sed 's!.\+github.com/\(.*\).git",!\1!g'               \
    | sort                                                  \
    | uniq                                                  \
    > packages.list

This yields a file which contains github repository names on github.com like this:

Seldaek/monolog
symfony/symfony
...

2. Generate satis.json

Based on this list of packages, which now contains pretty much all of the packages I regularly use, I created a PHP template script that generates a satis.json for me:

#!php
{
    "name": "Github libraries",
    "url": "http://internal-satis-url-here",
    "homepage": "http://internal-satis-url-here",
    "repositories": [
        <?php
        $i = 0;
        foreach (file('php://stdin') as $package) {
            if ($i ++ > 0) {
                echo ", \n";
            }
            echo "          ", json_encode(
                array(
                    'type' => 'vcs',
                    'url' => sprintf('https://github.com/%s.git', trim($package))
                )
            );
        }
        ?>
    ]
}

Calling this script with the generated packages.list as input will render the satis.json file I will use to generate the packages.json:

#!shell
php satis.json.php < ./packages.list > ./satis.json

Now, it's just a matter of generating the packages.json with satis:

#!shell
satis update ./satis.json .

and copying the rendered packages.json to some URL available to your machine, lets says http://satis.local/

3. Disable packagist by default

You can disable packagist and enable your local repository by default, by adding it to ~/.composer/config.json:

#!json
{
    "repositories": [
        {
            "type": "composer",
            "url": "http://satis.local/"
        },
        {
            "packagist": false
        }
    ]
}

4. Automating

You can simply automate this build process with a Makefile or any other tool you prefer for build automation. Makefiles make sense as they are pretty good at handling source file dependencies:

install: packages.json 
    scp index.html packages.json host:/var/www/satis

clean:
    rm packages.json satis.json

packages.json: satis.json
    satis update satis.json .

satis.json: packages.list
    php satis.json.php < packages.list > satis.json

The end result:

In a sample project I tested, a full composer update went from 3 minutes to 18 seconds, which still leaves to be desired for, but is nonetheless a 1000% performance improvement. Worth the try, if you ask me!

Happy composing!


< Return to main page


You're looking at a very minimalistic archived version of this website. I wish to preserve to content, but I no longer wish to maintain Wordpress, nor handle the abuse that comes with that.