Package management. No really. How hard can it be?

We had a little discussion at work some time ago, again, involving package management. No, I’m not talking about the .deb versus .rpm debate. I am talking about source dependencies for JavaScript and PHP projects. But the problem holds equally well for OS package management or build dependencies.

Currently, we are using a fine blend of Composer, Bower and NPM. Coincidentally, another one is peeking around the corner, because Composer uses a tool internally, which uses a manifest.json to identify the source files as coming from some source repository, namely github, so the package is able to update itself1.

Thank heavens, we’re at least agreeing on a common definition format being JSON. At least, the syntax of it, not the semantics…

The knights who say ‘NIH!’

NIH. The “Not invented here” syndrome. There are plenty of tools out there, but lets just do it again, but now the way we think is best. It’s called a syndrome, which should give you an idea of it’s benefits. Syndromes usually don’t constitute good health. That counts for something.

We fail to see that effectively all these tools do the exact same thing. Some file is loaded. It is checked for dependencies on other packages. This is done recursively, ending up in a list of selectable versions, then this list is narrowed down using some kind of version resolution scheme, such as preferring stable over development and explicit version dependencies. When this is complete, a list of installable packages remains which can be downloaded from some kind of source, usually some site hosting the packages or github. Not that hard, right? Right.

There is one big “but”. The description of dependencies are part of the packages, i.e., their source distribution. So now, the source tree must have some kind of declaration of which dependencies it has. And here we go. We are building a PHP project, so we must have some tool that does the job, in PHP. Also, we’re building front end javascript and sass, so we need some tool doing the exact same thing for front end assets. And, last but not least, we tend to use NPM for Javascript tooling that is not hosted in some bower registry. Hence, Composer, Bower and NPM.

Awful. We fail to see that the one thing we need to publish in our source code is the dependencies. If we can all agree on some kind of standard for this, the choice of package management can be mine, whether it is composer, npm, bower, dselect, yum, what-freaking-ever-pm I set my mind on today. But the knights who say NIH, don’t usually sit down to think things through. And they tend to squable over futilities. Me included.

Knights of the round table, then?

Here’s my thought. Let’s think of a standard. And lets use that in any package manager we use. Now, we get rid of all composer.json, package.json, bower.json and whatever Python, Ruby, Erlang or Brainfuck version there is out there and let’s just hold hands and sing a mantra:

A package manager exists to download stuff in the right version for me. Downloading is easy. Dependency resolution is easy (though it might take some time if the graph is big). Agreeing on how to make these dependencies explicit seems to be hard. But it really isn’t. All you need is a file saying “I need this package in that version”. It could look just like this:

symfony/symfony: 2.6.*
jquery: 2.*
node-sass: *@stable

Kinda looks like YML, doesn’t it. But let’s set the preferences of serialization format X versus Y aside. Let’s agree that at least one format is useful. Or, in RFC style, any package manager implementing the spec COULD support multiple file formats reflecting the same structure. It MUST implement at least one.

Now, we need some tool to translate these package names to download locations. Wait. That doesn’t need to be a tool. It could be a hosted service… No wait. Don’t we already know something like that? Like a sources.list in Debian? Or yum.repos in Fedora? Then, finally, we need a way to identify where the packages need to be stored, once downloaded. Whether that’s ./node_modules, ./vendor, /usr/local, /, or "C:\Program Files" really is just Yet Another Config Value.

We don’t need more package managers. We need a specification. A standard. And it really shouldn’t be hard. Hell, I think I almost summed it up already. Once we do that, we can let go of the NIH, get ourselves a pair of halves of a coconut shell2 and start prancing around whinnying and enjoy ourselves just a little bit better.

  1. If anyone might have read my previous post on Gödels incompleteness theorem, you might recognize the fact that adding more to the “system” not necessarily is a good idea. 

  2. This, by the way, works really well. 

Posted in Development | Tagged , | Leave a comment

Why Gödels incompleteness theorem should be software engineering 101

I am a developer. That’s why I know how us folk think. We want clean and nice solutions for problems, because that’s what we do best: provide elegant programs that solve actual everyday problems. We also tend to overrate elegancy. Especially in web development -my field of expertise-, when projects and programs are relatively small for relatively small budgets and always with end results that have a short life span, we think about reusability and tend towards abstract and generic solutions that provide more than a solution, it provides a way of life, progression for the industry as a whole, a solution to even more problems the client nor project manager ever thought of or could even imagine they would ever encounter. We are smart, we know what problems they will face eventually so we prepare them for this. Even more so, we prepare our solution for this, so ultimately the only thing we have to do is flick a switch from now on, and everything is working for everything and we never have to write one line of code again. From now on all we have to do is configure. Which isn’t like coding. At all. You know the feeling, right?

Read More »

Posted in Development | Tagged , | Leave a comment

Building OpenGL SDK: compile error “X11/extensions/XInput.h”

If you get this error:

src/freeglut_internal.h:111:39: fatal error: X11/extensions/XInput.h: No such file or directory

This might save you some time searching for the correct dev packages to install. The package it is in in the Debian repositories is called x11proto-input-dev, but it contains the file XI.h which contains the correct headers freeglut is including. Symlink the file and you’re good to go.

cd /usr/include/X11/extensions && sudo ln -s XI.h XInput.h

HTH :)

Posted in Development, Linux & BSD | Tagged , | 5 Comments

Composer local package mirroring: Press the pedal to the metal.

In my previous blog post, I told you about hosting local package repositories for composer. Me, or to be honest, my colleagues, weren’t too excited about the performance gains. So I decided to dig in a bit deeper.

Read More »

Posted in Development | Tagged , , , , | 6 Comments

Composer: create a local package repository to improve speed

When you’re developing a lot of symfony projects with a relatively large set of dependencies, sooner or later you’ll get annoyed by performance issues. By default, composer uses to get package metadata (what versions are available and where to get them from). All packages are part of by default, but 9 times out of 10, you’ll only need a fraction of that.

So, to gain performance, I decided to generate a local package repository which only contains the package info for the libraries I regulary use. Read More »

Posted in Development | Tagged , , | 1 Comment

Flow charts in code: enter graphviz and the “dot” language

If you’re like me, you like gui’s as long as they don’t push you in a direction other than your train of thought. Whenever the tool tends to distract you from the task you are performing, you get annoyed. Stuff like “why am I searching for such an over-obvious functionality”, or “why didn’t they think of making the clickable area a bit bigger”, or simply “aaaarggghhh, it crashed on me again”.

Read More »

Posted in Development, Linux & BSD | Tagged , , | 5 Comments

PHP: We are getting slow and sluggish, and we’re lazy and arrogant about it.

In my most recent blog, I wrote about how I feel that too much of the world’s logic is coming onto the shoulders of PHP, these days. Today, I’m will be showing you why and how PHP’s powers could be harnessed better and more. We, as PHP web developers, should be absolutely fully aware that we’re allowing insane amounts of processing power to do too much work when it’s absolutely unnecessary to do so dynamically.

Read More »

Posted in Development | Tagged , | 8 Comments

PHP in the web development world: are we doing it all wrong?

Some thoughts and ponderings on how “the frameworks out there” might not do it just as right as they should.

The UNIX principle applied?

Something that is currently overlooked in a large part of the PHP community is that not all design patterns are necessarily implemented in PHP code. Your webserver can be considered the front controller of your application too. Routes can be defined using your filesystem and webserver configuration. Not everything has to be Object Oriented to be well organised and well structured. To gain perspective, here are some ideas to think outside the box, and not have OO patterns drive you beyond reason.

Read More »

Posted in Development | Tagged , , , | 2 Comments

Tetris in HTML and Javascript

First, I saw the documentary Ecstacy of Order. Then I thought of great things, like building 3D Tetris in HTML5 canvas. I started it, just typing away, of course only Vanilla JS. But after a few hours, having the simple Tetris basics covered, I got bored and stopped.

But wait, there’s still hope! I thought I could write a post about me thinking of awesome stuff, getting the basics working and then never finishing it. So I started typing this post, but halfway through, I decided that the basics are good enough. And that the basics are worth posting too.

Sorry if I disappointed you. Have fun reading the code ;) The gameplay is not that interesting.

  1. The controls are A & Z for rotating, < and > for moving. You can also use your cursor keys if you use Firefox. Didn’t test it in IE, btw. 

Posted in Development | Tagged , | Leave a comment

Symfony2: A two piece puzzle.

If you’re getting started with Symfony2, you’ll get something running pretty quickly. The AcmeBundle contains some of the basic features you’ll need when writing a Symfony2 app. The basics lie in the MVC paradigm, so there is some model (Doctrine ORM), some view (Twig templates) and some controllers.

However, none of that is the real core of Symfony2. The most interesting part lies in two main components of the framework. First, the HttpFoundation component and second, the Service container.

Read More »

Posted in Development | Tagged , , | Leave a comment