What the web should really look like, or: what we might learn from cryptocoins and accountants.

Given any system, anytime, anywhere, state is a derivation of some sequence of events. Always. It is like that in the real world, as we already know that events can never be exactly reproduced without replaying them in pure isolation from whatever else that might influence it. In practice, we just try to marginalize the outer effects when trying to reproduce something. This is true for all social science as well as quantum physics and everything in between. In other words: what we see, what we observe is a series of facts, strung together by our minds as a series of events. Many philosophers have argued that causation in itself is a product of our mind. Especially Leibniz has some interesting thoughts about this. Suffice to say that a common view on causation is that the series of facts are empirically witnessed, but that any connection between these facts is a mere product of our mind: we construe causality in our minds1.

State, on the other hand, is whatever “is” at a given moment in time. A photograph, for example, is more or less a record of the state all objects were in at that point in time. Or, you could pause a video and observe this state of whatever was filmed. But state is not what’s important. Whatever is happening is important. Or, even better: whatever caused things to happen, and what actions people take might cause other things to happen. John picks up an axe. Will he murder again? Rita looks away. Will she love him ever again? Stuff like that.

Back to the digital world. We have known for a long time that state is complicated. Because if some object holds state, it means that that object can change and we must be very keen on trusting the object to either be responsible for it’s own state by receiving messages or events or being told by other objects that it’s state should change. This means that there is always a question of “responsibility”. This effectuated principles such as Law of Demeter / Principle of Least Knowledge, Separation of Concerns and the Single Responsibility Principle. In object oriented design, we tried to emphasize this aspect with stuff like Design by Contract and Encapsulation.

Lately we found out that it’s good to do Composition over inheritance, moving from complex software architecture to be more “Pure”, which means: less “Stateful”. In other words: we decided that Declarative is the new black. Yeah. That last one is mine. From a few years back. Ever since that post, people start picking up on it everywhere. Well, no… not really. Well, they do, but it had nothing to do with my influence on the world’s events. Or …. did it? Let’s be reasonable. It probably didn’t 😉 But then again: causality is not something we can be certain about; we can only guess. So my truth is that everyone is starting to pick up on Scala and Haskell. Because I mentioned it. Prove me wrong. 😉

Fact of the matter is: we turned the world inside out. We thought that state was the thing that was real and should be captured and displayed. We wanted to show € 40,- as our account balance. We didn’t bother with thinking about how it got to that amount. Whenever we heard “subtract 20,-“, we just did. That is what we did while managing state. If you’re lucky and there was time, we kept a log. The log hopefully did it’s job, when things go wrong. If we’re lucky, we can reproduce what happened.

When I talk about this in terms of account balance, you’re surely thinking: that’s ridiculous. I would never build it that way. No bank would work that way. Finance doesn’t work that way.

True. Why not? Because it needs to be correct. Any time, all the time. Correct and validated. There you go. Whenever a change is initiated for an account balance, the change is recorded and validated. Same for cryptocoins. All cryptocurrencies work that way. It is a big long event log of things that changed. And everyone is able to validate the history of events. This is not a log you keep when you’re travelling, forgetting to write down most things of what you have seen. It is a truthful representation of things that happened, that we can play back. We can go back to square one, repeat all events, and the end result will be the same.

This is what event sourcing is. And this is why declarative is the new black. And why state is evil. And why caching is ugly and unreliable. State is the result of whatever happened. A mere snapshot. A photograph. Not reality. Reality is whatever happens and has led to a certain state. State is whatever is observed at a point in time. That’s why state is true only for whoever watches.

It can not be a coincidence that ephemeral is a word that pops up more than often lately. We want to be able to replay things. We want Event Sourcing, Immutable acyclic directed graphs as the backbone of our vcs (ahem, git, ahem), binlogs, Dockerfiles, Makefiles, declarative builds. State is not interesting. How we ended up at a state is interesting. And, it only comes to mean something whenever someone is watching, or, when it starts to have an effect on anything outside it’s being. And that’s why the observer effect is probably one of the most complex things to study right now. As long as no-one watches, things are simple. They, as you probably know, can be2 in all thinkable states anyway, right? 😉

So, without observing state, or even better: construing state for the purpose of observing facts, the facts don’t mean anything. At least, not to the observer.

There are two lessons here:

  1. If you say that state is a single source of truth 3, think again. State is a representation of truth. The source of truth is events following a trusted starting state.
  2. If you keep state somewhere, can you be sure that is a result of an event? If not, either you should rely on a starting point (trusted state) or disregard it.

In other words: avoid state wherever you can. Be declarative. Go functional. It is the path to truth. No matter which direction in time you replay it. There is no fake news if you have the facts that led to a certain state. There is only history. Recorded and validated. And acted upon.

Unfortunately, we cannot reconstruct history this way. Ultimately we would end up at some point that we have to say: this is how things were at a certain point in time and trust that. We’ve already learned this from Gödel.

But let’s at least try to not screw things up while we’re watching 😉

  1. This is no doubt fairly debatable, but bear with me for a second. 

  2. Note that “being” becomes a relative term here… Being, in relation to: what? To whom? 

  3. Something that happens a lot in lovely front-end land 

Posted in Miscellaneous | Leave a comment

Privacy at the dark side of the moon

Currently, when you visit any website you can be pretty sure that there is some advertising campaign going on. Your browser is checked for cookies on some remote domain, and we have no clear way of what information this remote domain is tracking, unless we’re willing to dive into a deep pit, unraveling the darkest patterns of information sharing. If we’re not known, we’re made known instantly. You did not have a cookie before, you have it now. Unless you really know what’s going on – but even then – you have no idea that a lot of servers are eavesdropping on your every move.

Google and Facebook being two of the largest providers of advertising “big” data and gatherers of your scurrying, slowly building up a profile of whatever you do. Were you not logged in at the time of visiting the website? If you log into your Facebook or Gmail account later, the pieces can be easily put together; your previous visits to website can (and will) be linked to your current login session, allowing the marketing machines to retroactively build up your profile. And it’s not even limited to using the same session. Since it’s statistics we’re dealing with, it is not that hard to throw in stuff like browser fingerprinting, IP addresses, or even just usage patterns such as search queries, following links, etc. Correlating this to other data in some degree of “sureness”, is still valid statistical information. Statistics, after all, always allows for some error.

It’s been argued a lot around me, including by myself, that it is a service to you as a customer. It could very well match whatever you were looking for anyway, and so it is helping you to get information on what’s out there. And that is another form of doing what the web was intended for: sharing information.

But, it’s not true. It’s guided information. Any guided information, whether it was correctly or incorrectly guided, is still guided. It bears value from The One (or The Entity) guiding the information. And might very well be close to censorship: you do not know what the motivation is for presenting the information to you, and why it’s presented in the shape or form it is.

Feeble attempts are being made by governments to protect us. In Europe for example, you need to provide consent for the website linking to advertisement domains. They must provide with a concise and clear privacy statement, stating for example that by using the website, you will need to allow the website to place tracking cookies and see whatever you’re doing. This usually is presented in the form that a website is doing this for your own good, and does this only to improve your experience. That might be true to some degree. For example: on a sales website, it makes much sense for it to only present the products you might actually be interested in, based on your profile. You could explain that as a service to your customers. But, it is not confined to the borders of the website you’re visiting if all of that data is relatable to whatever you were doing on another website.

There have been attempts for browser applications to protect you. From the earliest days on, Internet Explorer and Netscape Navigator have had options to “disable cookies”, for example. Internet Explorer’s standard setting at some point even was not to allow third party cookies (e.g.: cookies from foreign domains) causing all kinds of usability issues. All of the sudden, it was no longer possible to embed Youtube videos in your website, since Youtube wants to place functional – next to tracking – cookies. It is no coincidence that Youtube has not been making money since their founding. Money is not what youtube exists for to generate. It exists to generate information.

That is what the web has become. A big pool of information on you, what you do, what might appeal to you and how you would statistically fit into a pattern that would make it quite probable for you to “convert” to whatever the advertisements are targeted at, in most cases a buyer of some service or product. You are the product.

We, the people, have no freedom. We are addicted to the Internet, and even though a statement like that could be qualified as pretentious, I’d say we’re addicted to the Internet the same way we’re addicted to transportation. There is simply no way around it, there are very few people who can actually manage day-to-day life without. Of course, you could if you really wanted to, but there is no real reason to should want to.

Historically, freedom is closely tied to independence and equality. The dependence in this case would refer to the fact that we would yield a lot of possibilities and perks if we would let go of the use of internet. For example: our governments start to move to using internet (and e-mail) as a primary means of communication. If we want to know our rights on whatever matter there is, we will probably need to find out on the internet. Inequality in this case would refer to the fact that I as a website visitor do not nearly have the amount of information on the websites and companies active on the internet as vice versa. This is not an equal game.

Where do the agencies stand? Do we have a morale, an ethic code to follow? Do we have a responsibility to the ignorant among the website users, or do we simply stand by and see how we throw most of it on the streets, for the sake of pleasing our paying customers? Do we, as the scholars of this age, turn our backs when we see commerce turning into deceit? Or should we soothe and keep at a standpoint where we may assume that there is good in man, that no man will use any information for the wrong means?

Innuendo? You decide. I think true change is long overdue.

Posted in Uncategorized | Leave a comment

Package management. No really. How hard can it be?

We had a little discussion at work some time ago, again, involving package management. No, I’m not talking about the .deb versus .rpm debate. I am talking about source dependencies for JavaScript and PHP projects. But the problem holds equally well for OS package management or build dependencies.

Currently, we are using a fine blend of Composer, Bower and NPM. Coincidentally, another one is peeking around the corner, because Composer uses a tool internally, which uses a manifest.json to identify the source files as coming from some source repository, namely github, so the package is able to update itself1.

Thank heavens, we’re at least agreeing on a common definition format being JSON. At least, the syntax of it, not the semantics…

The knights who say ‘NIH!’

NIH. The “Not invented here” syndrome. There are plenty of tools out there, but lets just do it again, but now the way we think is best. It’s called a syndrome, which should give you an idea of it’s benefits. Syndromes usually don’t constitute good health. That counts for something.

We fail to see that effectively all these tools do the exact same thing. Some file is loaded. It is checked for dependencies on other packages. This is done recursively, ending up in a list of selectable versions, then this list is narrowed down using some kind of version resolution scheme, such as preferring stable over development and explicit version dependencies. When this is complete, a list of installable packages remains which can be downloaded from some kind of source, usually some site hosting the packages or github. Not that hard, right? Right.

There is one big “but”. The description of dependencies are part of the packages, i.e., their source distribution. So now, the source tree must have some kind of declaration of which dependencies it has. And here we go. We are building a PHP project, so we must have some tool that does the job, in PHP. Also, we’re building front end javascript and sass, so we need some tool doing the exact same thing for front end assets. And, last but not least, we tend to use NPM for Javascript tooling that is not hosted in some bower registry. Hence, Composer, Bower and NPM.

Awful. We fail to see that the one thing we need to publish in our source code is the dependencies. If we can all agree on some kind of standard for this, the choice of package management can be mine, whether it is composer, npm, bower, dselect, yum, what-freaking-ever-pm I set my mind on today. But the knights who say NIH, don’t usually sit down to think things through. And they tend to squable over futilities. Me included.

Knights of the round table, then?

Here’s my thought. Let’s think of a standard. And lets use that in any package manager we use. Now, we get rid of all composer.json, package.json, bower.json and whatever Python, Ruby, Erlang or Brainfuck version there is out there and let’s just hold hands and sing a mantra:

A package manager exists to download stuff in the right version for me. Downloading is easy. Dependency resolution is easy (though it might take some time if the graph is big). Agreeing on how to make these dependencies explicit seems to be hard. But it really isn’t. All you need is a file saying “I need this package in that version”. It could look just like this:

symfony/symfony: 2.6.*
jquery: 2.*
node-sass: *@stable

Kinda looks like YML, doesn’t it. But let’s set the preferences of serialization format X versus Y aside. Let’s agree that at least one format is useful. Or, in RFC style, any package manager implementing the spec COULD support multiple file formats reflecting the same structure. It MUST implement at least one.

Now, we need some tool to translate these package names to download locations. Wait. That doesn’t need to be a tool. It could be a hosted service… No wait. Don’t we already know something like that? Like a sources.list in Debian? Or yum.repos in Fedora? Then, finally, we need a way to identify where the packages need to be stored, once downloaded. Whether that’s ./node_modules, ./vendor, /usr/local, /, or "C:\Program Files" really is just Yet Another Config Value.

We don’t need more package managers. We need a specification. A standard. And it really shouldn’t be hard. Hell, I think I almost summed it up already. Once we do that, we can let go of the NIH, get ourselves a pair of halves of a coconut shell2 and start prancing around whinnying and enjoy ourselves just a little bit better.

  1. If anyone might have read my previous post on Gödels incompleteness theorem, you might recognize the fact that adding more to the “system” not necessarily is a good idea. 

  2. This, by the way, works really well. 

Posted in Development | Tagged , | Leave a comment

Why Gödels incompleteness theorem should be software engineering 101

I am a developer. That’s why I know how us folk think. We want clean and nice solutions for problems, because that’s what we do best: provide elegant programs that solve actual everyday problems. We also tend to overrate elegancy. Especially in web development -my field of expertise-, when projects and programs are relatively small for relatively small budgets and always with end results that have a short life span, we think about reusability and tend towards abstract and generic solutions that provide more than a solution, it provides a way of life, progression for the industry as a whole, a solution to even more problems the client nor project manager ever thought of or could even imagine they would ever encounter. We are smart, we know what problems they will face eventually so we prepare them for this. Even more so, we prepare our solution for this, so ultimately the only thing we have to do is flick a switch from now on, and everything is working for everything and we never have to write one line of code again. From now on all we have to do is configure. Which isn’t like coding. At all. You know the feeling, right?

Read More »

Posted in Development | Tagged , | Leave a comment

Building OpenGL SDK: compile error “X11/extensions/XInput.h”

If you get this error:

src/freeglut_internal.h:111:39: fatal error: X11/extensions/XInput.h: No such file or directory

This might save you some time searching for the correct dev packages to install. The package it is in in the Debian repositories is called x11proto-input-dev, but it contains the file XI.h which contains the correct headers freeglut is including. Symlink the file and you’re good to go.

cd /usr/include/X11/extensions && sudo ln -s XI.h XInput.h


Posted in Development, Linux & BSD | Tagged , | 6 Responses

Composer local package mirroring: Press the pedal to the metal.

In my previous blog post, I told you about hosting local package repositories for composer. Me, or to be honest, my colleagues, weren’t too excited about the performance gains. So I decided to dig in a bit deeper.

Read More »

Posted in Development | Tagged , , , , | 6 Responses

Composer: create a local package repository to improve speed

When you’re developing a lot of symfony projects with a relatively large set of dependencies, sooner or later you’ll get annoyed by performance issues. By default, composer uses packagist.org to get package metadata (what versions are available and where to get them from). All packages are part of packagist.org by default, but 9 times out of 10, you’ll only need a fraction of that.

So, to gain performance, I decided to generate a local package repository which only contains the package info for the libraries I regulary use. Read More »

Posted in Development | Tagged , , | 1 Response

Flow charts in code: enter graphviz and the “dot” language

If you’re like me, you like gui’s as long as they don’t push you in a direction other than your train of thought. Whenever the tool tends to distract you from the task you are performing, you get annoyed. Stuff like “why am I searching for such an over-obvious functionality”, or “why didn’t they think of making the clickable area a bit bigger”, or simply “aaaarggghhh, it crashed on me again”.

Read More »

Posted in Development, Linux & BSD | Tagged , , | 5 Responses

PHP: We are getting slow and sluggish, and we’re lazy and arrogant about it.

In my most recent blog, I wrote about how I feel that too much of the world’s logic is coming onto the shoulders of PHP, these days. Today, I’m will be showing you why and how PHP’s powers could be harnessed better and more. We, as PHP web developers, should be absolutely fully aware that we’re allowing insane amounts of processing power to do too much work when it’s absolutely unnecessary to do so dynamically.

Read More »

Posted in Development | Tagged , | 9 Responses

PHP in the web development world: are we doing it all wrong?

Some thoughts and ponderings on how “the frameworks out there” might not do it just as right as they should.

The UNIX principle applied?

Something that is currently overlooked in a large part of the PHP community is that not all design patterns are necessarily implemented in PHP code. Your webserver can be considered the front controller of your application too. Routes can be defined using your filesystem and webserver configuration. Not everything has to be Object Oriented to be well organised and well structured. To gain perspective, here are some ideas to think outside the box, and not have OO patterns drive you beyond reason.

Read More »

Posted in Development | Tagged , , , | 2 Responses