Peter Müller

Content-addressable Browser Caching

Tue, 24 May 2016 17:41:11 GMT

What if it were possible to use content-addressable caching in browsers instead of URL-based caching? The increased cache hits would be highly beneficial, especially for sites depending on 3rd party libraries via a CDN. It turns out this is possible. The subresource integrity specification gives us all we need. But it turns out it's not that simple.

None of the ideas I'm describing here are new, and none of them are concieved by me. I just happened to get this idea, like many others, and tried to get the standards bodies to implement it. This lead to a lot of historical digging because much of the knowledge about this is buried in mailing lists, blog posts that only exist in historical web archives and spec maintainers' heads. I'm writing this post in the hope that next time someone gets this great idea, they won't have to do the same work I just did.

Content-addressable

Addressing an asset based on its content rather than its location is not a new thing. Content-addressable storage has been around for a long time and is beneficial in situations where the content is known not to change. Instead of a location to reference the content, you use a cryptographic hash of the content, which uniquely identifies it.

This technique should not be new to modern web developers, as we are already partially buying into immutable files with our far-future-expires caching of files that have a hash of their content as part of their file name. A great method for both getting long time caching and full control over exactly when to use updated assets, because they change identity. You have probably also used content-addressable systems with git, bittorrent or bitcoin.

But even though parts of content-addressability have been adopted in web development file naming regimes, we are not reaping the full benefits of it in regards to caching, since browser caches are based on URLs instead of content identity.

This is where a new W3C specification that most people haven't heard of comes in.

Subresource Integrity

The subresource integrity specification describes a way for browsers to verify that an asset they have downloaded is indeed the asset that was expected. The developer declaring a cryptographic hash of the expected content of the downloaded asset. When the browser has fetched the asset it compares the declared cryptographic hash to that of the assets content. If there is a mismatch between the two hashes the browser will refuse to parse, evaluate and cache the asset.

Despite the choir of people that see no value in this on Reddit and hacker news, it's actually a really useful technique to secure your use of 3rd party dependencies or even your own assets. SSL will secure your transport, but doesn't guarantee that the asset has not been compromised on the originating server. There are lots of situations where the static assets you depend on are hosted on a server outside of your control, like a CDN, or even just on a different machine than your main web server.

So in order to gain this extra security you use subresource integrity. It's tedious declaring these hashes, but you can use tooling to help you do it.

The clever reader will now have noticed that content-addressability and subresource integrity have one thing in common; The cryptographic hash that uniquely identifies the content. So this was the idea I had:

Content-addressable Browser Cache

If a cryptographic hash uniquely identifies the content of an asset and subresource integrity hashes are declared by the developer, why not use these hashes as indexes for caching the asset instead of using the URL?

The benefits of doing this would be that you could leverage the cache across domains. Say you use jQuery 2.2.1 from a CDN on your site. If a visitor visits your site and doesn't have jQuery 2.2.1 on that URL in the browser cache it's a cache miss and a download. But what if the visitor had jQuery 2.2.1 from a different URL in the cache? The hashes of these two jQuery assets are identical. So if your site and the other site your visitor got jQuery 2.2.1 from both had declared an integrity hash, this would mean a cache hit because the identity hash is the same.

Awesome idea. I am by no means the first person to have it. The earliest mentions I could find about this was Douglas Crockford describing it in 2008 on a now dead URL but summarized by Mark Seaborn in his discussion of shared caching by hash. Obviously, many people had thought about this before me. So why hasn't it been implemented?

I got some hints on the whatwg IRC channel on where to read up on discussions about the idea. These are the reasons they had not to implement it:

Timing Attacks

A timing attack is a way of extrapolating knowledge about some otherwise non-accessible piece of data by timing the duration of an operation relating to it. In the case of content-addressable browser cache the piece of information that is otherwise hidden is the user's browser history and whether the user is logged in to certain services. The method to obtain this information is to time the duration of a HTTP fetch of a uniquely identified asset that only exists on the service an attacker is out to gain information about. If the fetch returns immediately it is very likely that it was because of a cache hit. If it takes a few hundred milliseconds or returns an error it's likely that the user hasn't used the probed service recently.

Leaking this sort of information is considered a security vulnerability. Similar vulnerabilities, like checking if a link has been visited with Element.match(':visited'), have been addressed to avoid enabling potential evil third parties to involuntarily track users.

The client side version of this timing attack using a fetch-operation is not much different than the old-school <script src="https://interestingservice/asset.js" onload="trackTime()"></script>.

However, if the attacker also controlled the server the user is visiting, which is likely when the attacker can run an attack script, a timing attack is not even needed. The attacker could include the relevant 3rd party assets with the correct subresource integrity, but have the URL be from the local server. If the visitor requests the page, but not the 3rd party asset, you can conclude from the server logs that the user has a history on the 3rd party page.

Cache Poisoning

Cache poisoning is the act of injecting malicious content into the user's cache under a trusted name. Being able to substitute a jQuery cache hit on a user's browser with a malicious script would be of great value to an evildoer.

Since subresource integrity would already provide a guarantee that the content to be cached matches a developer defined cryptographic hash, a cache poisoning attack is fairly difficult to execute. The attacker would have to brute force generate an attack script that matches the same cryptographic hash as the intended target asset to impersonate. This alone is difficult. Furthermore, this would have to be done for exactly the hash function that the specific site that the attacker wants to inject their script in. Choosing a strong enough hash function for your integrity hash would make this an almost impossible task in the first place.

On top of that, the attacker would have to lure a potential target user to a phishing site that references the malicious script with the integrity hash of the intended 3rd party asset to inject itself as in the cache. And of course the visitor must not have the asset already in cache, otherwise, the poisoning attack will not work.

From my point of view, this attack is potentially devastating, but a successful execution is practically impossible. SHA-256 has no known conflicts yet, and this attack depends on hitting a conflict for a very specific piece of content.

Alternatives

The shared cache based on content identity was very promising since it would create a collaborative caching ecosystem with no investment from the browsing user, and little investment from site developers. However, there are other ways to improve shared caching of 3rd party assets.

One solution is for the frontend community to come together and standardize on a single CDN so all sites reference the same URL's, thus increasing the chances of cache hits. This makes it a people problem rather than a technical one. For that reason I see very little chance of this ever happening.

Another solution is one that the browser users themselves can opt into. The Web Boost Chrome extension maintains a list of blessed URL's of known and popular 3rd party libraries and the content of the libraries. It intercepts requests to these URL's and serves a response from the plugins own distribution. The downside is that is the extension ever gets hacked or the author becomes a malicious actor it's very hard to know and avoid for the users that have this installed.

References

Check your link rot

Mon, 06 Apr 2015 23:10:29 GMT

When we develop websites we often take the greatest care that all details are right. People are checking the design, the implementation, the responsiveness, the performance, and hordes of people are clicking all the links so ensure that everything is as it should be. We congratulate each other, high five our team, put our work in production and move on to other things.

However our perfectly crafted website is now all but a snapshot of a distant past. As we move on, the pages stick around and become a part of the ever growing public record of the internet. But even though the site was perfect when concieved it's now in a new state. Decay. From now on things can only get worse.

Link-rot

Even though Cool URIs don't change, we ocasionally do link to bad URLs. Not that we know they are bad when we create our links. Maybe they have been good links all along. But then suddenly things happen. Companies go bankrupt, people lose interest in their blogs, domain renewal is forgotten, things are deleted, URLs change in a refactoring or one of a thousand other reasons or non-reasons people have, resulting in the URL you linked to being dead.

Worst case is having a page with good information, engaging users so they want to learn more, but when clicking they end up with a 404 or 500 error. You certainly want to avoid that.

Also more subtle things creep in all over the place, and might actually have been there from the start because manual tests don't reveal them.

Redirects can cause slow downs in page navigation. If you are not careful they might even slow down your own internal page navigation, or even worse your static assets. Having 301 or 302 redirects to your own navigation is detrimental to your page performance as you are essentially doubling or tripling each page requests time to first byte, also known as the bottleneck that causes your user to become impatient and switch context. If you have succeeded in actually showing your users the content they were looking for and they are makign the investment to navigate to other pages, you certainly don't want to keep them waiting.

Dealing with redirects

Some redirects are good and some are bad. As a general rule, if you and everyone else uses redirects correctly you should never have a link to a 301 redirect. 301's are supposed to be permanent, giving you a guarantee that the content will always be foudn at the redirect target and never at the URL you requestted. In the ideal case you should rewrite the URL in your html immediately when a 301 is encountered to speed up navigation.

302 temporary redirects are generally tougher to deal with. Try to reduce the amount you have inside your own site, but some probably can't be removed entirely. If you are running a multi language site you probably want content negotiation on your site root respond with 302 to redirect the client to the language specific version of your site. If you have a series of multiple 302 redirects you should take a look at your server setup though.

302 redirects by external parties are generally difficult to reason about and you probably can't avoid linking to them unless you do some research to figure out what the other party is doing and if it's safe to link directly to the redirect target.

There are lots of cases where it's easy to cause accidental redirect chains when we create web pages. Some times we type http://example.com in the URL and it gets redirected to http://example.com/ (trailing slash). That one is most certainly a pure waste of your visitors time.

Another common one these days is the target switching to TLS on their sites, causing a redirect from http to https. If you are really unlucky you'll hit a chain of http -- 301 --> https, no trailing slash -- 301 --> trailing slash, / -- 302 --> /en/. The first two could be completely avoided.

Tools to the rescue

Dealing with the thousands of links we all generate over a full career as web developers or copy writers is an impossible manual task. We need to use tools to help us keep our references fresh.

While dealing with this problem myself I wrote a quick tool to detect broken links. It turned out that I ended up getting a lot of extra functionality almost for free, so it wasn't much hassle to also start detecting redirect chains like the inefficient one from the last example. I figured that I am not the only one dealing with this problem, so I decided to finally put some polish on what started out as a quick hack, so it can be used by others as well.

The tool is Hyperlink.

I used Assetgraph, which I know quite well, to set up the scraping and static analysis of all pages and assets. Hyperlink starts our by populating your internal pages and assets, where internal is defined as anything on the same domain. When that part is done it starts scraping all outgoing URLs. Any warning or error encountered on the way is treated as an error condition, including inefficient redirect chains.

The current version (2.0.0 as of this writing) only looks at inefficient redirect chains for outgoing links, but an upcoming version will look at inefficient redirects in all types of URLs.

I decided to make the output of Hyperlink follow the Test Anything Protocol, which looks like this:

TAP version 13
# Crawling internal assets
ok 1 loading https://mntr.dk/
ok 2 loading https://mntr.dk/favicon.ico
ok 3 loading https://mntr.dk/assets/images/favicon/152.png
ok 4 loading https://mntr.dk/assets/images/favicon/144.png
ok 5 loading https://mntr.dk/assets/images/favicon/120.png
...
not ok 123 URI should have no redirects - http://www.milwaukeepolicenews.com/
  ---
    operator: error
    expected: "200 http://www.milwaukeepolicenews.com/"
    actual:   "ECONNRESET http://www.milwaukeepolicenews.com/"
  ...

The above example is reduced, but both shows successes and an error. If Hyperlink encounters any errors the exit code of the process will be the number of errors encountered. This means Hyperlink fits well into command line chains, Continuous Integration setups and similar tooling.

The TAP output is very close to human readable, but can be further improved by piping the output to any of these wonderful TAP formatters: tap-colorize, tap-difflet, tap-dot, tap-json, tap-min, tap-nyan, tap-spec, tap-xunit

Formatting can be done like so: hyperlink https://mntr.dk | tap-colorize

Jenkins integration

Obviously running command line tools manually is only fun the first few times, then the novelty subsides and you'd rather do other things. Luckily these days we can get robots to do our dirty work. Where I work we use Jenkins, but any CI setup should be able to easily do the same. Here's how I set up an automated task with Jenkins.

I want to keep my company site links fresh, but I don't need to run link checks on every deploy. On the other hand link rot can happen even when my site is not updated, so the ideal strategy for me is to set up a periodical build, which I chose in the "Build Triggers" section of my task. I chose to go with a daily check at 5am, a time where our CI is usually not busy. The cron schedule for this is simple: H 5 * * *

The "Build" section of the task is pretty simple. It's a single "Execute shell" block with this content:

npm install hyperlink tap-dot
node_modules/.bin/hyperlink https://podio.com/site -r | tee site.tap | node_modules/.bin/tap-dot

The only little piece of magic going on here is in tee site.tap. Tee is a very useful little tool that lets you write stdin to a file, while also piping it to stdout. This enables me to save the TAP report for later analysis but also get a useful log output in the log section of the running task. In my case I only care for the failing tests and to see things are running, so tap-dot is fine for me in the log output.

In the "Post-build Actions" section I added a Publish TAP Results block and configured it with site.tap. This generates a decent looking report I can use for later. This also generates a nice graph for me on the tasks status page, giving me an overview of changes over time.

I haven't set up any alert actions yet, as this is pretty new and there still might be false negatives that need to be weeded out. But it's a very good starting point and was quick to set up. So now I have a daily report of broken or inefficient links on my site including the ability to detect changes over time.

Looking into the future

I'm hoping to leverage the fact that Assetgraph is already modelling redirects as assets. This will let me detect all redirects from all URLs in the entire dependency graph. Eventually this could mean that it would be possible to either suggest patches or actually directly update the source files that have inefficient URLs, assuming they are static files and you run Hyperlink against the local files.

There are a few improvements to be done still. Most of them are about strengthening Assetgraph to support this specific use case better. It might be useful to be able to have the output copied to your clipboard for later use if you run the tool manually for example.

What other ideas do you have for a tool that helps you keep your link rot at bay?

Getting started with Assetgraph

Wed, 02 Jul 2014 00:15:00 GMT

When presented with the challenges of web performance optimization, or any other kind of manipulation of web sites and assets, it is helpful to have a good set of tools at your disposal. Assetgraph aims to be exactly this. A high level interface for your websites, yet still providing low level interfaces for individual assets. A toolkit that lets you build your own tools that fit very specificly to your individual needs.

I have spoken at lengths about how Assetgraph distinguishes itself from other build tools by not just being another unix tool configuration wrapper. In the following I will assume you have already heard me sing Assetgraphs praises. If you haven't, watch my talk from EmpireJS.

Assetgraph is a node module and this post assumes that you are relatively comfortable writing and executing node scripts. By the end you should have learned enough about Assetgraph to get your hands dirty and write your own tools with it.

If you want to see how easy it is to build tools that filter out unused files, inlines your images or rename files for optimal caching, you are in for a treat!

If you are more into just consuming a well tested out-of-the-box build tool, take a look at assetgraph-builder or its grunt-wrapper grunt-reduce.

NOTE: THIS ARTICLE DESCRIBES ASSETGRAPH VERSION 3, NEWER VERSIONS ARE AVAILABLE.

Assetgraph Vocabulary

Before we get started it's useful to get some vocabulary straight. If you're not easily confused you might want to skip to the part where you get your hands dirty.

Like many bigger projects Assetgraph has some project specific vocabulary. We've tried not to be too magical about the terms we chose, so hopefully you'll get what things are from their name. Sometimes the inherent properties that go with the names are non-obvious though. This is an attempt at an explanation.

Asset

An Asset in Assetgraph is a model of the contents of a file including its metadata. Assets have a bunch of base properties that are relevant to all asset types, like content-type, url, file name, file extension, loaded state, inlined or not. All Assets have a rawSrc getter and setter, giving you direct access to the raw file behind the asset. They also have a bunch of convenience methods like md5Hex, clone and replaceWith, along with a populate method to parse and find outgoing relations in the source code of the asset.

The most interesting things happen in the Asset constructors for more specific data types, like Html or JavaScript, where each Asset instance also has a highlevel instance of the Assets types data model. For HTML this is the DOM, modelled with jsdom. For JavaScript it's the uglify-js AST.

Using these highlevel interfaces you have the ability to manipulate each assets as you see fit, using familiar highlevel abstractions you would also find in the browser.

You might want to take a look at the full list of already implemented Asset types.

Relation

A Relation in Assetgraph defines the edges of the graph. They bind the Assets together and define what depends on what, and from where. Relations not only keep track of which file needs what other file. They also keep track of where exactly the relation came from. Be it a Html Script node src attribute or a CSS background image url token. Relations automatically update references when Assets move around, making the dependency graph stable at all times without broken links.

Relations have type, to, from and href properties that are highly relevant when querying the graph for them.

There are a bunch of convenience functions available for more lowlevel graph manipulation, which I'll skip for now as this is just an introduction.

Here is the full list of Assetgraph Relations.

Query

The Assetgraph Query model is a MongoDB inspired model where you can combine some simple boolean AND, OR, NOT statements into a powerful expression to specificly target only the Assets or Relations you are interested in.

Each query is a function that matches the properties of each Asset or Relation with the corresponding property or the object in the query. The query object can use strings, numbers, arrays, regexes, user defined functions and even recursive query objects to match the subject.

Some examples:

var query = require('assetgraph').query;

var simple = query({
  type: 'Html',
  isLoaded: true
});

var boolean = query.or(
  {
    url: /^http:/
  },
  {
    fileName: function(fileName) {
      return fileName.length > 8;
    }
  }
);

var nested = query({
  type: 'HtmlAnchor',
  from: {
    isInline: false
  },
  to: {
    isImage: true
  }
});

Most interactions with Assetgraph happen trough queries to the graph. It is recommended that you get to know the query model if you want to be an effective tool maker. Often times you'll see simple query objects being passed into Assetgraph methods or Transforms without the query() call. Assetgraph will automatically turn such objects into queries for your convenience.

Take a look at the Assetgraph Query source code.

Assetgraph

The Assetgraph is the instance that ties all of the above together. This is where the Assets and Relations are stored and where you can use a Query to find them all again. There are a bunch of convenience methods for pre-order and post-order traversal and of course the most used ones findAssets and findRelations.

Transform

Assetgraph is cool, but always diving into the lowlevel code of specific Asset syntax trees becomes bothersome pretty quickly. Transforms are highlevel functions that wrap these lowlevel calls, manipulating the Assetgraph instance in more convenient ways. Assetgraph is extensible, so you can write your own highlevel transforms that fit your specific needs. Assetgraph already comes preloaded with a lot of very useful transforms, most of which are written with web performance optimization in mind, but don't let yourself be limited by that!

There are some fine descriptions of most of the available core transforms in the Assetgraph README.

Transform Queue

The Transform Queue is what lets you chain a number of transforms together to form a pipeline that applies multiple transforms in order. While all installed transforms are available directly on the Transform Queue and Assetgraph instance for you convenience, they all return a reference to the Transform Queue they are in, enabling you to easily chain Transforms. There are a few convenience methods on the Transform Queue, like if, endif and queue (for quick inline transforms), the most important one is run.

If you don't .run() the Transform Queue, nothing will happen.

Minimum Assetgraph Lifecycle

While tools have great diversity, there will always be some common boilerplate that needs to be written in order to bootstrap them. The same goes for Assetgraph-based ones. This will be an explanation of how a bare minimum setup will look with Assetgraph.

First, it's important to remember that Assetgraph can only work with websites if their internal references are valid. This may sound like an obvious best practice, since that is the only way a website can actually be loaded in a browser. Sadly I need to point this out, as most existing web performance build chains actually set you up with non-working websites, that are then assembled by the tool or some configuration of a static file server. If you want to get the most out of Assetgraph, build your websites so they work in the browser with no magic in between. Incidentally this simplifies your workflow greatly and lessens the pain for front-end developers considerably, so I consider it best pratice.

Now, let's get started. Consider a website structure like this one:

app/
├── css
│   └── main.css
├── js
│   ├── main.js
│   └── jquery.js
├── images
│   └── logo.png
├── index.html
└── favicon.ico

Your web application is in a directory called app and you have som pretty basic scaffolding done already. A simple start for a simple introduction. Note that this is just an example. Assetgraph makes no assumptions about directory structure as it will infer it from your source code.

We start out with creating a script that can load the website into an Assetgraph instance:

var Assetgraph = require('assetgraph');

var graph = new AssetGraph({
  root: 'app'
});

The above creates an Assetgraph instance, configuring it with app as the web root. The root must be a string, and may be any valid file://, http://, https:// or ftp:// url. If no protocol is specified, a location on local disc is assumed, and the path is resolved as you would expect on the command line.

An Assetgraph instance in itself, without any data, is quite useless. So next up we're interested in actually loading data from our website into the graph. We can do this using the loadAssets transform. Loading your index.html into the graph can be done like so:

graph.loadAssets('index.html');

The loadAssets transform takes a range of inputs to make your life easier. The most useful to you now will be the string or array of strings. Each string, just like before, may be a full url or a protocol relative url in the previously described schemes. All relations in the graph will use the Assetgraph root to resolve paths, not the file system root. If you want to get more advanced with the loadAssets transform it might be useful to consult the source code for now.

Before we can run the script there is one more piece of boilerplate code that needs to be added. What we are doing when calling Assetgraph transforms with configuration parameters, is actually not executing them right away. Instead, we are appending them to a transform queue, which is what is returned from the transform call. To make this explicit in this example we save the return value in a new variable:

var queue = graph.loadAssets('index.html');

All transforms in the queue are run in the queue scope and will return the queue, making transforms chainable. All transforms will have the assetgraph instance passed to them as the first parameter. The loadAssets call you just added to your script, won't actually be run before the transform queue has been started. We do this using the run method:

queue.run();

This implementation detail is a bit counter intuitive and can bite you later, so make a note of it now and I will make a note on improving the API. If it hasn't changed by September 2014 you are hereby mandated to bug me about it on Github.

You can now run your script, and index.html will be loaded into the graph model. However this is not terribly exciting yet, since all that happens is reading a file into memory and not logging any output. So let's make it a bit more exciting by adding some logging to the console.

Logging and Debugging

Setting up logging is done on the Assetgraph instance, meaning it goes before the transform queue is run. Your script now looks like this:

var Assetgraph = require('assetgraph');

var graph = new AssetGraph({
  root: 'app'
});

graph.on('addAsset', function(asset) {
  console.log('addAsset', asset.toString());
});

var queue = graph.loadAssets('index.html');
queue.run();

We're hooking into the Assetgraph emitted event addAsset and logging some basic information about the event and the asset that was added to the graph. Try running your script now, and you should actually see some output in your console. There are more events you can hook into, to get some more insight into the internals of Assetgraph: addAsset, removeAsset, addRelation, removeRelation, beforeTransform, afterTransform.

Furthermore there are some different error levels that are especially useful to hook into in order to get some more useful information for debugging your code: info, warn, error. These last ones are conditions of increasing severity.

info is usually used when Assetgraph sees a potential error situation in your web application code, but a fix has already been applied. This could be trying to bundle scripts where one or more of them are leaking strict mode to the global scope, or exceeding the IE max style sheet rules number. Don't worry too much about info events.

If you see warn events you should take note, as these usually describe problems in your web application that have to be fixed by you. Things like parse errors or missing files that would cause a 404 response in production etc.

The error event is the most severe. This is usually only emitted when a completely non-recoverable error has been encountered, or a library throws unexpectedly. It usually makes sense to just stop running your script if you hit this one. It's also likely that when you get error level events that we'd like to hear about it, as it might indicate a missing error handling in Assetgraph. Please report these to us.

Let's spice it up with one more logging detail, writing some stats about the assets contained in the graph to stderror:

queue.writeStatsToStderror();

Populating the Dependency Graph

We've now arrived at the core functionality of Assetgraph. The arguably most powerful functionality is the ability to automatically and recursively traverse the dependencies of loaded assets. This, as they say, is where the magic happens, and what enables you to work with your web assets in their entire context without having to define large manifest files telling your tool what files you want included.

We are using the populate transform. The transform can be configured in a multitude of ways, in order to describe how to traverse the dependency graph, what to load, and more importantly, what not to load. Think of this as a web scraper. Let it scrape everything and you might end up copying the internet, so use with care.

Available options are:

{
    followRelations: <Assetgraph.query>,
    startAssets: <Assetgraph.query>,
    from: <Assetgraph.query>, // Alias for 'startAssets'
    stopAssets: <Assetgraph.query>,
    concurrency: <int> // Number of concurrent requests. >= 1
}

It should be obvious by now that it is useful to get to know the query syntax.

For now I'll assume that we are working with a web site on local disc and that we are only interested in loading assets into Assetgraph that are local as well. So we want to configure the populate transform to only follow urls that are relative or root relative, while excluding the ones that are absolute or protocol relative (ie. on a different domain).

We can do this like so:

queue.populate({
  followRelations: {
    hrefType: ['relative', 'rootRelative']
  }
});

This makes your final bootstrap script look like this:

var AssetGraph = require('assetgraph');

var graph = new AssetGraph({
  root: 'build'
});

graph.on('addAsset', function(asset) {
  console.log('addAsset', asset.toString());
});

var queue = graph.loadAssets('index.html');

queue.populate({
  followRelations: {
    hrefType: ['relative', 'rootRelative']
  }
});

// Put further transforms here

queue.writeStatsToStderr();

queue.run();

Congratulations, You have now successfully loaded the entirety of your local files, which you are referring to in your source code, into Assetgraph. You are now bootstrapped with all you need in order to work with your assets.

Normally a starter guide would end here, but I'll throw in some quick examples that might be of use to you, just so you get some ideas of what is possible.

Writing files to disc

Reading files from disc to memory is fun, but it's even more fun writing them back to disc. Assetgraph lets you rework your dependencies and assets in memory in the transform queue, but the only way you gain anything from that is by actually saving the changes.

To write your files back to disc we'll use the writeAssetsToDisc transform. The first argument is a query object, which we'll make pretty broad, only limiting it to assets that have been loaded (trying to write unloaded assets to disc won't work anyway).

The second argument is the root directory to write the files to. You can leave it blank, which will fall back to the Assetgraph instance root, meaning you are overwriting the files in their existing location. Might be useful, but normally you want to separate your source files from your output files. We're setting a new outRoot demo.

queue.writeAssetsToDisc(
  {
    isLoaded: true
  },
  'demo'
);

Congratulations, you have now copied all your referenced assets from one directory to another. While you think this might as well have been accomplished with cp -r app demo, the important point to note is your referenced assets. If you haven't somehow included a file on your website, it won't be in demo. Imagine how many unreferenced files get put into production every day by people forgetting about them. Even worse, imagine a bower component has a malicious php file with a rights escalation somewhere in a test directory, and you just copied it to your production server.

So see this as a useful step to only get the essentials on your production site. If something is missing now it's because you didn't refer to it. This could easily happen with error pages, favicon.ico, robots.txt or similar. If you want unreferenced files to explicitly be a part of the graph, make sure to include them in the loadAssets transform.

Inlining small images

Base64 encoding and inlining images. A tedious and stupid workflow. In development you want to work with your raw images to make it easier to maintain and debug, while in production you want to reduce http requests by bundling or inlining. Automation is the way and Assetgraph can help.

We'll limit ourselves to only images that are CSS backgrounds, as inlining content images requires some more specific knowledge of the context. I choose a completely arbitrary number for the file size of images to inline: 4096 bytes. Feel free to experiment on both accounts.

We're using the inlineRelations transform, which is dead simple. The only thing that is happening here is just a more complex query than I've shown before.

queue.inlineRelations({
  type: 'CssImage',
  to: {
    isLoaded: true,
    isInline: false,
    rawSrc: function(rawSrc) {
      return rawSrc.length < 4096;
    }
  }
});

File revving

File revving is a fancy word for revisioning files for optimal caching in the visitor's browser. The optimal way to serve static assets is with a far future cache expiry, since the fastest request is no request at all.

The optimal revisioning strategy is including a hash of the file content in the file name. If the file hasn't changed, the hash isn't changed, giving you the ability to optimally use the cache of the visitor's browser for unchanged files and only load the ones that have changed since the last visit.

The easiest way to set up cache headers for static assets is to put them all in the same directory, where the web server will append the correct cache header to the HTTP response.

If none of this made sense, then I highly encourage you to read up on optimal asset delivery for browsers. Your users will thank you.

This is by far the most complex example, but I want to show it because this is a place where Assetgraph shines compared to other build tools that do not have a dependency graph model at their core.

Our strategy for renaming the files in the right order will be post order traversal, renaming leaf nodes in the graph before branch nodes to assure that the hash we calculate is actually based on the correct file contents including hashed references to decendants.

First we need to craft a query that will only rename the files we want renamed. Some files might be static, but we still want them to keep their original url and take part in a different caching scheme, designed for rapid updates. Think of HTML pages, RSS feeds etc. I have come up with this query combination to target only the files that are safe to rename:

var query = AssetGraph.query;
var moveQuery = query.and(
  // Find all loaded an non-inlined assets
  // Except ones of the defined types and fileNames
  {
    isLoaded: true,
    isInline: false,
    type: query.not(['CacheManifest', 'Rss']),
    fileName: query.not(['.htaccess', 'humans.txt', 'robots.txt', 'favicon.ico'])
  },

  // Exclude HTML-files that are linked to
  query.not({
    type: 'Html',
    incomingRelations: function(relations) {
      return relations.some(function(rel) {
        return rel.type === 'HtmlAnchor';
      });
    }
  }),

  query.or(
    // Exclude initial assets from renaming
    query.not({
      isInitial: true
    }),
    // Include external HTML templates
    {
      type: 'Html',
      isFragment: true
    }
  )
);

The above is a distillation of a few years of iteration to try and define best practice for the most common use cases. I'd love to go into depth on this, but that's certainly not fit for a beginners guide. Copy paste this for now and return when you are more comfortable with your understanding of the graph model and the query model.

Now all that is left is to run the moveAssetsInOrder transform, which does our post order traversal for us. It takes a query as the first argument and a function as the second. The function is called once per asset and the expected return value is the new file name of the asset.

We're moving all revved files into /static and appending the first 10 chars of the hash of the file to the file name.

queue.moveAssetsInOrder(moveQuery, function(asset) {
  var targetUrl = '/static/';

  return targetUrl + asset.fileName.split('.').shift() + '.' + asset.md5Hex.substr(0, 10) + asset.extension;
});

Seems pretty easy right? Just like copying. Except, not. This is where the graph model comes in handy. When moving files by giving them a new url, all their incoming relations automatically get updated. So all the places where the old file name was referenced before are now correctly pointing at the new location. If you've ever tried to do this with unix tools, out of context of the website as a whole, you will know what a difficult feature this is. But here you have it, implemented in very few lines of code.

Now all that is left to do is configure your web server to serve all files from /static with far future expires cache headers. Look up how to in your relevant server manual or on StackOverflow.

Summing up

You've hopefully learned a bit more about Assetgraph now and are ready to get your hands dirty and try out new stuff on your own. At the very least I hope you've gained an understanding of the strengths and weaknesses of the paradigm and the toolset, so I am looking forward to getting grilled with relevant questions here, on Twitter or Github or on a conference we both attend over a beer ;)

I'm always asked for comparisons with other popular tools, like Grunt, Gulp, Broccoli or similar. Assetgraph is not one to one comparable, as is primarily focuses on fully functional references, while the other tools primarily deal with files. This enables the other tools to do whatever, while Assetgraph needs your page to actually work before you can unlock the full potential. This makes Assetgraph wel suited as a post processing tool that you apply to the already assembled page. If you use one of the other tools to achieve this assembly is up to you.

I'm also often asked about run time speed. Assetgraph is generally faster than Grunt, due to the limited file I/O. Assetgraph is generally slower than Gulp, since Gulp has you predefine your files and can run pipes in parallel while Assetgraph has to discover the files incrementally and runs transforms sequentially.

If you wish to use Assetgraph for web performance optimization it is my recommendation to make it a part of your deployment step, not your development loop. Web performance optimization is about transforming code to be machine optimized, while your development loop is about optimizing for humans. When you move these concerns out of the development loop you will see that your development speeds up, and the time it takes to run a build suddenly is of much lesser importance.

This is the final compilation of all examples, now prettified a bit:

var AssetGraph = require('assetgraph');
var query = AssetGraph.query;
var moveQuery = query.and(
        {
            isLoaded: true,
            isInline: false,
            type: query.not([
                'CacheManifest',
                'Rss'
            ]),
            fileName: query.not([
                '.htaccess',
                'humans.txt',
                'robots.txt',
                'favicon.ico'
            ])
        },
        query.not({
            type: 'Html',
            incomingRelations: function (relations) {
                return relations.some(function (rel) {
                    return rel.type === 'HtmlAnchor';
                });
            }
        }),
        query.or(
            query.not({
                isInitial: true
            }),
            {
                type: 'Html',
                isFragment: true
            }
        )
    );

var graph = new AssetGraph({
        root: 'build'
    });

graph.on('addAsset', function (asset) {
        console.log('addAsset', asset.toString());
    })
    .loadAssets('index.html');
    .populate({
        followRelations: {
            hrefType: ['relative', 'rootRelative']
        }
    })
    .inlineRelations({
        type: 'CssImage',
        to: {
            isLoaded: true,
            isInline: false,
            rawSrc: function (rawSrc) {
                return rawSrc.length < 4096;
            }
        }
    })
    .moveAssetsInOrder(moveQuery, function (asset) {
        var targetUrl = '/static/';

        return targetUrl + asset.fileName.split('.').shift() + '.' + asset.md5Hex.substr(0, 10) + asset.extension;
    })
    .writeAssetsToDisc({
        isLoaded: true
    }, 'demo')
    .writeStatsToStderr()
    .run();

I hope you found this introduction useful and now have some inspiration to get started with your own tools. I bet you can come up with some amazing ideas I've never thought about.

I love the Unix philosophy, but...

Sun, 23 Mar 2014 21:05:29 GMT

I attended the Edge Conference in London, for which Andrew Betts should receive high praises.

During the build system panel some remarks were made about the Unix philosophy in response to a question I asked. I did get a counter argument in, but the available time only allowed for tweet size comments, so I don't think I got my point across properly. I'll try to go a bit in depth on my reasoning here.

My question was this (somewhat rewritten by the moderator):

Most build scripts, including those generated by scaffolding, are a chain of several discrete steps. Do we need the complexity of having control of every step? Should we instead have opinionated build chains that just get us to our goal instead?

Most of the panel responded that they like tools that follow the unix philosophy. Small single purpose tools that do one thing only, do it really well, and are chainable using Unix pipes (or a comparable implementation).

I agree with this sentiment. I love these tools as well. I live on the command line, and never would have if these tools weren't so powerful and versatile, individually and in combination.

So the feedback from the panel was that a proper set of Unixy tools put together in the right order would be a better solution than an opinionated tool which keeps some of the internals hidden, reducing the amount of control and adding an aura of magic. Developers like control and dislike magic.

And this is where I feel the need for nuance. The simple tool chain works extremely well in cases where you have one file as input and one as output. It might have been through all sorts of compilations and minifications on its way through the pipe, but it's still just text. And text fits really well into the Unix philosophy. The same is true for a collection of files of similar type, the concatenation case.

These simple chains are great for tasks like pre-processing. Going from one type of source file to another, maybe even concatenating. It's fine for your development loop, where you have to keep things simple, maintainable and fast.

The production build case is an entirely different and much more complex beast.

In a production build case you are not modifying single files, or collections of files of the same type. You are manipulating a web of files, a dependency graph of nodes (files) and relations (href, src et al). If your build chain does not keep these relations intact throughout the process, your data integrity is broken, your web of files turns into a bunch of files without relation.

Since the Unix pipes standardize on a very simple interface, raw contextless text, they are an incredibly bad medium for keeping the relational integrity of this dependency graph intact. All of the currently popular build tools suffer from this problem, whether they write temporary files to disk or do incremental compilations in a stream. None of them are able to keep the integrity of the graph alive in a pipe, so they save state in temporary files.

You see the symptoms of this in some of the grunt tasks that, without realizing it, deal with graph problems. Problems with the file revision renaming tasks because the build chain loses the references to the files, forcing the use of regexp replacement, using patterns that do not match relative path references correctly. Problems with concatenation tasks because the inclusion order isn't taken into account, thus forcing the users to write manifest files, a duplication of information already available in a functional web page.

There are many more examples of bad pratice workarounds that plugin authors have to make because the Unix philosophy doesn't give you enough freedom of expression to work with data models that can be kept intact. Combine all of these hacks and workarounds and you get a 600 line grunt configuration file that might just work, but is quite difficult to understand.

I use Grunt as an example because it is the oldest and most widely used, but also because the problem's visibility is proportional to the configuration file size. None of the other competitors like Gulp or Broccoli have solved the dependency graph integrity problem. They might have simpler configuration, which I am a big fan of, but they require the same hacks to unbreak the state of individual assets built out of context.

The big problem with all of these approaches are that the transitional states that occur in the Yeoman/Grunt _tmp-directory are very often broken webs. They are files out of context, and might not even be servable as a web page in themselves at any given point in the build chain. This adds more complexity for the user of these tools, since it is left up to the user to configure how to get from a state of broken web to a state of functional web. Many of these transforms are non-interchangeable, non-reorderable and can generally break easily if misconfigured slightly. So do we really need fine grained control of each of them?

After the Unix tool response by the panel, I mentioned powershell because it allows to pass more than text in it's command line pipes. This is an amazingly powerful concept that enables developers to pass on entire data models with all their references intact.

While I usually hate on Microsoft for providing a pretty bad development platform out of the box on Windows, this thing is a real gem. One that is underappreciated and underutilized.

When we created Assetgraph, we set out to create a data model that could describe a web page in its entirety. It would be a platform for making all of these complex code base transformations easy, while still giving you a guarantee of dependency coherence. However, piping this data model through a Unix pipe would be the death of it, since all state would be lost.

Had Unix pipes provided capability of piping objects instead of only text, we wouldn't have had to create an entire plugin and pipeline architecture, forcing users to write things "the Assetgraph way". We wouldn't have to be competitors with "the Grunt way", "the Gulp way" or "the Broccoli way". We could instead have focused on strengthening the core data model and let users write decoupled Unixy functions that made use of this data model, adopting a philosophy of "web in, web out".

All of these implementations are workarounds to the limitations of the Unix pipe. Hack by hack we've dug ourselves deeper into our individual holes and have trench wars about which hack is best, yet we still salute Unix pipes as the end all be all of data transformation.

So, I love the Unix philosophy, but it leaves some things to be desired for the use case of front end building. The pipe is simply too restrictive.

PUSH, the post GET paradigm

Mon, 17 Mar 2014 02:25:29 GMT

Web performance these days is non-optional. If your web page is slow, you lose business. Our visitors are an impatient lot, and though they are not actively counting milliseconds, web developers have to, just in order to keep visitors' thoughts away from cat videos and flappy bird.

In our hunt for milliseconds we, the web developers, are going through a great ordeal just to keep up. We are following the 14 rules, we are fighting off marketing's ~~bigger is better~~ and we are setting up automated torture machines for our code. All for the milliseconds, all to keep our visitors' attention.

Needless to say, reshaping, contorting, torturing your code in order to conform to the 14 rules, also leaves you with something that is utterly unapproachable from a developer's standpoint. Well optimized production grade static assets are almost as far away from good development practices as you can possibly get. One module per file? Forget it, too expensive. Each image as a separate file? You must be crazy, go decode this base64 or pull out an image editor to reconstruct it from a sprite. And that JavaScript error you're trying to debug? Start unwrapping uglified concatenated code in your head, under time pressure of course; production is down you know...

Unmaintainable. Complex. Prone to errors.

We are being held hostage by the protocols that serve us. I hope you haven't developed Stockholm syndrome by now.

All hail SPDY/HTTP2

Our savior is here. SPDY promises deliverance from all the hardships we have endured. In the future we will all have gzipped HTTP headers, multiplexed HTTP streams mitigating a lot of hand shaking and slow-starts. All of this with keep-alive that works properly. Oh the rejoicing. And all that for just the cost of adding SSL.

Wait, what?

Oh yes, no SSL, no SPDY. So we're getting a lot of nice enhancements, but are also forced to accept the added overhead of a TLS handshake, a slow ordeal (tweakable though).

So suddenly it comes down to a tradeoff. HTTP handshake overhead versus TLS handshake overhead. Which one to choose depends on the nature of your site, and that may change over time. So no simple answers here. Jimmy Durante seems to sum up this situation pretty well.

And even in this new world of SPDY, our old rules of minification still apply. Concatenation still yields fewer requests. And while the overhead of each request is lower with SPDY, since we're multiplexing into the existing stream, latency isn't a thing that magically vanishes.

There may be a way to change the game though.

NOTE: HTTP/2 does not require SSL.

PUSHing the limits

SPDY offers us an interesting new tool, PUSH streams. Your SPDY enabled server is suddenly able to initiate new multiplexed streams during existing requests, sending stuff the client hasn't even asked for... yet. While this may sound a bit ominous, the gist of it is that if the server knows what assets the client is about to ask for, it is able to initiate a PUSH stream of those assets, storing them directly in the browser cache before the browser even knew it needed them.

That latency I told you about before, caused by GET request round trips? Magically vanished!

So SPDY PUSH is great news. Really great actually. If we can eliminate the overhead of initiating a GET request for an asset, the number of assets suddenly isn't an issue. If the number of assets isn't an issue, why even bother bundling?

Remember those build pipelines I talked about before? Unmaintainable? Complex? Prone to errors? Gone!

That's right. We don't need them any more. With all the latency overhead in transferring multiple assets gone, we can rethink our production code. You still want to minify code, but you can keep files separate.

Our HTTP optimized build systems bundled code in as few files as possible. The previous best strategy was to bundle and serve all static assets with a far future expires header, hoping for a cache hit on the next visit, and thus not even triggering a GET request for the assets at all. A good strategy under the circumstances. However, a single change in just a single module will change the entire bundle, thus triggering a re-download and transferring more bytes overall, with more GET round trips needed.

But imagine serving every asset individually. Suddenly a change to a single file means you only need to push that specific file to the browser cache. All the rest of the assets, the ones you haven't touched? Still cached. No re-download.

So you might rightfully ask, how can the server know if the browser has an asset cached already if the browser doesn't initiate a request for the asset, not sending any request headers for that particular file? Good question. It can't know. Instead the server just opens the flood gates by starting a PUSH stream for every asset you might need, and starts pumping. For the server and the performance hungry engineer, it's all about saturating the bandwidth as much as possible and getting that data over the wire.

Each PUSH request starts with a header, though. Properly configured, say with an ETag with a last modified timestamp or hash of the file content, the browser knows within the first few packets if it needs the rest of the transmission or not. The browser can simply hang up that particular stream. So the server might push a few packets in vain and stop when the browser tells it to. It's an optimistic and greedy approach. But it does leverage the cache properly, and the extra spent bytes don't cost a huge time overhead, since the stream termination request goes over the wire in the same time a normal HTTP GET round trip would take just to request some initial data from a server. It's a tradeoff, lower latency for higher data transfer.

So this is the theory. HTTP GET round trip latency removed, improved bandwidth saturation, better cache reuse. Assuming the server knows what to send.

So how do we teach our servers about that?

The context aware static file server

The bad news: there is no specified configuration format for telling your server what to push when a specific file is requested. The Nginx team at least came up empty handed when asked about it. So we are on our own for now. However many of us are running some sort of server that we have fine grained control over. Frameworks like express certainly make modifications like these easier.

So let's look at some strategies for making your static web server context aware. There are a few I can think of that could be used to seed a web server with enough contextual knowledge to start leveraging SPDY PUSH.

Referrer header mapping

Imagine that your server uses the referrer to record the relations between files over time. HTML --> CSS --> Image and so forth. Gradually the server can build up a relation graph and thus predict a set of other assets likely to be requested next when seeing a GET request for an HTML file. The downside here is that this functionality may take some time to warm up, and while it does, your visitors will be paying the price of multiple request latencies.

HAR file mapping

You might be able to build a server that knows how to interpret HAR files. These files are basically what you are looking at when using your favorite browser developer tools network tab. So you might be able to set up some preproduction step that automates a headless webkit to scrape all your pages and export a series of HAR files, which the server can use to reconstruct a dependency graph of your files.

Static analysis

This one is my favorite, and where I get to pitch one of my personal projects. Assetgraph can be used to statically analyze your web assets and create a graph model that contains all files and relations between them. Think of it as a browser that scrapes your entire site, except it only ever visits every resource once and can run directly on your file system. Having your static file server traverse the entire dependency graph on startup would seed it with all the contextual knowledge it needs for SPDY PUSH.

Making it happen

So I wanted to take a look at what is possible with the technology we have now, and see if we can get to the point where performance measurements and comparisons with existing solutions are actually possible. Needless to say, SPDY or HTTP2 aren't mainstream yet, but it's nice to be prepared.

I started out with the idea of using Assetgraph for all the graph knowledge. I have been a maintainer of the project for a few years, so this is my go to tool for anything web. From there I needed a server that could hook into this graph instance. I know a bit of node.js, so I found a wonderful module by Fedor Indutny called node-spdy. As luck would have it, node-spdy is compatible with express, which I have touched a few times before.

So after a few hours of hacking I came up with expush. A small proof of concept static file server that will search for html-files in the web root and auto discover any outgoing dependencies recursively, creating a dependency graph which the file server will look at before it falls back to serving files from the file system. Whenever an HTML-page is requested, the entire dependency sub graph from it is traversed and a PUSH request is initiated for each asset.

These are early days. It's quite buggy and has no finish at all, but it's enough to prove that this can be done. There are all sorts of bugs, like errors being thrown when reloading a page before the keep-alive dies and Chrome not actually supporting ETag cache header response properly, so every asset is pushed over the wire in its entirety. I do think that this experiment should be enough to do some initial speed tests and comparisons with various other web performance optimization setups.

As has hopefully been established by this post, latency is a big killer. If you're not convinced you should watch Paul Irish's Fluent 2014 keynote. I hope SPDY push can make a serious contribution to speeding up the web by eliminating a lot of latency issues.

I'd love to show you some numbers right now, but alas, I am not a benchmark expert. So if you are one, or know one, please poke me and lets see if we can get some numbers and science this thing up!