Compression of Static Assets

[daninus14]

I noticed that static assets are not compressed by default using caveman2/clack and hunchentoot.

This is something that I was expecting, and I haven't even found any configurations to enable it.

I would assume a modern server gzips all the static files on load and keeps a cache to use if the files have not been updated since the last compression, and just serves files from the cache and adds the appropriate header accordingly to avoid doing a compression on each serve

any thoughts on this? I always took this (compression) for granted so I never had to configure or implement it?

If we (probably I or a joint effort) were to add it, is this appropriate for a server to take care of, so a contribution to hunchentoot and woo, or a framework like clack?

[fukamachi]

This is a very good point to discuss.
It's possible to add compression support for all web servers. However, I think it's better to implement it in the Clack (or Lack) layer since the code would be almost the same.

[daninus14]

This is the protocol/API I'm thinking of developing. Can you please take a look and add any thoughts you have? I'd rather hash out the protocol/API before actually coding it to make this more efficient.

I'm copying below from my org files...

• Make a Compression of Static Files for Web Servers
  There are already libraries to do gzip. However Hunchentoot and Clack do not seem to support encoding. Need to add a header of encoding and a compression of the files.

There can be settings like

• *cache-directory*
• *check-if-modified* to have a lookup for the original file if the modified date is newer than whatever is saved in the Hash table of that filepath. Defaults to T but can be set to NIL to avoid the check for performance reasons.
• *compress-on-load* defaults to NIL which means that only when a file is requested, does it check if it has been compressed. If it has, just return the path to the cached compressed version; if it hasn't, then compress it, save it to the cache (with the added relative path cache_dir/filepath), and then return the cached compressed filepath. If the value is T, then on loading the library, it will compress all the files provided in *static-files-directory* and save them to the cache directory.
• *static-files-directory* useful only if the *compress-on-load* variable is set to T, in which case it will compress and add to the cache directory all the files in the provided path.
• The function (get-path-to-compressed-file filepath) should return NIL if there is no path. See *check-if-modified* above.
• (defun ensure-path-to-compressed-file (filepath &optional &key check-if-modified)) Will check if there is a path for a compressed version of the original pathname and return it, if there is none, it will then compress a file and save it and return the compressed file's path. See *check-if-modified* above. The optional parameter check-if-modified overrides the behavior set in the *check-if-modified*.

To use, the server/web app/clack would then call in the render function the (ensure-path-to-compressed-file filepath) whenever a filepath is provided, which would make the server render the cached compressed version instead of the non compressed version.

I think it's rather straightforward to implement once I described the protocol like this. Please let me know if there are any modifications you can think of. I did not look up how any implementation does this, so I may be missing a lot of things. Not sure if it's worth it to look up django or something comparable to see how they do it, or just do this quickly and add features as is needed later on.

[daninus14]

Just an update of the above mainly to add alternatives for the compression algorithm per Headers/Content-Encoding:

There are already libraries to do gzip. However Hunchentoot and Clack do not seem to support encoding. Need to add a header of encoding and a compression of the files.

There protocol can be as follows:

• *cache-directory*
• *check-if-modified* to have a lookup for the original file if the modified date is newer than whatever is saved in the Hash table of that filepath. Defaults to T but can be set to NIL to avoid the check for performance reasons.
• *compress-on-load* defaults to NIL which means that only when a file is requested, does it check if it has been compressed. If it has, just return the path to the cached compressed version; if it hasn't, then compress it, save it to the cache (with the added relative path cache_dir/filepath), and then return the cached compressed filepath. If the value is T, then on loading the library, it will compress all the files provided in *static-files-directory* and save them to the cache directory.
• *static-files-directory* useful only if the *compress-on-load* variable is set to T, in which case it will compress and add to the cache directory all the files in the provided path.
• The function (get-path-to-compressed-file filepath) should return NIL if there is no path. See *check-if-modified* above.
• (defun ensure-path-to-compressed-file (filepath &optional &key check-if-modified algorithm)) Will check if there is a path for a compressed version of the original pathname and return it, if there is none, it will then compress a file and save it and return the compressed file's path. See *check-if-modified* above. The optional parameter check-if-modified overrides the behavior set in the *check-if-modified*. The optional parameter algorithm overrides the behavior set in *compression-algorithm*
• *compression-algorithm* is a setting for which compression algorithm to use.
• *compressed-files-ht* this will keep a record of the files that have been compressed. It is of the form filepath -> (compressed-filepath algorithm)
• (get-algorithm-for-compressed-file filepath) if filepath is in *compressed-files-ht* then it returns the algortihm used to compress the file, otherwise returns NIL
• (get-http-header-for-compressed-file filepath) if filepath is in *compressed-files-ht* then it returns the appripriate http header to use for the file, otherwise returns NIL

To use, the server/web app/clack would then call in the render function the (ensure-path-to-compressed-file filepath) whenever a filepath is provided, which would make the server render the cached compressed version instead of the non compressed version.

Note that there is a gzip-stream library, but I think a previously compressed file will be best, and the servers already are responsible for how they want to deliver files with chunking etc.

[fukamachi]

It's more sophisticated than I expected. I thought it was fine with the simple caching mechanism.
A Lack middleware that replaces a pathname to a gzip file sounds good to me.

(builder
 (:static-encoder :cache-directory #P"/path/to/dir/")
 (:static :path "/public/"
          :root #P"static/")
 app)

[fukamachi]

Note that there is a gzip-stream library, but I think a previously compressed file will be best, and the servers already are responsible for how they want to deliver files with chunking etc.

It's great in some scenarios, but files may be temporarily generated, so caching can not be so reasonable in such cases.
Need to expire to reduce disk consumption.

However, I think writing for your own use is better at the beginning.

[daninus14]

It's more sophisticated than I expected. I thought it was fine with the simple caching mechanism. A Lack middleware that replaces a pathname to a gzip file sounds good to me.

So should I try to submit the middleware as a PR to lack or just make a standalone system which people can use as a lack middleware and get it via quicklisp?

(builder
 (:static-encoder :cache-directory #P"/path/to/dir/")
 (:static :path "/public/"
          :root #P"static/")
 app)

I'm a bit confused why is there a (:static) middleware in this example?

Also, the order of middlewares that gets executed is it top to bottom?

It's great in some scenarios, but files may be temporarily generated, so caching can not be so reasonable in such cases.
Need to expire to reduce disk consumption.

Interesting use case.

However, I think writing for your own use is better at the beginning.

So we can start with the simple use case and then when someone needs something more complicated, he can just go ahead and build upon whatever we make.

[daninus14]

Hi, so basically it's finished. It's usable, but there are a few features missing and assumptions taken.

I detailed the assumptions in the Readme of the projects.

There are three projects:

1. A general compression-cache project which can be added to any web server in general.
2. A lack middleware for compressing and caching assets in the server, which also sets caching headers for the client called lack-compression-cache
3. A lack middleware for rerouting requests called lack-rerouter

Please take a look at the code

[fukamachi]

Looks pretty good!

[daninus14]

Thanks. I've been using it and it works great so far 😄