Resize images on the fly with Nginx try_files

Nginx is fantastic as a high performance server that can serve static files very, very quickly with minimal, predictable use of resources. Static files are the CSS, Javascript and boilerplate images that make up the design of a website. As the name suggests, static files tend not to change very often. These days the chances are that they have already been optimised, minified and compressed before being deployed to the web server.

Nginx doesn’t handle requests for scripts like PHP or Python directly, but instead acts a reverse proxy and will pass requests to backend server. We tend to use php-fpm but the backend can be anything, even Apache.

Websites typically have a lot of images uploaded by users that are needed in lots of different variants (thumbnail, crop, large etc.). These user generated images obviously can’t be optimised and resized ahead of time like all the boilerplate images.

One solution is to generate all the different image variants when an image is uploaded, but this isn’t very flexible, can create a noticeable delay as all the variants are generated, and things can get complicated if you need to create a new variant for thousands and thousands of images.

There are lots of solutions that will resize images dynamically but these tend to send all requests for images to a script that will serve existing images (hopefully from a cache) or attempt to generate a new image on the fly, serve the new image and save it in a cache for future use. This approach is tried and tested but it is unnecessary overhead calling a script for every image request. It is much more efficient to serve images straight from the disk, something Nginx excels at, and not have to involve a script at all.

A nice solution is to use the Nginx try_files directive to serve an image directly, if it exists and only send the request to the backend if the image (variant) requested doesn’t exist.

From the documentation for try_files:

Checks the existence of files in the specified order and uses the first found file for request processing; the processing is performed in the current context. The path to a file is constructed from the file parameter according to the root and alias directives. It is possible to check directory’s existence by specifying a slash at the end of a name, e.g. “$uri/”. If none of the files were found, an internal redirect to the uri specified in the last parameter is made. For example:

location /images/ {
    try_files $uri /images/default.gif;
}
 
location = /images/default.gif {
    expires 30s;
}

In the above example if the specified image is missing, /images/default.gif is returned instead and has it’s ‘Expires’ header set to 30 seconds.

Simplified Nginx conf file

See how we can use this in practice

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
server {
    listen 80;
 
    access_log /var/log/nginx/example.com.access.log main;
    error_log /var/log/nginx/example.com.error.log;
    root /var/www/vhosts/example.com/htdocs;
    index index.php index.html;
    server_name example.com;
 
    # Check if the requested image exists and if not pass to @backend
    location ~* ^.+.(jpg|jpeg|png|gif)$ {
        try_files $uri @backend;
        add_header Pragma "public";
        add_header Cache-Control "public";
        expires     1y;
        access_log  off;
        log_not_found off;
     }
 
 
 
    # Other static files
    location ~* ^.+.(ico|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|tar|mid|midi|wav|bmp|rtf|js|eot|woff|svg|htc)$ {
        add_header Pragma "public";
        add_header Cache-Control "public";
        expires     1y;
        access_log  off;
        log_not_found off;
    }
 
    location / {
        # Check if a file exists, or route it to index.php.
        # Configure depending on application used
        try_files $uri $uri/ /index.php?$query_string;
    }
 
    location @backend {
        rewrite ^(.*)$ /images.php?url=$uri;
    }
 
    location ~\.php$ {
        try_files $uri =404;
        fastcgi_pass unix:/dev/shm/apache-php.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include /etc/nginx/fastcgi_params;
    }
 
}

In the solution above @backend passes requests to /images.php. Images.php bypasses the rest of the application and is simply contains code to necessary resize images on the fly. Write your own or make use of an existing library. Personally, I like Intervention Image.

Quickly duplicate a row in MySQL

A quick way of duplicating a row in a table without running into duplicate key errors or having to type out all the field names…

Not quite a one-liner, but very straightforward.

/*
Duplicate row 58 from mytable
*/
CREATE TEMPORARY TABLE tmptable  SELECT * FROM mytable WHERE id = 58;
 
/*
Change the unique key
*/
UPDATE tmptable SET id=0; 
 
/* 
Insert the duplicate row into the original table
*/
INSERT INTO mytable SELECT * FROM tmptable;
 
/* 
Drop the temporary table
*/
DROP TABLE tmptable;

Cache busting and Asset Management

Setting the scene…

As developers we are pulled in lots of conflicting directions; websites and applications have got bigger and more complicated, at the same time we expect these applications to be faster and faster. It is now standard practice to configure servers to instruct browsers to cache ‘static’ resources for very long periods of time and serve libraries such as jQuery from CDNs (Content Delivery Networks) – the rationale behind both of these practices is to try and ensure that these ‘static’ resources only need to be downloaded to the browser the first time a user requests the page.

The problem

Everything is good. The website is running fast and well and all those ‘static’ resources are safely cached away in the users browsers, reducing server load and bandwidth. One day the client comes in for a meeting. As a result you have to update some images and a stylesheet. You dutifully update the stylesheet and the images, upload the updated files to the server and email the client. A short time later the client phones up and says that they can’t see the changes.

What has happened? The problem is that you have already instructed the browser to cache the resources, therefore the browser isn’t downloading the updated files and the browser shows the old files.

‘Just click on settings in your browser and click the clear cache button’. This isn’t really an acceptable solution. Are you going to track down and tell every user of the website to clear their cache?

The solution

The simple solution is that every time you change a static resource (image, stylesheet, javascript file, etc.) you need to rename the file. In essence this is simple, but on a large project this can be extremely time consuming. Automating this process and causing the browser to request new files is known as ‘Cache busting’. 

For quite a while the traditional PHP solution has been use Minify https://code.google.com/p/minify/ (first released in 2007). In a nutshell Minify will append a timestamp in a query string to each of your files and can also compress and combine them (fewer downloads is good). Minify is still a great choice, but there are now some powerful alternatives. 

Assetic https://github.com/kriswallsmith/assetic is one alternative – it is a modern PHP asset management system, most commonly encountered in the context of Symphony2, although it is straightforward in any PHP application using composer https://getcomposer.org/ to manage dependencies. 

Assetic works as an asset management system, in a nutshell this means that it can bundle together and version (rename) files in order to act as a cache buster. In addition it can manipulate the content it acts upon – for example, it can Minify css and javascript or optimise images. Assetic can be set to operate in various ways, but will commonly export assets and save them with a new unique  filename. Because they are actual files on disk they can then be served very efficiently by the server.

This article was originally published at www.ab-uk.com

Installing Homebrew & Fabric on OSX 10.9 Mavericks

I’ve just upgraded my Mac at work to 10.9.1 Mavericks. No messing about – completely clean install on a formatted disc. Last year I wrote a quick post about installing Fabric on my Mac, this is an update of that post to reflect a few changes.

If you don’t know…

Fabric is a Python (2.5 or higher) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

I think Fabric is a good solution to manage the deployment of websites and applications – it is an integral part of my workflow these days.

Homebrew is a package manager for OSX and is an alternative to the commonly used MacPorts. You can find out a bit more about the differences between them in this thread on arstechnica.

Install Command Line Developer Tools

These were previously part of Xcode but now need to be installed separately, open a terminal and type:

xcode-select --install

…and follow the prompts.

Install Homebrew

The address of the download appears to have chnaged slightly since last year…

ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

…and follow the prompts.

When you see the successful installation message, type:

brew doctor

This will alert you to any problems that you need to resolve.

If everything is good you should see the following message:

Your system is ready to brew.

Install Git, Python and Fabric

brew install git
brew install python

You now need to edit your system paths so that Python can be found:

I edited my .bashrc file: nano ~/.bashrc and added the following line:

export PATH=/usr/local/bin:/usr/local/sbin:~/bin:$PATH

And then /etc/paths sudo nano /etc/paths so that usr/local/bin (the location of packages installed by Homebrew) is first:

/usr/local/bin
 
/usr/local/sbin
 
/usr/bin
 
/bin
 
/usr/sbin
 
/sbin

And finally install Fabric. There isn’t a Homebrew package available by default, so I just installed Fabric using pip – the pip package manager for Python is installed at as part of Python.

pip install fabric

Check if it works…

Let’s check and see if everything has worked…

$ fab
 
Fatal error: Couldn\'t find any fabfiles!
 
Remember that -f can be used to specify fabfile path, and use -h for help.
 
Aborting.

In this case the Fatal error is good. It means that Fabric has installed correctly and you now need to write some fab files.