Original post is here: eklausmeier.goip.de
- 1. Motivation
- 2. NGINX installation
- 3. Brotli compilation for Arch Linux
- 4. NGINX configuration
- 5. Caching
- 6. Deployment
Since mid of August I switched from Hiawatha web-server to NGINX web-server. I initially intended to use OpenLiteSpeed web-server. See Installing OpenLiteSpeed on Arch Linux, but installation and configuration of OpenLiteSpeed turned out to be complicated. I had previously experimented and used Lighttpd.
1. Motivation #
The author and maintainer of Hiawatha, Hugo Leisink, on 18-Feb-2019 stated on his weblog:
Many times, I wondered whether I should keep going on with the project or not, but somehow I always found a reason to continue. But not this time. Recently, a serious issue was found in the Hiawatha webserver and the fact that I didn't care much, made me realize that it's time to stop.
Clearly, Hiawatha will never support HTTP/2 or HTTP/3 ... new features will be based on what I need, not on what is needed for a webserver in general.
Over the years he did not change his opinion on that. So it clearly was time to find a web-server which is fully maintained and offers below functionality:
- Brotli compression
- HTTP/3 and QUIC
- URL rewriting
- Built-in caching like Varnish
In Set-Up Hiawatha Web-Server I compared the size of various web-servers.
| web-server | #header | #C source | LOC |
|---|---|---|---|
| Hiawatha 11.3 | 155 | 136 | 206,878 |
| NGINX 1.25 | 136 | 259 | 229,625 |
2. NGINX installation #
Installing NGINX is pretty simple as it is contained in the Extra-repository of Arch Linux. For installing the Brotli extension you need to install the NGINX source code, then download the Brotli module, and compile the module. Below comment from the original GitHub repository is important:
You will need to use exactly the same
./configurearguments as your Nginx configuration and append--with-compat --add-dynamic-module=/path/to/ngx_brotlito the end, otherwise you will get a "module is not binary compatible" error on startup. You can runnginx -Vto get the configuration arguments for your Nginx installation. Then
1$ cd nginx-1.25
2$ ./configure --with-compat --add-dynamic-module=/path/to/ngx_brotli
3$ make modules
A concrete example for installing brotli-1.0rc. Switch to root user and go to directory /usr/src/nginx. In below configure the majority of the command line is the from nginx -V.
1./configure --prefix=/etc/nginx --conf-path=/etc/nginx/nginx.conf --sbin-path=/usr/bin/nginx --pid-path=/run/nginx.pid --lock-path=/run/lock/nginx.lock --user=http --group=http --http-log-path=/var/log/nginx/access.log --error-log-path=stderr --http-client-body-temp-path=/var/lib/nginx/client-body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-cc-opt='-march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/nginx-mainline/src=/usr/src/debug/nginx-mainline -flto=auto' --with-ld-opt='-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=auto' --with-compat --with-debug --with-file-aio --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_degradation_module --with-http_flv_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-http_v3_module --with-mail --with-mail_ssl_module --with-pcre-jit --with-stream --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-compat --add-dynamic-module=/tmp/ngx_brotli-1.0.0rc
Now make modules and copy the two files
1cd objs
2cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
3cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/
3. Brotli compilation for Arch Linux #
Go to /tmp directory.
1git clone https://github.com/google/ngx_brotli.git
2cd ngx_brotli
3git submodule update --init
Go to NGINX source code: cd /usr/src/nginx, switch to root user.
1./configure `nginx -V 2>&1 | perl -ne 's/ --with-cc-opt='.+'//g; print $1 if /configure arguments: (.+)$/;'` --with-compat --add-dynamic-module=/tmp/ngx_brotli
2make
3cd objs
4cp -p ngx_http_brotli_filter_module.so /usr/lib/nginx/
5cp -p ngx_http_brotli_static_module.so /usr/lib/nginx/
6systemctl restart nginx
4. NGINX configuration #
For the special case w.r.t. body-size I had already written on this here: nginx: 413 Request Entity Too Large - File Upload Issue. The general structure of a NGINX configuration is like below:
some global configuration;
http {
server A {
list 80;
}
server B {
listen 443;
}
}
All the rewriting rules for port 80 and 443 are the same, just copied from the top server to the bottom server config.
#user http;
worker_processes 1;
error_log /var/log/nginx/error.log;
load_module /usr/lib/nginx/ngx_http_brotli_filter_module.so;
load_module /usr/lib/nginx/ngx_http_brotli_static_module.so;
events {
worker_connections 1024;
}
http {
root /srv/http;
index index.html;
client_max_body_size 15900M;
http2 on;
gzip on;
brotli on;
brotli_comp_level 10;
brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
fastcgi_cache_path /var/cache/nginx/ keys_zone=nginxpc:720m inactive=720m;
fastcgi_cache_key "$request_method$request_uri";
fastcgi_cache nginxpc;
fastcgi_cache_valid 720m;
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
log_format hiawatha_format '$remote_addr|$time_local|$status|$bytes_sent|$request|$http_referer|$http_user_agent|$host:$server_port|$https';
access_log /var/log/nginx/access.log hiawatha_format;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
http3 on;
http3_hq on;
types_hash_max_size 4096;
server {
listen 80;
server_name localhost;
rewrite "^(/*)$" "/blog" redirect;
rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
rewrite "^/(aux|blog|music|gallery)($|/.*)" "/rewrite/sndsaaze/public/index.php?/$1$2" last;
#charset koi8-r;
error_page 404 /rewrite/sndsaaze/public/index.php?/404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
location ~ \.php$ {
try_files $fastcgi_script_name =404;
# default fastcgi_params
include fastcgi_params;
# fastcgi settings
fastcgi_pass unix:/run/php-fpm/php-fpm.sock;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
# fastcgi params
fastcgi_param DOCUMENT_ROOT $realpath_root;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
}
location ~ ^/ttyd(.*)$ {
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://eklausmeier.goip.de:7681/$1;
}
}
server {
listen 443 quic;
listen 443 ssl;
server_name localhost;
rewrite "^(/*)$" "/blog" redirect;
rewrite "^/aux/search.php$" "/rewrite/sndsaaze/public/aux/search.php" last;
rewrite "^/(404\.html|feed\.xml|sitemap\.html|sitemap\.xml)$" "/rewrite/sndsaaze/public/index.php?/$1" last;
rewrite "^/(aux|blog|music|gallery)($|/.*)" "/rewrite/sndsaaze/public/index.php?/$1$2" last;
ssl_certificate /etc/hiawatha/eklausmeier.goip.de.pem;
ssl_certificate_key /etc/hiawatha/eklausmeier.goip.de.pem;
# From https://blog.qualys.com/product-tech/2013/08/05/configuring-apache-nginx-and-openssl-for-forward-secrecy
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";
location / {
# used to advertise the availability of HTTP/3
add_header Alt-Svc 'h3=":443"; ma=86400';
}
location ~ \.php$ {
try_files $fastcgi_script_name =404;
# default fastcgi_params
include fastcgi_params;
# fastcgi settings
fastcgi_pass unix:/run/php-fpm/php-fpm.sock;
fastcgi_buffers 8 16k;
fastcgi_buffer_size 32k;
# fastcgi params
fastcgi_param DOCUMENT_ROOT $realpath_root;
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
}
location ~ ^/ttyd(.*)$ {
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_pass http://eklausmeier.goip.de:7681/$1;
}
}
}
It is a common error to forget:
brotli_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
gzip_types application/xml image/svg+xml text/css text/csv text/javascript text/markdown text/plain text/vcard text/xml;
See the examples below, where this configuration has been forgotten:
Method to check for compression:
1curl -D - -H "Accept-Encoding: gzip,deflate,br" --write-out "%{size_download}\n" -o /tmp/prism-css.br http://localhost/jscss/prism.css
5. Caching #
By using fastcgi_cache with a quite large retention interval of 720 minutes (=12 hours), I keep already generated pages in the cache for a long time.
If you want to delete specific entries in the cache, the critical line is:
fastcgi_cache_key "$request_method$request_uri";
You can compute the file-name of the cache file by specifying request-method and URL like so:
1printf "GET/blog" | md5sum
or
1printf "GET/blog/2023/08-29-from-hiawatha-to-nginx" | md5sum
This will print the file-name which you can delete with rm. In our case in the directory /var/cache/nginx. See How to Setup FastCGI Caching with Nginx on your VPS.
6. Deployment #
With this very aggressive caching in place, the deployment of my blog changed. Previously, I generated all static files with
1php saaze -mortb /tmp/build
and then deployed via
1blogdeploy -p
Above deployment script essentially just removes the previous directories and replaces them with the newly generated ones. I did this for the "staging environment" and "production", i.e., on my work-PC and on the self-hosting PC.
Now I rarely generate all static files and use the dynamic mode of Simplified Saaze, i.e., Simplified Saaze generates the HTML file whenever it is actually accessed. Once it is generated then NGINX caches it. So, essentially the generation of the static files is deferred to the actual access time:
1$ ls -l /var/cache/nginx | wc
2 350 3143 26826