Web server configuration

Architecture

Our http(s) content is served using the following web servers:

1. Nginx
2. Varnish
3. Apache

The traffic flows in-order, ie: Internet -> nginx -> varnish -> apache

And then back out to the client in the reverse order: apache -> varnish -> nginx -> Internet

Why??

OSE's principles aims for simplicity--so you ask, "Why aren't we simply using only Apache? Why Nginx as well? And why Varnish?" Great question!

Before simplicity, OSE is radically committed to using FLOSS. We're also an ecologically-aware, low-budget, non-profit with limited financial & computational resources. Keeping this in mind, below are the reasons for the complexity described in this documentation:

1. Varnish is a cache. It's an essential component that allows us to serve a very high volume of requests on many websites from a single server. Unfortunately, the free version of Varnish does not speak https.
2. Nginx is our tls-terminator. It listens to our encrypted traffic over https & passes unencrypted http traffic internally onto varnish.
3. Nginx has great DOS protection and rate-limiting built-in.
4. Nginx being distinct from Apache gives us the ability to serve a SITE_DOWN vhost to users for a specific domain, while devs are still able to iterate & test changes to the backend Apache server. Note that we have only 1 dedicated host, and we don't have a load balancer.
5. Most people who use https + varnish specifically use Nginx to terminate their https. Therefore, there is better documentation & a user-support-base for this architecture.
6. History. I (Michael Altfield) came on-board in 2017 with only Apache running. I added https to protect our user's passwords that were being sent in cleartext, and--in doing so--I had to abandon CloudFront so a third-party didn't have our private keys. At the time, CF was our CDN cache, so I had to implement a self-hosted cache. I chose varnish, and then had to add nginx before it for https termination.

Nginx

This section will describe our use of the Nginx server in our web server configuration, which serves as our https terminator, basic DOS protection, and SITE_DOWN tool.

https

HPKP

HTTP Public Key Pinning (HPKP) can brick your domain if not done properly. For safety, 14 keys were pinned following the Let's Encrypt HPKP Best Practices Guide, including:

1. Two distinct, pre-generated backup keys' CSRs @ /etc/pki/tls/hpkpBackupKeys/
2. Our leaf certificate issued by Let's Encrypt using certbot @ /etc/letsencrypt/live/opensourceecology.org/cert.pem
3. The intermediate Let's Encrypt certificate that signed our certificate @ /etc/letsencrypt/live/opensourceecology.org/chain.pem
4. The Internet Security Research Group (ISRG) Root Certificate for Let's Encrypt
5. The IdenTrust Root Certificate, which cross-signed the Let's Encrypt Root Certificate
6. In case Let's Encrypt is no longer usable in the future, all root certificates & the root certificates of their cross-signers for CloudFlare, since they offer free certificates. This includes digicert, addtrust, globalsign, and gtecybertrust (now digicert)
7. In case Let's Encrypt is no longer usable in the future, all root certificates & the root certificates of their cross-signers for SSL.com, since they offer free certificates for 90 days.

Moreover, apache was configured with a report-uri, which can be checked on the server to debug potential client-side hpkp issues

report-uri="http://opensourceecology.org/hpkp-report"

Varnish

This section will describe our use of the Varnish server in our web server configuration, which serves as our in-memory cache.

Why Varnish?

Our biggest site is this wiki (running on Mediawiki). As of 2017, Wikipedia (the largest site running on Mediawiki) has chosen Varnish as their cache-of-choice, after experimenting with Squid & Nginx caching. If the biggest user of our biggest site's application backend is using Varnish, we should use it too. And I found good wordpress plugins that play nicely with Varnish as well.

Apache

This section will describe our use of the Apache server in our web server configuration, which serves as our backend application web server.

See Also

• Website
• OSE Server