Self-Hosted CDN Caching: Varnish and Nginx Reverse Proxy in Practice

Self-Hosted CDN Caching: Varnish and Nginx Reverse Proxy in Practice

Not every site needs a commercial CDN. When traffic scale, compliance, or cost structure demands a more controllable setup, a "self-hosted CDN cache layer" is a common choice: put a reverse-proxy cache in front of the origin, keep static content cached locally, reduce origin load, and speed up access.

1. First Decide: Should You Self-Host?

A self-hosted cache layer fits these scenarios:

• The origin already spans data centers, and you want to reduce fetch-path pressure with a local cache layer;
• Fine-grained cache logic is needed, e.g., special handling for internal systems or dynamic APIs;
• Compliance or data localization requires traffic not to pass through third-party CDNs;
• Cost-sensitive with controllable traffic, avoiding usage-billed commercial CDN fees.

It does not fit when you need global multi-node coverage, huge DDoS protection, or ultra-low-latency edge access. Those remain strengths of commercial CDN networks that self-hosting cannot easily replicate. For selection help, see CDN provider selection guide.

2. Comparing Main Self-Hosted Options

3. Nginx Proxy Cache in Practice

Nginx is the lowest-barrier and most common self-hosted option. Its caching relies on the proxy_cache module, storing backend responses on local disk.

3.1 Basic Configuration

http {
    # Define cache directory and shared memory zone
    proxy_cache_path /data/nginx/cache keys_zone=mycache:10m max_size=10g;

    server {
        proxy_cache mycache;

        location / {
            proxy_pass http://127.0.0.1:8000;
            # 200/302 cached 10 minutes, 404 cached 1 minute
            proxy_cache_valid 200 302 10m;
            proxy_cache_valid 404      1m;
            # Custom cache key (default includes scheme, method, host, URI)
            proxy_cache_key "$scheme$request_method$host$request_uri";
        }
    }
}

Key points:

• keys_zone defines the shared memory zone that stores cache-entry metadata;
• max_size caps total disk usage; the cache manager evicts by LRU beyond it;
• proxy_cache_valid sets validity per response status code;
• On startup, the cache loader gradually loads disk metadata into memory to avoid startup jitter.

Verifying the Cache Works

After configuring, use curl -I to inspect response headers. Nginx does not emit a cache-hit marker by default — add one line first:

add_header X-Cache-Status $upstream_cache_status;

Then request the same URL twice:

curl -I https://example.com/static/app.css

The first response shows X-Cache-Status: MISS; the second should show X-Cache-Status: HIT. If you see EXPIRED, BYPASS, or STALE, the hit logic is not behaving as expected, so revisit the cache key and validity settings. This single line is especially useful when troubleshooting the classic "I updated the page but still see the old version" failure.

3.2 Cache Purging

Nginx purges via the HTTP PURGE method:

map $request_method $purge_method {
    PURGE 1;
    default 0;
}
# In the location: proxy_cache_purge $purge_method;

curl -X PURGE "https://www.example.com/*"

Restrict which IPs may issue PURGE with geo to prevent malicious cache flushes.

3.3 Working with Other Pieces

An Nginx cache typically sits in front of the origin as a "local CDN cache layer," combined with SSL/TLS configuration and Nginx performance optimization to raise single-machine throughput. More site configs are in Nginx site configuration.

4. Advanced Use: Varnish and OpenResty

4.1 Varnish

Varnish is built specifically for HTTP caching with very high performance. It controls caching through VCL (Varnish Configuration Language), supporting cache-key rewriting, conditional caching, and coordination with backend health checks. It works well as a dedicated cache layer in front of app servers.

A minimal VCL looks like this:

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
}

sub vcl_recv {
    # Do not cache authenticated areas
    if (req.url ~ "^/account") {
        return (pass);
    }
    # Put the language cookie into the cache key
    if (req.http.Cookie ~ "lang=") {
        set req.http.X-Lang = regsub(req.http.Cookie, "^.*lang=([^;]+);?.*$", "\1");
        set req.hash += req.http.X-Lang;
    }
}

This example covers the two most common VCL needs: passing dynamic paths straight through (pass), and folding a language cookie into the cache key so the Chinese and English versions never mix caches.

4.2 OpenResty

OpenResty embeds Lua into Nginx, letting you implement auth, rate limiting, A/B splitting, and user-specific cache keys directly at the cache layer — an "all-in-one reverse proxy + cache + business logic" option.

5. 16IDC View: Landing a Self-Hosted Setup

In the small and mid-size sites 16IDC serves, the typical self-hosted form is a "single-machine Nginx proxy cache" or "a few nodes with simple DNS routing" — rarely a full Varnish/ATS cluster from day one. Some advice:

• Start with Nginx: prove the value with minimal changes before adding heavier components;
• Keep a CDN above the cache layer: self-hosted caching solves origin-to-datacenter acceleration; global distribution still benefits from commercial CDN;
• Design cache keys well: follow CDN cache strategy and cache key strategy to avoid caching dynamic content by mistake;
• Monitor everything: hit ratio, origin traffic, and disk usage all need monitoring, or problems become hard to locate.

The core value of self-hosting is control; the cost is that you operate it yourself. For most users, the pragmatic path is to clarify the CDN acceleration goal and budget first, then decide between commercial CDN, a self-hosted cache, or both.

FAQ

Can a self-hosted cache replace a commercial CDN? Not fully. A self-hosted cache accelerates the path between your origin and your data center and typically lives in only a handful of locations; the value of a commercial CDN is edge distribution across hundreds of PoPs. The two are complementary — many sites run "commercial CDN for global coverage plus a self-hosted cache for frequent origin fetches."

What if the cache directory keeps growing? First check whether the cache key is too fine-grained (splitting one resource into many copies by cookie or timestamp), then rely on max_size with LRU so the cache manager cleans up automatically. If the hit ratio stays low, the problem is usually the key or validity design, not disk space.

How do I quantify the benefit? Watch three metrics: cache hit ratio (aim for 90%+ on static assets), origin traffic volume, and average origin response time. Both nginx-module-prometheus and Varnish's varnishstat give you a live hit-ratio curve.

Source: https://docs.nginx.com/nginx/admin-guide/content-cache/content-caching/

Reference: Varnish official docs: https://varnish-cache.org/docs/