CDN Log Hit-Rate Analysis Script

CDN Log Hit-Rate Analysis Script

The single most useful number in CDN logs is the cache hit rate. When the hit rate stays below 70%, roughly one in three requests has to go back to the origin, which pushes up both bandwidth costs and origin load. Take a site that fetches 500GB from the origin each month and pays for traffic: lifting the hit rate from 60% to 90% saves roughly 300GB of origin traffic per month — not a trivial line item. This article provides a practical Python script that computes HIT/MISS ratios, pinpoints the URLs responsible for the most origin fetches, and surfaces the pages dragging your hit rate down. One caveat: a higher hit rate is not automatically better — if dynamic endpoints get cached too, users see stale data while the numbers look great. Judge the metric against your business.

What the cache status field tells you

Most CDN access logs include a cache status field. On Cloudflare, for example, every request log records a CacheStatus with values such as HIT, MISS, DYNAMIC, EXPIRED, REVALIDATED, and STALE; Alibaba Cloud CDN and Tencent EdgeOne expose similar fields under different names. The math itself is simple:

$$hit_rate = \frac{HIT}{HIT + MISS} \times 100%$$

Reference: Cloudflare log fields documentation https://developers.cloudflare.com/logs/reference/log-fields/

Reading the log format

The script above assumes standard combined-format logs, where fields are space-separated and the request line is wrapped in double quotes. CDN logs differ in field ordering: some put the cache status at the end of the request line, some in a dedicated field, and some emit JSON. Before writing the parser, run head -5 cdn.log and look at two sample lines to confirm where HIT/MISS markers appear, then decide between ' HIT ' and '\" HIT ' style matchers. For JSON-formatted logs, parse with json.loads and read the CacheStatus field directly — that is what Cloudflare's Logpush exposes.

The hit-rate analysis script

The script below reads a standard Nginx/CDN access log, classifies each request by cache status, prints the overall hit rate, and counts origin fetches per URL so you can immediately see which pages cost the most round trips:

from collections import Counter

def analyze_cdn_log(path: str):
    total = Counter()
    origin_fetch = Counter()   # origin fetches per URL

    with open(path, 'r', encoding='utf-8') as f:
        for line in f:
            if '" HIT ' in line:
                total['HIT'] += 1
            elif '" MISS ' in line:
                total['MISS'] += 1
                url = line.split('"')[1].split(' ')[1]
                origin_fetch[url] += 1

    hit = total['HIT']
    miss = total['MISS']
    all_requests = hit + miss
    ratio = hit / all_requests * 100 if all_requests else 0.0

    print(f'total: {all_requests}, HIT: {hit}, MISS: {miss}')
    print(f'hit rate: {ratio:.2f}%')
    print('Top 10 origin-heavy URLs:')
    for url, cnt in origin_fetch.most_common(10):
        print(f'  {cnt:>6}  {url}')

analyze_cdn_log('cdn.log')

Given log lines like this:

127.0.0.1 - - [01/Jul/2026:10:15:22 +0800] "GET /product/a HTTP/1.1" 200 1024 "MISS"
127.0.0.1 - - [01/Jul/2026:10:15:23 +0800] "GET /product/a HTTP/1.1" 200 1024 "HIT"

the script prints something like:

total: 18423, HIT: 15123, MISS: 3300
hit rate: 82.10%
Top 10 origin-heavy URLs:
   1204  /api/list?page=2
    986  /product/a

Why the hit rate drops — and how to investigate

Combined with the script output, work through these in order. The sequence matters: rule out counting problems first, then cache keys, and only then TTLs — so you don't churn the cache policy into a worse state:

1. Dynamic endpoints polluting the numbers. /api/ routes usually should not be cached. Exclude them from the statistics or mark them with no-store.
2. Query strings fragmenting the cache. /product/a?from=wechat and /product/a?from=seo produce different cache keys, so the same content gets cached multiple times. Consolidate parameters with a sensible cache key strategy.
3. TTLs too short. If the HTML TTL is too small, hot pages expire and refetch constantly. Lengthen the TTL and enable an origin shield to cut origin traffic sharply.
4. Missing cache rules. Static assets like images, CSS, and JS contribute almost nothing to the hit rate if they never get a cache rule.

Making analysis a routine task

The script above works when you're debugging an incident, but it's better to make it routine: run it every morning via cron, append the hit rate to a result file, and get notified by email or chat when it drops below a threshold. It's simple — format the output of analyze_cdn_log as one line and schedule it with cron:

# Run daily at 8am, append output to a result file
0 8 * * * cd /opt/cdn-stats && python3 analyze.py >> hit-rate.log

Paired with an alerting tool, a hit rate below 80% for three consecutive days triggers a reminder, and rising origin traffic is caught before it turns into a bandwidth bill.

FAQ

Why does the script's hit rate not match the CDN console? Consoles usually bucket DYNAMIC, STALE, and other states separately, while the script only counts HIT and MISS. To align exactly, include EXPIRED and REVALIDATED in the denominator. Is a 100% hit rate always good? Not necessarily — if dynamic endpoints are cached, the numbers look great but users see stale data. What if the log is huge? Analyze only the last 24 hours of a rotated daily log, or pass a time range through sys.argv.

Best practices

• Feed the hit rate into your log analysis monitoring and alert when it drops below 70%.
• Focus on the top origin-heavy URLs first; fixing the 20% of pages that generate the most origin traffic usually brings the rate back above 90%.
• Revisit your cache strategy whenever the site's structure changes.
• Origin traffic costs money too, so hit-rate tuning is a direct lever for CDN cost control.

For broader CDN selection and setup, browse the CDN acceleration category.