Security Log Auditing Basics: Building a Traceable Log System

Introduction to Security Log Auditing

Security log auditing is the cornerstone of security operations (SecOps). It involves the systematic collection, storage, analysis, and review of logs generated by systems, networks, applications, and security devices to detect security incidents, meet compliance requirements, and support post-incident forensics.

Why Log Auditing is Critical

• Security Threat Detection: By analyzing abnormal patterns in logs (such as multiple failed logins, access during unusual hours, privileged operations), you can detect intrusion signs before attacks cause substantial damage
• Compliance Requirements: PCI DSS, ISO 27001, SOC 2, GDPR, China's Cybersecurity Law, and other regulations all explicitly require log auditing and retention
• Incident Response: After a security incident, logs are the core evidence for reconstructing the attack chain, determining the scope of impact, and tracing the source of the attack
• Operational Troubleshooting: Log auditing helps operations teams track system anomalies, performance bottlenecks, and configuration errors

According to Verizon's 2026 Data Breach Investigations Report, 65% of data breaches involved insufficient logging or missing log monitoring—in other words, if log systems were complete, more than half of these breaches could have been detected earlier or completely avoided.

Log Sources and Classification

Major Log Sources

Log Level Standards

Follow RFC 5424 syslog priority standards:

Log Audit Architecture Design

Centralized Log Architecture

Modern log audit systems typically use ELK (Elasticsearch + Logstash + Kibana) or Loki + Grafana technology stacks to build a centralized log platform:

Log Sources → Filebeat/Fluentd (Log Collection)
                   ↓
              Kafka/Redis (Buffer Queue)
                   ↓
           Logstash/Fluentd (Parsing & Filtering)
                   ↓
        Elasticsearch / Loki (Storage & Indexing)
                   ↓
            Kibana / Grafana (Visualization & Alerts)

Key Architecture Considerations

1. Log Volume Estimation: Estimate daily log volume based on business scale. A medium-scale web application (100K daily PV) generates approximately 5-20 GB of logs per day, requiring storage capacity and retention strategy planning
2. Retention Strategy: Hot data (last 7 days) on SSD for query speed; warm data (7-90 days) on HDD; cold data (> 90 days) archived to object storage (S3, OSS)
3. Log Immutability: For compliance audits (e.g., PCI DSS log requirements), use log signing or WORM storage to ensure logs cannot be modified or deleted during the retention period
4. High Availability Design: Log collectors and buffer queues should be redundantly deployed to avoid audit blind spots from single log source failures

Log Normalization

Logs from different sources have varying formats. It is recommended to standardize into a unified format:

{
  "timestamp": "2026-07-18T08:30:00.123Z",
  "source": "web-server-01",
  "type": "access_log",
  "level": "INFO",
  "message": "GET /api/orders HTTP/1.1 200 1024",
  "fields": {
    "client_ip": "203.0.113.42",
    "method": "GET",
    "path": "/api/orders",
    "status_code": 200,
    "response_time_ms": 245,
    "user_agent": "Mozilla/5.0 ..."
  }
}

Log Analysis Strategies

Establishing Baselines

Establish baselines for normal log patterns to effectively detect anomalies. Baseline metrics include:

• Average requests per minute per port
• Average response time per API endpoint
• Average login frequency per user
• Error rate baseline per service

Typical Detection Scenarios

Alert Strategy

• Avoid alert fatigue: Classify alerts (Critical > Warning > Info). Critical alerts notify via phone/PagerDuty, Warning sends email, Info logs to ticketing system
• Set noise suppression: Automatically merge identical alerts triggered repeatedly within a set time window
• Regular alert review: Review alert hit rate and false positive rate weekly, adjust rules to reduce false positives

Major Log Audit Tools Comparison

Compliance Audit Requirements

Different compliance standards have specific log audit requirements:

Conclusion

Security log auditing is not simply "recording logs"—it is a systematic discipline spanning architecture design, log normalization, analysis strategies, alert management, and compliance auditing. A well-established log audit system can provide early warnings before security incidents occur, deliver real-time situational awareness during incidents, and support forensic analysis after incidents.

For SMBs, it is recommended to start with ELK or Grafana Loki to build a basic audit platform, combined with open source SIEM tools like Wazuh for automated security event detection. As the business scales, gradually migrate to commercial solutions like Datadog or Splunk, or cloud vendors' managed log services.