ELK Log Analysis Platform Setup: Elasticsearch + Logstash + Kibana Hands-on Guide

ELK Log Analysis Platform Setup: Elasticsearch + Logstash + Kibana Hands-on Guide

The ELK Stack (Elasticsearch + Logstash + Kibana) is the gold standard for enterprise log analysis: Elasticsearch stores and full-text searches the logs, Logstash filters, transforms, and structures them, Kibana provides search, visualization, and dashboards, and a lightweight Filebeat handles collection — together forming a complete pipeline from "log produced" to "analyzed."

When Do You Need ELK

If your logs are still handled fine with grep and tail, you don't need ELK yet. It's worth introducing when any of these appears: logs from multiple apps or machines need centralized search; you need aggregation stats by error type, latency, or source; business teams (ops, support, R&D) should self-serve logs instead of always asking the backend; or you want log-based alerting (e.g., a spike in error logs).

Conversely, for a single host, small log volume, and "just needs to be searchable," lighter options like server log monitoring or Loki cost less. ELK's real cost is resources: a minimal stack needs at least 4-8GB of RAM, and ES nodes need dedicated disk I/O once volume grows.

1. Architecture Design

Log Sources → Filebeat → Logstash → Elasticsearch → Kibana
  |             |           |             |            |
  App Logs   Lightweight  Filter/     Store/Index   Visualization
              Collection   Transform

2. Docker Compose Deployment

version: '3'
services:
  elasticsearch:
    image: elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data

  logstash:
    image: logstash:8.12.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

  kibana:
    image: kibana:8.12.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

volumes:
  es-data:

Resource & Initialization Notes

• In a single-node test environment, give ES 1-2GB heap (ES_JAVA_OPTS) and never exceed half of physical memory;
• On Linux, raise vm.max_map_count or ES fails to start with max virtual memory areas vm.max_map_count [65530] is too low — run sysctl -w vm.max_map_count=262144;
• On first boot ES generates security certificates; Kibana connects to http://elasticsearch:9200 by default, and if security is enabled you must configure credentials in environment variables.

3. Logstash Configuration

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

4. Filebeat Configuration

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log
  fields:
    app: nginx
    env: production

output.logstash:
  hosts: ["logstash:5044"]

Verification & Troubleshooting

Once configured, verify the whole pipeline in order:

1. Confirm Filebeat is reading files: filebeat test output — it should reach Logstash on port 5044;
2. Check Logstash logs for grok parse errors: docker logs logstash;
3. Confirm indices are created: curl http://localhost:9200/_cat/indices?v — you should see indices like logs-2026.xx.xx;
4. Create an Index Pattern (e.g., logs-*) in Kibana, then search for a known access log in Discover to verify field parsing.

Common gotchas: a mismatched grok pattern drops logs out of ES (you'll see a _grokparsefailure tag); a misparsed time field makes logs sort by ingestion time instead of log time. Tune the grok pattern against one real log line in Logstash before bulk ingestion.

5. Key Features

A Real Troubleshooting Case

A business line ingested about 5GB of access logs per day. One night the alerting system saw 502s jump from a normal 0.2% to 8%. The on-call engineer used status:502 in Kibana Discover to lock a 30-minute window, then grouped by upstream_addr with a terms bucket — 90% of the 502s clustered on a single backend IP. A look at that node's response-time curve confirmed the database connection pool was exhausted. The whole diagnosis took under 10 minutes; with grep across machines it would have taken half an hour just to find the files. This "symptom to root cause" search path is exactly where ELK outshines plain log files. For more advanced alerting and visualization pairings, see Prometheus + Grafana basics.

Kibana Usage Essentials

Kibana's three most-used capabilities are Discover (full-text search), Visualize (charts), and Dashboard (boards). A typical scenario: 502s spike at 3 AM — search status:502 in Discover with the time range narrowed, then aggregate by upstream with a terms bucket to find which backend is failing. For more complex queries, switch to KQL, e.g., response_time > 3000 and status >= 500.

Best Practices

• Shard indices by day (logs-%{+YYYY.MM.dd}) and use ILM (Index Lifecycle Management) to roll over and delete old indices automatically, so logs never fill the disk;
• At higher volume, prefer a Filebeat → Kafka → Logstash buffered pipeline so log spikes don't overwhelm Logstash;
• Mask or drop sensitive fields (phone numbers, ID numbers) in Logstash before they reach ES;
• Set retention by need: audit logs usually 180 days, ordinary access logs 30 days is enough.

FAQ

How much memory should ES get? Elastic's guidance: heap no more than half of physical memory, and never above 30-31GB (beyond that, compressed pointers lose their benefit). For log-type workloads, more shards beat a bigger heap.

Logstash eats too much memory? Check for java or ruby filters in the pipeline (the most memory-hungry), and raise pipeline.batch.size to trade latency for throughput.

Is a single node enough? A single node fits scenarios with up to a few GB of logs per day; beyond that, split ES into three nodes and deploy Kibana and Logstash separately.

How do I choose between ELK and Loki? Loki trades cheap storage for query capability — great when you only need to find logs. ELK fits aggregation analysis, complex queries, and rich dashboards. See Loki log aggregation for a comparison.

How do I fix garbled Unicode/Chinese in logs? Check two places: whether filebeat.inputs declares encoding: utf-8, and whether Logstash converts field encoding before writing to ES. Most cases come from the collector reading the file with the wrong default encoding.

Reference: Elastic official docs — https://www.elastic.co/guide/index.html ; Filebeat reference — https://www.elastic.co/guide/en/beats/filebeat/current/index.html