ELK Log Analysis Platform Setup: Elasticsearch + Logstash + Kibana Hands-on Guide

ELK Log Analysis Platform Setup: Elasticsearch + Logstash + Kibana Hands-on Guide

The ELK Stack (Elasticsearch + Logstash + Kibana) is the gold standard for enterprise log analysis.

1. Architecture Design

Log Sources → Filebeat → Logstash → Elasticsearch → Kibana
  |             |           |             |            |
  App Logs   Lightweight  Filter/     Store/Index   Visualization
              Collection   Transform

2. Docker Compose Deployment

version: '3'
services:
  elasticsearch:
    image: elasticsearch:8.12.0
    environment:
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data

  logstash:
    image: logstash:8.12.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    depends_on:
      - elasticsearch

  kibana:
    image: kibana:8.12.0
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

volumes:
  es-data:

3. Logstash Configuration

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "logs-%{+YYYY.MM.dd}"
  }
}

4. Filebeat Configuration

filebeat.inputs:
- type: log
  paths:
    - /var/log/nginx/access.log
  fields:
    app: nginx
    env: production

output.logstash:
  hosts: ["logstash:5044"]

5. Key Features