Server Operation & Maintenance Guide: daily management, monitoring, and troubleshooting

Server Operation & Maintenance Guide: daily management, monitoring, and troubleshooting

Server maintenance is the foundation of reliable website operations. Over 60% of website outages can be prevented through standardized maintenance procedures. This guide covers daily management and troubleshooting techniques.

A good operations team often appears to be "doing nothing" — because incidents are resolved before users notice. This guide assumes you manage 1-10 Linux servers and covers the full loop from daily checks and alert configuration to locating common faults. Master these, and most "the site suddenly won't load" problems can be diagnosed within 30 minutes.

1. Daily Checklist

Daily Checks

# System load
uptime
top -bn1 | head -5
free -h

# Disk
df -h
du -sh /var/log/

# Network
ping -c 4 google.com
ss -tlnp

# Services
systemctl status nginx
systemctl status mysql
systemctl status php8.3-fpm

Reading uptime output is a basic skill: load average: 0.42, 0.31, 0.28 are the average load over the past 1, 5, and 15 minutes. On a single-core machine, a load above 1.0 means the CPU is queuing, and sustained values above the core count warrant investigation; if the 1-minute load is far above the 15-minute load, it is usually a traffic spike or slow queries — grab top first to see which process is taking the hit.

Weekly Maintenance

# Updates
apt update && apt upgrade -y

# Check errors
journalctl -p err -b
tail -100 /var/log/nginx/error.log

# Database maintenance
mysqlcheck -o --all-databases

# SSL expiry check
openssl s_client -connect example.com:443 -servername example.com < /dev/null 2>/dev/null | openssl x509 -noout -dates

Monthly

• Audit user accounts and SSH keys
• Review login attempts
• Rotate and compress logs
• Test backup restoration
• Evaluate resource usage trends

2. Monitoring Setup

UptimeRobot

Free website monitoring service:

1. Register at UptimeRobot
2. Add monitor → HTTP(s) type
3. Set check frequency (5 min free)
4. Configure notifications (Email/Slack/Telegram)

Self-hosted: Prometheus + Grafana

version: '3.8'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
  node-exporter:
    image: prom/node-exporter
    ports:
      - "9100:9100"

The most common alerting mistake is alerting on everything. Disk at 80%, CPU at 90% — fifty notifications a day make people numb, and real incidents get drowned out. Alert only on events that genuinely need a human: disk below 20% free, service down, certificates expiring within 7 days, backup failures. Use dashboards to watch trends for everything else.

3. Troubleshooting

Website Down Flowchart

User reports site down
  │
  ├─→ Is server online?
  │    ├─→ ping IP
  │    ├─→ SSH reachable?
  │    └─→ Contact data center
  │
  ├─→ Web service running?
  │    ├─→ systemctl status nginx
  │    ├─→ ss -tlnp | grep :80
  │    └─→ Check error logs
  │
  ├─→ Database running?
  │    ├─→ systemctl status mysql
  │    ├─→ mysqladmin ping
  │    └─→ Check disk space
  │
  ├─→ Firewall issues?
  │    ├─→ iptables -L -n
  │    ├─→ ufw status
  │    └─→ Check cloud security groups
  │
  └─→ SSL certificate?
       ├─→ openssl s_client
       └─→ certbot certificates

High CPU/Memory

top -o %CPU
top -o %MEM
ps aux --sort=-%cpu | head

Disk Cleanup

find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null
journalctl --vacuum-time=7d
docker system prune -af
apt autoremove -y
apt autoclean

A Real Troubleshooting Case

Before a flash-sale night, an e-commerce site got reports that "adding to cart is very slow". The operations engineer first checked load — not high; then ss -tlnp — port 80 was fine; then the database, where SHOW PROCESSLIST revealed many Sleep connections exhausting max_connections. Root cause: an oversized connection pool with no wait_timeout, and slow requests were burning through connections. Temporarily raising max_connections restored service, then tightening the pool and adding connection reuse in the app layer fixed it for good. The common thread in cases like this: first confirm the service is up, then check resources, and only then suspect application logic — get the order right and half the problem is already solved.

4. Backup Strategy

#!/bin/bash
BACKUP_DIR="/backup"
DATE=$(date +%Y%m%d)

# MySQL backup
mysqldump --all-databases --single-transaction | gzip > ${BACKUP_DIR}/mysql_${DATE}.sql.gz

# Website files
tar -czf ${BACKUP_DIR}/www_${DATE}.tar.gz /var/www/

# Config files
tar -czf ${BACKUP_DIR}/etc_${DATE}.tar.gz /etc/nginx/ /etc/mysql/ /etc/php/

# Delete backups older than 30 days
find ${BACKUP_DIR} -name "*.gz" -mtime +30 -delete

# Offsite backup
rclone copy ${BACKUP_DIR} remote:backup-bucket/ --progress

Backup Strategy

5. Performance Tuning

cat >> /etc/sysctl.conf << 'EOF'
# Network tuning
net.core.somaxconn = 1024
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_tw_reuse = 1

# File handle limits
fs.file-max = 100000
EOF

sysctl -p

Tune these to match your workload: a server mainly running an Nginx reverse proxy benefits from a larger somaxconn and tcp_max_syn_backlog; a high-concurrency API benefits from tcp_tw_reuse to cut TIME_WAIT buildup. sysctl -p applies changes immediately, but make them during a low-traffic window and watch for a day or two so a traffic spike does not reveal a new bottleneck.

Reference: Linux performance analysis tools https://www.brendangregg.com/linuxperf.html; systemd manual https://www.freedesktop.org/software/systemd/man/systemd.html; Prometheus docs https://prometheus.io/docs/introduction/overview/

16IDC Takeaway

The core of server maintenance isn't about fixing problems fast — it's about preventing them. Establish monitoring (even simple UptimeRobot + Telegram), automated backups (backup is only useful if you can restore), and an operations manual from day one. These basics will solve 90% of issues before users notice.