Security Incident Response: From Detection to Review

Security Incident Response: From Detection to Review

A security incident is not a question of "whether" but "when". NIST SP 800-61, the Computer Security Incident Handling Guide, models incident response as a continuously improving lifecycle: Preparation, Detection and Analysis, Containment/Eradication/Recovery, and Post-Incident Activity. For small and mid-sized sites without a dedicated security team, the point is not to copy the process verbatim, but to write down clearly what to do in the first 30 minutes after an anomaly is spotted.

The incident response lifecycle

• Preparation: build a response team, prepare tools and contact lists, and write a plan.
• Detection and Analysis: identify anomalies, confirm whether a real incident occurred, and assess scope and severity.
• Containment, Eradication and Recovery: isolate affected systems, remove malicious programs, and restore business.
• Post-Incident Activity: review root causes, improve defenses, and capture lessons learned.

A timeline you can actually follow

Mapping the lifecycle to concrete minutes keeps the plan from staying on paper. Here is a reference timeline that works for a single server or a small fleet; adjust it to your team size:

Preparation: write the checklist first

• Define contacts (owner, hosting provider, cloud vendor, legal/regulatory points of contact).
• Prepare forensics and isolation tools plus a clean backup environment.
• Front-load logging and monitoring: integrate Monitoring and Alerts and Security Log Auditing so there is enough audit trail.

Common signals for detection and analysis

• Anomalous logins: unfamiliar IPs, logins outside working hours, or a surge of failed root attempts (see SSH Hardening).
• Unusual server behavior: CPU/bandwidth spikes, mining processes, or abnormal outbound connections.
• File anomalies: website files tampered with, new backdoor scripts, or a ransomed database.
• External reports: alerts from cloud vendors or security platforms, abnormal search-engine indexing, or user reports of page hijacking.

Once a signal appears, a few commands confirm whether it is a real incident (do not kill anything yet):

last -i -20                 # recent login sources
ss -tunap | head -40        # current outbound sessions
top -c -b -n 1 | head -25   # highest CPU processes
lsof -p <PID> | head -30    # files opened by a suspicious PID

A real scenario: the 3 a.m. miner

Say your monitoring fires at 3 a.m.: a web server pegs at 100% CPU. top shows a process named kworkerds eating 300% CPU — the name mimics a kernel thread, but its path is /tmp/.X11-unix/. Do not kill it. Capture the ps -ef output and the outbound addresses from ss -tunap first, then isolate the host as described below. These miners usually enter through an unhardened Redis, a vulnerable plugin, or weak passwords. When eradicating, besides deleting files, check crontab -l, systemctl list-unit-files, and /etc/ld.so.preload so the daemon cannot relaunch itself.

Containment, eradication, and recovery

1. Contain: cut external network access to affected servers, freeze accounts, and preserve the scene (collect evidence before handling).
2. Snapshot: take a snapshot or disk copy of affected systems for later analysis (see the Backup Strategy).
3. Eradicate: locate and remove malicious programs, fix vulnerabilities (combined with Vulnerability Scanning), and rotate all affected credentials.
4. Recover: rebuild systems from a clean backup, verify business and security configuration, restore gradually, and observe for a while to confirm no recurrence.

What to preserve for forensics

Small-site forensics does not need expensive tools; the point is "keep things as they are." At minimum, keep: a disk snapshot or dd image of the affected host, system and access logs under /var/log, a process and connection inventory, and sha256sum hashes of tampered files. These are both your investigation trail and evidence for your cloud vendor, regulators, or insurers. Copy the scene with rsync or a vendor snapshot before touching anything, then you can experiment freely.

Post-incident review

A review is not about assigning blame but about understanding "why it happened and how to prevent it". Use the Incident Postmortem Template to capture the document, then return to the Security Hardening category to fill the weak spots.

What a usable plan should include

An executable plan should contain at least:

• Severity definitions (P0/P1/P2) and an escalation path;
• Clear role ownership and contact lists (cloud vendor, hosting provider, legal counsel);
• Standard handling steps and a tool list (isolation commands, forensics tools, a clean recovery environment);
• External communication wording and notification templates (users, regulators, media);
• Recovery acceptance criteria and review scheduling.

Common pitfalls

• In the panic, deleting data or reinstalling immediately, losing forensic evidence;
• Restoring service without eradicating the backdoor, only to be breached again weeks later;
• Not rotating affected credentials, letting attackers back in with old passwords;
• No rehearsal, so the process and tools prove unusable when it really happens.

Run a tabletop or simulation exercise every six months so the process and contacts genuinely work.

FAQ

• Do we really need a SIEM? Not necessarily. Start by retaining system logs, access logs, and cloud monitoring; add centralized collection when volume grows.
• No dedicated staff after hours? Cover it with alert escalation and an on-call rota — a few extra phone calls beat making a unilateral call at 3 a.m.
• How long should logs be kept? At least 30 days; extend to 180 days or more for compliance.

16IDC Observation

For small and mid-sized sites, the most important outputs of incident response are "a plan you can actually follow" and "enough historical logs". Rather than chasing complex SIEM systems, make monitoring, logging, and backup solid first, so that when an incident happens you have traces to investigate and backups to recover from.

Source: https://csrc.nist.gov/pubs/sp/800/61/r3/final
Reference: NIST SP 800-61 Rev. 3 https://csrc.nist.gov/pubs/sp/800/61/r3/final
Reference: CISA Incident Response Plan https://www.cisa.gov/resources-tools/resources/incident-response-plan