PROVIDER Β· the classic tool for column-wise text processing
AWK
AWK is one of the three great Unix text tools, splitting input into lines and columns with pattern-action rules so log statistics and CSV column work take a single command on any server.
Pros
Cons
Product Information
Detailed Notes
Overview
AWK was created in 1977 at Bell Labs by Alfred Aho, Peter Weinberger, and Brian Kernighan, one of the three great Unix text tools, splitting input into lines and columns and applying pattern-action rules to filter and aggregate.
For Programming Languages & Frameworks work the value is instant statistics from logs and reports, since counting access logs by status code, summing and averaging CSV columns, or extracting and deduplicating column N takes one command without writing files, and gawk adds arrays and functions for moderate data reshaping.
Recognise the boundaries, because nested JSON and multi-line records make awk unreadable where Python is better, implementations differ between gawk, mawk, and busybox awk with slimmed versions common in containers, and float precision plus character encoding need care.
Core Strengths
- Line and column handling suits rapid log and table statistics
- One-liners cover grouping, counting, summing, deduplication, and formatting
- Available on every Unix and Linux system with no dependencies
- Combines with grep, sed, sort, and uniq into strong pipelines
Product Ecosystem
Syntax
Pattern and action rules with BEGIN and END blocks and built-in NR, NF, FS, and OFS variables.
Data handling
Associative arrays, string functions such as split, substr, and gsub, and numeric maths.
Composition
Pipelines with grep, sort, and uniq accomplish complex statistics.
Drawbacks
- Nested JSON and multi-line records make syntax unwieldy
- Implementations differ across gawk, mawk, and busybox
- Float precision and UTF-8 handling have pitfalls
- Readability trails Python for complex reshaping
Use Cases
- Log statistics and reports (β β β β β ) - One command delivers
- CSV and TSV column work (β β β β β ) - Extract and aggregate
- Ad hoc ops investigation (β β β β β ) - Built into servers
- Structured data transformation (β β ) - Prefer Python or jq
Pricing
| Plan | Price | Scope |
|---|---|---|
| gawk and awk implementations | $0 | GPL, entirely free |
| Related text tools | $0 | sed, grep, and sort ship with the OS |
| Commercial support | With OS support | Via distribution vendors |
Entirely free with no installation cost, and the small learning investment pays off long term, since understanding NR, NF, and associative arrays saves writing throwaway scripts.
FAQ
-
How do awk and sed divide work? Awk excels at columns and statistics while sed handles per-line substitution and deletion, often chained in one pipeline.
-
Is it slow on large files? Awk streams with constant memory handling tens of gigabytes sequentially, where disk I/O is usually the bottleneck.
-
gawk versus awk? gawk is the fullest GNU implementation with libraries and richer functions, while mawk is faster with fewer features and common in containers.
Code Examples
Top 10 IPs from an Nginx log
# top 10 IPs by request count
awk '{print $1}' access.log | sort | uniq -c | sort -rn | head -10
# count 5xx errors per hour
awk '$9 ~ /^5/ {split($4, t, ":"); print t[2]":00"}' access.log | sort | uniq -c
AWK splits fields on whitespace where $1 is the client IP, $4 the timestamp, and $9 the status code.
Aggregation script with header
#!/usr/bin/awk -f
BEGIN { FS = "\t"; print "host", "requests", "bytes" }
{ req[$1]++; bytes[$1] += $2 }
END { for (h in req) printf "%s,%d,%d\n", h, req[h], bytes[h] }
Run with awk -f stats.awk access.tsv for a stable operations reporting tool.
Community & Resources
Communities and official channels
- GNU AWK Manual β Authoritative syntax and function reference
- GitHub onetrueawk/awk β The original AWK implementation
- Stack Overflow #awk β Answers for one-liner problems
Learning resources and tools
- The AWK Programming Language β The authors' book online
- AWK Tutorial (Chinese) β Chinese introduction with examples
- explainshell β Explains each part of a command