Splunk

index=messaging sourcetype=app_log status=FAILED
| stats count by error_code, account_id, host
| sort - count
| head 20

--- Follow-up: drill into top error ---

index=messaging sourcetype=app_log status=FAILED error_code="TIMEOUT_504"
| transaction account_id maxspan=5m
| table _time, account_id, message_id, host, duration_ms
| sort - duration_ms

index=messaging sourcetype=app_log
| bin _time span=5m
| stats count AS msg_count by _time, src_ip, account_id
| eventstats avg(msg_count) AS avg_count stdev(msg_count) AS stdev_count by account_id
| eval threshold = avg_count + (3 * stdev_count)
| where msg_count > threshold
| eval spike_ratio = round(msg_count / avg_count, 2)
| table _time, account_id, src_ip, msg_count, avg_count, spike_ratio
| sort - spike_ratio

index=messaging sourcetype=app_log
| bin _time span=1m
| stats count AS total,
        count(eval(status="FAILED")) AS failures by _time
| eval error_rate = round(100 * failures / total, 2)
| where error_rate > 5 AND total > 100
| stats latest(error_rate) AS current_rate,
        latest(total) AS requests_per_min

# Alert Settings:
# Trigger: number of results > 0
# Throttle: 10 minutes
# Action: PagerDuty / Slack webhook
# Severity: Critical

## Index Strategy

- Separate indexes by data sensitivity (security vs. ops)
- Use `index=messaging` not `index=*` in production searches
- Set retention per index based on compliance requirements

## Performance Tips

- Always bound time range (`earliest=-1h` minimum)
- Use `tstats` for metrics instead of raw searches when possible
- Extract fields at index time for high-cardinality fields
- Avoid leading wildcards: `*error*` is slow; use `error*` instead

## Common Fields to Extract

| Field | Regex / Method |
|-------|----------------|
| account_id | `EXTRACT account_id=(?<account_id>\d+)` |
| message_id | `EXTRACT message_id=(?<message_id>[A-Z0-9-]+)` |
| duration_ms | `EXTRACT duration=(?<duration_ms>\d+)ms` |