Real-Time Anomaly Detection for Web Traffic with Grafana and ClickHouse
This post describes a real-time anomaly detection system for web traffic that we built with Grafana, ClickHouse and Python. It runs in production and detects DDoS attacks, bot campaigns and credential stuffing.
The short version: it processes 50,000 requests per second, raises an alert within 90 seconds, and produces 85% fewer false positives than the static thresholds it replaced.
Why thresholds weren't enough
Most monitoring alerts on static thresholds. That doesn't work well anymore. Attacks adapt to stay under the limits, and legitimate traffic has become more complicated too, with global users, mobile devices and very different usage patterns. The hard part is telling a harmless spike from an attack.
Ratios instead of raw volume
Instead of watching raw traffic volume, we compute ratios that describe behavior. These stay stable when legitimate traffic grows and change when something unusual happens.
Sessions and identity
- Unique IPs per URI (IP_URI_ratio): how many different sources hit a given resource.
- Sessions per IP (Session_IP_ratio): one IP opening many sessions usually means automation.
- Session duration deviation (Duration_anomaly): session lengths compared with the baseline.
Request patterns
- Requests per endpoint per minute (RPM_endpoint): sudden traffic concentrated on one resource.
- URI diversity per session (URI_diversity): navigation that doesn't look like a real user.
- User agent diversity (UA_diversity): many agent strings from few sources, typical of bot networks.
Errors
- 5xx rate per IP segment (Error_rate_IP): sources that are exhausting server resources.
- 404 patterns (NotFound_pattern): reconnaissance and vulnerability scanning.
- Response time outliers (Response_outlier): resource exhaustion attacks.
Architecture
There are three components.
Grafana for visualization and alerting. It connects to ClickHouse natively, so queries run in under a second. Dashboards show anomalies with confidence intervals, detected events are annotated on the timeline, and alerts are routed by severity.
ClickHouse as the time-series engine. Its columnar storage suits analytical queries. Logs stream in from the web servers in real time, materialized views keep the metrics aggregated continuously, and the history provides the baselines.
A Python detection service, running in containers. It computes the metrics, builds feature vectors, and scores and classifies anomalies with confidence intervals.
Detection algorithms
No single algorithm catches everything, so the service runs several:
- Statistical: z-scores for quick outlier detection, rolling quantile thresholds with an adaptive baseline, and seasonal decomposition for time patterns.
- Machine learning: Isolation Forest for multivariate anomalies, Prophet for deviations from seasonal patterns, and one-class SVM for novelty detection.
Each algorithm's output gets a confidence weight, and the outputs are checked against each other before an alert is raised. Requiring agreement between them is what keeps false positives low without losing sensitivity.
Results
- Detection latency: 30–90 seconds from event to alert.
- Throughput: 50,000 requests per second across several application instances.
- False positives: 85% lower than with static thresholds.
- Queries: 95% of dashboard queries return in under a second.
- Storage: 85% lower cost thanks to ClickHouse's columnar compression.
- Footprint: 2 CPU cores and 4 GB RAM for the whole detection service, with less than 1% added latency for the application. Storage grows linearly with traffic and is kept in check by retention policies.
| Metric | Thresholds | Our system | Change |
|---|---|---|---|
| False positive rate | 23.4% | 3.5% | 85% reduction |
| Detection latency | 5-15 minutes | 30-90 seconds | 75% improvement |
| Attack coverage | 34% | 89% | 162% increase |
| Operational overhead | High | Low | Significant reduction |
Case 1: credential stuffing
A coordinated campaign hit the login endpoints from distributed IP addresses. Several indicators fired together:
- Session_IP_ratio rose 847% above its baseline.
- URI_diversity showed the traffic was focused on authentication endpoints.
- 203 different user agents came from just 23 IP addresses.
- The login success rate stayed at zero while attempts went up.
The attack was detected and mitigated automatically within 2 minutes of starting, before it degraded service for legitimate users. The post-incident analysis confirmed a coordinated bot network using compromised credential databases.
Case 2: application-layer DDoS
This attack targeted the search function, which is expensive to run, while keeping overall request volume normal. Total traffic looked fine; what gave it away was the per-endpoint view:
- Response_outlier flagged server resources running out.
- Error_rate_IP showed the errors concentrated on a small set of sources.
We applied dynamic rate limiting to the affected endpoints only. Availability stayed above 99% throughout the attack, with little impact on users.