Monitoring and Observability for Production Web Apps

Most web applications are built without a serious observability strategy. They work fine until something goes wrong at 2am, and then the on-call engineer spends an hour grepping through server logs, refreshing dashboards, and trying to correlate timestamps across three different systems. This is not a tooling problem — it is a design problem. Observability needs to be built into your application from the beginning, not bolted on after the first production incident.

Monitoring is watching known failure modes — CPU over 90%, error rate over 1%, response time over 500ms. Observability is the system's ability to answer questions you did not know you would need to ask. A highly observable system lets you understand novel failure modes from the outside, by examining the outputs the system produces: logs, metrics, and traces.

The Three Pillars: Logs, Metrics, Traces

Logs: Events in Time

Logs are timestamped records of discrete events. They are the most familiar signal — every application produces them by default — but they are also the most frequently misused. The problem with most application logs is that they are written for humans to read in development, not for machines to query in production.

The shift from unstructured to structured logging is the single highest-leverage improvement most teams can make. Instead of:

logger.info(f"User {user_id} logged in from {ip_address}")

Write:

logger.info("user.login", extra={
    "user_id": user_id,
    "ip_address": ip_address,
    "event": "user.login"
})

Structured logs (typically JSON) are machine-parseable, filterable, and aggregatable. In Elasticsearch, Loki, or CloudWatch Logs Insights, you can query event = "user.login" AND ip_address = "x.x.x.x" instantly across millions of lines. In Python, the structlog library makes structured logging straightforward. In Node.js, pino outputs JSON by default.

Metrics: Aggregated Numbers Over Time

Metrics are numerical measurements sampled at intervals — request rate, error count, database query duration, cache hit ratio, memory usage. The four golden signals (from Google's Site Reliability Engineering book) are the starting point for any application metric strategy:

• Latency: how long requests take to complete
• Traffic: demand on the system — requests per second
• Errors: the rate of failed requests — HTTP 5xx, application exceptions
• Saturation: how full your resources are — CPU, memory, disk, queue depth

Prometheus is the de facto standard for metrics collection in cloud-native environments. It scrapes metrics from instrumented services via HTTP at regular intervals and stores them in a time-series database. Grafana provides dashboards and alerting on top of Prometheus data. For Python applications, prometheus-client exposes a /metrics endpoint. django-prometheus automatically instruments request durations, database query counts, and cache operations.

Traces: Following a Request Across Services

Distributed traces record the path of a single request through a system — from the load balancer through the application layer to each database query and external API call. Each segment is a "span," and spans are linked by a common trace ID that propagates in request headers. Tracing answers questions that logs and metrics cannot: why did this specific request take 2.3 seconds when the median is 80ms?

OpenTelemetry has become the vendor-neutral standard for trace instrumentation. It works with Jaeger, Tempo, Honeycomb, and Datadog. For Python, opentelemetry-sdk with auto-instrumentation covers Django, requests, SQLAlchemy, and Redis out of the box.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("payment.amount", amount)
    span.set_attribute("payment.currency", currency)
    result = payment_gateway.charge(amount, currency)

Alerting: Page on Symptoms, Not Causes

Alert fatigue is one of the leading causes of missed incidents in production systems. The principle: page on symptoms, not causes. Alerting on error rate above 1% is a symptom — it directly affects users. Alerting on CPU above 80% is a cause — it might lead to a problem, but by itself it is not one.

Your core alert set should start with:

• HTTP 5xx error rate above threshold for 5 minutes
• P95 latency above threshold for 5 minutes
• Health check endpoint returning non-200 for 2 minutes
• Disk space below 10%
• Deployment failure or rollback trigger

Every alert should have a runbook entry that explains what the alert means and what to check first. An alert without a runbook is an alert that creates confusion, not resolution.

Error Tracking: Grouping Exceptions Intelligently

Error tracking platforms like Sentry, Rollbar, or Honeybadger specifically capture exceptions, group them by fingerprint (so you see "this error occurred 342 times" rather than 342 individual log lines), and annotate them with context: which user, which release, which URL, what the request parameters were. Sentry integrates with Django in three lines and captures unhandled exceptions automatically. It also supports performance monitoring and can alert on regression rates — new errors introduced by a specific deployment.

A Minimal Production Stack

For a small-to-medium production application, a practical stack that covers all three pillars without excessive complexity:

• Structured logging: structlog → Loki (self-hosted) or CloudWatch (managed)
• Metrics: prometheus-client + Prometheus + Grafana
• Traces: OpenTelemetry → Tempo (self-hosted) or Honeycomb (managed)
• Error tracking: Sentry (free tier covers most small applications)
• Uptime monitoring: Better Uptime or UptimeRobot for external health checks

The Grafana stack (Loki + Prometheus + Tempo + Grafana) is particularly compelling because a single Grafana instance provides unified access to logs, metrics, and traces with cross-signal correlation built in. You can click on a log line and jump directly to the trace for that request, or click on a metric spike and see related logs in the same time window.

At AgiCAD, I set up the minimal Grafana stack as a standard part of every client project's infrastructure. The cost of running Prometheus, Loki, and Tempo alongside a medium-traffic application is minimal — a single 2 GB RAM VM handles all three comfortably. The time saved in the first serious production incident pays for a year of infrastructure costs, and the confidence that comes from knowing you will see problems before users report them is a meaningful improvement for the engineering team. The most important thing is to start: add structured logging to your next deployed feature, set up a basic Prometheus scrape, add Sentry. Each piece compounds the others.