Modern services rarely fail in a single, obvious way. A slow API may be caused by application code, database contention, pod throttling, a failing dependency, or an overloaded Kubernetes node. Effective monitoring must connect these layers instead of presenting each one in isolation.
Datadog can collect Kubernetes infrastructure metrics, application traces, logs, runtime data, events, and custom metrics. The challenge is not enabling every feature. It is creating a consistent telemetry model that helps engineers answer three questions quickly:
- Is the service meeting its reliability objectives?
- Which requests, dependencies, or resources explain a regression?
- What changed before the problem began?
This guide builds that model around a Spring Boot service running in Kubernetes. It covers deployment, instrumentation, tagging, dashboards, alerts, security, cost control, and the mistakes that commonly reduce signal quality.
Reference architecture
A typical Kubernetes installation runs a Datadog Agent on every node as a DaemonSet and a Cluster Agent as a Deployment. The node Agents collect container and host telemetry, receive application traces and DogStatsD metrics, and forward logs. The Cluster Agent handles cluster-level collection and coordinates selected features centrally.
flowchart LR
U[Client] --> S[Spring Boot service]
S -->|Traces and runtime metrics| A[Node Agent]
S -->|Custom metrics via DogStatsD| A
S -->|Logs to stdout| C[Container runtime]
C --> A
K[Kubernetes API] --> CA[Cluster Agent]
A --> D[Datadog]
CA --> D
The application should not contain a Datadog API key. It emits telemetry to the local Agent, while the Agent authenticates with Datadog. This reduces secret exposure and decouples the application from Datadog's intake endpoints.
Install the Datadog Agents correctly
For production clusters, use the official Datadog Helm chart or Datadog Operator rather than maintaining hand-written DaemonSet manifests. Both approaches make upgrades and configuration changes easier to review.
Create the API key through your normal secret-management workflow. Avoid placing it in a Git repository, Helm values file, shell history, or application environment. The Helm chart can reference an existing Kubernetes Secret.
The following abridged values illustrate the important decisions; verify property names against the chart version you pin:
datadog:
apiKeyExistingSecret: datadog-api-key
site: datadoghq.com
clusterName: production-us-east-1
logs:
enabled: true
containerCollectAll: false
apm:
portEnabled: true
processAgent:
processCollection: false
kubeStateMetricsCore:
enabled: true
clusterAgent:
enabled: true
admissionController:
enabled: true
mutateUnlabelled: false
Install a pinned chart version so that deployments are reproducible:
helm repo add datadog https://helm.datadoghq.com
helm repo update
helm upgrade --install datadog datadog/datadog \
--namespace datadog \
--create-namespace \
--version <tested-chart-version> \
--values values.yaml
Important production choices include:
- Set a unique cluster name. Otherwise, data from multiple clusters becomes difficult to distinguish.
- Select the correct Datadog site. The value depends on the Datadog organization and region.
- Pin and test upgrades. Agent and chart updates can change resource consumption, defaults, or supported features.
- Define Agent requests and limits. Observe actual usage before tightening limits; an under-provisioned Agent can drop or delay telemetry.
- Use tolerations where required. The DaemonSet should run on relevant nodes, including tainted node pools, if workloads there must be monitored.
- Limit log collection deliberately. Collecting every container log often creates noise and unnecessary ingestion cost.
After installation, verify that an Agent is ready on each expected node, the Cluster Agent is healthy, and Kubernetes nodes and workloads appear under the intended cluster name.
Establish a unified tagging model
Tags are the join keys between metrics, traces, logs, deployments, and infrastructure. Inconsistent tags are one of the most expensive observability mistakes because they make correlated data look unrelated.
Datadog's unified service tagging model uses three primary tags:
env: deployment environment, such asproductionservice: stable logical service name, such ascheckout-apiversion: deployed artifact or image version, preferably an immutable build identifier
Apply these as Kubernetes labels on both the Deployment and pod template:
metadata:
labels:
tags.datadoghq.com/env: production
tags.datadoghq.com/service: checkout-api
tags.datadoghq.com/version: "2025.03.18-7f2c9ab"
spec:
template:
metadata:
labels:
tags.datadoghq.com/env: production
tags.datadoghq.com/service: checkout-api
tags.datadoghq.com/version: "2025.03.18-7f2c9ab"
Keep tag values predictable and bounded. Good tags include team, region, cluster, and availability_zone. User IDs, request IDs, timestamps, full URLs, and exception messages are poor metric tags because they produce unbounded cardinality.
A useful rule is: if the number of possible values grows with traffic, customers, or time, do not use the field as a metric tag. Keep high-cardinality identifiers in logs or trace span attributes, where they can be searched without multiplying metric series.
Instrument Spring Boot with APM
Datadog's Java tracing library instruments common frameworks and clients, including Spring MVC, JDBC, and supported HTTP clients. It is loaded as a Java agent, so basic tracing usually requires little or no application code.
In Kubernetes, the Datadog admission controller can inject the Java tracing library. Label injection is safer than mutating every pod in the cluster:
spec:
template:
metadata:
labels:
tags.datadoghq.com/env: production
tags.datadoghq.com/service: checkout-api
tags.datadoghq.com/version: "2025.03.18-7f2c9ab"
admission.datadoghq.com/enabled: "true"
annotations:
admission.datadoghq.com/java-lib.version: "<tested-java-tracer-version>"
spec:
containers:
- name: checkout-api
image: registry.example.com/checkout-api:7f2c9ab
env:
- name: DD_LOGS_INJECTION
value: "true"
- name: DD_RUNTIME_METRICS_ENABLED
value: "true"
- name: DD_PROFILING_ENABLED
value: "false"
Pin a tracer version that has passed integration and load testing. Avoid a floating latest version: automatic changes to instrumentation inside a production JVM should be treated like dependency upgrades.
Runtime metrics add JVM visibility such as heap, garbage collection, and thread activity. Continuous profiling can provide deeper CPU and allocation analysis, but it should be enabled intentionally after considering overhead, security requirements, and account configuration.
Automatic instrumentation is a starting point, not a license to create spans for every method. Add custom spans only around meaningful operations such as pricing calculation, inventory reservation, or interactions with an unsupported dependency. Excessive spans increase ingestion volume and make traces harder to read.
Publish useful custom metrics
Infrastructure and APM data do not describe every business condition. A checkout service may need metrics for accepted orders, rejected payments, inventory conflicts, or queue depth.
Micrometer is the natural instrumentation facade for Spring Boot. One option is its StatsD registry configured with Datadog flavor, sending metrics to the node Agent over DogStatsD:
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-statsd</artifactId>
</dependency>
management:
statsd:
metrics:
export:
enabled: true
flavor: datadog
host: ${DD_AGENT_HOST}
port: 8125
The precise Spring Boot property set depends on the Boot and Micrometer versions, so validate it against the versions in the service. If node-local networking is configured manually, the node IP can be exposed through Kubernetes:
- name: DD_AGENT_HOST
valueFrom:
fieldRef:
fieldPath: status.hostIP
The Agent's DogStatsD endpoint must also be configured to accept this traffic. Prefer Unix domain sockets where supported by your deployment model because they avoid opening a node port, although they require mounting the Agent socket into the application pod.
Application metrics should describe decisions and outcomes:
Counter.builder("checkout.orders.completed")
.tag("payment_method", paymentMethod)
.register(meterRegistry)
.increment();
Timer.Sample sample = Timer.start(meterRegistry);
try {
pricingClient.calculate(cart);
} finally {
sample.stop(Timer.builder("checkout.pricing.duration")
.register(meterRegistry));
}
Only use payment_method as a tag if it comes from a small controlled set. Do not tag the counter with customer_id, order_id, or raw error text.
Before creating a custom metric, check whether an existing HTTP, JVM, trace-derived, or Kubernetes metric already answers the question. Every custom metric should have an owner, a documented purpose, and at least one dashboard, alert, or investigation workflow that uses it.
Make logs correlate with traces
Spring Boot applications should normally write logs to stdout and stderr; the container runtime and Agent handle collection. Avoid application-side log shippers unless there is a specific requirement, because multiple collectors can duplicate events.
Structured JSON logs are easier to parse reliably than free-form text. Include stable fields such as logger name, level, service, environment, and application-specific event type. With Java trace-log correlation and DD_LOGS_INJECTION=true, supported logging frameworks can include trace and span identifiers through their mapped diagnostic context.
The resulting investigation flow is more valuable than any isolated signal:
sequenceDiagram
participant E as Engineer
participant M as Monitor
participant T as Trace
participant L as Logs
participant K as Kubernetes
M->>E: Error-rate alert
E->>T: Open affected service traces
T->>L: Filter logs by trace identifier
L->>K: Compare pod, node, and deployment events
Do not log secrets, authorization headers, session tokens, or complete request bodies. Redact sensitive fields before they leave the application. Exclusion filters can reduce ingestion, but they are not a substitute for preventing sensitive data from being logged.
Use multiline processing cautiously. Java stack traces need to remain a single event, but incorrect rules can merge unrelated log lines. JSON logging generally avoids this ambiguity.
Build dashboards around operational questions
A dashboard should support a decision, not merely display available metrics. Start with one service overview containing:
- Request rate, error rate, and latency percentiles
- Availability or SLO status
- Top endpoints by latency and errors
- Dependency latency and failures
- JVM heap, garbage collection, threads, and CPU
- Pod restarts, readiness, CPU throttling, and memory pressure
- Replica count and deployment version
- Relevant business outcomes
Add template variables for env, service, version, cluster, and region. This allows the same dashboard to support production incidents, canary comparisons, and staging investigations.
Percentiles are usually more useful than average latency. An average can remain stable while a significant minority of users experience severe delays. Compare latency with request volume and error rate so that low-traffic anomalies are not mistaken for widespread impact.
Deployment markers are equally important. If errors increased immediately after version 7f2c9ab appeared, engineers should see that relationship without consulting a separate release system.
Alert on symptoms before causes
User-visible symptoms should page people; likely causes should provide context. A high CPU monitor alone is often noisy because high utilization can be healthy. A sustained increase in failed requests or exhausted latency budget is closer to customer impact.
A practical monitor hierarchy is:
- SLO or symptom monitors: availability, error rate, or latency burn.
- Service diagnostics: dependency failures, queue growth, or thread pool exhaustion.
- Platform diagnostics: unavailable replicas, crash loops, memory pressure, or node failures.
Use multi-window burn-rate alerts for SLOs when possible. A fast-burn alert detects severe incidents quickly, while a slow-burn alert finds persistent degradation without paging on short spikes.
Every actionable monitor should include:
- A plain-language description of the impact
- Service and environment context
- Dashboard and runbook links
- Ownership and routing information
- A recovery condition
Group notifications carefully. Grouping by pod can send dozens of alerts for one deployment failure. Grouping only by service may hide a regional failure. Choose dimensions that match how responders can act.
Control cost and telemetry volume
More telemetry is not automatically better telemetry. Datadog usage can grow through indexed logs, custom metric cardinality, ingested and retained traces, and enabled products.
Manage volume at the source:
- Collect logs only from required workloads.
- Exclude routine health-check and readiness logs when they add no diagnostic value.
- Keep metric tags bounded.
- Avoid custom spans around trivial methods.
- Configure trace ingestion and retention intentionally rather than assuming every trace must be retained.
- Set realistic log retention and indexing rules.
- Review usage by team and service regularly.
Sampling is a tradeoff. Lower ingestion reduces cost and noise, but overly aggressive sampling can remove rare failures. Preserve errors and important transactions while sampling repetitive successful traffic according to service volume and investigation needs.
Common pitfalls and better alternatives
Do not expose credentials to application pods
Only Agents and approved integration components should need the Datadog API key. Application keys are for Datadog API operations and should not be distributed across the cluster.
Do not monitor only Kubernetes
Healthy pods do not prove that users receive correct or timely responses. Combine infrastructure health with traces, service metrics, logs, and business outcomes.
Do not use mutable service names
Names such as checkout-api-v2 or checkout-api-green fragment service history. Keep service:checkout-api stable and represent deployments with the version tag.
Do not alert on every transient event
A single pod restart may be harmless during a rollout. Alert when restart frequency, unavailable replicas, or resulting service impact crosses a meaningful threshold.
Do not ignore Agent health
Monitoring has its own failure modes. Watch Agent readiness, restart count, resource pressure, collection errors, and telemetry delivery status. Missing data should not be interpreted as healthy data.
Do not enable everything cluster-wide on day one
Begin with a representative service and define success criteria: useful traces, correlated logs, controlled cardinality, actionable monitors, and acceptable overhead. Expand using reviewed defaults rather than one-off configurations.
A production rollout checklist
Before declaring the setup complete, confirm that:
- The Agent runs on every intended node and the Cluster Agent is healthy.
- Cluster, environment, service, and version tags are consistent.
- A test request produces a trace with expected downstream spans.
- Trace identifiers appear in correlated application logs.
- JVM and selected business metrics are visible.
- Sensitive fields are absent or redacted.
- Dashboards support filtering by deployment version and cluster.
- Alerts route to the owning team and link to a runbook.
- A controlled failure triggers and then recovers the expected monitor.
- Telemetry volume and cardinality are reviewed after realistic traffic.
- Agent, chart, and tracer versions are pinned and included in upgrade testing.
Conclusion
A strong Datadog implementation is built around correlation and operational discipline, not the number of enabled integrations. Deploy Agents consistently, keep credentials out of applications, standardize env, service, and version, and instrument Spring Boot with a combination of automatic tracing and carefully selected business metrics.
Then optimize for action: structured logs linked to traces, dashboards organized around service health, SLO-oriented alerts, bounded tag cardinality, and explicit telemetry budgets. When these foundations are in place, engineers can move from an alert to the affected request, log event, deployment, pod, and node without assembling the incident by hand.
