High-Cardinality Metrics Pitfalls

Introduction

High-cardinality metrics can provide valuable insights into system performance and user behavior. However, they come with significant challenges that can impact observability, data storage, and analysis.

Key Concepts

• Cardinality: Refers to the uniqueness of values in a dataset. High-cardinality metrics can have thousands or millions of unique values.
• Metrics: Numerical values that represent data points over time, often used for monitoring and observability.
• Observability: The capability to measure and understand the internal states of a system based on external outputs (metrics, logs, traces).

Common Pitfalls

• Excessive Storage Costs: High-cardinality metrics can lead to increased costs due to the volume of data stored.
• Performance Degradation: Querying large datasets can slow down performance and increase latency.
• Data Aggregation Challenges: Aggregating high-cardinality data can lead to incomplete or misleading insights.

Best Practices

1. Use Tagging Wisely: Limit the number of tags you use in metrics to avoid excessive cardinality.
2. Aggregate Where Possible: Aggregate data at a higher level to reduce cardinality without losing essential insights.
3. Monitor Storage Costs: Keep an eye on the costs associated with storing high-cardinality metrics and adjust as necessary.

Example Metric with High Cardinality

# Example of a high-cardinality metric in Prometheus
http_requests_total{method="GET", status="200", user_id="12345"}
            

High-Cardinality Decision Flowchart

graph TD;
    A[Start] --> B{Do you need high-cardinality metrics?};
    B -- Yes --> C[Define necessary tags];
    B -- No --> D[Use aggregated metrics];
    C --> E[Monitor storage and performance];
    D --> E;
    E --> F[End];
        

FAQ