Recorder History Optimization

Status: ✅ IMPLEMENTED Last Updated: 2025-12-07

Overview

This document describes the implementation of _unrecorded_attributes for Tibber Prices entities to prevent Home Assistant Recorder database bloat by excluding non-essential attributes from historical data storage.

Reference: HA Developer Docs - Excluding State Attributes

Implementation

Both TibberPricesSensor and TibberPricesBinarySensor implement _unrecorded_attributes as a class-level frozenset to exclude attributes that don't provide value in historical data analysis.

Pattern

class TibberPricesSensor(TibberPricesEntity, SensorEntity):
    """tibber_prices Sensor class."""

    _unrecorded_attributes = frozenset(
        {
            "description",
            "usage_tips",
            # ... more attributes
        }
    )

Key Points:

• Must be a class attribute (not instance attribute)
• Use frozenset for immutability and performance
• Applied automatically by Home Assistant's Recorder component

Categories of Excluded Attributes

1. Descriptions/Help Text

Attributes: description, usage_tips

Reason: Static, large text strings (100-500 chars each) that:

• Never change or change very rarely
• Don't provide analytical value in history
• Consume significant database space when recorded every state change
• Can be retrieved from translation files when needed

Impact: ~500-1000 bytes saved per state change

2. Large Nested Structures

Attributes:

• periods (binary_sensor) - Array of all period summaries
• data (chart_data_export) - Complete price data arrays
• trend_attributes - Detailed trend analysis
• current_trend_attributes - Current trend details
• trend_change_attributes - Trend change analysis
• volatility_attributes - Detailed volatility breakdown

Reason: Complex nested data structures that are:

• Serialized to JSON for storage (expensive)
• Create large database rows (2-20 KB each)
• Slow down history queries
• Provide limited value in historical analysis (current state usually sufficient)

Impact: ~10-30 KB saved per state change for affected sensors

Example - periods array:

{
    "periods": [
        {
            "start": "2025-12-07T06:00:00+01:00",
            "end": "2025-12-07T08:00:00+01:00",
            "duration_minutes": 120,
            "price_mean": 18.5,
            "price_median": 18.3,
            "price_min": 17.2,
            "price_max": 19.8
            // ... 10+ more attributes × 10-20 periods
        }
    ]
}

3. Frequently Changing Diagnostics

Attributes: icon_color, cache_age, cache_validity, data_completeness, data_status

Reason:

• Change every update cycle (every 15 minutes or more frequently)
• Don't provide long-term analytical value
• Create state changes even when core values haven't changed
• Clutter history with cosmetic changes
• Can be reconstructed from other attributes if needed

Impact: Prevents unnecessary state writes when only cosmetic attributes change

Example: icon_color changes from #00ff00 to #ffff00 but price hasn't changed → No state write needed

4. Static/Rarely Changing Configuration

Attributes: tomorrow_expected_after, level_value, rating_value, level_id, rating_id, currency, resolution, yaxis_min, yaxis_max

Reason:

• Configuration values that rarely change
• Wastes space when recorded repeatedly
• Can be derived from other attributes or from entity state

Impact: ~100-200 bytes saved per state change

5. Temporary/Time-Bound Data

Attributes: timestamp, next_api_poll, next_midnight_turnover, last_api_fetch, last_cache_update, last_turnover, last_error, error

Reason:

• timestamp is the rounded-quarter reference time used at the moment of the state write — it's stale as soon as the next update fires and has no analytical value in history
• next_api_poll, next_midnight_turnover etc. are only relevant at the moment of reading; they're superseded by the next update
• Similar to entity_picture in HA core image entities

Note: The entity's native_value (the actual price/state) is always recorded by HA as the entity state itself — independently of _unrecorded_attributes. So excluding timestamp does not create a gap in the time-series; the state row already carries the recording timestamp.

Impact: ~200-400 bytes saved per state change

Example: next_api_poll: "2025-12-07T14:30:00" stored at 14:15 is useless when viewing history at 15:00

6. Relaxation Details

Attributes: relaxation_level, relaxation_threshold_original_%, relaxation_threshold_applied_%

Reason:

• Detailed technical information not needed for historical analysis
• Only useful for debugging during active development
• Boolean relaxation_active is kept for high-level analysis

Impact: ~50-100 bytes saved per state change

7. Redundant/Derived Data

Attributes: price_spread, volatility, diff_%, rating_difference_%, period_price_diff_from_daily_min, period_price_diff_from_daily_min_%, period_count_total, period_count_remaining

Reason:

• Can be calculated from other attributes
• Redundant information
• Doesn't add analytical value to history

Impact: ~100-200 bytes saved per state change

Example: price_spread = price_max - price_min (both are recorded, so spread can be calculated). period_count_remaining = period_count_total - period_position (both components are recorded).

Attributes That ARE Recorded

These attributes remain in history because they provide essential analytical value:

Time-Series Core

• All price values - Core sensor states (the entity's native_value is always recorded separately)

Diagnostics & Tracking

• cache_age_minutes - Numeric value for diagnostics tracking over time
• updates_today - Tracking API usage patterns

Data Completeness

• interval_count, intervals_available - Data completeness metrics
• yesterday_available, today_available, tomorrow_available - Boolean status

Period Data

• start, end, duration_minutes - Core period timing
• price_mean, price_median, price_min, price_max - Core price statistics
• period_position - Position of current period in the day's sequence
• period_count_today, period_count_tomorrow - How many periods per day (useful in automations)

High-Level Status

• relaxation_active - Whether relaxation was used (boolean, useful for analyzing when periods needed relaxation)

Expected Database Impact

Space Savings

Per state change:

• Before: ~3-8 KB average
• After: ~0.5-1.5 KB average
• Reduction: 60-85%

Daily per sensor:

Sensor Type	Updates/Day	Before	After	Savings
High-frequency (15min)	96	~290 KB	~140 KB	50%
Low-frequency (6h)	4	~32 KB	~6 KB	80%

Most Impactful Exclusions

1. periods array (binary_sensor) - Saves 2-5 KB per state
2. data (chart_data_export) - Saves 5-20 KB per state
3. trend_attributes - Saves 1-2 KB per state
4. description/usage_tips - Saves 500-1000 bytes per state
5. icon_color - Prevents unnecessary state changes

Real-World Impact

For a typical installation with:

• 80+ sensors
• Updates every 15 minutes
• ~10 sensors updating every minute

Before: ~1.5 GB per month After: ~400-500 MB per month Savings: ~1 GB per month (~66% reduction)

Implementation Files

• Sensor Platform: custom_components/tibber_prices/sensor/core.py

  • Class: TibberPricesSensor
  • 46 attributes excluded

• Binary Sensor Platform: custom_components/tibber_prices/binary_sensor/core.py

  • Class: TibberPricesBinarySensor
  • 29 attributes excluded

When to Update _unrecorded_attributes

Add to Exclusion List When:

✅ Adding new description/help text attributes ✅ Adding large nested structures (arrays, complex objects) ✅ Adding frequently changing diagnostic info (colors, formatted strings) ✅ Adding temporary/time-bound data (timestamps that become stale) ✅ Adding redundant/derived calculations

Keep in History When:

✅ Core price/timing data needed for analysis ✅ Boolean status flags that show state transitions ✅ Numeric counters useful for tracking patterns ✅ Data that helps understand system behavior over time

Decision Framework

When adding a new attribute, ask:

1. Will this be useful in history queries 1 week from now?

   • No → Exclude
   • Yes → Keep

2. Can this be calculated from other recorded attributes?

   • Yes → Exclude
   • No → Keep

3. Is this primarily for current UI display?

   • Yes → Exclude
   • No → Keep

4. Does this change frequently without indicating state change?

   • Yes → Exclude
   • No → Keep

5. Is this larger than 100 bytes and not essential for analysis?

   • Yes → Exclude
   • No → Keep

Testing

After modifying _unrecorded_attributes:

1. Restart Home Assistant to apply changes
2. Check Recorder database size before/after
3. Verify essential attributes still appear in history
4. Confirm excluded attributes don't appear in new state writes

SQL Query to check attribute presence:

SELECT
    state_id,
    attributes
FROM states
WHERE entity_id = 'sensor.tibber_home_current_interval_price'
ORDER BY last_updated DESC
LIMIT 5;

Maintenance Notes

• ✅ Must be a class attribute (instance attributes are ignored)
• ✅ Use frozenset for immutability
• ✅ Only affects new state writes (doesn't purge existing history)
• ✅ Attributes still available via entity.attributes in templates/automations
• ✅ Only prevents storage in Recorder, not runtime availability

Long-Term Statistics Optimization (state_class)

This is a second, independent mechanism that controls writes to the HA statistics and statistics_short_term tables — distinct from _unrecorded_attributes, which only affects the state_attributes table.

Why This Matters

• The state_attributes table (controlled by _unrecorded_attributes) is auto-purged after ~10 days
• The statistics/statistics_short_term tables (controlled by state_class) are never auto-purged — they grow unbounded

This makes state_class=TOTAL on many sensors the primary cause of long-term database bloat for users with long-running installations.

HA Constraint: MONETARY Device Class

For sensors with device_class=SensorDeviceClass.MONETARY, only two state_class values are valid:

state_class	Statistics written	Frontend effect
TOTAL	✅ Yes — unbounded growth	Statistics line-chart on entity detail page
None	❌ No	States timeline only (History panel, "Show More")
MEASUREMENT	❌ Blocked by hassfest	—

MEASUREMENT causes a hassfest validation error for MONETARY sensors, leaving only TOTAL or None.

Implementation Decision

Only 3 of 26 MONETARY sensors keep state_class=TOTAL — those where long-term history is genuinely useful:

Sensor	Reason
current_interval_price	Long-term price trend (weeks/months)
current_interval_price_base	Required for Energy Dashboard
average_price_today	Seasonal daily average tracking

All other 23 MONETARY sensors use state_class=None:

• Forecast/future sensors (next_avg_*h)
• Daily snapshots (lowest/highest_price_today/tomorrow)
• Rolling windows (trailing/leading_24h_*)
• Next/previous interval sensors

Effect of state_class=None:

• ✅ Short-term state history (States timeline, ~10 days) still works normally
• ✅ Templates, automations, and attributes are unaffected
• ❌ Statistics line-chart removed from entity detail page for these sensors
• ❌ No writes to statistics/statistics_short_term tables

Expected Impact

Going from 26 → 3 sensors writing to the statistics tables:

• ~88% reduction in statistics table writes
• Prevents the primary cause of long-term database bloat
• Existing statistics data is retained (only new writes stop)

Relationship to _unrecorded_attributes

These are two independent mechanisms targeting different tables:

Mechanism	Table affected	Purged?	Controls
_unrecorded_attributes	state_attributes	✅ ~10 days	Which attributes are stored per state write
state_class=None	statistics, statistics_short_term	❌ Never	Whether long-term statistics are written at all

Both optimizations work together. _unrecorded_attributes reduces the size of each state write; state_class=None eliminates an entire category of unbounded writes.

Implementation File

• custom_components/tibber_prices/sensor/definitions.py — state_class set per-sensor in SensorEntityDescription

References

• HA Developer Docs - Excluding State Attributes
• Implementation PR: [Link when merged]
• Related Issue: [Link if applicable]