Version: Next 🚧

Recorder History Optimization

Status: ✅ IMPLEMENTED Last Updated: 2025-12-07

Overview

This document describes the implementation of _unrecorded_attributes for Tibber Prices entities to prevent Home Assistant Recorder database bloat by excluding non-essential attributes from historical data storage.

Reference: HA Developer Docs - Excluding State Attributes

Implementation

Both TibberPricesSensor and TibberPricesBinarySensor implement _unrecorded_attributes as a class-level frozenset to exclude attributes that don't provide value in historical data analysis.

Pattern

class TibberPricesSensor(TibberPricesEntity, SensorEntity):
    """tibber_prices Sensor class."""
    
    _unrecorded_attributes = frozenset(
        {
            "description",
            "usage_tips",
            # ... more attributes
        }
    )

Key Points:

Must be a class attribute (not instance attribute)
Use frozenset for immutability and performance
Applied automatically by Home Assistant's Recorder component

Categories of Excluded Attributes

1. Descriptions/Help Text

Attributes: description, usage_tips

Reason: Static, large text strings (100-500 chars each) that:

Never change or change very rarely
Don't provide analytical value in history
Consume significant database space when recorded every state change
Can be retrieved from translation files when needed

Impact: ~500-1000 bytes saved per state change

2. Large Nested Structures

Attributes:

periods (binary_sensor) - Array of all period summaries
data (chart_data_export) - Complete price data arrays
trend_attributes - Detailed trend analysis
current_trend_attributes - Current trend details
trend_change_attributes - Trend change analysis
volatility_attributes - Detailed volatility breakdown

Reason: Complex nested data structures that are:

Serialized to JSON for storage (expensive)
Create large database rows (2-20 KB each)
Slow down history queries
Provide limited value in historical analysis (current state usually sufficient)

Impact: ~10-30 KB saved per state change for affected sensors

Example - periods array:

{
  "periods": [
    {
      "start": "2025-12-07T06:00:00+01:00",
      "end": "2025-12-07T08:00:00+01:00",
      "duration_minutes": 120,
      "price_mean": 18.5,
      "price_median": 18.3,
      "price_min": 17.2,
      "price_max": 19.8,
      // ... 10+ more attributes × 10-20 periods
    }
  ]
}

3. Frequently Changing Diagnostics

Attributes: icon_color, cache_age, cache_validity, data_completeness, data_status

Reason:

Change every update cycle (every 15 minutes or more frequently)
Don't provide long-term analytical value
Create state changes even when core values haven't changed
Clutter history with cosmetic changes
Can be reconstructed from other attributes if needed

Impact: Prevents unnecessary state writes when only cosmetic attributes change

Example: icon_color changes from #00ff00 to #ffff00 but price hasn't changed → No state write needed

4. Static/Rarely Changing Configuration

Attributes: tomorrow_expected_after, level_value, rating_value, level_id, rating_id, currency, resolution, yaxis_min, yaxis_max

Reason:

Configuration values that rarely change
Wastes space when recorded repeatedly
Can be derived from other attributes or from entity state

Impact: ~100-200 bytes saved per state change

5. Temporary/Time-Bound Data

Attributes: next_api_poll, next_midnight_turnover, last_api_fetch, last_cache_update, last_turnover, last_error, error

Reason:

Only relevant at moment of reading
Won't be valid after some time
Similar to entity_picture in HA core image entities
Superseded by next update

Impact: ~200-400 bytes saved per state change

Example: next_api_poll: "2025-12-07T14:30:00" stored at 14:15 is useless when viewing history at 15:00

6. Relaxation Details

Attributes: relaxation_level, relaxation_threshold_original_%, relaxation_threshold_applied_%

Reason:

Detailed technical information not needed for historical analysis
Only useful for debugging during active development
Boolean relaxation_active is kept for high-level analysis

Impact: ~50-100 bytes saved per state change

7. Redundant/Derived Data

Attributes: price_spread, volatility, diff_%, rating_difference_%, period_price_diff_from_daily_min, period_price_diff_from_daily_min_%, periods_total, periods_remaining

Reason:

Can be calculated from other attributes
Redundant information
Doesn't add analytical value to history

Impact: ~100-200 bytes saved per state change

Example: price_spread = price_max - price_min (both are recorded, so spread can be calculated)

Attributes That ARE Recorded

These attributes remain in history because they provide essential analytical value:

Time-Series Core

timestamp - Critical for time-series analysis (ALWAYS FIRST)
All price values - Core sensor states

Diagnostics & Tracking

cache_age_minutes - Numeric value for diagnostics tracking over time
updates_today - Tracking API usage patterns

Data Completeness

interval_count, intervals_available - Data completeness metrics
yesterday_available, today_available, tomorrow_available - Boolean status

Period Data

start, end, duration_minutes - Core period timing
price_mean, price_median, price_min, price_max - Core price statistics

High-Level Status

relaxation_active - Whether relaxation was used (boolean, useful for analyzing when periods needed relaxation)

Expected Database Impact

Space Savings

Per state change:

Before: ~3-8 KB average
After: ~0.5-1.5 KB average
Reduction: 60-85%

Daily per sensor:

Sensor Type	Updates/Day	Before	After	Savings
High-frequency (15min)	96	~290 KB	~140 KB	50%
Low-frequency (6h)	4	~32 KB	~6 KB	80%

Most Impactful Exclusions

periods array (binary_sensor) - Saves 2-5 KB per state
data (chart_data_export) - Saves 5-20 KB per state
trend_attributes - Saves 1-2 KB per state
description/usage_tips - Saves 500-1000 bytes per state
icon_color - Prevents unnecessary state changes

Real-World Impact

For a typical installation with:

80+ sensors
Updates every 15 minutes
~10 sensors updating every minute

Before: ~1.5 GB per month After: ~400-500 MB per month Savings: ~1 GB per month (~66% reduction)

Implementation Files

Sensor Platform: custom_components/tibber_prices/sensor/core.py
- Class: TibberPricesSensor
- 47 attributes excluded
Binary Sensor Platform: custom_components/tibber_prices/binary_sensor/core.py
- Class: TibberPricesBinarySensor
- 30 attributes excluded

When to Update _unrecorded_attributes

Add to Exclusion List When:

✅ Adding new description/help text attributes ✅ Adding large nested structures (arrays, complex objects) ✅ Adding frequently changing diagnostic info (colors, formatted strings) ✅ Adding temporary/time-bound data (timestamps that become stale) ✅ Adding redundant/derived calculations

Keep in History When:

✅ Core price/timing data needed for analysis ✅ Boolean status flags that show state transitions ✅ Numeric counters useful for tracking patterns ✅ Data that helps understand system behavior over time

Decision Framework

When adding a new attribute, ask:

Will this be useful in history queries 1 week from now?
- No → Exclude
- Yes → Keep
Can this be calculated from other recorded attributes?
- Yes → Exclude
- No → Keep
Is this primarily for current UI display?
- Yes → Exclude
- No → Keep
Does this change frequently without indicating state change?
- Yes → Exclude
- No → Keep
Is this larger than 100 bytes and not essential for analysis?
- Yes → Exclude
- No → Keep

Testing

After modifying _unrecorded_attributes:

Restart Home Assistant to apply changes
Check Recorder database size before/after
Verify essential attributes still appear in history
Confirm excluded attributes don't appear in new state writes

SQL Query to check attribute presence:

SELECT 
    state_id,
    attributes
FROM states 
WHERE entity_id = 'sensor.tibber_home_current_interval_price'
ORDER BY last_updated DESC 
LIMIT 5;

Maintenance Notes

✅ Must be a class attribute (instance attributes are ignored)
✅ Use frozenset for immutability
✅ Only affects new state writes (doesn't purge existing history)
✅ Attributes still available via entity.attributes in templates/automations
✅ Only prevents storage in Recorder, not runtime availability

References

HA Developer Docs - Excluding State Attributes
Implementation PR: [Link when merged]
Related Issue: [Link if applicable]

Overview​

Implementation​

Pattern​

Categories of Excluded Attributes​

1. Descriptions/Help Text​

2. Large Nested Structures​

3. Frequently Changing Diagnostics​

4. Static/Rarely Changing Configuration​

5. Temporary/Time-Bound Data​

6. Relaxation Details​

7. Redundant/Derived Data​

Attributes That ARE Recorded​

Time-Series Core​

Diagnostics & Tracking​

Data Completeness​

Period Data​

High-Level Status​

Expected Database Impact​

Space Savings​

Most Impactful Exclusions​

Real-World Impact​

Implementation Files​

When to Update _unrecorded_attributes​

Add to Exclusion List When:​

Keep in History When:​

Decision Framework​

Testing​

Maintenance Notes​

References​

Overview

Implementation

Pattern

Categories of Excluded Attributes

1. Descriptions/Help Text

2. Large Nested Structures

3. Frequently Changing Diagnostics

4. Static/Rarely Changing Configuration

5. Temporary/Time-Bound Data

6. Relaxation Details

7. Redundant/Derived Data

Attributes That ARE Recorded

Time-Series Core

Diagnostics & Tracking

Data Completeness

Period Data

High-Level Status

Expected Database Impact

Space Savings

Most Impactful Exclusions

Real-World Impact

Implementation Files

When to Update _unrecorded_attributes

Add to Exclusion List When:

Keep in History When:

Decision Framework

Testing

Maintenance Notes

References