We are excited to share insights from our latest research published in Data-Centric Engineering (Cambridge University Press): “A training workbench based on transient building models for creating intelligent energy operators” (DOI: 10.1017/dce.2025.10016).
Why Do We Need Intelligent Energy Operators?
Modern buildings are intricate thermodynamic systems with significant thermal inertia, non-linear thermal storage dynamics, and volatile external driving forces (solar radiation, outdoor temperature, internal gains, and fluctuating electricity tariffs).
Traditional Rule-Based Control (RBC) or classic PID systems are rigid: they cannot anticipate future events or adjust to dynamic tariff structures without burdensome manual re-tuning.
While Deep Reinforcement Learning (DRL) offers an appealing paradigm—learning optimal control policies through interaction—training DRL agents directly on physical buildings is fundamentally infeasible due to safety risks, equipment wear, and sample inefficiency.
The Training Workbench Architecture
To bridge this critical gap, we formulated a modular training workbench that couples:
- Transient High-Fidelity Building Models: Accurately replicating multi-zone heat transfer, thermal storage, and HVAC plant equipment.
- Standardized DRL Environments: An OpenAI Gym / Gymnasium compatible API allowing rapid integration with leading algorithms (PPO, SAC, DQN).
- Multi-Objective Generalized Reward Functions: Formulations that jointly penalize thermal discomfort (predicted mean vote / temperature violation hours) and electrical cost.
# Conceptual reward function structure
def calculate_operator_reward(indoor_temp, comfort_band, energy_cost, alpha=0.6):
comfort_penalty = max(0, comfort_band.min - indoor_temp) + max(0, indoor_temp - comfort_band.max)
cost_penalty = energy_cost
reward = - (alpha * comfort_penalty + (1.0 - alpha) * cost_penalty)
return reward
Empirical Validation & Next Steps
Using our testbench, autonomous operators learned supervisory schedules that achieved significant energy cost reductions while preserving strict thermal comfort limits during peak heating hours.
Our ongoing work focuses on deploying these trained policies into physical test facilities, including Tecnalia’s KUBIK experimental building.
Read the full paper on Cambridge University Press.
