HERON

Hierarchical Environments for Realistic Observability in Networks

A domain-agnostic multi-agent reinforcement learning (MARL) framework for hierarchical control systems.

Getting Started

Quick start guide with examples

Getting Started

Key Concepts

Core abstractions and design principles

Key Concepts

Quick Examples

HERON Core

from heron.agents import CoordinatorAgent, FieldAgent
from heron.protocols.vertical import SetpointProtocol
from heron.messaging.memory import InMemoryBroker

# Create hierarchical agents
broker = InMemoryBroker()
field_agent = FieldAgent(agent_id='device_1', level=1, broker=broker)
coordinator = CoordinatorAgent(
    agent_id='coordinator_1',
    level=2,
    subordinates=[field_agent],
    protocol=SetpointProtocol(),
    broker=broker
)

obs = coordinator.observe(global_state)
action = coordinator.act(obs)

PowerGrid Case Study

from powergrid.envs import MultiAgentMicrogrids

env = MultiAgentMicrogrids(config={
    'network': 'ieee13',
    'num_microgrids': 2,
    'mode': 'centralized'
})

obs, info = env.reset()
for step in range(96):
    actions = {agent: policy(o) for agent, o in obs.items()}
    obs, rewards, dones, truncs, info = env.step(actions)

Coordination Protocols

from heron.protocols.vertical import PriceSignalProtocol
from heron.protocols.horizontal import ConsensusProtocol

# Vertical: parent -> subordinate coordination
vertical = PriceSignalProtocol(initial_price=50.0)

# Horizontal: peer-to-peer coordination
horizontal = ConsensusProtocol(max_iterations=10)

RLlib Training

from ray.rllib.algorithms.ppo import PPOConfig
from powergrid.envs import MultiAgentMicrogrids

config = (
    PPOConfig()
    .environment(MultiAgentMicrogrids)
    .training(lr=3e-4, train_batch_size=4000)
)
algo = config.build()
algo.train()

Key Features

Hierarchical Agents

3-level agent hierarchy: System, Coordinator, Field agents

Agents

Feature-Based State

Composable Features with visibility control

Core

Coordination Protocols

Vertical (setpoint, price) and horizontal (P2P, consensus)

Protocols

Message Broker

Extensible pub/sub for distributed execution

Messaging

Dual Execution Modes

Centralized training, distributed deployment

Centralized vs Distributed Execution

Case Studies

Ready-to-use implementations (PowerGrid, and more)

Case Studies

Architecture

HERON Framework (Domain-Agnostic)
+-- heron/
|   +-- agents/          # Hierarchical agent abstractions
|   +-- core/            # State, Action, Observation, Feature
|   +-- protocols/       # Vertical & Horizontal coordination
|   +-- messaging/       # Message broker interface
|   +-- envs/            # Base environment interface
|
+-- Case Studies
    +-- power/           # PowerGrid case study
        +-- agents/      # GridAgent, DeviceAgent
        +-- features/    # Electrical, Storage features
        +-- networks/    # IEEE test systems
        +-- envs/        # MultiAgentMicrogrids