A Learning-based Optimal Market Bidding Strategy for Price-Maker Energy Storage

by   Mathilde D. Badoual, et al.

Load serving entities with storage units reach sizes and performances that can significantly impact clearing prices in electricity markets. Nevertheless, price endogeneity is rarely considered in storage bidding strategies and modeling the electricity market is a challenging task. Meanwhile, model-free reinforcement learning such as the Actor-Critic are becoming increasingly popular for designing energy system controllers. Yet implementation frequently requires lengthy, data-intense, and unsafe trial-and-error training. To fill these gaps, we implement an online Supervised Actor-Critic (SAC) algorithm, supervised with a model-based controller – Model Predictive Control (MPC). The energy storage agent is trained with this algorithm to optimally bid while learning and adjusting to its impact on the market clearing prices. We compare the supervised Actor-Critic algorithm with the MPC algorithm as a supervisor, finding that the former reaps higher profits via learning. Our contribution, thus, is an online and safe SAC algorithm that outperforms the current model-based state-of-the-art.



There are no comments yet.


page 6


How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization

Deterministic-policy actor-critic algorithms for continuous control impr...

Deep Reinforcement Learning for Long Term Hydropower Production Scheduling

We explore the use of deep reinforcement learning to provide strategies ...

FORK: A Forward-Looking Actor For Model-Free Reinforcement Learning

In this paper, we propose a new type of Actor, named forward-looking Act...

Reinforcement Learning for Mixed-Integer Problems Based on MPC

Model Predictive Control has been recently proposed as policy approximat...

Model-based actor-critic: GAN + DRL (actor-critic) => AGI

Our effort is toward unifying GAN and DRL algorithms into a unifying AI ...

Infer Your Enemies and Know Yourself, Learning in Real-Time Bidding with Partially Observable Opponents

Real-time bidding, as one of the most popular mechanisms for selling onl...

Online Inventory Management with Application to Energy Procurement in Data Centers

Motivated by the application of energy storage management in electricity...
This week in AI

Get the week's most popular data science and artificial intelligence research sent straight to your inbox every Saturday.