Welfare Maximization Algorithm for Solving Budget-Constrained Multi-Component POMDPs

by   Manav Vora, et al.

Partially Observable Markov Decision Processes (POMDPs) provide an efficient way to model real-world sequential decision making processes. Motivated by the problem of maintenance and inspection of a group of infrastructure components with independent dynamics, this paper presents an algorithm to find the optimal policy for a multi-component budget-constrained POMDP. We first introduce a budgeted-POMDP model (b-POMDP) which enables us to find the optimal policy for a POMDP while adhering to budget constraints. Next, we prove that the value function or maximal collected reward for a b-POMDP is a concave function of the budget for the finite horizon case. Our second contribution is an algorithm to calculate the optimal policy for a multi-component budget-constrained POMDP by finding the optimal budget split among the individual component POMDPs. The optimal budget split is posed as a welfare maximization problem and the solution is computed by exploiting the concave nature of the value function. We illustrate the effectiveness of the proposed algorithm by proposing a maintenance and inspection policy for a group of real-world infrastructure components with different deterioration dynamics, inspection and maintenance costs. We show that the proposed algorithm vastly outperforms the policy currently used in practice.


Explainable Deterministic MDPs

We present a method for a certain class of Markov Decision Processes (MD...

Solving POMDPs by Searching the Space of Finite Policies

Solving partially observable Markov decision processes (POMDPs) is highl...

Optimal Inspection and Maintenance Planning for Deteriorating Structures through Dynamic Bayesian Networks and Markov Decision Processes

Civil and maritime engineering systems, among others, from bridges to of...

Models and algorithms for skip-free Markov decision processes on trees

We introduce a class of models for multidimensional control problems whi...

Dynamic maintenance policy for systems with repairable components subject to mutually dependent competing failure processes

In this paper, a repairable multi-component system is studied where all ...

Searching k-Optimal Goals for an Orienteering Problem on a Specialized Graph with Budget Constraints

We propose a novel non-randomized anytime orienteering algorithm for fin...

Optimal Inspection of Network Systems via Value of Information Analysis

This paper develops computable metrics to assign priorities for informat...

Please sign up or login with your details

Forgot password? Click here to reset