Positivity-Hardness of Optimization Problems on Markov Decision Processes

Core Concepts

The Positivity problem, a well-known number-theoretic problem whose decidability status has been open for decades, is polynomial-time reducible to the threshold problems for the optimal values of various quantities in Markov decision processes, including termination probabilities of one-counter MDPs, satisfaction probabilities of energy objectives, conditional and partial expectations, and conditional value-at-risk for accumulated weights.

Abstract

The paper investigates a series of optimization problems for one-counter Markov decision processes (MDPs) and integer-weighted MDPs with finite state space. These problems include:

Termination probabilities and expected termination times for one-counter MDPs
Satisfaction probabilities of energy objectives
Conditional and partial expectations
Satisfaction probabilities of constraints on the total accumulated weight
Computation of quantiles for the accumulated weight
Conditional value-at-risk for accumulated weights
Although algorithmic results are available for some special instances, the decidability status of the decision versions of these problems is unknown in general.
The paper demonstrates that these optimization problems are inherently mathematically difficult by providing polynomial-time reductions from the Positivity problem for linear recurrence sequences. This problem is a well-known number-theoretic problem whose decidability status has been open for decades. The reductions rely on the construction of MDP-gadgets that encode the initial values and linear recurrence relations of linear recurrence sequences. These gadgets can be adjusted to prove the various Positivity-hardness results.
The key steps are three direct reductions from the Positivity problem to the threshold problems for the maximal termination probability of one-counter MDPs, the maximal partial expectation, and the maximal conditional value-at-risk. Further chains of reductions establish Positivity-hardness for the full series of optimization problems under investigation.

Stats

None.

Quotes

None.

Key Insights Distilled From

Positivity-hardness results on Markov decision processes

by Jakob Piriba... at arxiv.org 04-08-2024

https://arxiv.org/pdf/2302.13675.pdf

Deeper Inquiries

What other classes of MDPs or related models could be investigated for Positivity-hardness of optimization problems

In addition to the classes of MDPs explored in the paper, other models that could be investigated for Positivity-hardness of optimization problems include recursive MDPs, MDPs with multiple weight functions, and partially observable MDPs. Recursive MDPs, for example, introduce a form of recursion that can lead to more complex decision-making processes. MDPs with multiple weight functions allow for the consideration of multiple objectives or criteria, adding another layer of complexity to the optimization problems. Partially observable MDPs involve uncertainty about the system's state, which can further complicate the decision-making process and the optimization of objectives. Exploring these classes of MDPs could provide valuable insights into the inherent mathematical difficulties of solving optimization problems in these more complex models.

How might the techniques used in this paper be extended to handle more expressive models or objectives beyond the ones considered

The techniques used in the paper could be extended to handle more expressive models or objectives by adapting the construction of MDP-gadgets to encode the specific characteristics of the new models or objectives. For example, for models with recursive elements, the gadgets could be designed to capture the recursive nature of the system and its impact on the optimization problems. Similarly, for models with multiple weight functions, the gadgets could be modified to account for the interactions between different objectives and their corresponding weights. By tailoring the gadgets to the specific features of the models or objectives under consideration, the techniques could be applied to a wider range of scenarios, providing insights into the Positivity-hardness of optimization problems in these more expressive settings.

Are there any natural decision problems for standard finite-state MDPs with a single weight function and single objective that are known to be undecidable

For standard finite-state MDPs with a single weight function and single objective, there are no known natural decision problems that are undecidable. Undecidability results have been established for more expressive models, such as recursive MDPs, MDPs with multiple weight functions, or partially observable MDPs. However, for the specific class of standard finite-state MDPs with a single weight function and single objective, the decidability status of decision problems remains open, with algorithmic results available for some special instances but no known undecidability results. Further research in this area could shed light on the computational complexity of decision problems in these standard MDPs.