Online Convex Optimization with Unbounded Memory: Regret Bounds and Applications

Core Concepts

This work introduces a generalization of the online convex optimization (OCO) framework that allows the loss in the current round to depend on the entire history of past decisions. It provides matching upper and lower bounds on the policy regret in terms of the time horizon and a quantitative measure of the influence of past decisions on present losses.

Abstract

The paper introduces a generalization of the online convex optimization (OCO) framework, called "Online Convex Optimization with Unbounded Memory", that captures long-term dependence of the current loss on past decisions.

Key highlights:

• Defines the notion of p-effective memory capacity (Hp) that quantifies the maximum influence of past decisions on present losses.
• Proves an O(√(HpT)) upper bound on the policy regret and a matching (worst-case) lower bound.
• As a special case, proves the first non-trivial lower bound for OCO with finite memory, and improves existing upper bounds.
• Demonstrates the broad applicability of the framework by deriving regret bounds for online linear control and an online variant of performative prediction.

The paper first formalizes the problem setup, making assumptions about the feedback model, loss functions, and the dynamics of the history space. It then presents two algorithms, one using follow-the-regularized-leader (FTRL) and another combining FTRL with a mini-batching approach.

The key technical contributions are:

1. Defining the notion of p-effective memory capacity (Hp) and using it to derive tight upper and lower bounds on the policy regret.
2. Specializing the results to the case of OCO with finite memory, proving the first non-trivial lower bound and improving existing upper bounds.
3. Applying the framework to two diverse problems - online linear control and online performative prediction - and deriving improved regret bounds.

The paper concludes by discussing future research directions, including extensions to unknown dynamics and bandit feedback settings.

Stats

The paper does not contain any explicit numerical data or statistics. It focuses on theoretical analysis and regret bounds.

Quotes

"In many applications the loss of the learner depends not only on the current decisions but on the entire history of decisions until that point."
"We introduce the notion of p-effective memory capacity, Hp, that quantifies the maximum influence of past decisions on present losses."
"We prove an O(√(HpT)) upper bound on the policy regret and a matching (worst-case) lower bound."

Key Insights Distilled From

Online Convex Optimization with Unbounded Memory

by Raunak Kumar... at arxiv.org 04-02-2024

https://arxiv.org/pdf/2210.09903.pdf

Deeper Inquiries

How can the framework be extended to handle unknown dynamics, where the learner does not know the operators A and B

To handle unknown dynamics where the learner does not know the operators A and B, the framework can be extended by incorporating techniques from online learning with unknown linear operators. One approach could involve using online learning algorithms that are robust to uncertainty in the dynamics. The learner could adaptively estimate the unknown operators based on the observed data and update its decisions accordingly. This adaptive approach would require the algorithm to maintain a belief or distribution over possible operators and update this belief as new data is received. By incorporating techniques from Bayesian online learning or reinforcement learning with function approximation, the framework could adapt to unknown dynamics and still make effective decisions.

What are the implications of considering non-linear, but decaying, dependence of the history on past decisions

Considering non-linear, but decaying, dependence of the history on past decisions introduces challenges due to the non-convex nature of the problem. Non-linear dependencies can lead to complex interactions between past decisions and current losses, making it harder to optimize the regret function. However, by carefully designing the framework to handle non-linearities and leveraging techniques from non-convex optimization, it may be possible to capture the decaying influence of past decisions effectively. This could involve using advanced optimization algorithms that can handle non-convex functions and incorporating regularization techniques to prevent overfitting to the history.

Can the techniques developed in this paper be applied to further improve regret bounds for other variants of online linear control problems

The techniques developed in this paper can potentially be applied to improve regret bounds for other variants of online linear control problems. By extending the framework to handle different types of control policies, dynamics, or loss functions, the same principles of capturing long-term dependencies on past decisions can be applied. For example, by considering different classes of controllers, incorporating additional constraints, or modifying the data distribution assumptions, the framework can be adapted to various scenarios in online linear control. By carefully analyzing the problem structure and leveraging the weighted norms and p-effective memory capacity concept, it may be possible to derive tighter regret bounds for a broader range of online linear control variants.