To tackle hypervolume identifiability issue, we propose to rethink MOBO from the perspective of non-Markovian RL via sequence modeling. We propose BOFormer, which leverages the sequence modeling capability of the Transformer architecture and thereby minimizes the generalized temporal difference loss. BOFormer comprises two distinct networks as shown above: The upper network functions as the policy network, utilizing the historical data and the Q-value predicted by the target network to estimate the Q-values for action selection. The lower network serves as the target network, responsible for constructing Q-values for past observation-action pairs.
$Q$-Augmented Representation
Define
\begin{align*}
y^{(i)*}_{t}:=\max\limits_{1\leq j\leq t}y^{(i)}_{i}, \forall i \in [1,\cdots,K]
\end{align*}
as the best observed function value of $i$-th objective up to time $t$. The observation-action pair for $x$ can be denoted by
\begin{equation*}
o_t(x) \equiv \Big\{\mu_t^{(i)}(x), \sigma_t^{(i)}(x),y_t^{(i)*} ,\frac{t}{T}\Big\}_{i\in [K]},
\end{equation*}
where $\mu_t^{(i)}(x):=\mathbb{E}[f_i(x)\rvert \mathcal{F}_t]$ and $\sigma_t^{(i)}(x):=\sqrt{\mathbb{V}[f_i(x)\rvert \mathcal{F}_t]}$ are the posterior predictive distribution under Gaussian Process (GP) prior.
BOFormer as an Acquisition Function for MOBO
In BOFormer, we use the normalized hypervolume improvement as the reward, $\textit{i.e.}$,
\begin{equation*}
r_t:=\frac{\text{HV}(\mathcal{X}_t)-\text{HV}(\mathcal{X}_{t-1})}{\text{HV}(\mathcal{X^*})-\text{HV}(\mathcal{X}_{t})}.
\end{equation*}
Then, $h_t$, the history up to time $t$, is the concatenation of past observation-action pair representation defined as follows:
\begin{align}
h_t = \left\{\mu_j^{(i)}(x_j), \sigma_j^{(i)}(x_j), y_{j-1}^{(i)*}, j/t, r_i, Q_{\bar{\theta}}\right\}_{i\in[k], j\in[t-1]} \label{eq: state action}.
\end{align}
In non-Markovian RL, $Q_{\bar{\theta}}$ can be defined recursively, where
\begin{align*}
Q_{\bar{\theta}}(h_t,o_t(x_t)):= Q_{\bar{\theta}}\left( \left\{ o_j(x_j), r_j, Q_{\bar{\theta}}(h_{j-1},o_{j-1}(x_{j-1})) \right\}_{j=1}^{t-1}, o_t(x_t)\right).
\end{align*}
Then, BOFormer selects the next sample point by letting $x_t = \text{argmax}_{x\in\mathcal{X}}Q_{\bar{\theta}}(h_t,o_t(x))$.
Off-Policy Learning and Prioritized
Trajectory Replay Buffer
We extend the concept of Prioritized Experience Replay (PER) and introduce the Prioritized Trajectory Replay Buffer (PTRB). The detailed modifications are as follows: (i) Elements pushed into this buffer are entire trajectories $\tau = \{o_i(x_i),r_i\}_{i=1}^T$. (ii) The TD-error considered in PER is replaced by $\delta(Q_{\theta_t},\tau)$, which is the summation of the TD-error of the policy network for all transitions in this trajectory, $\textit{i.e.}$,
\begin{align}
\delta(Q,\tau) & := \sum_{i=1}^{T-1} \big( Q\left( h_i, o_i(x_i)\right) -
\big(r_i + \gamma \max_{x\in\mathbb{X}}Q_{\bar{\theta}}(h_{i+1}, o_{i+1}(x))\big) \big)^2.
\end{align}
Let $\mathcal{B}$ denote the batch sampled from PTRB. The loss function of BOFormer is defined as $L(\theta):=\sum_{\tau \in \mathcal{B}} \delta(Q_{\theta},\tau)$. For the training data, BOFormer is trained solely on synthetic GP functions, which are relatively cheap to generate (compared to real-world functions), and then be deployed to optimize unseen testing functions.
