text:
\section{Introduction.}\label{intro}

\section{Introduction}
Over the past few decades, various day-to-day dynamical models have been proposed to capture the impacts of commuters' day-to-day adjustments of their travel choices on the evolution of network traffic flow patterns. These models provide a better understanding of traffic volatility and uncertainty and offer insights on mitigating them. Via the discussion of the convergence of the day-to-day traffic dynamics to stationarity, researchers also hope to establish some behavioral justification for the premise of traffic network equilibrium, a premise widely adopted for the analysis and modeling of transportation systems \citep{Sheffi1985UrbanTN}.

The travel choice of interest in the literature is mostly route or departure time. For the former, a school of studies models its outcome as an aggregate traffic flow adjustment process. For instance, \cite{smith1984stability} assumed that commuters ending up with a path of higher cost may switch to another lower-cost path on the following day, at a rate that is proportional to the cost difference. Other examples of this school include, among others, \cite{zhang1996local,nagurney1997projected,yang2009day, he2010link,han2012link, guo2013discrete} and \cite{guo2015link}.  Usually, deterministic use equilibrium (DUE) is the stationary state of these day-to-day dynamics. Since the cost in these dynamics refers to the actual travel cost, these models essentially assume that commuters possess perfect information about the whole system. As a remedy, others consider cost perception error in their flow adjustment process, which is usually assumed to follow the multinomial logit model \citep{smith2016route, xiao2019day}. Another school of studies explicitly models the perception and update of travel costs and the resulting route choices \citep{cantarella1995dynamic, watling1999stability,bie2010stability,cantarella2013day, xiao2015combined, xiao2016physics, ye2021day, hazelton2022emergence}. For instance, \cite{watling1999stability} described a process that the actual and perceived cost on the day determines the perceived cost for the following day as well as the corresponding route choice. Usually, stochastic user equilibrium (SUE) is the stationary state for these two types of dynamics. Furthermore, when considering bounded rationality in the adjustment process \citep{guo2011bounded, ye2017rational}, the stationary state becomes boundedly rational user equilibrium (BRUE) \citep{lou2010robust,di2013boundedly}. Under some conditions such as monotonicity, the stability of the equilibrium (DUE, SUE or BRUE) is typically ensured, suggesting that the system will reach the equilibrium starting from an arbitrary initial flow pattern. In comparison, the modeling of day-to-day departure time choice has received less attention. Although models previously developed for day-to-day route choice may be applicable for modeling departure time choice, the stability of the equilibrium is hard to ensure as the monotonicity of the cost function does not hold anymore \citep{iryo2008analysis, iryo2019instability}. Indeed, \cite{guo2018we} revealed the non-convergence of five dynamics widely used in the literature. Subsequently, models with extra behavioral rules have been proposed to ensure stability. For example, \cite{jin2020stable} assumed that drivers make decisions on both arrival and departure times while \cite{jin2021stable} required that drivers only switch to adjacent departure time intervals.

An implicit behavioral assumption commonly made in the literature is commuters being myopic. As \cite{he2016marginal} pointed out, commuters' rationality is limited to the marginal benefit defined as the difference in perceived route costs, without prediction or anticipation of others' responses.
To consider more strategic thinking of commuters, \cite{he2012modeling} proposed a prediction-correction framework, in which commuters will predict the link flows of the next day based on the link flow on the day and previous prediction, and use the updated prediction to correct their perceived link costs. \cite{he2016marginal} further advanced the dynamical model with one-step strategic thinking where commuters can predict others' responses to the current traffic condition and include this marginal cost in their decision-making for the next day. In all these models, commuters make their travel choices for the next day and do not look far ahead. More specifically, they are incapable of optimizing their decisions beyond the next day.

The models with no or limited strategic commuters are well justifiable for representing existing conditions because human drivers are boundedly rational and may not be capable of making strategic and foresighted decisions \citep{simon1955behavioral}. On the contrary, this paper considers a different, somewhat futuristic scenario with strategic and forward-looking commuters. We observe that commuters are increasingly connected through rapidly-changing communications and vehicular technologies. Connectivity allows drivers to access various decision-support technologies such as a navigation app that assist them in making travel decisions. Such decision support is expected to become even stronger in the coming decades with the development and deployment of driving automation when drivers feel more comfortable with rendering more driving and travel agency to machine. In the connected and automated mobility era, commuters (connected drivers or automated vehicles) will possess strong computation capability and their travel decisions can be algorithmic and more strategic. Instead of being myopic, commuters can optimize their decision sequences for a longer time range. The first advantage of being farsighted is that it can help better balance exploration and exploitation when the system dynamics, which depend on other commuters' behavior, are unknown and need to be learned. Moreover, even with perfect information, being strategic and farsighted ensures a lower total cost compared to being myopic because there is inertia or cost associated with switching travel choices. It is thus intriguing to investigate how the behavior of, and interaction among strategic commuters would dictate the traffic pattern evolution, and whether Wadropian equilibrium would ever arise in the evolution process. The latter is critical for the inquiry as to whether the notion of Wadropian equilibrium remains relevant for planning future connected and automated mobility systems.

This paper represents our first attempt in this endeavor. We consider an idealized scenario with homogeneous commuters, which implicitly ensures perfect information. The day-to-day travel choice problem is modeled as a mean field game (MFG) \citep{huang2006large,lasry2007mean}, where the decision-making process of individual commuters is explicitly modeled; each commuter makes optimized decisions instead of following a heuristics or behavioral rule. On the ground of the proposed model, we investigate the day-to-day evolution pattern by examining the mean field equilibrium (MFE) where a commuter cannot reduce their overall cost by changing their policy sequence. We then conduct a thorough analysis of the properties of the MFE such as its existence and uniqueness. We also prove that under mild conditions, the MFE will converge to stationarity regardless of the initial flow pattern.

To the best of our knowledge, this paper is the first to model and analyze day-to-day traffic dynamics with strategic and farsighted commuters. The proposed  modeling framework can be customized for various travel choices, advancing our understanding of traffic volatility and stability in the connected and automated mobility era.
In addition, our work connects the MFE and the conventional Wardropian equilibrium concept by examining the physical interpretation of the stationarity of the MFE. The remainder of this paper is organized as follows. Section \ref{model} presents the MFG-based day-to-day dynamical model. Section \ref{prop} investigates the properties of the model, followed by the application of the model for specific travel choices in Section \ref{dege}. Section \ref{numer} presents numerical examples, and lastly Section \ref{conc} concludes the paper.

\section{Day-to-day dynamical model}
\label{model}
We consider a group of homogeneous, perfectly strategic commuters who are capable of making a sequence of day-to-day travel choices to minimize their total travel cost over a planning horizon. We model their travel choices and interactions as a mean field game or MFG, which refers to a repeated game played by an infinite number of infinitesimal players \citep{huang2006large,lasry2007mean}. Since each player is infinitesimal, they have no influence on the population in the MFG. In addition, they will only respond to the population rather than any individual player. The benefit of adopting the MFG framework is to use the "smoothing" effect of a large number of players to avoid investigating complicated mutual interactions \citep{ameli2022departure}. It is worth noting that such a treatment has long been adopted in the transportation network equilibrium analysis. Recent examples include \cite{chen2016optimal,lin2021credit} among others. We also note that the MFG framework has been applied in various transportation applications such as dynamic routing \citep{huang2021dynamic, cabannes2021solving, shou2022multi}, with-in day departure time choice modeling \citep{ameli2022departure}, pedestrian motion modeling \citep{aurell2019modeling}, and electric vehicles charging coordination \citep{tajeddini2018mean}.

\subsection{Model setting}
\label{MFGmodel}
A general modeling framework is presented and analyzed in this section without focusing on a specific travel choice. We consider a planning period of $N$ days, with a day $n\in \mathcal{N}=\left\{0,1,...,N-1 \right\}$. The set of available travel choices is $\mathcal{S} = \left\{s_1, ..., s_M \right\}$. Since on each day, each commuter will select one of the $M$ travel choices, the choice can be viewed as the state of the commuter on the day. In this sense, the set $\mathcal{S}$ becomes a (finite) state space of commuters. The distribution of states on the whole population is called the mean field (MF) distribution. On day $n$, the MF distribution is denoted as $\mu_n \in \mathcal{P}(\mathcal{S})$, where $\mathcal{P}(\mathcal{S})$ refers to the probability mass function defined on the state space. We use a compact notation without subscript $\mu = \left\{\mu_n \right\}_{n\in \mathcal{N}}\in \mathcal{M} $ to denote the MF distribution sequence across the time period $\mathcal{N}$, where $\mathcal{M}$ refers to the domain of all possible MF distribution sequences.

On each day, commuters can change to another option or stay with the previous one. We consider the action $a_n$ on day $n$ is to pick the travel option for day $n+1$. Subsequently, the day-to-day travel dynamics can be modeled as a Markov chain, and the action space $\mathcal{A}$ is identical to the state space $\mathcal{S}$. In this model, we seek a time-varying, feedback control policy $\pi_n(a|s): \mathcal{S} \times \mathcal{S} \rightarrow [0,1]$, where $\pi_n(a|s)=P(a_n=a|s_n=s)$.
We use a similar notation $\pi = \left\{\pi_n \right\}_{n\in \mathcal{N}} \in \Pi$ to represent the policy sequence in the whole period, where $\Pi$ is the domain for all possible policy sequences. For simplicity, we denote that $\pi_n(\cdot|s) \in \mathcal{P}(\mathcal{S}), \pi_n\in \mathcal{S}\times\mathcal{P}(\mathcal{S})$. Meanwhile, the system dynamic is $P(s_{n+1}=s'|s_n=s, a_n=a)=p(s'|s,a)=\left\{
\begin{aligned}
1 & , & s'=a \\
0 & , & \text{else}
\end{aligned}
\right.$. It means that although commuters can choose their actions from a stochastic policy, i.e., a distribution, if they choose $s'$ for the next day, their next state will always be $s'$.

On each day $n$, each commuter will experience a cost, which is modeled by $c(s,s',\mu_n,\pi_{n,s})=f(s,\mu_n)+g(s,s',\pi_{n,s})$, where $\pi_{n,s} = \pi_n(\cdot|s)$. The first cost $f(s,\mu_n): \mathcal{S} \times \mathcal{P}(\mathcal{S}) \rightarrow \mathbb{R}$ is the daily travel cost corresponding to travel choice $s$. For example, it is usually travel time in the context of route choice or the combination of travel time and schedule delay for departure time choice \citep{Vickrey1969}. We use the second cost $g(s,s',\pi_{n,s}): \mathcal{S} \times \mathcal{S} \times \mathcal{P}(\mathcal{S}) \rightarrow \mathbb{R}$ to reflect other behavioral considerations for choice preference. For example, it can contain user inertia, which refers to commuters' reluctance in adjusting their states \citep{mahmassani1987boundedly, zhang2015modeling, liu2017interactive}; the inertia can be represented by a general distance function  $d(s,s'): \mathcal{S} \times \mathcal{S} \rightarrow \mathbb{R}$, where $s$ and $s'$ are travel choices in adjacent days. Besides, as introduced in Section \ref{sentropy}, we can use this cost component to reflect an error term in the value function such as perception error. For example, an entropy term can be added like $g(s,s',\pi_{n,s}) = d(s,s')+ \frac{1}{\theta} \ln \pi_{n,s}(s')$ to model the error term that follows an independent and identically distributed Gumbel distribution. In this paper, we attempt to establish a general modeling framework without specifying the form of $f$, $g$, and $d$. Nevertheless, we consider distinct instances with or without the entropy term $\frac{1}{\theta} \ln \pi_{n,s}(s')$ given its relevance in the literature.

We now define the metrics for the distribution space $\mathcal{M}$ and policy space $\Pi$ to facilitate the discussion later. We first metrize  $\mathcal{P}(\mathcal{S})$ with the distance $d_f$:
\begin{equation}
    d_f(\mu,\nu) = \max_{s\in \mathcal{S}} |\mu(s)-\nu(s)| \quad  \mu,\nu \in \mathcal{P}(\mathcal{S})
\end{equation}
With the distance $d_f$, we define the metrics for $\mathcal{M}$ and $\Pi$ with sup metrics \citep{cui2021approximately}:
\begin{equation}
    d_M(\mu,\mu') = \max_{n\in \mathcal{N}} d_f(\mu_n, \mu'_n) \quad  \mu,\mu' \in \mathcal{M}
\end{equation}
\begin{equation}
    d_P(\pi,\pi') = \max_{n\in \mathcal{N}} \max_{s\in \mathcal{S}} d_f(\pi_n(\cdot|s),\pi_n'(\cdot|s)) \quad  \pi,\pi' \in \Pi
\end{equation}

Equipped with these metrics, we have the following results, whose proof is provided in Appendix.
\begin{proposition}
\label{pcms}
The metric spaces $(\mathcal{M}, d_M)$ and $(\Pi, d_P)$ are complete metric spaces.
\end{proposition}

\subsection{Individual behavior }
\label{sopt}
It is considered that, with an anticipation of an MF distribution sequence $\mu\in \mathcal{M}$ that represents how the population behaves, each commuter solves the following optimal control problem to find their optimal policy sequence $\pi^*$ with respect to $\mu$
\begin{equation*}
    \min_{\pi\in \Pi} J^\mu(\pi) = E \left[ \sum_{n=0}^{N-1} c(s_n, a_n, \mu_n, \pi_{n,s_n} ) | s_0 \sim \mu_0, a_n \sim \pi_{n,s_n}, s_{n+1}=a_n \right]
\end{equation*}
where $\mu_0$ is the initial state distribution.

To characterize the optimality, for the given $\mu$, the value function of a policy sequence $\pi$ and the optimal value function on day $n$ are defined respectively as follows:
\begin{equation*}
    V^{\mu, \pi}_n(s) := E \left[ \sum_{k=n}^{N-1} c(s_k, a_k, \mu_k, \pi_{k,s_k}) | s_n=s, a_k \sim \pi_{k,s_k}, s_{k+1}=a_k \right]
\end{equation*}
\begin{equation*}
    V^{\mu}_n(s) := \inf_{\pi\in \Pi} E \left[ \sum_{k=n}^{N-1} c(s_k, a_k, \mu_k, \pi_{k,s_k}) | s_n=s, a_k \sim \pi_{k,s_k}, s_{k+1}=a_k \right]
\end{equation*}

$\pi$ is optimal with respect to $\mu$ if and only if $V^{\mu, \pi}_n(s) = V^{\mu}_n(s)$ for all $s\in \mathcal{S}$ and $n \in \mathcal{N}$. Note that in our model, the value function of a state is the expected total cost (rather than reward) starting from the state. Thus, a state with a higher value is less preferable.

For any given value function $V$, we can define two Bellman equations, for the policy and optimal value functions respectively, by the following two Bellman operators:
\begin{equation*}
    \mathcal{G}_{\mu}^{\pi} V(s) = \sum_{s'\in \mathcal{S}} \pi(s'|s) \left(c(s, s', \mu, \pi_{n,s})+V(s') \right)
\end{equation*}
\begin{equation*}
    \mathcal{G}_{\mu} V(s) = \inf_{\pi} \sum_{s'\in \mathcal{S}} \pi(s'|s) \left(c(s, s', \mu, \pi_{n,s})+V(s') \right)
\end{equation*}
where $\mu \in \mathcal{P}(\mathcal{S})$, $\pi \in \mathcal{S}\times \mathcal{P}(\mathcal{S})$. Note that here $\mu$ and $\pi$ refer to the MF distribution and policy on one day rather than the whole sequences. Hereinafter, we will call the former policy Bellman operator and the latter optimal Bellman operator or simply Bellman operator.

If a policy sequence $\pi\in \Pi$ is optimal with respect to an MF distribution sequence $\mu\in \mathcal{M}$, then for all $n\in \mathcal{N}$ and $s\in \mathcal{S}$, there must be
\begin{equation*}
    \scrG_{\mu_n} V^{\mu}_{n+1}(s) = \scrG_{\mu_n}^{\pi_n} V^{\mu}_{n+1}(s) = V^{\mu}_n(s)
\end{equation*}

Note that for $\mu\in\mathcal{P}(\mathcal{S})$, the Bellman operator on a value function $V$, $\scrG_{\mu} V$, essentially suggests, given $V$ as the value function for the next day, how well commuters can do for each state in the previous day. Consider the case that if commuters switch $V$ to another value function $V'$, an observation is that $\scrG_{\mu} V'$ cannot be too distant from $\scrG_{\mu} V$, since commuters can always use the optimal policy for $V$ as a backup. Proposition \ref{lcg} characterizes this property formally. We skip its proof as it comes from Proposition 4(a) in \cite{gomes2010discrete}.

\begin{proposition}
\label{lcg}
For any two given value functions, $V$ and $V'$, and any MF distribution $\mu\in\mathcal{P}(\mathcal{S})$, let $\pi\in\mathcal{S}\times \mathcal{P}(\mathcal{S})$ be the optimal policy determined by $V$ and $\mu$, then we have
\begin{equation*}
    \scrG_{\mu} V'(s) \leq \scrG_{\mu} V(s)+\sum_{x\in \mathcal{S}} \pi(x|s) [V'(x)-V(x)]
\end{equation*}
\end{proposition}

Consider the fact that $\scrG_{\mu} V(s) = f(s,\mu)+ \sum_{x\in \mathcal{S}} [g(s,x,\pi_s)+V(x)]\pi_s(x)$, where $\pi$ is the optimal policy. It is easy to see that $\frac{\partial \scrG_{\mu} V(s)}{\partial V(x)} = \pi_s(x)$. In this sense, Proposition \ref{lcg} actually implies the concavity of the Bellman operator with respect to the value function, as elaborated in the following corollary. Note that the concavity holds entirely due to the strategic behaviors and does not require any structure of the cost function.

\begin{corollary}
For any $\mu\in \mathcal{P}(\mathcal{S})$, $\scrG_{\mu} V$ is concave with respect of $V$.
\end{corollary}

\subsection{Entropy penalization}
\label{sentropy}
The optimal policy each day depends on the MF distribution on that day and the value function on the next day. Even for strategic commuters, it may be still too strong to assume that they will always have and will act to the perfect value information of each state. To address this, we associate with the value function an error term $\delta$, stemming from commuters' perception biases or errors:
\begin{equation*}
    U(s) = V(s) + \delta
\end{equation*}
Consider when the daily travel cost and the inertia cost are considered, if $\delta$ follows i.i.d. Gumbel distribution where the variance is $\frac{\pi^2}{6\theta}$, for any MF distribution $\mu\in \mathcal{P}(\mathcal{S})$ and state $s$, the optimal probability of taking action $s'$ is
\begin{align*}
    \pi(s'|s) = \frac{e^{-\theta (f(s,\mu)+d(s,s')+V(s'))}}{\sum_{x\in \mathcal{S}} e^{-\theta (f(s,\mu)+d(s,x)+V(x))}}
\end{align*}

It is sufficed to show that this policy is equivalent to the optimal policy considering the entropy penalization term in the behavioral cost as shown in the following proposition. The equivalency can be easily proved by solving $\scrG_{\mu_n} V_{n+1}(s)$. The uniqueness in the proposition holds because the formulation of $\scrG_{\mu_n} V_{n+1}(s)$ becomes a strictly convex optimization problem due to the entropy term.
\begin{proposition}
When the cost function is $c(s,s',\mu_n,\pi_{n,s}) = f(s,\mu_n)+d(s,s')+ \frac{1}{\theta} \ln \pi_n(s'|s)$, the optimal policy for any day $n\in \mathcal{N}$ and state $s\in \mathcal{S}$ is uniquely determined by
\begin{equation}
\label{eqlogit}
    \pi_n(s'|s) = \frac{e^{-\theta (f(s,\mu_n)+d(s,s')+V_{n+1}(s'))}}{\sum_{x\in \mathcal{S}} e^{-\theta (f(s,\mu_n)+d(s,x)+V_{n+1}(x))}}
\end{equation}
where $V_{n+1}$ is the value function on day $n+1$.
\end{proposition}

\begin{remark}
On the ground of the equivalency we just prove, we can incorporate the entropy penalization into the behavioral cost $g(s,s',\pi_{n,s}) = d(s,s')+ \frac{1}{\theta} \ln\pi_{n,s} (s')$ if we wish to consider the error term regarding the value function. Note that the behavioral cost may appear in other forms if we consider other distributions of the error term.
\end{remark}

Substitute the optimal policy in Equation \ref{eqlogit} in the Bellman equation, we can easily obtain the following results.
\begin{equation}
\label{eqblm}
    \scrG_{\mu_n} V(s) = -\frac{1}{\theta} \ln \left[ \sum_{x\in \mathcal{S}} e^{- \theta (f(s,\mu_n)+d(s,x)+V(x))}  \right] = f(s,\mu_n) - \frac{1}{\theta} \ln \left[ \sum_{x\in \mathcal{S}} e^{- \theta (d(s,x)+V(x))}  \right]
\end{equation}

\subsection{Population behavior}
For a given initial distribution $\mu_0$, when all commuters act optimally according to the same policy $\pi$ due to the homogeneity, the MF distribution sequence is uniquely determined. We call this MF distribution sequence induced by $\pi$, which can be calculated by the flow conservation equation. For a given policy $\pi\in \mathcal{S}\times\mathcal{P}(\mathcal{S})$ and an MF distribution $\mu\in \mathcal{P}(\mathcal{S})$, we can define the operator
\begin{equation*}
    \mathcal{K}_{\pi} \mu(s) = \sum_{s'\in \mathcal{S}} \mu(s') \pi(s'|s)
\end{equation*}

Note that $\mathcal{K}_{\pi}$ essentially specifies the resulting distribution on the following day, given the policy and MF distribution on the previous day. Thus, if a policy sequence $\pi\in \Pi$ can induce an MF distribution sequence $\mu\in \mathcal{M}$, for any $n$ and $s\in \mathcal{S}$, there must be
\begin{equation*}
    \scrK_{\pi_n} \mu_n(s) = \mu_{n+1}(s)
\end{equation*}

Note that multiplying the equation in Proposition \ref{lcg} by $\mu(s)$ on both sides and summing up all states yield another important corollary using the operator we just defined:
\begin{corollary}
For any two given value functions, $V$ and $V'$, and any MF distribution $\mu\in\mathcal{P}(\mathcal{S})$, let $\pi$ be the optimal policy determined by $V$ and $\mu$, then we have
\begin{equation}
\label{ecg}
    \sum_{s\in \mathcal{S}} \mu(s) (\mathcal{G}_{\mu} V'(s)-\mathcal{G}_{\mu} V(s)) \leq \sum_{s\in \mathcal{S}} (V'(s)-V(s)) \mathcal{K}_{\pi} \mu(s)
\end{equation}
\end{corollary}

\section{Mean field equilibrium}

\label{prop}
In this section, we analyze the properties of the proposed model without referring to specific choice scenarios. We will further discuss them for specific travel choices in Section \ref{dege}.

\subsection{Definition}
\label{MFE}
The way we analyze the evolution pattern of the whole population is to find the mean field equilibrium or MFE (to avoid confusion we later use MFEs to refer to mean field equilibria) in the game. The MFE is a special self-consistent state that satisfies the following two conditions: 1) commuters anticipate the MF distribution sequence to be $\mu^*$ and act according to the optimal policy sequence $\pi^*$; 2) the policy sequence $\pi^*$ induces the distribution sequence that is exactly the same as $\mu^*$. In this self-consistent situation, every commuter is acting optimally with respect to the true system evolution. Thus, no one can reduce their total cost by changing policy any day. Owing to the homogeneity assumption for commuters, finding the MFE can be entirely offline. These strategic commuters will find the MFE at the beginning of the planning horizon and then act optimally according to the MFE. Also, note that the MFE is a different concept than the conventional Wardropian user equilibrium. The former is a state of day-to-day evolution and thus contains a sequence of policies, while the latter refers to a within-day stationary state.

Formally, we first denote the mapping from an MF distribution sequence to its optimal policy sequence as $\Phi: \mathcal{M} \rightarrow 2^\Pi$ and the mapping from a policy sequence to its induced MF distribution sequence starting from a specific $\mu_0$ as $\Psi: \Pi \rightarrow \mathcal{M}$. The image space of $\Phi$ is the power set $2^{\Pi}$ because the optimal policy on each day may not be unique. We now define the MFE as follows \citep{cui2021approximately}:
\begin{definition}
A pair $(\pi,\mu)$ is called an MFE if
\begin{equation*}
    \pi = \Phi(\mu), \quad \mu = \Psi(\pi)
\end{equation*}
\end{definition}

\subsection{Existence}
\label{sexist}
We first make two assumptions.
\begin{assumption}
\label{acc}
The cost function $f(s,\mu)$ is continuous with respect to any $\mu \in \mathcal{P}(\mathcal{S})$ for all $s\in \mathcal{S}$.
\end{assumption}

\begin{assumption}
\label{accg}
$\scrG_{\mu}^{\pi} V(s)$ is convex and continuous to $\pi$ for all $s\in \mathcal{S}$, $V\in \mathbb{R}^M$ and $\mu \in \mathcal{P}(\mathcal{S})$.
\end{assumption}

The first assumption is mild, while the second essentially requires that the mapping $\pi_s \rightarrow \sum_{s'\in\mathcal{S}} \pi_s(s') c(s,s',\mu,\pi_s)$ is convex and continuous, which entirely depends on the cost component that is related to $\pi_s$. For example, it holds when $c(s,s',\mu, \pi_s) = f(s,\mu)+ d(s,s')$ and $ f(s,\mu)+d(s,s')+ \frac{1}{\theta} \ln\pi_{n,s} (s')$. There is no requirement on the daily travel cost $f$ or the adjustment cost $d$.

With the two assumptions, we can prove the existence of the MFE by extending the proof in \cite{cui2021approximately} to incorporate the behavioral cost.
\begin{theorem}
\label{temfe}
Under Assumption \ref{acc} and \ref{accg}, for any given $\mu_0$, there exists at least one MFE $(\pi, \mu)$ that starts from $\mu_0$.
\end{theorem}

\proof{Proof of Theorem \ref{temfe}}
We identify $\mathcal{M}$ with simplex $\mathcal{S}_{|\mathcal{N}|(|\mathcal{S}|-1)} \subseteq \mathbb{R}^{|\mathcal{N}|(|\mathcal{S}|-1)}$ and $\Pi$ with simplex $\mathcal{S}_{|\mathcal{N}||\mathcal{S}|(|\mathcal{S}|-1)} \subseteq \mathbb{R}^{|\mathcal{N}||\mathcal{S}|(|\mathcal{S}|-1)}$. Define the mapping $\Gamma: \mathcal{S}_{|\mathcal{N}||\mathcal{S}|(|\mathcal{S}|-1)} \rightarrow 2^{\mathcal{S}_{|\mathcal{N}||\mathcal{S}|(|\mathcal{S}|-1)}}$, which maps from a policy sequence $\pi$ to the optimal policy sequences with respect to the MF distribution sequence induced by $\pi$. As in \cite{cui2021approximately}, $\Gamma$ maps to a non-empty set.

We then show that the mapping is convex. For any policy $\hat{\pi}$, suppose $\pi, \pi' \in \Gamma(\hat{\pi})$. Let us denote the MF distribution sequence induced by $\hat{\pi}$ as $\mu$ and the optimal value function as $\left\{V_{n}\right\}_{n\in \mathcal{N}}$. Then, for all $n\in \mathcal{N}$ and $s\in \mathcal{S}$, $\scrG_{\mu_n}^{\pi_n} V_{n+1}(s) = \mathcal{G}_{\mu_n}^{\pi_n'} V_{n+1}(s) = \scrG_{\mu_n} V_{n+1}(s)$. According to Assumption \ref{accg}, for any $\lambda \in [0,1]$, if we denote $\tilde{\pi} = \lambda \pi +(1-\lambda) \pi'$, there must have $\mathcal{G}_{\mu_n}^{\tilde{\pi}_n} V_{n+1}(s) \leq \scrG_{\mu_n} V_{n+1}(s)$. Meanwhile, according to the definition of the optimal Bellman operator, we can get $\mathcal{G}_{\mu_n}^{\tilde{\pi}_n} V_{n+1}(s) \geq \scrG_{\mu_n} V_{n+1}(s)$, which indicates $\tilde{\pi} \in \Gamma(\hat{\pi})$. Since the mapping is also closed and bounded, it is compact.

Finally, we show that the mapping has a closed graph. Consider any sequence such that $(\pi^j, \hat{\pi}^j) \rightarrow (\pi, \hat{\pi})$ with $\hat{\pi}^j \in \Gamma(\pi^j)$, where $\pi^j, \hat{\pi}^j\in \Pi$. We want to prove that $\hat{\pi} \in \Gamma(\pi)$. Due to the optimality, we have that for all $n$ and $s$
\begin{equation*}
    \mathcal{G}_{\Psi(\pi^j)}^{\hat{\pi}_n^j} V_{n+1}(s) \leq \mathcal{G}_{\Psi(\pi^j)}^{\pi'} V_{n+1}(s) \qquad \forall \pi' \in \mathcal{S} \times \mathcal{P}(\mathcal{S})
\end{equation*}
Similar to \cite{cui2021approximately}, we have the continuity of $\Psi$. Besides, based on the two assumptions, we also have the continuity of the mapping $\pi \rightarrow \mathcal{G}_{\Psi(\pi)}^{\pi'} V(s) $ as well as the mapping $\pi' \rightarrow \mathcal{G}_{\Psi(\pi)}^{\pi'} V(s) $. Therefore, let $j \rightarrow \infty$ on both sides, and we have the following equation for all $n\in \mathcal{N}$ and $s\in \mathcal{S}$.
\begin{equation*}
    \mathcal{G}_{\Psi(\pi)}^{\hat{\pi}_n} V_{n+1}(s) \leq \mathcal{G}_{\Psi(\pi)}^{\pi'} V_{n+1}(s) \qquad \forall \pi' \in \mathcal{S} \times \mathcal{P}(\mathcal{S})
\end{equation*}

Therefore, $\hat{\pi} \in \Gamma(\pi)$, which shows that the mapping has a closed graph. By Kakutani's fixed point theorem, there exists at least one fixed point $\pi^*$ such that $\pi^* \in \Gamma (\pi^*)$. Then the pair $(\pi^*, \Psi(\pi^*))$ is an MFE by definition.
\Halmos

As previously analyzed in Section \ref{MFE}, all the commuters will act according to the MFE. Since Assumption \ref{acc} and \ref{accg} are generally satisfied in various travel choice scenarios, the existence of MFE makes such an equilibrium notion more appealing for analyzing day-to-day traffic dynamics, as it provides a benchmark against which various improving transportation policies can be compared.

\subsection{Uniqueness}
\label{ssumfe}
When the cost function has better properties in monotonicity, we can further prove the uniqueness of the MFE, which is illustrated in Theorem \ref{tgumfe}.

\begin{assumption}
\label{gawm}
The cost function $f(s,\mu)$ is strictly monotone with respect to $\mu$, i.e., for any MF distribution $\mu,\mu' \in \mathcal{P}(\mathcal{S})$ such that $\mu\neq \mu'$, we have:
\begin{equation*}
    \sum_{s\in \mathcal{S}}[f(s,\mu)-f(s,\mu')]\left[\mu(s)-\mu'(s)\right] > 0
\end{equation*}
\end{assumption}

\begin{theorem}
\label{tgumfe}
Given Theorem \ref{temfe} and with Assumption \ref{gawm}, for any given initial distribution $\mu_0$, if $(\pi,\mu)$ and $(\pi',\mu')$ are both MFEs, then there must have $\mu_n(s) = \mu_n'(s)$ for all $s\in\mathcal{S},n\in\mathcal{N}$.
\end{theorem}

\proof{Proof of Theorem \ref{tgumfe}}
Suppose two pairs $(\pi,\mu)$ and $(\pi',\mu')$ are both MFEs. We have
\begin{align*}
    &J^{\mu}(\pi)-J^{\mu}(\pi')+J^{\mu'}(\pi')-J^{\mu'}(\pi) \\
    =& \sum_{n=0} ^{N-1}  \sum_{(s,a) \in \mathcal{S} \times \mathcal{A}} \left[\mu_n(s)\pi_n(a|s)c(s,a,\mu_n,\pi_{n,s}) - \mu_n'(s)\pi_n'(a|s)c(s,a,\mu_n,\pi_{n,s}') \right.\\
    & \left. + \mu_n'(s)\pi_n'(a|s)c(s,a,\mu_n',\pi_{n,s}') -\mu_n(s)\pi_n(a|s)c(s,a,\mu_n',\pi_{n,s}) \right] \\
    =& \sum_{n=0} ^{N-1}  \sum_{(s,a) \in \mathcal{S} \times \mathcal{A}} \left(\mu_n(s)\pi_n(a|s) - \mu_n'(s)\pi_n'(a|s) \right) \left(f(s,\mu_n)-f(s,\mu'_n)  \right) \\
    =& \sum_{n=0} ^{N-1}  \sum_{s \in \mathcal{S}} \left(\mu_n(s) - \mu_n'(s) \right) \left(f(s,\mu_n)-f(s,\mu'_n)  \right)\geq 0
\end{align*}

Note that $J^{\mu}(\pi)$ refers to the case where the whole population evolves according to $\mu$, and the representative commuter also evolves according to $\mu$ (induced by the policy $\pi$) as well. On contrary, $J^{\mu}(\pi')$ refers to the case where the population evolves according to $\mu$, but the representative commuter evolves according to $\mu'$ (induced by $\pi'$). Meanwhile, since $(\pi,\mu)$ is a MFE, $J^{\mu}(\pi)-J^{\mu}(\pi') \leq 0$. For the same reason, $J^{\mu'}(\pi')-J^{\mu'}(\pi) \leq 0$. Thus, we have $J^{\mu}(\pi)-J^{\mu}(\pi')+J^{\mu'}(\pi')-J^{\mu'}(\pi) \leq 0$. Therefore, all the equalities hold, and $\mu_n(s)=\mu'_n(s)$ for all $s\in\mathcal{S},n\in\mathcal{N}$ due to the strictly monotone cost. \Halmos

\subsection{Convergence to stationarity}
\label{scs}
This section analyzes the long-term behavior of the system when the planning horizon goes to infinity. To do so, we first define stationary MFE (s-MFE), which is the stationary solution of the system. The s-MFE can be viewed as a special evolution pattern that the policy, as well as the MF distribution on each day, remains invariant. We then analyze the physical interpretation of the s-MFE and connect it to the conventional Wardropian equilibrium. Finally, we prove the convergence from the MFE to the s-MFE, which indicates that the policy, as well as the MF distribution, will gradually become stationary.

\subsubsection{Stationary MFE}
Before defining the s-MFE, we need another two assumptions.

\begin{assumption}
\label{auop}
The Bellman equation $\scrG_{\mu}^{\pi} V(s)$ is continuous and strictly convex to $\pi$ for all $s \in \mathcal{S}, V \in \mathbb{R}^M$ and $\mu \in \mathcal{P}(\mathcal{S})$.
\end{assumption}

\begin{assumption}
\label{abbc}
There exists $C>0$ such that for all $s\in \mathcal{S}$, $\pi_s \in \mathcal{P}(\mathcal{S})$ and $\mu \in \mathcal{P}(\mathcal{S})$
\begin{equation*}
    \vert \sum_{s'\in \mathcal{S}} g(s,s',\pi_s) \pi_s(s') \vert \leq C, \quad 0\leq f(s,\mu) \leq C, \quad 0\leq d(s,s')\leq C
\end{equation*}
\end{assumption}

Assumption \ref{auop} is a stronger version of Assumption \ref{accg}, having more requirements on the cost component related to $\pi_s$. However, it still does not restrict the daily travel cost $f$ and the adjustment cost $g$. For example, it still holds for the entropy penalized model proposed in the last section, where $c(s,s',\mu,\pi_{n,s}) = f(s,\mu)+d(s,s')+\frac{1}{\theta} \ln \pi_{n,s}(s')$, regardless of the formulation of $f$ and $d$. However, the counterpart  without the entropy penalization component does not satisfy this new assumption. This assumption ensures the uniqueness of the optimal policy given $\mu$ and $V$ since the optimization becomes strictly convex. Assumption \ref{abbc} provides a bound for the cost functions, which is natural in our setting. The following proposition shows that Assumption \ref{abbc} also provides a bound for the difference of value functions between states. We leave the proof in Appendix.
\begin{proposition}
\label{pbgv}
When Assumption \ref{abbc} holds, for any value function $V$, MF distribution $\mu \in \mathcal{P}(\mathcal{S})$, and states $s,s'$, we have
\begin{equation}
\label{ebgv}
    \vert \scrG_{\mu} V(s)-\scrG_{\mu} V(s') \vert \leq 3C
\end{equation}
\end{proposition}

For the entropy penalized model, i.e. $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta} \ln \pi_s(s')$, a direct implication of Proposition \ref{pbgv} is that the MF distribution should have a positive lower bound for all states on all days, as shown in the following corollary. Still, this result does not have any requirement on the formulation of $f$ and $d$ or the initial distribution. We also leave the proof in Appendix.

\begin{proposition}
\label{lbmu}
If Assumption \ref{abbc} holds and $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta} \ln \pi_s(s')$, for all MFEs $(\pi,\mu)$, there must exist a constant $\omega>0$ such that $\mu_n(s) \geq \omega$ holds for all $n \geq 1$ and $s\in \mathcal{S}$
\end{proposition}

We are now ready to provide a formal definition of the stationary MFE based on the concept of stationary solutions introduced by \cite{gomes2010discrete}.
\begin{definition}
Under Assumption \ref{auop}, a pair $(\bar{V},\bar{\mu})$, where $\bar{\mu} \in \mathcal{P}(\mathcal{S})$ and $\bar{V}\in \mathbb{R}^M$, is called an s-MFE if it satisfies the following conditions for all $s\in \mathcal{S}$:
\begin{itemize}
    \item There exists a constant $\bar{\lambda}$ such that $\scrG_{\barmu} \bar{V}(s) = \bar{V}(s)+ \bar{\lambda}$
    \item $\scrK_{\barpi}\bar{\mu}(s) = \bar{\mu}(s)$, where $\bar{\pi}$ is the unique optimal policy determined by $\bar{V}$ and $\bar{\mu}$.
\end{itemize}
\end{definition}

\begin{remark}
The first condition implies that the policy is time-invariant. Note that it is the relative value of the value function of different states that determines the optimal policy. If the Bellman operator only adds the same constant to the value function of each state, the relative value of each state would remain the same and thus the policy should be time-invariant. The second condition ensures the stationarity of the MF distribution.
\end{remark}

\begin{remark}
Consider when $\mu_0=\bar{\mu}$ and the final value $V_N = \bar{V}$, the pair $(\pi,\mu)$ such that $\pi_n=\bar{\pi}, \mu_n = \bar{\mu}$ for all $n\in \mathcal{N}$ is an MFE by definition. Meanwhile, $(\bar{V}, \bar{\mu})$ is also an s-MFE. In this sense, an s-MFE is a special MFE that is stationary but requires some specific initial distributions and final values.
\end{remark}

Below we discuss the existence and uniqueness of the s-MFE. The next proposition is adopted from Theorem 3 in \cite{gomes2010discrete} that discusses the existence of the s-MFE, and we thus skip its proof.

\begin{proposition}
\label{gpesfme}
When Assumption \ref{acc}, \ref{auop}, and \ref{abbc} hold, there always exists an s-MFE.
\end{proposition}

For the uniqueness of the s-MFE, we need to slightly strengthen the monotonicity assumption of the cost function from strictly monotone to strongly monotone.

\begin{assumption}
\label{gasm}
The cost function $f(s,\mu)$ is strongly monotone, i.e., for any MF distribution $\mu,\mu' \in \mathcal{P}(\mathcal{S})$, we have:
\begin{equation*}
    \sum_{s\in \mathcal{S}}[f(s,\mu)-f(s,\mu')]\left[\mu(s)-\mu'(s)\right] \geq \eta d_f(\mu,\mu')^2
\end{equation*}
where $\eta$ is a positive constant.
\end{assumption}

The next proposition discusses the uniqueness of s-MFE and is similar to Proposition 7 in \cite{gomes2010discrete}. Since we are using different norms, the proof has some differences and is thus provided in Appendix.
\begin{proposition}
\label{gpusfme}
Given Proposition \ref{gpesfme} and with Assumption \ref{gasm}, if $(\bar{V}^1,\bar{\mu}^1)$ and $(\bar{V}^2,\bar{\mu}^2)$ are both s-MFE, there must have $\bar{\mu}^1 = \bar{\mu}^2$.
\end{proposition}

\subsubsection{Physical interpretation of the s-MFE}
\label{sspi}
So far we have analyzed the properties of the s-MFE. It is intriguing to ask what the s-MFE actually represents besides being a special stationary solution. In this section, we discuss the physical interpretation of the s-MFE and connect them to Wardropian equilibrium in a specific context where the entropy term $\frac{1}{\theta} \ln \pi_s(s')$ is included in the cost function as discussed in Section \ref{sentropy}.

In a simple case when user inertia is not considered, i.e., $g(s,s',\pi_s) = \frac{1}{\theta} \ln \pi_s(s')$, the following proposition illustrates its equivalency to the conventional logit-based stochastic user equilibrium (SUE) \citep{Sheffi1985UrbanTN}. We leave the proof to Appendix.

\begin{proposition}
\label{gpssue}
When $g(s,s',\pi_s) = \frac{1}{\theta} \ln \pi_s(s')$, let the MF distribution $\bar{\mu}$ denote the logit-based SUE distribution and $\bar{V}(s) = f(s,\bar{\mu})$ for all $s\in \mathcal{S}$. Then the pair $(\bar{V},\bar{\mu})$ is an s-MFE.
\end{proposition}

\begin{remark}
    Note that we use $\bar{\mu}$ as a distribution, rather than a traffic flow pattern.

\end{remark}

For the cases with user inertia, we will show that the s-MFE can be interpreted as an instance of the state-dependent stochastic user equilibrium (SDSUE) proposed in \cite{castaldi2017stochastic, castaldi2019stochastic}. To elaborate, we first present a definition of the SDSUE.

\begin{definition} Suppose the perceived cost of switching from state $j$ to $k$ where $j,k\in \mathcal{S}$ is $U_j^k = V_j^k + \delta_j^k$, where $V_j^k$ is the deterministic cost and $\delta_j^k$ is the random term. For simplicity let us consider an identical and independently distributed term so that $U_j^k = V_j^k + \delta$. Consider a choice model (e.g. logit model) $P_j^k = P_j^k(V_j^k)$, then $\left\{q_k\right\}_{k \in \mathcal{S}}$ is defined as an SDSUE if it satisfies $q_k = \sum_{j\in \mathcal{S}} q_j P_j^k$ \quad $\forall j\in \mathcal{S}$.
\end{definition}

The existence of SDSUE is generally ensured \citep{castaldi2017stochastic, castaldi2019stochastic}. Consider the case when $V_j^k = V^k+d_j^k$ where $d_j^k$ represents the inertia cost of changing from state $j$ to $k$ \citep{castaldi2017stochastic}, the SDSUE can be viewed as a special type of SUE that considers inertia cost into the deterministic cost. When $d_j^k=0$ for all state $j,k$, it collapses back to the conventional SUE. Following this line, the next proposition illustrates that the s-MFE can be viewed as an instance of SDSUE. We omit the proof as it is trivial.

\begin{proposition}
\label{pssdsue}
Consider an s-MFE pair $(\bar{V},\bar{\mu})$ that determines a unique optimal policy $\bar{\pi}$. Then, when using $d(s,s')+\bar{V}(s')$ as the deterministic cost of changing from $s$ to $s'$, $\bar{\mu}$ is the corresponding SDSUE.
\end{proposition}

Recall that in Proposition \ref{gpssue}, when there is no inertia, $\bar{\mu}$ is equivalent to the SUE distribution and $\bar{V}$ corresponds to the daily travel cost. In Proposition \ref{pssdsue}, we build the connection of $\bar{\mu}$ with SDSUE for the case with inertia. One may wonder what is the physical interpretation of $\bar{V}$ in this case. In fact, it is not equivalent to the daily travel cost anymore. If we think $f(s,\bar{\mu})$ as a daily measurement of the performance of state $s$, $\bar{V}$ can be treated as a farsighted measurement. Consider if $f(x,\bar{\mu})>f(y,\bar{\mu})$, indicating state $y$ is more preferable in the sense of the daily travel cost, the superiority of state $y$ should be more dominant for strategic commuters. As they can plan for the future, choosing $x$ not only incurs more travel costs immediately but also causes one more likely to switch to other states, inducing more inertia costs in the future. The following proposition elaborates on this point in more detail in a specific case where $d(s,s')=\epsilon \cdot \mathbf{1}_{s \neq s'}$. We leave the proof to Appendix.

\begin{proposition}
\label{pvnf}
When $g(s,s',\pi_s)=d(s,s')+\frac{1}{\theta} \ln \pi_s(s')$ and $d(s,s')=\epsilon \cdot \mathbf{1}_{s \neq s'}$, for an s-MFE pair $(\bar{V},\bar{\mu})$, if $\bar{V}(x)>\bar{V}(y)$, there must have $\bar{V}(x)-\bar{V}(y) > f(x,\bar{\mu})-f(y,\bar{\mu}) > \bar{V}(x)-\bar{V}(y)-\epsilon$
\end{proposition}

\subsubsection{Convergence to stationarity}
\label{sscs}

We are now ready to examine how the MFE could converge to the s-MFE. First, let's consider a simpler case that there is no inertia cost and thus the behavioral cost solely stems from the error term, i.e., $g(s,s',\pi_{n,s}) = \frac{1}{\theta} \ln \pi_{n,s}(s') $. We have shown that in this case, the s-MFE is equivalent to the logit-based SUE. Denote the optimal value function as $\left\{V_{n}\right\}_{n\in \mathcal{N}}$, then the unique optimal policy for all $n$ is
\begin{equation*}
    \pi_n(s'|s) = \frac{e^{-\theta (f(s,\mu_n)+V_{n+1}(s'))}}{\sum_{x\in \mathcal{S}} e^{-\theta (f(s,\mu_n)+V_{n+1}(x))}} = \frac{e^{-\theta V_{n+1}(s')}}{\sum_{x\in \mathcal{S}} e^{-\theta V_{n+1}(x)}}
\end{equation*}

As can be seen, the policy has nothing to do with the previous state $s$. As a result, with flow conservation, we have the following equation for all $n \geq 1$:

\begin{equation}
\label{evmu}
V_{n}(s) + \frac{1}{\theta} \ln \mu_{n}(s) = -\frac{1}{\theta} \ln \sum_{x\in \mathcal{S}} e^{-\theta V_{n}(x)}
\end{equation}

The result describes the relationship between the MF distribution and the value function. We show its connection with SUE in the following proposition.
\begin{proposition}
\label{trsue}
For any initial distribution $\mu_0$, if $g(s,s',\pi_{n,s}) = \frac{1}{\theta} \ln \pi_{n,s}(s')$, then the MFE reaches the logit-SUE on the second day.

\end{proposition}

Furthermore, recall that logit-SUE is the s-MFE in this case, which indicates that the system stays in logit-SUE for the rest of the period. Following this case, if we further let $\theta\rightarrow \infty$, then the cost function does not have adjustment cost as well as the entropy term, which captures commuters who have perfect information regarding the value function on each day. It is proved that when $\theta \rightarrow \infty$, the SUE flow converges to deterministic user equilibrium (DUE) flow \citep{mamun2011select}. As a result, in such a case the system is expected to reach DUE on the second day and stays there for the rest of the period, as shown in the following corollary. The proof for it is skipped.

\begin{corollary}
When $\theta \rightarrow \infty$, the MFE reaches the DUE on the second day.
\end{corollary}

The above results largely hinge on the property that without adjustment cost, the optimal policy only depends on the state of the next day rather than the previous state. However, since the adjustment cost depends on both the previous and subsequent states, this property no longer holds when user inertia is considered. Intuitively, rather than immediately reaching stationarity on the second day, the convergence should take longer time due to inertia.

Recall that in Section \ref{sopt}, we discuss that the Bellman operator is concave with respect to the value function. To prove the convergence in general cases, we need to further strengthen the concavity of the Bellman operator. Generally, strong concavity is defined using norms such as $L_2$-norm. Unfortunately,
if a constant is added to the value of all states, the value function would not have essential change but its $L_2$-norm does increase. To overcome this issue, a new norm for the value function is defined \citep{gomes2010discrete}. As can be seen, for a given value function $V$, if there is another value function $V'$ that satisfies $V'(s) = V(s)+\lambda$ where $\lambda$ is a constant, then $\Vert V \Vert _\# = \Vert V' \Vert _\#$.
\begin{equation*}
    \Vert V \Vert := \sqrt{\sum_{s\in \mathcal{S}} V(s)^2}
\end{equation*}
\begin{equation*}
    \Vert V \Vert _\# := \inf_{\lambda \in \mathbb{R}} \Vert V+\lambda \Vert
\end{equation*}

We are now ready to propose a new assumption to strengthen the concavity of the Bellman operator, using the norm defined above.

\begin{assumption}
\label{ascg}
For all MFEs $(\pi,\mu)$ and the corresponding value function $\left\{V_n\right\}_{n\in\mathcal{N}}$, there exists a constant $\phi>0$ such that the following equation holds for any $n\in \mathcal{N}$ and any value function $V'\in \mathbb{R}^M$
\begin{equation*}
    \sum_{s\in \mathcal{S}} \mu_n(s) (\mathcal{G}_{\mu_n} V'(s)-\mathcal{G}_{\mu_n} V_{n+1}(s)) - \sum_{s\in \mathcal{S}} (V'(s)-V_{n+1}(s)) \mathcal{K}_{\pi_n} \mu_n(s) \leq -\phi \Vert V_{n+1} - V' \Vert ^2_\#
\end{equation*}
\end{assumption}

Assumption \ref{ascg} essentially requires that the Bellman operator is strongly concave around points that are MFEs. The strong concavity of other points is not required. An intuitive interpretation for this assumption is that, if $\Vert V_{n+1}-V' \Vert _\#>0$, indicating the two value functions are different in the relative sense, the optimal policy for $V'$ should be different from $\pi_n$. Furthermore, the more different these two value functions are, the more diverse the two optimal policies will be. As can be seen, this assumption does not hold when $g(s,s',\pi_s) = 0$. In this case, as long as there exists the same state that can minimize the two value functions, they share the same optimal policy. However, this assumption holds for our entropy penalized model, which is further elaborated in the following proposition. We leave the proof in Appendix. Note that this assumption still does not have any requirement on the formulation of $f$ and $d$.

\begin{proposition}
\label{pesc}
If Assumption \ref{abbc} holds and $c(s,s',\mu,\pi_s) = f(s,\mu)+d(s,s')+\frac{1}{\theta} \ln \pi_s(s')$, Assumption \ref{ascg} always holds regardless of the formulation of $f$ and $d$.

\end{proposition}

To make it easier to follow, with a slight abuse of notation, let us denote the time horizon now as $\mathcal{N}=\left\{-N,...,0,..., N \right\}$. In this case, there are in total $2N+1$ days in the period. We change the notation to better illustrate the middle part of the horizon (i.e., $n=0$). The two notations of the horizon are the same in the asymptotic sense when $N\rightarrow \infty$. The following theorem proves the exponential convergence of the MFE to the s-MFE, which follows the steps of Theorem 7 in \cite{gomes2010discrete}. However, we are using different formulations regarding the strong concavity of the Bellman operator, which results in differences in the proof. The proof is provided in Appendix.
\begin{theorem}
\label{gtcue}
Given Theorem \ref{tgumfe} and Proposition \ref{gpusfme}, for any initial distribution $\nu \in \mathcal{P}(\mathcal{S})$, consider the MFE $(\pi,\mu)$ such that $\mu_{-N} = \nu$ and let $\left\{V_n\right\}_{n\in\mathcal{N}}$ be the corresponding value function. If Assumption \ref{ascg} further holds, when $N\rightarrow \infty$, we have
\begin{equation*}
    V_0 \rightarrow \bar{V}  \qquad \mu_0 \rightarrow \bar{\mu}
\end{equation*}
where $(\bar{V},\bar{\mu})$ is the unique s-MFE. Besides, the convergence occurs at an exponential speed.
\end{theorem}

\begin{remark}
The theorem illustrates that the policy, as well as the MF distribution, will converge to stationarity.
\end{remark}

Below we briefly summarize all the conditions we have on $f$ and $g$. As we have discussed when we propose Assumption 2,4 and 7, these assumptions totally depend on the behavioral cost $g$, especially the cost component related to $\pi_s$, e.g., the entropy term $\frac{1}{\theta} \ln \pi_s(s')$, rather than the adjustment cost $d(s,s')$), which does not have any requirement on the formulation of the daily travel cost $f$ and the inertia $d$. Note that Assumption 2,4,7 hold for the entropy penalized model where $g(s,s',\pi_s)=d(s,s')+\frac{1}{\theta} \pi_s(s')$.

\begin{table}[ht]
\caption{Summary of conditions}\label{tcond}
\begin{center}
\begin{tabular}{l|c|l}
                                      & Assumption   & Condition      \\ \hline

\multirow{4}{*}{Dailty travel cost $f$} &  1 & Continuous  w.r.t to $\mu$              \\
                                      &  3 & Strictly monotone w.r.t to $\mu$          \\
                                      &  5 &
                                      Bounded\\
                                      &  6 & Strongly monotone w.r.t to $\mu$          \\ \hline
\multirow{4}{*}{Behavioral cost $g$}    &  2 & Ensure $\scrG_{\mu}^{\pi} V$ to be convex and continuous w.r.t $\pi$ \\
                                      &  4 & Ensure $\scrG_{\mu}^{\pi} V$ to be strictly convex and continuous w.r.t $\pi$  \\
                                      &  5 & Bounded          \\
                                      &  7 & Ensure $\scrG_{\mu} V$ to be strongly concave w.r.t $V$ around MFEs
\end{tabular}
\end{center}
\end{table}
Table \ref{ttheory} summarized the results we have derived.
\begin{table}[ht]
\caption{Summary of theoretical results}\label{ttheory}
\begin{center}
\begin{tabular}{l|l|l}
                            & Interpretation                    & Required conditions \\ \hline
Theorem 1                   & Existence of the MFE                 & Assumption 1,2      \\
Theorem 2                   & Uniqueness of the MFE                & Assumption 1,2,3    \\
Proposition 5               & Existence of the s-MFE               & Assumption 1,4,5    \\
Proposition 6               & Uniqueness of the s-MFE              & Assumption 1,4,5,6  \\
Proposition 10 & The MFE reaches the s-MFE on the 2nd day       & Assumption 1, $g(s,s')=\frac{1}{\theta}\ln \pi_s(s')$    \\
Theorem 3                   & The MFE exponentially converges to the s-MFE & Assumption 1,4,5,6,7
\end{tabular}
\end{center}
\end{table}

\section{Specific choice scenarios}
\label{dege}
The above sections have developed and analyzed a general day-to-day evolution model. However, some conditions and results may not hold for or be applicable to specific scenarios of travel choice. In this section, we apply our general model to the choice of route and departure time respectively to derive specific results and connect them with those in the literature, whenever possible.

\subsection{Route choice}
\label{bprcase}
To ensure homogeneity, consider the case that all the commuters have the same origin and destination (OD). The state space $\mathcal{S}$ refers to all the possible paths connecting the OD pair, and $\mu_n(s)$ implies the proportion of commuters choosing path $s$ on day $n$. Therefore, for a fixed demand $\xi$, the flow on path $s$ on day $n$ is $\xi \mu_n(s)$. Denote the link set as $\mathcal{L}$. Then, the flow on link $l \in \mathcal{L}$ is $v(l,\mu_n): \mathcal{L} \times \mathcal{P}(\mathcal{S}) \rightarrow \mathbb{R}$, where $v(l,\mu_n)=\sum_{s\in \mathcal{S}} \xi \mu_n(s) \delta_{l,s}$. $\delta_{l,s}$ equals $1$ if link $l$ is on route $s$, and $0$ otherwise. Let $t_l\left(v(l,\mu_n)\right)$ denote the link travel cost. Then, the path travel cost is $f(s,\mu_n) = \sum_{l \in \scrL} t_l\left(v(l,\mu_n)\right) \delta_{l,s}$

The model is applicable to any inertia behavior since we do have no requirement in the form of $d(s,s')$. For example, for commuters currently choosing $s$, their adjustment cost can be $d(s,s')=\epsilon \cdot \mathbf{1}_{s \neq s'}$, implying that commuters will receive the same penalty when switching to a different route. Alternatively, we can use a distance function that is inversely proportional to the overlap between $s$ and $s'$, which indicates that switching to a more familiar route will induce less inertia cost. We consider the error in the value function by including the entropy term. Thus, we have $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta} \ln \pi_s(s')$. Apparently, Assumption \ref{acc} and \ref{accg} are always satisfied. Thus, the existence of the MFE is ensured in the current setting of day-to-day route choice.

\subsubsection{Unique link flow evolution}
In Section \ref{ssumfe}, we use the strict monotonicity condition to ensure the uniqueness of the MF distribution sequence. However, since the daily travel cost $f(s,\mu)$ refers to the path travel cost, its strict monotonicity may not necessarily hold for a general network. Therefore, we modify the assumption and assume that the link travel cost is strictly monotone.

\begin{assumption} [Modified version of Assumption \ref{gawm}]
\label{awm}
The link cost function $t_l(v)$ is strictly monotone for all link $l \in \scrL$.
\end{assumption}

The assumption is mild and has been widely adopted in the literature. Considering the MF distribution essentially corresponds to the path flow, an intuition is that with the new assumption, the MF distribution sequence may not be unique anymore, but they should share the same link flow evolution. The following proposition modifies Theorem \ref{tgumfe} and formalizes the intuition. We leave the proof to Appendix.

\begin{proposition}
\label{tumfe}
When Assumption \ref{acc}, \ref{accg} and \ref{awm} hold, for any given initial distribution, all MFEs have the same link flow evolution.
\end{proposition}

Further, note that with Assumption \ref{awm}, the path travel cost may still be strictly monotone for some networks, such as a single OD network connected by parallel routes or the small grid network shown in Figure \ref{fnet}, adopted from \cite{he2016marginal}. Consequently, the MFE is unique.

\subsubsection{Convergence to stationarity}

Since our analysis in Section \ref{sscs} for the case without the adjustment cost only requires the existence of the MFE, which is satisfied in the current setting. As a result, when user inertia is not considered, the day-to-day route choice will always reach the logit-SUE or DUE immediately on the second day, regardless of the structure of the cost function $f(s,\mu)$.

However, for a more general case with inertia, the convergence requires that $f(s,\mu)$, i.e., path travel cost, is strongly monotone. This is a strong requirement. Similar to the above, if link travel costs are strongly monotone, the strong monotonicity of path travel costs holds for networks whose path-link incidence matrix is full row rank (See Proposition \ref{ppsm} in Appendix for detailed proof). It is worth noting that some commonly used cost functions such as the BPR function are not generally strongly monotone \citep{lu2010stability}. However, it is still reasonable to consider the BPR function to be strongly monotone in our model setting when the cost function contains the entropy term, i.e., $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta} \pi_s(s')$. The reason is that, according to Proposition \ref{lbmu}, there exists a lower bound $\omega$ for all possible MF distributions in every MFE and s-MFE, which indicates that all of the link flow also has a lower bound $\xi\omega$, and the BPR function is strongly monotone in this range $[\xi\omega,\infty)$. In other words, the formulation in the range $[0, \xi\omega)$ does not influence the resulting MFE and s-MFE.

In summary, for small and sparse networks whose path-link incidence matrix is full row rank, it is still reasonable to consider the path travel time strongly monotone with link travel time functions of the BPR type. Therefore, the exponential convergence to stationarity still holds. Compared to the stability analysis results in the literature, a system with strategic commuters reaches stationarity immediately on the second day or converges exponentially, for the case without and with adjustment costs respectively.

\subsection{Departure time choice}
\label{pqcase}

Let's consider a discrete version of the classic Vickrey bottleneck model for the morning commute problem \citep{Vickrey1969}. The state space $\mathcal{S}$, in this case, refers to the allowable departure time window, and each state $s$ is a time slice in the window. Denote the length of the departure time window as $L$ hours, then the length of each time slice is $\frac{L}{M}$ hours. Similarly, $\mu_n(s)$ refers to the proportion of commuters who departs at $s$ on day $n$, which is equivalent to the departure rate at time $s$. Denote the total number of commuters is $\xi$, then the number of commuters that departs at time $s$ on day $n$ is $\xi\mu_n(s)$. Without losing generality, we ignore the free-flow travel time, and thus travel time experienced by commuters only consists of the queuing delay at the bottleneck. The discrete version of the travel time function proposed by \cite{han2013partial} is adopted:
\begin{align*}
    T(s,\mu_n) = \sum_{x\in \mathcal{S}, x\leq s}\frac{\mu_n(x)}{C_b}-s-\min_{y \in \mathcal{S}, y \leq s} \left\{ \sum_{x\in \mathcal{S}, x\leq y}\frac{\mu_n(x)}{C_b}-y \right\}
\end{align*}
where $T(s,\mu_n)$ refers to the travel time in day $n$; $C_b$ refers to the "normalized" bottleneck capacity, which is $C_b = \frac{C_b^*}{\xi} \frac{L}{M}$, where $C_b^*$ is the true capacity in the unit of commuters per hour, and $y$ refers to the nearest time that the inflow is below the capacity. Suppose all commuters share the same desired arrival time $r$. The scheduling cost is included in the travel cost function:
\begin{align*}
    f(s,\mu_n) := \alpha T(s,\mu_n)+\beta [r-s-T(s,\mu_n)]_+ + \gamma [s+T(s,\mu_n)-r]_+
\end{align*}
where $\alpha, \beta, \gamma$ are the penalty coefficients for travel time, early arrival and late arrival, and $[r-s-T(s,\mu_n)]_+= \max\left\{ r-s-T(s,\mu_n), 0\right\}$. We consider the adjustment cost to be a distance function, $d(s,s')=\epsilon \cdot |s-s'|$.

The following proposition shows that the travel cost considering scheduling cost satisfies Assumption \ref{acc} and \ref{accg}. We leave the proof to Appendix. As a result, for any initial departure time distribution, the day-to-day departure time choice problem always has an MFE. However, due to the complex structure of the cost function, it remains an open question whether the MFE is unique, or whether the MFE converges to the stationarity.
\begin{proposition}
\label{pbmc}
Assumption \ref{acc} and \ref{accg} hold for the cost function in the day-to-day departure time choices problem.
\end{proposition}

Note that in some more special cases, we can continue to derive more results. For example, if the travel cost does not contain scheduling cost, and the travel time only depends on the current inflow without the historical effect in terms of the queueing model, we can still prove the uniqueness of the MFE. The analysis for these cases resembles the analysis in Section \ref{bprcase} and will not be discussed in detail.

\subsubsection{Convergence to stationarity}

Specifically, in the case of no adjustment cost, as per the analysis in Section \ref{sscs}, the system will reach SUE or DUE immediately on the second day. This result contrasts with those in the literature. More specifically, although the day-to-day departure-time choice dynamics for myopic commuters usually cannot converge to an equilibrium state \citep{guo2018we}, our results show that for strategic commuters who do not suffer user inertia, the system can reach equilibrium immediately on the second day.

Below we summarize the connection between our model and the conventional Wardropian equilibrium in both route and departure time choices for the entropy penalized model when $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta}\ln\pi_s(s')$.

\begin{table}[ht]
\caption{Connection with Wardropian equilibrium when $g(s,s',\pi_s) = d(s,s')+\frac{1}{\theta}\ln\pi_s(s')$}\label{tres}
\begin{center}
\begin{tabular}{l|l|l|l}
                                 & Choice scenario               & Results                                         & Required conditions      \\ \hline
\multirow{2}{*}{Without inertia} & Route           & Reaching SUE/DUE on the second day                 & Assumption 1    \\
                                 & Departure time  & Reaching SUE/DUE on the second day                 & Assumption 1    \\ \hline
With inertia                     & Route           &
\makecell[l]{Exponential convergence to an \\ instance of SDSUE}  & Assumption 1,5,6
\end{tabular}
\end{center}
\end{table}

\section{Numerical examples}
\label{numer}
In this section, we provide numerical examples to illustrate the proposed model. We first introduce the numerical algorithm used to compute the approximate MFE and then present two experiments for day-to-day route and departure time choice problems.

\subsection{Numerical algorithm}
Finding an MFE is essentially to solve a fixed point problem. We seek an MF distribution sequence $\mu$ such that $\mu = \Psi(\Phi(\mu))$ using fictitious play  \citep{perrin2020fictitious, elie2020convergence}, as presented in Algorithm \ref{algofp}. It is an iterative scheme that computes the average MF flow induced by the past optimal responses and its corresponding policy sequence as follows. By this way, the policy sequence $\bar{\pi}^j$ can always induce the MF flow $\bar{\mu}^j$ at every iteration  \citep{perrin2020fictitious}
\begin{equation*}
    \bar{\mu}^j_n = \frac{j-1}{j} \bar{\mu}^{j-1}_n + \frac{1}{j} \mu^j_n \quad \forall n
\end{equation*}

\begin{equation*}
    \bar{\pi}_n^j(a|s) = \frac{\sum_{i=0}^j \mu_n^i(s) \pi_n^i(a|s)}{\sum_{i=0}^j \mu^i_n(s)} \quad \forall n,s,a
\end{equation*}

\begin{algorithm} [ht]
	\caption{Fictitious Play} \label{algofp}
	\hspace*{\algorithmicindent} \textbf{Input}: An initial distribution $\mu_0$; An initial policy sequence $\pi^0$; Calculate the MF flow $\bar{\mu}^0$ induced by $\pi^0$  starting from $\mu_0$ \\
	\hspace*{\algorithmicindent} \textbf{Output}: $(\bar{\pi}^J, \bar{\mu}^J)$
	\begin{algorithmic}[1]
		\For {$j=1,2,\ldots, J$}
		    \State Compute $\pi^{j}$, the optimal response policy against $\bar{\mu}^{j-1}$
		    \State Compute $\mu^{j}$, the MF flow induced by $\pi^{j}$ starting from  $\mu_0$
		    \State Update $\bar{\mu}^j$, the average MF flow
		    \State Update $\bar{\pi}^j$, the policy that generates $\bar{\mu}^j$
		\EndFor
	\end{algorithmic}
\end{algorithm}

Exploitability is used to measure the convergence performance of the algorithm \citep{perrin2020fictitious}. For a pair $(\pi,\mu)$ that satisfies $\pi = \Phi(\mu)$, the exploitability is defined as $\phi(\pi,\mu) = J^\pi(\mu) - \min_{\pi'\in \Pi} J^{\pi'}(\mu)$, which characterizes how close between $\pi$ and the optimal policy.

\subsection{Day-to-day route choice}

The proposed model is applied to a three-by-three grid network taken from \cite{he2016marginal}, as shown in Figure \ref{fnet}. With a link flow $v(l)$ for $l\in \mathcal{L}$, the travel time is
\begin{equation*}
    t_l(v(l)) = t_l^0 \left[ 1+b_l \left( \frac{v(l)}{c_l} \right)^4 \right]
\end{equation*}
where the coefficients of the travel time function are randomly generated and provided in Table \ref{tbprc}. All commuters travel from node 1 to 9, and the total inflow is 2,000. Commuters have 6 paths to choose. Path-link relationship is given in Table \ref{tpl}. The planning horizon contains $N=30$ days, with an initial MF distribution $\mu_0 = [0.1,0.1,0.5,0.1,0.1,0.1]$. Lastly, the behavioral cost is $g(s,s',\pi_s) = \epsilon \cdot \textbf{1}_{s\neq s'} + \frac{1}{\theta}\ln\pi_s(s')$. As previously analyzed in Section \ref{bprcase}, the path travel cost can be considered to be strongly monotone in this case.
\begin{figure}[!ht]
  \centering
  \includegraphics[width=0.3\columnwidth]{pic/9node12link.png}
  \caption{Test network with nine nodes and twelve links.}\label{fnet}
\end{figure}
\begin{table}[ht]
\caption{Link travel cost information}\label{tbprc}
\begin{center}
\begin{tabular}{c|ccc|c|ccc}
Link & $c_a$  & $b_a$   & $t_a^0$ & Link & $c_a$  & $b_a$   & $t_a^0$ \\ \hline
0    & 600 & 0.23 & 15 & 6    & 500 & 0.16 & 17 \\
1    & 600 & 0.29 & 12 & 7    & 500 & 0.24 & 19 \\
2    & 600 & 0.22 & 14 & 8    & 500 & 0.18 & 11 \\
3    & 500 & 0.18 & 12 & 9    & 800 & 0.19 & 17 \\
4    & 900 & 0.21 & 14 & 10   & 700 & 0.23 & 10 \\
5    & 600 & 0.2  & 17 & 11   & 600 & 0.16 & 16
\end{tabular}
\end{center}
\end{table}

\begin{table}[ht]
\caption{Path-link relationship}\label{tpl}
\begin{center}
\begin{tabular}{c|l}
Path &   \quad Link      \\ \hline
0    & 0, 1, 4, 9   \\
1    & 0, 3, 6, 9   \\
2    & 0, 3, 8, 11  \\
3    & 2, 7, 10, 11 \\
4    & 2, 5, 8, 11  \\
5    & 2, 5, 6, 9
\end{tabular}
\end{center}
\end{table}

First, we consider a general case with $\epsilon=\theta=1$, where both adjustment costs and the error term are considered. Results are shown in Figure \ref{fnet_e1t1}. As can be seen from Figure \ref{fnet_e1t1}(a), the algorithm converges well to the MFE since the exploitability reaches near 0. Figure \ref{fnet_e1t1}(b) shows the MF distribution of all 30 days, which represents the evolution pattern. The six dots on each curve represents the MF distribution on that day. Here we use curves connecting dots on the same day to better demonstrate the evolution. A drastic adjustment happens on day 0 when the MF distribution change to the green curve from the blue one. Later, the adjustment becomes mild. The orange curve, the red curve, and the curves on all the following days are almost overlapping, which indicates that the distribution is converging and gradually stabilized. The travel cost $f(s,\mu_n)$ and the augmented cost $f(s,\mu_n)+\frac{1}{\theta} \ln \mu_n$ when $n=29$ (the last day) are shown in Figure \ref{fnet_e1t1}(c). Neither travel costs nor augmented costs are equal for all routes, which shows that the final distribution is neither logit-SUE nor DUE.

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=1\columnwidth]{pic/net_e1t1.png}}
    {Day-to-day route choices dynamics in a grid network with $\epsilon=\theta=1$.\label{fnet_e1t1}}
    {}
\end{figure}

We can also check the policy sequence of the MFE. Figure \ref{fpolicy} shows $\pi_n$ for $n=0$ and $n=29$, where the $i$th row, $j$th column depicts the probability of choosing $j$ as the next option when the current choice is $i$. The action with denser color indicates a higher probability. By definition, row $i$ reflects $\pi_n(x|s=i), x\in\mathcal{S}$, which is the action distribution when the current state is $i$, while column $j$ reflects $\pi_n(s=j|x), x\in\mathcal{S}$, which is the inflow from other states to state $j$. Note that the summation of one column may not be 1. From the first column in Figure \ref{fpolicy}(a), many commuters will switch to route 1, causing a large jump in flow on route 1 in Figure \ref{fnet_e1t1}(b). On contrary, from the third column, commuters on other routes will seldom switch to route 3, even the commuters currently on route 3 have a high probability of changing to other routes. Therefore, the MF distribution of route 3 drops drastically. Figure \ref{fpolicy}(b) shows the equilibrium policy on the last day. Results show that the policy is more concentrated on the current choice. However, due to the entropy penalization, the policy cannot be fully concentrated. Another observation is that $\pi_{29}(s|s)$ is the same for all state $s$, and the probability of switching to other choices is uniformly distributed.

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=0.8\columnwidth]{pic/policy.png}}
    {Equilibrium policy on the first and the last day when $\epsilon=\theta=1$.\label{fpolicy}}
    {}
\end{figure}

On contrary, when $\epsilon=0, \theta=1$, representing the case without adjustment costs, the evolution pattern is different. Similarly, the MF distributions on all 30 days are plotted in Figure \ref{fnet_e0t1}(b), from which we can find that all the curves since day 1 are overlapping. It indicates that the MF distribution reaches stationarity on day 1 and does not change anymore. From Figure \ref{fnet_e0t1}(c), the augmented costs are equal for all 6 routes, which indicates that the stationary distribution corresponds to the logit-SUE flow. This result also agrees with the analysis in Section \ref{scs}.

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=1\columnwidth]{pic/net_e0t1.png}}
    {Day-to-day route choices dynamics in a grid network with $\epsilon=0, \theta=1$.\label{fnet_e0t1}}
    {}
\end{figure}

To show the results when $\theta$ is gradually increasing, we increase $\theta$ to 20 while still maintaining $\epsilon=0$. From Figure \ref{ftheta}(c), for larger $\theta$ (lower variance), the travel cost and the augmented travel cost are nearly overlapping, which indicates that the final MF distribution is converging to DUE.

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=1\columnwidth]{pic/net_e0t20.png}}
    {Day-to-day route choices dynamics in a grid network with $\epsilon=0, \theta=20$.\label{ftheta}}
    {}
\end{figure}

\subsection{Day-to-day departure time choice}

We use the bottleneck model setting in \cite{guo2018day}. The total number of commuters is $6,000$, and the capacity of the bottleneck is $c=3,000$ vehicles per hour. The penalty for travel time, early arrival and late arrival are $\alpha=10, \beta=5, \gamma=15$, respectively. The departure time window on each day is $[0,3]$ hours, which is further discretized into $M=40$ slices. Thus, the length of each slice (i.e. each state) is $0.075$ hours or 4.5 minutes. The desired arrival time for all commuters is $r=2$. The planning horizon contains $N=30$ days, and the MF distribution is initialized in uniform distribution. Here we use $g(s,s',\pi_s)=\epsilon \cdot |s-s'|+\frac{1}{\theta} \ln \pi_s(s')$. We set the inertia weight $\epsilon=1$ so that the adjustment cost and the daily travel cost are comparable. Besides, since the concept of SUE is not widely used in departure time choice problems, we set $\theta=20$ to represent a relatively low variance in the perception error. Results are shown in Figure \ref{fdepart_e1t20}. As before, we plot the MF distribution on all 30 days in Figure \ref{fdepart_e1t20}(b). From the figure, we can observe that the adjustments do not stop, indicating the system does not converge to stationarity in the period. The travel cost and the augmented travel cost are fairly close on the last day, which indicates that $\theta=20$ is large enough to clear the influence of the perception error.

When there are no adjustment costs and $\theta=20$, results are shown in Figure \ref{fdepart_e0t20}. Similarly, all the curves except the initial one are overlapping, which indicates that the MF distribution reaches the orange curve on the second day and stays invariant for the rest days. This result also agrees with the analysis in Section \ref{scs}. The discrepancy among the travel costs for all chosen departure times is around 8\%, which is fairly small. We attribute this discrepancy to the computational effort because the exploitability measure remains more than 8 in the final iteration and the algorithm does not exactly locate the MFE. In fact, it is proven that the algorithm can converge to the MFE with monotone cost functions \citep{perrin2020fictitious}. Since the cost function adopted here for day-to-day departure time choice is not monotone, the convergence of the algorithm is not guaranteed. We leave the development of a better algorithm to further investigation.

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=1\columnwidth]{pic/depart_e1t20.png}}
    {Day-to-day departure time choice dynamics with $\epsilon=1, \theta=20$.\label{fdepart_e1t20}}
    {}
\end{figure}

\begin{figure}[ht]
    \FIGURE
    {\includegraphics[width=1\columnwidth]{pic/depart_e0t20.png}}
    {Day-to-day departure time choice dynamics with $\epsilon=0, \theta=20$.\label{fdepart_e0t20}}
    {}
\end{figure}

\section{Conclusion}
\label{conc}
In this research, we consider day-to-day travel choices of perfectly strategic and farsighted commuters. Different from existing studies that use heuristic rules to aggregately model day-to-day adjustment of travel choices, we explicitly capture commuters' strategic decision-making process. To do so, we model the process as an MFG and cast the travel choices of commuters as an optimal control problem. We then examine the equilibrium of the MFG as the evolution pattern. General properties are analyzed without focusing on any specific travel choice. The results are further used to derive corresponding properties for day-to-day route choices and departure time choices. Various numerical experiments are conducted to better illustrate the proposed model.

Broadly speaking, our study illustrates that the notion of Wardropian equilibrium may remain relevant in modeling future transportation networks even with strategic traveling agents, as the connection between the day-to-day traffic dynamics and Wardropian equilibrium is established as summarized in Table \ref{tres}. Furthermore, by accurately characterizing the day-to-day evolution using the concept of MFE, the MFE-based analysis may eventually constitute a serious alternative to the equilibrium paradigm for analyzing a transportation system.

For future studies, it is important to first relax the homogeneity assumption. One way to accommodate heterogeneous commuters is to consider multi-type MFG, where players are classified into multiple types, and players of the same type are still assumed to be homogeneous. Besides, when it comes to the heterogeneous setting, it is important to consider cases with imperfect and incomplete information, because players of one type may not know the proportion of each type and detailed information (i.e. OD, cost preference, and actions) of other types. It is interesting to investigate how to incorporate the learning process in the framework, where agents update their policies while observing the information based on their daily experiences. Another direction to extend the current work is to incorporate myopic agents into the model. The behavior of the myopic agents can be modeled aggregately using, e.g., the proportional swap rule \citep{smith1984stability}, and can be further treated as a common noise in the strategic decision making process.

\ACKNOWLEDGMENT{%
The work described in this paper was partly supported by research grants from National Science Foundation (CMMI-1854684 and CMMI-1904575) and the USDOT University Transportation Center for Connected and Automated Transportation (CCAT).

\bibliographystyle{informs2014trsc}

redpajama_set_name: RedPajamaArXiv


On the occasion of their first European hotel opening in Paris, the Shangri-La Group had a new visual identity developed. The resulting design aesthetically associates the hotel group's Chinese origin with the originalarchitecture of the Roland Bonaparte mansion. A harmoniously balanced colour concept with playful illustrations marks the different areas of the hotel, such as restaurants, bars and meeting rooms. The result is a self-contained identity inspired by the Belle Époque.


The Friends organise events throughout the year to raise funds for The Vale & Downland Museum. Activities include second-hand book sales, outings, talks and other fundraising events.
The Friends of the Vale & Downland Museum take data privacy very seriously and you can be assured all of your information is stored in accordance with the General Data Protection Regulation. Please click here to read more.
An annual subscription to the Friends of the Vale & Downland Museum is just £6.50 for individuals and £10 for families.
Click here to download the Friends of VDM Application Form.


Question 1: Get the algebraic expression in the following cases using variables, constants and arithmetic operations.
Subtraction of z from y.
One half of the sum of numbers x and y.
One fourth of the product of numbers p and q.
Numbers x and y both squared and added.
Number 5 added to three times the product of numbers m and n.
Sum of numbers a and b subtracted from their product.
Question 2: Identify the terms and their factors in the following expressions. Show the terms and factors by tree diagrams.
Answer: Term xy2 and factor 5; term x2y and factor 7.
Answer: Term x and factor – 4; term y and factor 5.
Answer: Term y and factor 6; term y2 and factor 3.
Answer: Term xy and factor 1; term x2y2 and factor 2.
Answer: Term pq and factor 1; term q and factor 1.
Answer: Term ab and factor 1.2; term b and factor 2.4; term a and factor 3.6.
Answer: Term x and factor ¾ .
Answer: Term t2 and coefficient – 3.
Answer: Term t and coefficient 1; term t2 and coefficient 1; term t3 and coefficient 1.
Answer: term x and coefficient 1, term xy and coefficient 2; term y and coefficient 3.
Answer: term m and coefficient 100; term n and coefficient 1000.
Answer: term p2q2 and coefficient – 1; term pq and coefficient 7.
Answer: Term a and coefficient 1.2; term b and coefficient 0.8.
Answer: r2 and coefficient 3.14.
Answer: Term l and coefficient 2; term b and coefficient 2.
Answer: Term y and coefficient 0.1; term y2 and coefficient 0.01.
Question 5: Identify terms which contain x and give the coefficient of x.
Question 6: Identify the terms which contain y2 and give the coefficient of y2.
Question 7: Classify into monomials, binomials and trinomials.
Question 8: State whether a given pair of terms is of like or unlike terms.


text:
var Navbar = require('./navbar');
require('chai').should();

describe('Navbar', () => {

  before(() => {
    this.empty = new Navbar();
    this.single = new Navbar([
      { title: 'Hello', route: 'hello' }
    ]);
    this.generic = new Navbar([
      { title: 'Foo', route: 'foo' },
      { title: 'Bar', route: 'bar' },
      { title: 'Baz', route: 'baz' }
    ]);

    this.activeEmpty = new Navbar([], 'none');
    this.activeSingle = new Navbar([
      { title: 'Hello', route: 'hello' }
    ], 'hello');
    this.activeGeneric = new Navbar([
      { title: 'Foo', route: 'foo' },
      { title: 'Bar', route: 'bar' },
      { title: 'Baz', route: 'baz' }
    ], 'bar');
  });

  it('should have a create static method which instantiates a new Navbar', () => {
    Navbar.should.contain.keys('create');
    Navbar.create().constructor.should.equal(Navbar);
  });

  it('should have goto, toggle and render methods', () => {
    Navbar.prototype.should.contain.all.keys('goto', 'toggle', 'render');
  });

  it('should be collapsed when created', () => {
    this.generic.state.collapsed.should.equal(true);
  });

  it('should have menu items from constructor call', () => {
    this.empty.state.items.should.eql([]);
    this.single.state.items.should.eql([{ title: 'Hello', route: 'hello', active: false }]);
    this.generic.state.items.should.eql([
      { title: 'Foo', route: 'foo', active: false },
      { title: 'Bar', route: 'bar', active: false },
      { title: 'Baz', route: 'baz', active: false }
    ]);
    this.activeEmpty.state.items.should.eql([]);
    this.activeSingle.state.items.should.eql([{ title: 'Hello', route: 'hello', active: true }]);
    this.activeGeneric.state.items.should.eql([
      { title: 'Foo', route: 'foo', active: false },
      { title: 'Bar', route: 'bar', active: true },
      { title: 'Baz', route: 'baz', active: false }
    ]);
  });

  it('should toggle collapsed state', () => {
    this.generic.state.collapsed.should.equal(true);
    this.generic.toggle();
    this.generic.state.collapsed.should.equal(false);
    this.generic.toggle();
    this.generic.state.collapsed.should.equal(true);
  });

  it('should change the active property of an item when goto is called', () => {
    this.generic.state.items.should.eql([
      { title: 'Foo', route: 'foo', active: false },
      { title: 'Bar', route: 'bar', active: false },
      { title: 'Baz', route: 'baz', active: false }
    ]);
    this.generic.goto('baz');
    this.generic.state.items.should.eql([
      { title: 'Foo', route: 'foo', active: false },
      { title: 'Bar', route: 'bar', active: false },
      { title: 'Baz', route: 'baz', active: true }
    ]);
  });

  it('should call render when toggle is called', (done) => {
    this.generic.render = (state) => {
      state.collapsed.should.equal(false);
      done();
    };
    this.generic.toggle();
  });

  it('should call render when goto is called', (done) => {
    this.generic.render = (state) => {
      state.items.should.eql([
        { title: 'Foo', route: 'foo', active: true },
        { title: 'Bar', route: 'bar', active: false },
        { title: 'Baz', route: 'baz', active: false }
      ]);
      done();
    };
    this.generic.goto('foo');
  });
});

redpajama_set_name: RedPajamaGithub


Two states plan to put marauding primates on the pill in an effort to curb human-monkey conflict.
Monkeys better tone down their 'business', or else some states in India might put them on the pill.
The most commonly seen Indian monkey, the Rhesus Macaque, can charm some but can be a pain for others with their marauding ways, belligerence and lack of fear of humans.
They are not often violent, but they do eat farmers' crops, ransack homes and harass humans in their quest for a meal.
Clashes between monkeys and humans are on the rise in Uttarakhand and Himachal Pradesh, according to government officials. As a result, the two states are considering fertility drugs to combat the problem.
"No crops are growing because these monkeys are taking it all. They are increasing in numbers, troubling farmers," Himachal Pradesh Forest Minister Thakur Singh Bharmouri told Al Jazeera.
India banned exporting monkeys for biomedical research in 1978. The country used to be the world's largest exporter of monkeys. Since then, the country has had trouble with their burgeoning population. Rampant urbanisation encroaching on their habitats has added to the problem.
The lush northern states of Himachal Pradesh and Uttarakhand have been particularly vulnerable to conflicts with monkeys as jungles invariably adjoin farmlands.
Himachal Pradesh has already performed more than 68,000 vasectomies on monkeys since the state started the country's first sterilisation programme in 2006.
Officials estimate there are about 300,000 monkeys in the state, all of which they aim to sterilise.
Government employees in several parts of the state either capture the monkeys or pay Rs 500 (about $8) as bounty to people who catch and bring them to centres where they perform the operation.
But while the number of monkeys has reduced, the situation is not improving, said Bharmouri.
Monkeys are getting more aggressive and the forest department is looking for other ideas, he said, adding "progress has been made but I still need to protect people".
Both Himachal Pradesh and Uttarakhand are working with an advisory organisation, the Wildlife Institute of India (WII), on ways to control the number of monkeys.
The institute has recommended that Uttarakhand also start performing vasectomies, and that both states research the feasibility of anti-fertility drugs, among other ideas.
"We are still looking at all the material. We need to find out how to deliver them [anti-fertility drugs]. Also, how do we identify the individuals we've already given it to?" PC Tyagi, a landscape management specialist at the WII, told Al Jazeera.
"They need to make a feasible plan."
Tyagi could not specify what type of drugs could be used and emphasised that the idea is still in its early stages.
The states must also keep a record of how many monkeys there are and where they live so they can analyse and keep track of the impact, he said.
Researchers at the California National Primate Research Center are also working with the WII and the Indian government to help resolve the human-monkey conflict.
The center says fertility drugs have not been extensively tested on non-human primates.
But Brenda McCowan, the center's programme leader, said the contraceptive, porcine zona pellucida, administered by an injection, has worked well with horses and deer.
"Should such drugs be considered as one tool to address monkey overpopulation, the plan would be to approach this cautiously by conducting safety and efficacy trials before considering widespread use," McCowan said.
Wildlife officials in the capital of New Delhi are also interested in sterilising monkeys because they say relocation efforts to a local sanctuary are not working.
Delhi struggles with employing monkey-catchers who are supposed to bring the animals to the wildlife sanctuary where officials say some 20,000 of them are fed daily. Wildlife officials also say the monkeys are expensive to feed and they do not stay in the sanctuary, often returning to their places of origin.
While contraceptives could be effective, animal activists say existing procedures are not carried out properly.
Himachal Pradesh has been under fire because the sterilisations have resulted in hundreds of monkey deaths, either from poor capturing conditions or untrained veterinarians.
Uttarakhand has looked into performing vasectomies in the past, but the state was confronted with vast opposition and did not follow through with the plans.
Member of Parliament and prominent animal rights activist Maneka Gandhi doubts the effectiveness of sterilisation, vasectomies and contraceptives.
"It may remove reproductive ability, but will it stop them from coming into the cities for food? No," she told Al Jazeera.
"It will simply result in breaking up families so that more people are attacked. It will also result in a large number of monkey deaths."
McCowan, however, said the center has more than a decade of experience studying macaque behavior and that if done properly, contraception could be a positive solution.
"Such an approach would be used to compliment the already robust efforts of the monkey sterilisation programmes," she said.


text:
Google Announces 1B Total Android Activations, Names Next Version 'KitKat'
Chris Velazco	@chrisvelazco / 6 years
Apple made waves this morning when it announced its oft-rumored September 10 event, but Google isn't giving up the day's limelight without a fight — according to a post on Sundar Pichai's Google+ page, more than 1 billion Android devices have been activated to date. That's not exactly a shock considering the sort of momentum we've seen in activations over the past few months. CEO Larry Page said there were 1.5 million Android activations a day back in July and Android device activations hit the 900 million mark earlier this year.
Even more puzzling than that is the name of the next version of Android, which bucks the long-standing trend of generic dessert names. As it turns out, Android 4.4 is going to be called "KitKat" and not "Key Lime Pie" as earlier thought, though at this point there's still no official word on when we can expect to see the build go live. Kit Kats are of course a popular chocolate treat made by Nestle, and at this point we're trying to determine how much this crazy little deal is worth, but the promotional blitz has already begun.
UPDATE: Google has confirmed to the BBC that the idea for the name originated with them, and that no money is changing hands as part of the deal.
Nestle has already kicked off a bizarre cross-promotion strategy that will bestow Nexus 7s and Google Play credits on those who find specially branded Kit Kat candy bars emblazoned with the Android logo, à la Willy Wonka. Yes, this is really happening. Apparently the agreement was finalized behind closed doors at this year's Mobile World Congress in Barcelona, and all parties involved amazingly managed to keep the name under wraps (I slay me) for six months despite the fact that Nestle has been churning out that Android-laden packaging for 50 million candy bars.
Though the name itself is enough to get Android fans and mobile pundits talking, Google may be looking to expand its scope with this latest update. The newly anointed Android KitKit landing page claims that it's the company's goal "to make an amazing Android experience available for everybody," which may hint at an expansion into different hardware segments. Google already has some skin in the smartphone and tablet games thanks to its acquisition of Motorola Mobility and hardware partnerships with Asus and LG, but it's possible we could see Android KitKit powering a smartwatch — a move made partially possible by Google's quiet acquisition of WIMM Labs last year.
[The life-sized Android KitKat featured in Sundar Pichai's image was created by Themendous.]

redpajama_set_name: RedPajamaCommonCrawl


Note that Small-cap and Mid-cap Indexes have buyback yields that are generally smaller than the Large-cap Indexes.
Roth's source (S&P Dow Jones Indices) claims that the buy-back yield last year was 2.91%. I did not realize it was so high. Dividend yield was 2.16%.
2.16% + 2.91% + 1.3% = 6.37% expected real return (absent any change in valuations, i.e. Bogle's "speculative return").
This is higher than most forecasters expect, but is in line with long term historic returns. Maybe the "new normal" angst is misplaced.
I expect that the buyback yield is more fragile than the dividend yield. Companies seem loathe to cut dividends but buybacks come and go in tune with the economy. Maybe the buyback term should have a discount factor for this fragility.
Those of us following the Bill Bernstein formula for liability matching portfolios were taught to count on only half of our equities' dividends, in real terms, in hard times, since that is the worst cut in the historic record. I think the high buyback yield makes it less likely that dividends will be cut severely in the next downturn. In the 2008 crash real dividends were cut by 25%, per the Shiller data set.
It seems appropriate to use net buybacks in the Gordon equation, but it should be net of dilution. After dilution, the buyback yield is close to zero.
Note that the appendix linked there is a separate document from the main paper which also has an appendix.
I don't understand the article. Why do investors have to "give themselves permission" to cash out the "buyback yield." Is this really helping anyone understand how to take income from an investment portfolio, a problem that just is not a problem in the first place as far as I am concerned.
I don't believe this is correct. Both the dividend yield and the earnings growth rate factors used by Mr. Bogle in his simple expected value formula are per share. In other words, the net effect of share repurchases are already included in the dividend and earnings growth data.
Dividend Yield: "12 month dividend per share/price."
Earnings Growth Rate: "Percentage change in 12 month earnings per share."
So it wouldn't be proper to use the buyback yield in Mr. Bogle's formula or the Gordon Equation — the net effect of the share buybacks are already included in the per share data to begin with.
Simplegift wrote: I don't believe this is correct. Both the dividend yield and the earnings growth rate factors used by Mr. Bogle in his simple expected value formula are per share. In other words, the net effect of share repurchases are already included in the dividend and earnings growth data.
Since buybacks change over time, wouldn't it be better to try to include them somehow? If we ignore them, but use the historical earnings growth rate from a time with much lower buybacks, then can't we underestimate future returns? Similar for dilution, though maybe in the other direction if it's increased over time.
The truth is I'm not comfortable enough with the Gordon Model to properly argue this with you… but I can try. I believe I was once able to convince myself that for a single company with characteristics that never change, expected returns = starting yield + growth. But once we allow a dynamic world, it seems we need to make assumptions that I don't understand why it's ok to make. So I just take the model as a better-than-nothing guess and trust the academics at RA and AQR to do their best in their assumptions. Maybe over time I can better understand.
By the way, I still have a window up on your other thread, and hope to read the article to see if it helps me understand your simple chart.
My point is that the effect of net share buybacks are already included in the per share dividend and earnings growth data reported for the S&P 500 Index. If net share buybacks are positive in a given year and decrease the number of shares outstanding, the denominator in the per share calculation is decreased. And vice-versa: If net share buybacks are negative, the shares outstanding increase. So no adjustments are needed for buybacks in the historical, per share data.
Also, as you point out, this is the reason it's important to use net share buybacks in any repurchase calculations.
Simplegift wrote: My point is that the effect of net share buybacks are already included in the per share dividend and earnings growth data reported for the S&P 500 Index.
For r = y + g, it makes sense to me that we would use current y (dividend yield). But should we use current g? Doesn't it move all over the place? I thought most predictions use a long term average.
If we were to see for example, net buybacks after dilution permanently rise to 1%, is it safe to ignore this while using the long term earnings growth rate? Shouldn't we expect a higher EPS growth than in the past?
In the extreme, companies could stop paying dividends altogether and buy back shares instead. Using the old g would seem to be incorrect. Adding net buybacks to r while using the old g might be better than doing nothing.
I wasn't able to read Mr. Roth's article due to the paywall, but if this is a useful mechanism, I think it would be necessary to distinguish buybacks funded by revenue earned by the company from buybacks funded by taking on additional debt. In the low interest rate environment, there has been discussion of many companies issuing debt to fund buybacks.
Wouldn't debt-funded buybacks would have to be excluded from this? In a total return calculation, I think the market would have priced in the present value of the debt change to a stock price, so it would not be a concern, but if using buybacks and dividends as some sort of measure of sustainable income earned by the company, I'd be inclined to think it would matter.
But the article seems to be incoherent about what use the "mechanism" is supposed to have in any context having to do with an investor "getting" income.
I think it would be necessary to distinguish buybacks funded by revenue earned by the company from buybacks funded by taking on additional debt. In the low interest rate environment, there has been discussion of many companies issuing debt to fund buybacks.
lazyday wrote: For r = y + g, it makes sense to me that we would use current y (dividend yield). But should we use current g? Doesn't it move all over the place? I thought most predictions use a long term average.
Now I see where you're coming from. Right, annual EPS growth rates jumps all over the place from year to year — so one has to use a smoothed long-term average. And the net effect of all the share buybacks in recent years has been to increase the EPS growth rates over historical averages.
Since share buybacks only began in 1981 (they were illegal prior to 1981, I believe), one has to believe their net effect has been to increase earnings growth rates in recent decades — along with many other factors. But the net effect of these buybacks is already included in the EPS growth data above.
PS. History tells us that earnings growth rates of 10%+ are certainly not sustainable long-term (buybacks or no buybacks)!
Simplegift wrote: Since share buybacks only began in 1981 (they were illegal prior to 1981, I believe), one has to believe their net effect has been to increase earnings growth rates in recent decades — along with many other factors. But the net effect of these buybacks is already included in the EPS growth data above.
So I guess your solution is to use more recent g data, and avoid fouling the per-share assumptions of the model.
Haven't gotten to the formula yet.
the level of payouts that investors receive.
AQR uses two different expected returns r=y+g predictions that handle the problem of increasing buybacks.
First method, found near the top of p3 of the appendix, is essentially to replace dividend yield with .5/CAPE, or half the earnings yield. Historical payout ratio isn't far from 50%, so it's like using dividends and pretending the payout ratio didn't change. Assuming reinvested earnings make the same returns as the stock market (r), it shouldn't matter if earnings go to dividends, buybacks, or investment. This method produced a 3.7% real return prediction using end of 2016 data.
Second method, later on p3, is to replace dividend yield with net payout yield (dividend yield + 10 yr smoothed net buybacks after share issuance) and for g to use a rate for not earnings growth but total payouts. g is the average of (A) consensus forecast gdp (note 11 below) and (B) if I understand correctly, long term historical payout growth + (EPS growth since 1970 - long term EPS growth). B is considered to be EPS growth + dilution, see note 12. This method produced a 4.8% prediction.
dividend yield, net buyback yield, and aggregate GDP growth.