# Tagged Questions

A stochastic process satisfying the Markov property: the distribution of the future states given the value of the current state does not depend on the past states. Use this tag for general state space processes (both discrete and continuous times); use (markov-chains) for countable state space ...

1answer
55 views

### Canonical Markov Process

Let $X$ be a canonical, right-continuous Markov process with values in a Polish state space $E$, equipped with Borel-$\sigma$-algebra $\mathcal{E}$ and we assume that $t\rightarrow E_{X_{t}}f(X_{s})$ ...
1answer
37 views

### three-state Markov chain

a male and a female go to a 2-table restaurant on the same day. each day the male sits at one or the other of the 2 tables, starting at the table 1, with a Markov chain transition matrix: ...
1answer
45 views

### Rewriting Markov process

Let $X$ be a Markov proces with state space $(E,\mathcal{E})$with initial distribution $\nu$ and transition function $P_{t}$, so $$E_{\nu}(f(X_{t+s})\mid\mathcal{F}_{s})=P_{t}f(X_{s})$$ Suppose that ...
1answer
25 views

### Question on Markov chains of expected number of states

I am confused with an statement from my probability book that has to do with Markov chains. I hope someone could clarify that, if possible....Consider a Markov chain for which $P_{11}=1$ and ...
2answers
37 views

### Diffusion process. Distribution vs transition probability.

I need confirmation on the following problem: Take a SDE of the form: \begin{equation} dX_t=a(X_t,t)dt+b(X_t,t)dW_t \end{equation} where all the conditions, such that the solution $X_t$ is defined ...
1answer
30 views

### General State Space Markov Chain

I am having some difficulty understanding some early results of Markov Chain theory on a general state space. We have a function (Kernel) $K:E \times E \rightarrow \mathbb{R}$, and a distribution ...
0answers
20 views

### Variability in estimations over a non-ergodic/non-regular Markov process

Imagine we have a non-ergodic/non-regular Markov Process with with $n$ states. Among these $n$ states, there are $k$ absorbing states. For each of the $n-k$ non-absorbing states, it is not possible ...
1answer
42 views

### Identity in Markov Processes

I want to know if my reasoning here is correct, it seems simple enough but I just want clarification (I am considering the proof that if a Markov process satisfies the detailed balance condition, then ...
0answers
26 views

### Probability - How to calculate Xt by time t

A large group of cars are waiting for a car wash, and the line is infinitely long, always a car in line waiting to be washed, and that the staff can only tend to 1 car at a time. The business has 1 ...
0answers
58 views

### Markov Chains Worked Example (Stirzaker)

I have a Markov Chain with state space the non-negative integers. The rules of the M.C. are that when it is in state $i \neq 0$, it moves to one of {${0,1,2,\ldots,i+1}$} with probability $1/(i+2)$ ...
0answers
21 views

### On discrete-time stochastic attractivity of linear systems

Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$. Consider a continuous function $f: \mathbb{R}^n \rightarrow \mathbb{R}^p$. Assume that $f(0) = 0$, and that there ...
1answer
32 views

### Have there some discrete-time continuous-state Markov processes been studied?

I have seen discrete-time discrete-state Markov processes (such as random walks), continuous-time discrete-state Markov processes (such as Poisson processes), and continuous-time continuous-state ...
0answers
36 views

### Continuous time markov chain

Jobs arrive at a central computer according to a $PP(\lambda)$. The job processing times are i.i.d. $\exp(\mu)$. The computer processes them one at a time in the order of arrival. The computer is ...
1answer
46 views

### On discrete-time stochastic attractivity

Let $m$ be a probability measure on $Y \subseteq \mathbb{R}^p$, so that $m(Y)=1$. Consider a function $f: \mathbb{R}^n \times Y \rightarrow \mathbb{R}^n$, continuous on the first arguments, ...
2answers
75 views

### Probability of Extinction in a simple Birth and Death Process

We are asked to show that the probability of extinction $\zeta=\lim_{t\to \infty} P\left(X(t)=0\right)$ given by: \zeta=\begin{cases}1&\text{if }\lambda\le \mu,\\ \left(\frac \mu\lambda ...
1answer
41 views

### Metropolis Hastings definition - Proving $\pi(x)$ is the invariant density of our transition matrix

I'm currently working through the proof of the Metropolis-Hastings algorithm, and using two sources: page 328, section 3 page 1704-1705 I have a good understanding of most of the proof until ...
0answers
23 views

### Single evaluation for using exponential sampling until past a point

I am trying to improve an algorithm that looks like the following (and am getting stumped): I am provided with a starting time, rate, and a target time. I then use an exponential distribution to ...
0answers
83 views

### Why Markov matrices always have 1 as an eigenvalue

Also called stochastic matrix. Let $A=[a_{ij}]$ - matrix over $\mathbb{R}$ $0\le a_{ij} \le 1 \forall i,j$ $\sum_{j}a_{ij}=1 \forall i$ i.e the sum along each column of $A$ is 1. I ...
1answer
38 views

### Hitting times of Markov chain/process have always finite moments?

Consider an irreducible ergodic Markov chain on a finite state space $\Omega$. Then any state is positive recurrent and this should suffice to conclude that the mean hitting time of state \$s \in ...
2answers
42 views

### Specifying differential equation that describes a particular set of dynamics.

There are $S$ individuals who are susceptible to infection, and $I$ who are infectious. $S + I = N$, where $N$ is the total size of the population. Each infectious transmit the disease to a ...
1answer
51 views

### Showing a process is not markov

I keep searching but I can't find any place that gives a good method of showing a process is NOT Markov. The definition I am using is that for every $s<t$ and $g$ bounded borel there is $f$ borel ...
1answer
32 views

### Amount of information a hidden state can convey (HMM)

In this paper (Products of Hidden Markov Models, http://www.cs.toronto.edu/~hinton/absps/aistats_2001.pdf), the authors say that: The hidden state of a single HMM can only convey log K bits of ...
2answers
35 views

### How do you explain $f(x_4|x_3)f(x_3|x_2)f(x_2|x_1)f(x_1) = f(x_4,x_3,x_2,x_1)$?

Let $x_1=x(n_1)$, $x_2=x(n_2)$, $x_3=x(n_3)$ and $x_4=x(n_4)$ be random Markov processes $(n_1 < n_2 < n_3 < n_4)$. I don't understand the identity given below on their probability density ...
2answers
83 views

### Is first order moving average a Markov process?

Given first order moving average $$x(n) = e(n) + ce(n-1)$$ where $e(n)$ is a sequence of Gaussian random variables with zero mean and unit variance which are independent of each other, and $c$ is ...
0answers
105 views

### Is there monotone class theorem used in one of these steps?

IN Rogers & Williams "Diffusions, Markov Process and Martingales" they introduce the resolvent as: $$R_\lambda f(x):=\int_{[0,\infty)}e^{-\lambda t}P_tf(x)dt=\int_ER_\lambda(x,dy)f(y)$$ where ...
1answer
27 views

### A book on finite state continuous time Markov chain

I want to read in detail about finite state continuous time Markov chain. Can anybody suggest a book which deal this topic in detail?
1answer
49 views

### Random Process derived from Markov process

I have a query on a Random process derived from Markov process. I have stuck in this problem for more than 2 weeks. Let $r(t)$ be a finite-state Markov jump process described by ...
2answers
94 views

### Expected value of stochastic process

I have the following problem: $X_1,X_2,...$ are positive identically distributed random variables with the distribution function $F(x) :=P(X_n \leq x)$ and we assume that $F(0)<1$ for all $n$. Let ...
0answers
47 views

### Conditional distributions of (higher-order) autoregressive Markov processes

If we specify an $p$-th order autoregressive process in discrete time by its transition distribution $F_{t|t-1,\ldots,t-p}$, what can be said about lower order conditional distribution where we ...
2answers
37 views

### Question on MIT Markov Matrices video

Markov matrices are pretty new to me and I'm a little rusty with my linear algebra. My question stems from watching this video from YouTube on Markov matrices. For those who wish to skip the video, ...
1answer
33 views

### Showing a certain random process is a Markov Process

I have the following example of a random process: A person has two houses, house A and house B in which he can stay, we denote by $X_{i}\in\left\{ A,B\right\}$ the house he stayed in on the i-th day ...
2answers
39 views

### How to prove the existence of the limit of Markov transition matrix?

Does the limit of a Markov transition matrix $M$: $$\lim_{n\to\infty}M^n$$ always exist? And if yes, how to prove it?
1answer
192 views

### Motivation of Feynman-Kac formula and its relation to Kolmogorov backward/forward equations?

Kolmogorov backward/forward equations are pdes, derived for the semigroups constructed from the Markov transition kernels. Feynman-Kac formula is also a pde corresponding to a stochastic process ...
3answers
229 views

### Finding the transition probability matrix, two switches either on or off..

Each of two switches is either on or off during a day. On day n, each switch will independently be on with probability [1+number of on switches during day n-1]/4 For instance, if both switches are on ...
1answer
74 views

### Markov Process with Stationary Distribution

I have the following problem: If I have a markov process with stationary distribution. The state space for the MP is integers. I also know that $P_{i,j}>0$ for all i and j. It is also given that ...
0answers
35 views

### Markov Model Brainteaser

An orangutan and a chimpanzee each sit at a computer typing 1 character per second. The orangutan chooses each character independently from $S$ with probability $\frac{1}{27}$. The chimpanzee follows ...
0answers
53 views

### A different Markov property definition

In Shreve's Stochastic Calculus in Finance, the Markov property is defined as Definition 2.3.6. Let $(\Omega,\mathcal F,P)$ be a probability space, let $T$ be a fixed positive number, and let ...
0answers
82 views

### What are the definitions of a diffusion process and a jump process?

I have seen following different definitions of a diffusion process and of a jump process. I was wondering how they are actually defined? Also are diffusion processes and jump processes necessarily ...
0answers
32 views

### Is HMM discriminative or generative?

Wikipedia "An HMM can be considered as the simplest dynamic Bayesian network." Here. "In probability and statistics, a generative model is a model for randomly generating observable data, ...
0answers
54 views

### How to prove ergodic property from aperiodicity and positive recurrence

How to prove that in case of an irreducible, aperiodic and positive recurrent Markov Chain time average along sample paths is equal to the ensemble average ? i.e. \lim_{n\to \infty ...
1answer
125 views

### Multidimensional infinitesimal generator of a jump-diffusion

Let $X=\{X_t\}_{t\geq0}$ be an $n$-dimensional Markov process, defined by the SDE $$dX_t = \mu(t, X_t) \, dt + \sigma(t,X_t) \, dB_t+\beta(t-,X_{t-}) \, dN_t,$$ where $\mu, \sigma$ and $\beta$ are ...
1answer
113 views

### Markov Chain Transition Intensity Conversion

I have a question about converting a 3-state discrete state, continuous-time, markov chain to a 2-state. My 3-state model has states: Well (state 1), Ill (state 2) and Dead (state 3). ...
0answers
76 views

### Question on Conditional expectation

Let $X_1$ and $X_2$ be two random variables on $(\Omega,\mathcal{B},P)$. Suppose there is a function $g:\mathcal{B}\times\mathbb{R}\rightarrow[0,1]$ such that for any $x$, $g(\cdot,x)$ is a ...
2answers
34 views

### Different limiting distributions but they both satisfy same equations

I needed to find the limiting distribution of the matrix $$\pmatrix{ 0 & 0 & 1 \\ 1 & 0 & 0 \\ \frac{1}{2} & \frac{1}{2} & 0}$$ Instead of $\pi$ I'll use $A, B$ and $C$ ...
0answers
37 views

### HMM - how to calculate p(x[t]=i)?

Suppose we have an HMM, where $y_t$ are the observations and $x_t$ are the latent states: $p(y_1,\ldots,y_T,x_1,\ldots,x_T) = p(x_1)\prod_{t=1}^Tp(x_{t+1}|x_t)p(y_t|x_t)$ Suppose we already used the ...
1answer
46 views

### HMM as special case of MRF

I have learned that any Hidden Markov Model (HMM) can be described as a special case of a Markov Random Field (MRF) model. However, AFAIK, the dependencies in a HMM are directed, while the ...
1answer
63 views

### Markov Process: Show that the minimum time taken to get back to state $1$ is $(0.5)^{k-1}$

Suppose that the chain is intitially in state $1$, i.e $P(X_0 = 1) = 1$. Let $\tau$ denote the time of first returen to state $1$, i.e $$\tau = \min\{n > 0: X_N = 1\}.$$ Show that ...
1answer
51 views

### geometric sum - weighted random walk

I am trying to model the following sum: $\sum_{i=0}^{n}{W_i \alpha^{i}}$ where $\alpha \in[0, 1)$ and $W_n$ takes values 0 or 1 and may be modeled as a markow chain or for simplicity as a binary ...
1answer
66 views

### Is this Markov?

Consider a process $\{X_n, n\geq 0\}$ with state space $S=\{0,1,2\}$ s.t.  P(X_{n+1}=j | X_n=i, X_{n-1}=i_{n-1}, \dots, X_0=i_0)=\begin{cases} P_{ij}^I \ \ \ n \ \mbox{ even},\\ P_{ij}^{II} \ \ \ ...
1answer
58 views

### Fixed point of transition kernel generates martingale

Let $P^{h}, h \geqslant 0$ be a transition kernel for some homogenous Markov process $X_t$, $\mathbb{E}|X_t|<\infty$: $$P_{X_{t+h},X_t}(A,B) = \int\limits_{A}P^h(x,B)P_{X_t}(dx)$$ where ...
{"extraction_info": {"found_math": true, "script_math_tex": 0, "script_math_asciimath": 0, "math_annotations": 0, "math_alttext": 0, "mathml": 0, "mathjax_tag": 0, "mathjax_inline_tex": 100, "mathjax_display_tex": 8, "mathjax_asciimath": 0, "img_math": 0, "codecogs_latex": 0, "wp_latex": 0, "mimetex.cgi": 0, "/images/math/codecogs": 0, "mathtex.cgi": 0}, "config": {"markdown_headings": true, "markdown_code": true, "boilerplate_config": {"ratio_threshold": 0.18, "absolute_threshold": 10, "end_threshold": 15, "enable": true}, "remove_buttons": true, "remove_image_figures": true, "remove_link_clusters": true, "table_config": {"min_rows": 2, "min_cols": 3, "format": "github"}, "remove_chinese": true, "remove_edit_buttons": true, "extract_latex": true}, "english_confidence": 0.9005563259124756, "perplexity_flag": "middle"}