Notes on stationarity

In the usual statistical problems, there are two fundamental concepts: binning and stationarity. These sound trivial but it is worth it to investigate them using probability theory.

Binning gives rise to partition functions and entropy via event multiplicities and the multinomial distribution, resp.; stationarity gives rise to ergodicity and time scales.

Binning pertains to the fact that we collect data typically on repeated instances (sampling) and necessarily into discrete events (we can only measure to finite precision), and stationarity pops up because we need to assume that during this experiment the logical connections between the samples and our model do not vary with time; otherwise we cannot analyze the samples within the context of that one model.¹ In other words, stationarity is basically the consistency of the “causal mechanism” underlying the data — see exchangeable sequences (Jaynes 2003, 576).

Philosophy

Stationarity is theoretically well-defined only in the framework of stochastic processes. In the stochastic context, stationarity means that all pdfs involved are time-invariant and depend only on time differences. I think there’s no way of avoiding to read up on this subject a bit deeper if the concept is to be grasped. (Actually, the definition just written down could be incorrect.)

So let’s be outside of the stochastic framework. Flandrin (1989): “Loosely speaking, a signal $y(t)$ is considered to be stationary during a window $W$ when its ‘relevant properties’ remain the same throughout all time in $W$.” [Presumably, the relevant properties are some statistics associated with the (process generating the) signal.] Then,

The analysis and the processing of nonstationary signals call for specific tools which go beyond Fourier analysis. This is clear from the definition itself of the Fourier transform, which does not preserve [intrinsically] any time dependence and, hence, which cannot provide [by itself] any information regarding either a time evolution of spectral characteristics or a possible occurrence of time localized events. [Words in square brackets mine]

The FT can however provide a time evolution of the spectrum by use of a sliding window over the data. One says that the time-dependent spectrum thus produced is accurate when the signal is stationary during each window — i.e. the actual spectral content changes slowly enough to be resolved within each window. Seems reasonable. But let’s look closer at the above “loosely speaking” definition by subjecting it to the following scenarios.

1. If $y(t)$ stays relatively constant, then it would be judged as stationary by people not interested in the small signal wiggles;
2. If $y(t)$ repeats itself several times, i.e., it is quasi-periodic,² we’d call it stationary;
3. If the relevant property to us is only the envelope amplitude, then any amount of FM does not influence the stationarity of the signal to us;
4. Suppose persons A and B both look at $W$, observing the same $y(t)$, and suppose A and B are both interested in the same “relevant properties” of the signal. Then presumably they would agree on the signal’s stationarity, be it yes or no. But assume that person B now receives the information that in adjacent windows $\{W'\}$ the signal looks almost the same. B would then conclude that she is observing one quasi-period of $y(t)$ in $W$ and that the signal is stationary (because of point 2) in the time period that comprises the set of windows $\{W'\}, W$. This extra information can make A and B disagree.

Points 2 and 4 tell us that by most standards, quasi-periodicity signals stationarity, as some “constancy” persists and repeats through time. When we succeed in parametrizing the waveform of one period of such quasi-periodic signals the “relevant properties” that remain the same throughout time are simply these parameters. One slick way of parametrizing quasi-periodic signals is the Fourier Transform. Thus for such signals, and if our interests pertain to quasi-periodic characteristics, the FT summarizes everything about $y(t)$ of interest to us.

Presume, however, that while we’re still interested in quasi-periodic signals, we are told that the signal’s periodicity changes or disappears at a time somewhere in $W$. Then we can no longer be sure that a FT of $y(t)$ during $W$ captures the spectral content well; it will be contaminated by the change in the signal’s periodicity. But the problem is that in principle we can never know this from the FT’s output alone: the FT assumes that the input signal is one period of a perfectly periodic signal; it can deal with imperfections (quasi-periodicity) and additive noise but not with sudden changes in (or lack of) periodicity. The FT will always interpret these changes as being part of a larger, periodic waveform.³

The bottom line is that in signal processing the concept of stationarity depends on our subjective interests and is thus not an intrinsic property of the signal. However, the fiction that it somehow is an intrinsic property of the signal is afforded by (a) the common interests of signal processing people, namely spectral content, and (b) the aversion towards such subjective considerations by most of the statistical community. (Note that maintaining point (b) in any theory must result in a very awkward handling of the concept of information.)

An example of “relevant properties” that are not spectral content is the case of a noise signal. From Jaynes (2003), p. 208:

In practice, common sense usually tells us that any observed fine details of past noise are irrelevant for predicting fine details of future noise, but that coarser features, such as past mean square values, may be expected reasonably to persist, and thus be relevant for predicting future mean square values. Then our probability assignment for future noise should make use only of those coarse features of past noise which we believe to have this persistence. That is, it should have maximum entropy subject to the constraints of the coarse features that we retain because we expect them to be reproducible. Probability theory becomes a much more powerful reasoning tool when guided by a little common sense judgment of this kind about the real world, as expressed in our choice of a model and assignment of prior probabilities. [Emphasis mine]

Thus noise observed in the window $W$, defined by a set of samples $\{e_i\}$, is called stationary if it is believed that the underlying distribution $p(e|\theta)$ that is generating the noise samples doesn’t change within $W$. Within the same class of distributions, such change can be parametrized as a change in the distribution parameters $\theta$. In some cases, we can think of $p(e|\theta)$ as a real physical mechanism generating the noise (as in Johnson noise, where the central limit theorem produces the Gaussian distribution), but in most cases (e.g. cocktail party or environmental noise) the noise distribution $p(e|\theta)$ represents only our information about it. For example, if we believe that during small windows a good model of the noise is that it (1) averages out to zero, (2) has a constant power $s^2$ and (3) is memoryless, then according to the maximum entropy principle, assigning⁴

However, we note that the properties $(\mu, \sigma)$ can still be represented in terms of spectral content — indeed, the time and frequency domains are equivalent. However, note that the spectral content of noise can change considerably while still exhibiting the same average and power. So we are not really interested in its spectrum, although the quantities of interest are necessarily still expressible in spectral terms.⁵

As a last note, checking whether the relevant properties (call them $\theta$) stay constant during a given $W$ requires the subdivision of $W$ into $I$ smaller intervals of lengths $n_i > 1$ because we need to check whether the estimators ($\hat{\theta} \equiv \hat{\theta}(y_i)$) stay more or less constant; but this intrinsically requires that those $n_i$ are large enough to make the estimators for $\theta$ work. Given a certain division $\{n_i\}$, a strong variability of the associated $\hat{\theta}_i$ (i.e. the estimates made in each $i$th interval) points to non-stationarity, or it could point to insufficiently large $\{n_i\}$. It’s even worse: we must assume stationarity to hold during each of the $I$ intervals for the theoretical validity of the obtained estimates $\hat{\theta}_i$ (because the estimator functions are derived from the assumed underlying stationary distribution) — meaning that testing for the signal’s stationarity using empirical estimates of the relevant properties (e.g., $(\hat{\mu}, \hat{\sigma})$ in the stationary noise case discussed above) requires that we first establish that the signal be stationary! This is the same problem that we noted in the FT discussion above: the principle that we’re not able to determine from the FT spectrum that the signal is stationary, i.e. that the FT was appropriate in the first place.

Model stationarity

To try and formalize stationarity I propose the following concept of stationarity called model stationarity:

Given an observed signal $y(t)$ and a model $H$ with parameters $\omega$ and model function $f(t; \omega)$. Then $y(t)$ is called $H$-stationary relative to $H$ when the deviation $[y(t) - f(t; \omega)]$ is attributed to noise.

In other words, we call a signal model stationary relative to $H$ if we believe that the particular model $H$ captures the signal sufficiently well. Indeed, us attributing the deviations from the model to noise is saying that we have modeled the interesting part (for our purposes) well enough; we choose to be ignorant about that portion of the signal we designate as noise, which might or might not be considered to have a respectable physical origin (e.g. thermal noise).

If we cannot find any model $H'$ such that $y(t)$ is $H'$-stationary to it, we’ll usually want to chop up the signal into regions $R$ that are $H_R$-stationary. Note that this chopping procedure can also be modeled fully probabilistically by constructing a hierarchical model of some sorts, where the sought regions $R$ are model parameters. The goal is then to infer the $(R, H_R)$ pairs.

The model parameters $\omega$ are typically constrained to be independent of time — indeed, this is the essence of stationarity in the usual sense — but this is not actually necessary in our formulation. In practice, though, the time dependence of $\omega$ is approached via chopping up the signal: in each region $R$ the model parameters are constants.

Nonstationary signals that change too fast for FT analysis can be approached with the general technique of wavelet decompositions (e.g. Gabor filters). Roughly speaking, a wavelet decomposition (WD) is an expansion of an arbitrary function (the signal) into smooth localized contributions (basis functions) labeled by a scale and a position pa­rameter (time in our case). The time dependence of the basis functions allows the WD to react faster to local fast spectral changes in the signal during the given window $W$.

But in our formulation it is clear that nonstationary signals could be considered stationary to the WD model we’ll call $H_W$. But then what is $H_W$? Presumably we are still interested in spectral content, as WDs are time-dependent generalizations of the FT transform. However, as long as we cannot precisely state which model we are considering (i.e. how we model $y(t)$ by means of $f(t; \omega)$), we cannot decide whether the signal might be considered model stationary relative to $H_W$. As a more serious consequence, we cannot know whether the signal isn’t changing too fast for our WD; thus we cannot know whether the WD spectra are valid.

Probability theory allows us to quantify model stationarity because we can compare how well two hypotheses $H, H'$ fit a signal, and use it to quantify the plausibility that the deviations are attributable to noise, conditional on any hypothesis. Because the power spectrum of the FT has been derived as a sufficient statistic in probability theory, we can thus precisely evaluate model stationarity for the FT. Before one can do the same for other transformations such as WDs, we first have to derive them in the context of a specific model $H \equiv H_W$ before we can answer any stationarity-related question. As we saw, this is important because it is impossible to check stationarity using transformations that necessarily assume stationarity; thus it is hard to know whether during our chopping procedures our spectra, parameter estimates, etc. still make sense.

Indeed, in practice we can only answer whether “the deviation $[y(t) - f(t; \omega)]$ is attributable to noise” in the context of another model, $H'$. In the context of FTs (we’ll set $H \equiv H_P$), the other model could be white Gaussian noise or a chirped model. Another possibility lies in the incorporating of the chopping procedure into the model; two models can then differ only in the regions $R$ they’re considering, while still having the same function form $f(t; \omega)$. In any case, using model testing we can compare the probability of different models in terms of model stationarity, in function of the chopping procedure. In this way we can discover appropriate time scales in the signal, e.g. chop up $y(t)$ into the stationary regions $R =$ (attack, quasi-periodic, decay) provided we have models $H_A, H_P, H_D$ for these three signal forms.

with $\phi$ the Gaussian distribution, represents the state of knowledge [$I \equiv$ the above points (1-3)] most honestly, and the only quantities affecting our further inferences will be precisely the data images of the constraints (1-2) (See Jaynes 2003, 520).

Go to ▥ previous post in research series.