Method of moments (statistics)

For the technique used to prove convergence in distribution, see Method of moments (probability theory).

In statistics, the method of moments is a method of estimation of population parameters. The same principle is used to derive higher moments like skewness and kurtosis.

It starts by expressing the population moments (i.e., the expected values of powers of the random variable under consideration) as functions of the parameters of interest. Those expressions are then set equal to the sample moments. The number of such equations is the same as the number of parameters to be estimated. Those equations are then solved for the parameters of interest. The solutions are estimates of those parameters.

The method of moments was introduced by Pafnuty Chebyshev in 1887 in the proof of the central limit theorem. The idea of matching empirical moments of a distribution to the population moments dates back at least to Pearson.^[1]

Method

Suppose that the parameter $\theta$ = ($\theta _{1},\theta _{2},\dots ,\theta _{k}$) characterizes the distribution $f_{W}(w;\theta )$ of the random variable $W$. Suppose the first $k$ moments of the true distribution (the "population moments") can be expressed as functions of the $\theta$s:

${\begin{aligned}\mu _{1}&\equiv \operatorname {E} [W]=g_{1}(\theta _{1},\theta _{2},\ldots ,\theta _{k}),\\[4pt]\mu _{2}&\equiv \operatorname {E} [W^{2}]=g_{2}(\theta _{1},\theta _{2},\ldots ,\theta _{k}),\\&\,\,\,\vdots \\\mu _{k}&\equiv \operatorname {E} [W^{k}]=g_{k}(\theta _{1},\theta _{2},\ldots ,\theta _{k}).\end{aligned}}$

Suppose a sample of size $n$ is drawn, resulting in the values $w_{1},\dots ,w_{n}$. For $j=1,\dots ,k$, let

${\widehat {\mu }}_{j}={\frac {1}{n}}\sum _{i=1}^{n}w_{i}^{j}$

be the j-th sample moment, an estimate of $\mu _{j}$. The method of moments estimator for $\theta _{1},\theta _{2},\ldots ,\theta _{k}$ denoted by ${\widehat {\theta }}_{1},{\widehat {\theta }}_{2},\dots ,{\widehat {\theta }}_{k}$ is defined to be the solution (if one exists) to the equations:^[2]

${\begin{aligned}{\widehat {\mu }}_{1}&=g_{1}({\widehat {\theta }}_{1},{\widehat {\theta }}_{2},\ldots ,{\widehat {\theta }}_{k}),\\[4pt]{\widehat {\mu }}_{2}&=g_{2}({\widehat {\theta }}_{1},{\widehat {\theta }}_{2},\ldots ,{\widehat {\theta }}_{k}),\\&\,\,\,\vdots \\{\widehat {\mu }}_{k}&=g_{k}({\widehat {\theta }}_{1},{\widehat {\theta }}_{2},\ldots ,{\widehat {\theta }}_{k}).\end{aligned}}$

The method described here for single random variables generalizes in an obvious manner to multiple random variables leading to multiple choices for moments to be used. Different choices generally lead to different solutions [5], [6].

Advantages and disadvantages

The method of moments is fairly simple and yields consistent estimators (under very weak assumptions), though these estimators are often biased.

It is an alternative to the method of maximum likelihood.

However, in some cases the likelihood equations may be intractable without computers, whereas the method-of-moments estimators can be computed much more quickly and easily. Due to easy computability, method-of-moments estimates may be used as the first approximation to the solutions of the likelihood equations, and successive improved approximations may then be found by the Newton–Raphson method. In this way the method of moments can assist in finding maximum likelihood estimates.

In some cases, infrequent with large samples but less infrequent with small samples, the estimates given by the method of moments are outside of the parameter space (as shown in the example below); it does not make sense to rely on them then. That problem never arises in the method of maximum likelihood^[3] Also, estimates by the method of moments are not necessarily sufficient statistics, i.e., they sometimes fail to take into account all relevant information in the sample.

When estimating other structural parameters (e.g., parameters of a utility function, instead of parameters of a known probability distribution), appropriate probability distributions may not be known, and moment-based estimates may be preferred to maximum likelihood estimation.

Alternative method of moments

The equations to be solved in the method of moments (MoM) are in general nonlinear and there are no generally applicable guarantees that tractable solutions exist. But there is an alternative approach to using sample moments to estimate data model parameters in terms of known dependence of model moments on these parameters, and this alternative requires the solution of only linear equations or, more generally, tensor equations. This alternative is referred to as the Bayesian-Like MoM (BL-MoM), and it differs from the classical MoM in that it uses optimally weighted sample moments. Considering that the MoM is typically motivated by a lack of sufficient knowledge about the data model to determine likelihood functions and associated a posteriori probabilities of unknown or random parameters, it is odd that there exists a type of MoM that is Bayesian-Like. But the particular meaning of Bayesian-Like leads to a problem formulation in which required knowledge of a posteriori probabilities is replaced with required knowledge of only the dependence of model moments on unknown model parameters, which is exactly the knowledge required by the traditional MoM [1],[2],[5]–[9]. The BL-MoM also uses knowledge of a priori probabilities of the parameters to be estimated, when available, but otherwise uses uniform priors.

The BL-MoM has been reported on in only the applied statistics literature in connection with parameter estimation and hypothesis testing using observations of stochastic processes for problems in Information and Communications Theory and, in particular, communications receiver design in the absence of knowledge of likelihood functions or associated a posteriori probabilities [10] and references therein. In addition, the restatement of this receiver design approach for stochastic process models as an alternative to the classical MoM for any type of multivariate data is available in tutorial form at the university website [11, page 11.4]. The applications in [10] and references demonstrate some important characteristics of this alternative to the classical MoM, and a detailed list of relative advantages and disadvantages is given in [11, page 11.4], but the literature is missing direct comparisons in specific applications of the classical MoM and the BL-MoM.

Examples

An example application of the method of moments is to estimate polynomial probability density distributions. In this case, an approximating polynomial of order $N$ is defined on an interval $[a,b]$. The method of moments then yields a system of equations, whose solution involves the inversion of a Hankel matrix.

Proving the central limit theorem

Let $X_{1},X_{2},\cdots$ be independent random variables with mean 0 and variance 1, then let $S_{n}:={\frac {1}{\sqrt {n}}}\sum _{i=1}^{n}X_{i}$. We can compute the moments of $S_{n}$ as$$E[S_{n}^{0}]=1,E[S_{n}^{1}]=0,E[S_{n}^{2}]=1,E[S_{n}^{3}]=0,\cdots$$Explicit expansion shows that$$E[S_{n}^{2k+1}]=0;\quad E[S_{n}^{2k}]={\frac {{\binom {n}{k}}{\frac {(2k)!}{2^{k}}}}{n^{k}}}={\frac {n(n-1)\cdots (n-k+1)}{n^{k}}}(2k-1)!!$$where the numerator is the number of ways to select $k$ distinct pairs of balls by picking one each from $2k$ buckets, each containing balls numbered from $1$ to $n$. At the $n\to \infty$ limit, all moments converge to that of a standard normal distribution. More analysis then show that this convergence in moments imply a convergence in distribution.

Essentially this argument was published by Chebyshev in 1887.

Uniform distribution

Consider the uniform distribution on the interval $[a,b]$, $U(a,b)$. If $W\sim U(a,b)$ then we have

$\mu _{1}=\operatorname {E} [W]={\frac {1}{2}}(a+b)$

$\mu _{2}=\operatorname {E} [W^{2}]={\frac {1}{3}}(a^{2}+ab+b^{2})$

Solving these equations gives

${\widehat {a}}=\mu _{1}-{\sqrt {3\left(\mu _{2}-\mu _{1}^{2}\right)}}$

${\widehat {b}}=\mu _{1}+{\sqrt {3\left(\mu _{2}-\mu _{1}^{2}\right)}}$

Given a set of samples $\{w_{i}\}$ we can use the sample moments ${\widehat {\mu }}_{1}$ and ${\widehat {\mu }}_{2}$ in these formulae in order to estimate $a$ and $b$.

Note, however, that this method can produce inconsistent results in some cases. For example, the set of samples $\{0,0,0,0,1\}$ results in the estimate ${\widehat {a}}={\frac {1}{5}}-{\frac {2{\sqrt {3}}}{5}},{\widehat {b}}={\frac {1}{5}}+{\frac {2{\sqrt {3}}}{5}}$ even though ${\widehat {b}}<1$ and so it is impossible for the set $\{0,0,0,0,1\}$ to have been drawn from $U({\widehat {a}},{\widehat {b}})$ in this case.

See also

• Generalized method of moments
• Decoding methods

References

References needing to be wikified

[4] Pearson, K. (1936), "Method of Moments and Method of Maximum Likelihood", Biometrika 28(1/2), 35–59.

[5] Lindsay, B.G. & Basak P. (1993). “Multivariate normal mixtures: a fast consistent method of moments”, Journal of the American Statistical Association 88, 468–476.

[6] Quandt, R.E. & Ramsey, J.B. (1978). “Estimating mixtures of normal distributions and switching regressions”, Journal of the American Statistical Association 73, 730–752.

[7] https://real-statistics.com/distribution-fitting/method-of-moments/

[8] Hansen, L. (1982). “Large sample properties of generalized method of moments estimators”, Econometrica 50, 1029–1054.

[9] Lindsay, B.G. (1982). “Conditional score functions: some optimality results”, Biometrika 69, 503–512.

[10] Gardner, W.A., “Design of nearest prototype signal classifiers”, IEEE Transactions on Information Theory 27 (3), 368–372,1981

[11] Cyclostationarity