A PhD-level course at EMAp.

To compile the slides, you'll need to do

pdflatex -interaction=nonstopmode --shell-escape bayes_stats

a few times to get it right.

• Probability theory with measure. Jeff Rosenthal's book, A First Look at Rigorous Probability Theory, is a good resource.
• Classical Statistics at the same level as Mathematical Statistics. For a review, I suggest Theory of Statistics by Mark Schervish.

• The Bayesian Choice (BC) by Christian Robert will be our main source.
• A first course in Bayesian statistical methods (FC) by Peter Hoff is a good all-purpose introduction.
• Theory of Statistics (SV) by Mark Schervish is a comprehensive reference.
• Bayesian Theory (BT) by José Bernardo and Adrian Smith is a technical behemoth, suitable for use as a reference guide.

• An overview of computing techniques for Bayesian inference can be found here.
• See Esteves, Stern and Izbicki's course notes.
• Rafael Stern's excellent course.
• Principles of Uncertainty by the inimitable J. Kadane is a book about avoiding being a sure loser. See this review by Christian Robert.
• Bayesian Learning by Mattias Vilani is a book for a computation-minded audience.
• Michael Betancourt's website is a treasure trove of rigorous, modern and insightful applied Bayesian statistics. See this as a gateway drug.
• Awesome Bayes is a curated list of bayesian resources, including blog posts and podcasts.

Guido Moreira suggested topics, exercises and exam questions.

• The LaplacesDemon introductory vignette gives a very nice overview of Bayesian Statistics.
• What is a statistical model? by Peter McCullagh gives a good explanation of what a statistical model is. See also BC Ch1.
• There are a few Interpretations of Probability and its important to understand them so the various schools of statistical inference make sense.
• WHAT IS BAYESIAN/FREQUENTIST INFERENCE? by Larry Wasserman is a must read in order to understand what makes each inference paradigm tick.
• This cross-validated post has a very nice, measure-theoretic proof of Bayes's theorem.

• Berger and Wolpert's 1988 monograph is the definitive text on the Likelihood Principle (LP).
• See this paper By Franklin and Bambirra for a generalised version of the LP.
• As advanced reading, one can consider Birnbaum (1962) and a helpful review paper published 30 years later by Bjornstad.
• Michael Evans has a few papers on the LP. See Evans, Fraser & Monette (1986) for an argument using a stronger version of CP and Evans, 2013 for a flaw with the original 1962 paper by Birnbaum.

• David Alvarez-Melis and Tamara Broderick were kind enough to provide an English translation of De Finetti's seminal 1930 paper.
• Heath and Sudderth (1976) provide a simpler proof of De Finetti's representation theorem for binary variables.

• The SHeffield ELicitation Framework (SHELF) is a package for rigorous elicitation of probability distributions.
• John Cook provides a nice compendium of conjugate priors by Daniel Fink.

Required reading

• Hidden Dangers of Specifying Noninformative Priors is a must-read for those who wish to understand the counter-intuitive nature of prior measures and their push-forwards.
• The Prior Can Often Only Be Understood in the Context of the Likelihood explains that, from a practical perspective, priors can be seen as regularisation devices and should control what the model does rather than what values the parameter takes.
• Penalising Model Component Complexity: A Principled, Practical Approach to Constructing Priors shows how to formalise the idea that one should prefer a simpler model by penalising deviations from a base model.

Optional reading

• The Case for Objective Bayesian Analysis is a good read to try and put objective Bayes on solid footing.

• The paper The Federalist Papers As a Case Study by Mosteller and Wallace (1984) is a very nice example of capture-recapture models. It is cited in Sharon McGrayne's book "The Theory That Would Not Die" as triumph of Bayesian inference. It is also a serious contender for coolest paper abstract ever.
• This post in the Andrew Gelman blog discusses how to deal with the sample size (n) in a Bayesian problem: either write out a full model that specifies a probabilistic model for n or write an approximate prior pi(theta|n).

Required reading

• In their seminal 1995 paper, Robert Kass and Adrian Raftery give a nice overview of, along with recommendations for, Bayes factors.

Optional reading

• This paper by Christian Robert gives a nice discussion of the Jeffreys-Lindley paradox.
• Jaynes's 1976 monograph Confidence Intervals vs Bayesian Intervals is a great source of useful discussion. PDF.

• This paper by Lavine and Schervish provides a nice "disambiguation" for what Bayes factors can and cannot do inferentially.
• Yao et al. (2018) along with ensuing discussion is a must-read for an understanding of modern prediction-based Bayesian model comparison.

• The entry on the Encyclopedia of Mathematics on the Bernstein-von Mises theorem is nicely written.
• The integrated nested Laplace approximation (INLA) methodology leverages Laplace approximations to provide accurate approximations to the posterior in latent Gaussian models, which cover a huge class of models used in applied modelling. This by Thiago G. Martins and others, specially section 2, is a good introduction.

• Ever wondered what to do when both the number of trials and success probability are unknown in a binomial model? Well, this paper by Adrian Raftery has an answer. See also this discussion with JAGS and Stan implementations.
• This case study shows how to create a model from first (physical) principles.

• See Reporting Bayesian Results for a guide on which summaries are indispensable in a Bayesian analysis.
• Visualization in Bayesian workflow is a great paper about making useful graphs for a well-calibrated Bayesian analysis.

Disclaimer: everything in this section needs to be read with care so one does not become a zealot!

• See Jayne's monograph above.
• See Frequentism and Bayesianism: A Practical Introduction for a five-part discussion of the Bayesian vs Orthodox statistics.
• Why isn't everyone a Bayesian? is a nice discussion of the trade-offs between paradigms by Bradley Efron.
• Holes in Bayesian statistics is a collection of holes in Bayesian data analysis, such as conditional probability in the quantum real, flat and weak priors, and model checking, written by Andrew Gelman and Yuling Yao.
• Bayesian Estimation with Informative Priors is Indistinguishable from Data Falsification is paper that attempts to draw a connection between strong priors and data falsification. Not for the faint of heart.