Flexible Bayesian Regression Modelling

By David Nott

Flexible Bayesian Regression Modeling is a step-by-step guide to the Bayesian revolution in regression modeling, for use in advanced econometric and statistical analysis where datasets are characterized by complexity, multiplicity, and large sample sizes, necessitating the need for considerable flexibility in modeling techniques. It reviews three forms of flexibility: methods which provide flexibility in their error distribution; methods which model non-central parts of the distribution (such as quantile regression); and finally models that allow the mean function to be flexible (such as spline models). Each chapter discusses the key aspects of fitting a regression model. R programs accompany the methods.

This book is particularly relevant to non-specialist practitioners with intermediate mathematical training seeking to apply Bayesian approaches in economics, biology, finance, engineering and medicine.

• Introduces powerful new nonparametric Bayesian regression techniques to classically trained practitioners
• Focuses on approaches offering both superior power and methodological flexibility
• Supplemented with instructive and relevant R programs within the text
• Covers linear regression, nonlinear regression and quantile regression techniques
• Provides diverse disciplinary case studies for correlation and optimization problems drawn from Bayesian analysis ‘in the wild’

• Language: English
• Publisher: Academic Press
• Release date: Oct 30, 2019
• ISBN: 9780128158630

Book preview

States

Preface

Yanan Fan     

David Nott     

Michael S. Smith     

Jean-Luc Dortet-Bernadet     

The 2000s and 2010s have seen huge growth in Bayesian modelling, which now finds application in fields as diverse as engineering, law, medicine, psychology, astronomy, climate science and philosophy. Much of the increase in popularity is due to advances in Bayesian computation, most notably Markov chain Monte Carlo methods. The availability of general and easily applicable simulation-based computational algorithms has made it easier to build more realistic models, involving greater complexity and high dimensionality.

Introductory textbook accounts of Bayesian regression inference often focus on rather inflexible parametric models. When planning this book, we wanted to bring together, in a single volume, a discussion of Bayesian regression methods allowing three types of flexibility: flexibility in the response location, flexibility in the response-covariate relationship, and flexibility in the error distributions. The aim is to produce a collection of works accessible to practitioners, while at the same time detailed enough for interested researchers in Bayesian methods. Software implementing the methods in the book is also available.

Chapters 1 and 2 cover quantile regression. These are methods where inferential interest may lie away from the mean, in noncentral parts of the distribution. Quantile methods do not specify an error model, and are therefore challenging to implement in the Bayesian setting. Chapters 3 and 4 cover regression using Dirichlet process (DP) mixtures to flexibly capture the unknown error distribution. In Chapter 3, DP mixtures are considered in an ordinal regression setting, where the relationship between the covariates and response is modelled flexibly via density regression. In Chapter 4, DP mixtures are used for time series. Chapter 5 extends to regression with multivariate response, using the copula approach to handle mixed binary-continuous responses. Chapters 6 and 7 cover scalable Bayesian modelling using variational Bayesian inference: in Chapter 6, variational inference is described in detail for various spline-based models to flexibly model the covariate-response relationship in the mean. Chapter 7 develops a variational algorithm for count response data, in the presence of variable selection. Finally, Chapters 8 and 9 showcase some of the flexibility of the Bayesian methods when models incorporate shape constraints. The chapters of the book often deal with quite specialised and complex models and data types, but some general themes emerge from the discussion. The reader will obtain an understanding of the basic modelling and computational building blocks which are fundamental to successful new applications of modern and flexible Bayesian regression methods.

Each of the chapters is written in an easy to follow, tutorial style, with the aim to encourage practitioners to take advantage of powerful Bayesian regression methodology. Computer codes are available for each chapter at the website https://www.elsevier.com/books/flexible-bayesian-regression-modelling/fan/978-0-12-815862-3

Wherever appropriate, the chapters contain instructions on how to use the codes.

We are proud to be able to bring together a book containing the latest developments in flexible Bayesian methods. We warmly thank all the contributors to this project.

Chapter 1

Bayesian quantile regression with the asymmetric Laplace distribution

J.-L. Dortet-Bernadeta; Y. Fanb; T. Rodriguesc    aInstitut de Recherche Mathématique Avancée, UMR 7501 CNRS, Université de Strasbourg, Strasbourg, France

bSchool of Mathematics and Statistics, University of New South Wales, Sydney, NSW, Australia

cDepartamento de Estatistica, Universidade de Brasilia, Brasília, Brazil

Abstract

We give in this chapter an overview of the use of the asymmetric Laplace distribution for Bayesian quantile regression. We describe linear and nonlinear models in this context and provide R code and instructions for their use. We give several examples, including additive models, and present some postprocessing procedures able to correct known potential limitations of the asymmetric Laplace approach.

Keywords

quantile regression; asymmetric Laplace distribution; postprocess; additive models

Chapter Outline

1.1  Introduction

1.2  The asymmetric Laplace distribution for quantile regression

1.2.1  A simple and efficient sampler

1.2.2  Quantile curve fitting

1.2.3  Additive models

1.3  On coverage probabilities

1.4  Postprocessing for multiple fittings

1.5  Final remarks and conclusion

References

1.1 Introduction

Following the seminal work by Koenker and Bassett [15] quantile regression has been recognised in recent years as a robust statistical procedure that offers a powerful alternative to ordinary mean regression. This type of regression has proven its interest and its effectiveness in many fields where the data contain large outliers or when the response variable has a skewed or multimodal conditional distribution. It has been also successfully applied to regression problems where the interest lies in the noncentral parts of the response distribution, often found in the environmental sciences, medicine, engineering and economics.

Let τ. Let X be a d. The linear τth quantile regression model specifies the conditional distribution of a real response variable Y as

(1.1)

, and for a noise variable ϵ whose τ. Equivalently, we can write the τth quantile of the conditional distribution of Y . In the case of a single real covariate x, this linear model (1.1) encompasses the model of the τth quantile regression curve

(1.2)

, so that

(1.3)

, represent the locations of K knot points (see Hastie and Tibshirani [12]). Typically, the degree P is set to 3 here since using cubic splines gives a curve that looks sufficiently smooth to the human eye.

be n . If the distribution of the noise variable ϵ in the case of model (1.2), is typically carried out by solving the minimisation problem

(1.4)

otherwise (see Koenker and Bassett replaces the traditional quadratic loss used for mean regression. In this frequentist semiparametric setting, test procedures are usually based on asymptotic arguments or resampling techniques; see Koenker [14] for details and properties of the approach. Bayesian treatment of quantile regression has long appeared as a challenging task, mainly owing to the need to specify a likelihood. In the 2000s and 2010s, the asymmetric Laplace (AL) error model has emerged as a popular solution to this problem, largely due to its flexibility and simplicity, and the fact that the corresponding maximum likelihood estimate is the solution of the minimisation problem (1.4).

In this chapter, we give an overview of the use of the AL distribution for Bayesian quantile regression. More precisely, we start by briefly presenting this distribution in Section 1.2, and we describe the estimation of the regression parameters with the help of a simple Gibbs sampler, as proposed in Kozumi and Kobayashi [17] or Reed and Yu [23]. Then we focus in some more detail on the quantile curve fitting problem and describe a possible extension of the sampler that allows random knots and knot selection. We illustrate all these points on several examples by using R functions that are publicly available.

In the following sections we discuss two potential problems that arise with the use of the AL error model. Firstly we present the problem of the coverage probabilities that has been tackled recently, for example in Yang et al. [36]. Secondly, since the quantile curves corresponding to several τ levels are fitted separately, they may cross, violating the definition of quantiles. We describe how this problem can be overcome using a simple postprocessing procedure. Note that we do not consider here the use of a likelihood that is capable of simultaneously fitting several quantile curves; this approach is covered by Chapter 2 of this book. Finally we conclude with a short discussion.

1.2 The asymmetric Laplace distribution for quantile regression

The centred AL distribution with scale parameter σ, has density

(1.5)

is the check-function used in the minimisation problem (1.4). Clearly, for any σ, if we use this AL distribution to model the error ϵ via a random walk Metropolis–Hastings algorithm. They noticed that, on simulations, the resulting estimation is satisfactory even when the data do not arise from the AL distribution. The good behaviour of this Bayes estimate is studied more theoretically in Sriram et al. [31], who established posterior consistency for the linear quantile regression estimates and gave the rate of convergence in the case of AL distribution misspecification.

1.2.1 A simple and efficient sampler

A desirable feature of the AL distribution is that it can be decomposed as a scale mixture of Normals (Kotz et al. [16]). Let

(1.6)

where V and U . Then ϵ denotes the normal distribution with mean μ . Based on this representation, Kozumi and Kobayashi [17] or Reed and Yu [23] proposed a simple and efficient Gibbs sampler.

is such that

is generalised inverse Gaussian and the full conditional distribution of σ is inverse gamma. See Kozumi and Kobayashi [17] for more details on these conditional distributions and the corresponding Gibbs sampler. They also provide an extension of the work using double-exponential priors on the regression parameters and to the analysis of Tobit quantile regression. Other extensions of the sampler include quantile binary regression (Benoit and Van den Poel [5]), ordinal quantile regression (Rahman [21] or Alhamzawi [1]) and lasso quantile regression (Alhamzawi et al. [4] or Li et al. [18]). Note also that Tsionas [33] gives an alternative version of the Gibbs sampler, less appealing in practice since each component of the regression vector is updated separately.

The Gibbs sampler described in Kozumi and Kobayashi preschool children and has been used to search for reference ranges to help diagnose immunodeficiency in infants (Isaacs et al. [13]). More precisely, we consider IgG as the response variable and the age of the children as the predictor. Following previous studies, a quadratic model is used to fit the data due to the expected smooth change of IgG with age. If the data are in the file ‘igg.dta’, we can proceed as follows to fit the quadratic model for each decile, using chains of length 2000 and a burn-in period of 500, fitting nine equally spaced quantiles from 0.1 to 0.9:

.

The raw Markov chain Monte Carlo (MCMC) samples are contained in the output of bayesQR, and typing

, with the following output in R:

These MCMC draws, for instance, are useful for checking MCMC convergence, choosing burn-in period and calculating credible bounds for other quantities of interest. To plot the data and the nine estimated quantile curves we can use

where the resulting plot is given in Fig. 1.1.

Figure 1.1 Growth chart of serum concentration of immunoglobulin-G for young children. Quantile regression using a quadratic model and bayesQR for τ  = 0.1,…,0.9.

On the whole, for these intermediate τ levels, the quantile lines resulting from this fitted quadratic model appear satisfactory. Nevertheless we will see later that some problems arise when we fit the quantile lines at many levels and at more extreme τ levels.

1.2.2 Quantile curve fitting

As noted in the introduction, the quantile regression curve model (1.2) with spline functions (1.3) can be handled under the framework of linear quantile regression. It is thus tempting to use in this setting the AL error model and to fit the quantile regression curve with the help of the previous Gibbs sampler. Nevertheless, in the curve fitting setting, one also has to consider the appropriate specifications of parameters such as the number and position of the knots.

Chen and Yu [8] provide a Bayesian inference on this model, where the number of knots and their location are automatically selected. Their method relies on a reversible jump Markov chain Monte Carlo (RJMCMC) algorithm which, under the prior specifications they use, needs to compute an approximation of the ratio of marginal likelihoods. For a Bayesian inference using natural cubic splines see Thompson et al. [32]. We present here an alternative method for quantile curve fitting that allows for both random knots and knot selection with a strategy that avoids the use of RJMCMC.

Under the representation (1.3), fitting the curve consists of estimating the number of knots K, represent the corresponding knots. The quantile regression curve model (1.2) can be written as the linear model

(1.7)

is defined by

(1.8)

denotes the unit vector of size n. Following Fan et al. , such that

. In practice, such intervals can be defined by either using prior information on regions where a knot is suspected or, in the absence of such prior information, an equal partition of the range may be adopted.

We denote by γ and by z . We consider as prior distribution on γ . Each possible value for γ gives a model of the form corresponding to nonzero entries in z, if W , then the conditional distribution of Y given W is multivariate normal

(1.9)

Conditionally on W, we use the following decomposition of the joint prior distribution of the unknown