Survey

* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project

Survey

Document related concepts

Data assimilation wikipedia , lookup

Principal component regression wikipedia , lookup

Instrumental variables estimation wikipedia , lookup

Interaction (statistics) wikipedia , lookup

Discrete choice wikipedia , lookup

Lasso (statistics) wikipedia , lookup

Time series wikipedia , lookup

Coefficient of determination wikipedia , lookup

Transcript

Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Logistic Regression James H. Steiger Department of Psychology and Human Development Vanderbilt University Multilevel Regression Modeling, 2009 Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Logistic Regression 1 2 3 4 5 6 7 Introduction The Logistic Regression Model Some Basic Background An Underlying Normal Variable Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Logistic Regression and Retrospective Studies Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Introduction In this lecture we discuss the logistic regression model, generalized linear models, and some applications. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Probability Theory Background Before beginning our discussion of logistic regression, it will help us to recall and have close at hand a couple of fundamental results in probability theory. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable A Binary 0,1 (Bernoulli) Random Variable I Suppose a random variable Y takes on values 1,0 with probabilities p and 1 − p, respectively. Then Y has a mean of E (Y ) = p and a variance of σy2 = p(1 − p) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Proof I Proof. 1 Recall from Psychology 310 that the expected value of a discrete random variable Y is given by E (Y ) = K X yi Pr(yi ) i=1 That is, to compute the expected value, you simply take the sum of cross-products of the outcomes and their probabilities. There is only one nonzero outcome, 1, and it has a probability of p. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Proof II 2 When a variable Y takes on only the values 0 and 1, then Y = Y 2 . So E (Y ) = E (Y 2 ). But one formula for the variance of a random variable is σy2 = E (Y 2 ) − (E (Y ))2 , which is equal in this case to σy2 = p − p 2 = p(1 − p) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Conditional Distributions in the Bivariate Normal Case If two variables W and X are bivariate normal with regression line Ŵ = β1 X + β0 , and correlation ρ, the conditional distribution of W p given X = a has mean β1 a + β0 and standard deviation σ = 1 − ρ2 σw . If we assume X and W are in standard score form, then the conditional mean is µw |x =a = ρa and the conditional standard deviation is p σ = 1 − ρ2 Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable An Underlying Normal Variable It is easy to imagine a continuous normal random variable W underlying a discrete observed Bernoulli random variable Y . Life is full of situations where an underlying continuum is scored “pass-fail.” Let’s examine the statistics of this situation. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable An Underlying Normal Variable As a simple example, imagine that: 1 2 3 The distribution of scores on variable W has a standard deviation of 1, but varies in its mean depending on some other circumstance There is a cutoff score Xc , and that to succeed, an individual needs to exceed that cutoff score. That cutoff score is +1. What percentage of people will succeed if µw = 0? Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable An Underlying Normal Variable Here is the picture: What percentage of people will succeed? An Underlying Normal Variable −3 −2 −1 0 Xc 1 2 3 W Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable An Underlying Normal Variable Suppose we wished to plot the probability of success as a function of µw , the mean of the underlying variable. Assuming that σ stays constant at 1, and that Wc stays constant at +1, can you give me an R expression to compute the probability of success as a function of µw ? (C.P.) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Plotting the Probability of Success The plot will look like this: 0.6 0.4 0.2 0.0 Pr(Success) 0.8 1.0 > curve (1 -pnorm (1 ,x ,1) , -2 ,3 , + xlab = expression ( mu [ w ]) , ylab = " Pr ( Success ) " ) −2 −1 0 1 2 3 µw Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Plotting the Probability of Success Note that the plot is non-linear. Linear regression will not work well as a model for the variables plotted here. In fact, a linear regression line will, in general, predict probabilities less than 0 and greater than 1! Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Plotting the Probability of Success We can generalize the function we used to plot the previous figure for the general case where Wc is any value, and µw and σw are also free to vary. Pr.Success ← f u n c t i o n ( mu_w , sigma_w , cutoff ) {1 -pnorm ( cutoff , mu_w , sigma_w )} curve ( Pr.Success (x ,2 ,1) , -3 ,5 , xlab = expression ( mu [ w ]) , ylab = " Pr ( Success ) when the cutoff is 2 " ) 0.8 0.6 0.4 0.2 0.0 Pr(Success) when the cutoff is 2 1.0 > + > + + −2 0 2 4 µw Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Extending to the Bivariate Case Suppose that we have a continuous predictor X , and a binary outcome variable Y that in fact has an underlying normal variable W generating it through a threshold values Wc . Assume that X and W have a bivariate normal distribution, are in standard score form, and have a correlation of ρ. We wish to plot the probability of success as a function of X , the predictor variable. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Predicting Pr(Success) from X We have everything we need to solve the problem. We can write π(x ) = Pr(Y = 1|X = x ) = Pr(W > Wc |X = x ) Wc − µW |X =x = 1−Φ σW |X =x ! Wc − ρx = 1−Φ p 1 − ρ2 ! ρx − Wc = Φ p 1 − ρ2 Multilevel Logistic Regression (1) (2) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Predicting Pr(Success) from X Note that the previous equation can be written in the form π(x ) = Φ(β1 x + β0 ) (3) Not only is the regression line nonlinear, but the variable Y is a Bernoulli variable with a mean that changes as a function of x , and so its variance also varies as a function of x , thus violating the equal variances assumption. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Basic Background An Underlying Normal Variable Predicting Pr(Success) from X However, since Φ( ) is invertible, we can write Φ−1 (Pr(Y = 1|X = x )) = Φ−1 (µY |X =x ) = β1 x + β0 = β0x This is known as a probit model, but it is also our first example of a Generalized Linear Model, or GLM. A GLM is a linear model for a transformed mean of a variable that has a distribution in the natural exponential family. Since x might contain several predictors and very little would change, the extension to multiple predictors is immediate. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Binary Logistic Regression Suppose we simply assume that the response variable has a binary distribution, with probabilities π and 1 − π for 1 and 0, respectively. Then the probability density can be written in the form f (y) = π y (1 − π)1−y y π = (1 − π) 1−π = (1 − π) exp y log Multilevel π 1−π Logistic Regression (4) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Binary Logistic Regression The logit of Y is the logarithm of the odds that Y = 1. Suppose we believe we can model the logit as a linear function of X , specifically, Pr(Y = 1|X = x ) 1 − Pr(Y = 1|X = x ) = β1 x + β0 logit(π(x )) = log Multilevel Logistic Regression (5) (6) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Binary Logistic Regression The logit function is invertible, and exponentiating both sides, we get π(x ) = Pr(Y = 1|x ) exp(β1 x + β0 ) = 1 + exp(β1 x + β0 ) 1 = 1 + exp(−(β1 x + β0 )) 1 = 1 + exp(−β 0 x ) = µY |X =x (7) Once again, we find that a transformed conditional mean of the response variable is a linear function of X . Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Extension to Several Predictors Note that we wrote β1 x + β0 as β 0 x in the preceding equation. Since X could contain one or several predictors, the extension to the multivariate case is immediate. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Binomial Logistic Regression In binomial logistic regression, instead of predicting the Bernoulli outcomes on a set of cases as a function of their X values, we predict a sequence of binomial proportions on I occasions as a function of the X values for each occasion. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Binomial Logistic Regression The mathematics changes very little. The i th occasion has a probability of success π(xi ), which now gives rise to a sample proportion Y based on mi observations, via the binomial distribution. The model is π(xi ) = µY |X =xi = Multilevel 1 1 + exp −β 0 xi Logistic Regression (8) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Parameters How would we interpret the estimates of the model parameters in simple binary logistic regression? Exponentiating both sides of Equation 5 shows that the odds are an exponential function of x. The odds increase multiplicatively by exp(β1 ) for every unit increase in x . So, for example, if β1 = .5, the odds are multiplied by 1.64 for every unit increase in x . Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Characteristics of Logistic Regression 0.6 0.4 0.2 0.0 logit−1(x) 0.8 1.0 Logistic regression predicts the probability of a positive response, given values on one or more predictors The plot of y = logit−1 (x ) is shaped very much like the normal distribution cdf It is S-shaped, and you can see that the slope of the curve is steepest at the midway point, and that the curve is quite linear in this region, but very nonlinear in its outer range −6 −4 −2 0 2 4 6 x Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Parameters If we take the derivative of π(x ) with respect to x , we find that it is equal to βπ(x )(1 − π(x )). This in turn implies that the steepest slope is at π(x ) = 1/2, at which x = −β0 /β1 , and the slope is β1 /4. In toxicology, this is called LD50 , because it is the dose at which the probability of death is 1/2. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coefficients 1 2 3 4 Because of the nonlinearity of logit−1 , regression coefficients do not correspond to a fixed change in probability In the center of its range, the logit−1 function is close to linear, with a slope equal to β/4 Consequently, when X is near its mean, a unit change in X corresponds to approximately a β/4 change in probability In regions further from the center of the range, one can employ R in several ways to calculate the meaning of the slope Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coefficients An Example Example (Interpreting a Logistic Regression Coefficient) Gelman and Hill (p. 81) discuss an example where the fitted logistic regression is Pr(Bush Support) = logit−1 (.33 Income − 1.40) Here is their figure 5.1a. ● ● ● ●● 1.0 ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.6 0.4 0.2 ● ● ● ● ● ● ● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● Income ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ●● ● ● ● ● ● ● ● ● ● 4 ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●●● ● ●● ● ● ● ●● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ●●● ● ● ● ● ● ● ● ●● ● ● ● ● 3 ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●●● ● ● ●● ● ● ● ●● ● ● ● ●●● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● 2 ● ● ● ● ● ● ● ● ● ● ●●● ● ● ●● ● ● ● ● ● ●●● ●● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1 (poor) ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 0.0 Pr (Republican vote) 0.8 ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ●● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ●● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ●● ● ● ● ● ● ● ● ● ● 5 (rich) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coeffients An Example Example (Interpreting a Logistic Regression Coefficient) The mean value of income is > mean( income , na.rm = TRUE ) [1] 3.075488 Around the value X = .31, the probability is increasing at a rate of approximately β/4 = .33/4 = .0825. So we can estimate that on average the probability that a person with income level 4 will support Bush is about 8% higher than the probability that a person with income level 3 will support Bush. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coeffients An Example Example (Interpreting a Logistic Regression Coefficient) We can also employ the inverse logit function to obtain a more refined estimate. If I fit the logistic model, and save the fit in a fit.1 object, I can perform the calculations on the full precision coefficients using the invlogit() function, as follows > i n v l o g i t ( c o e f ( fit.1 )[1] + c o e f ( fit.1 )[2] * 3) (Intercept) 0.3955251 > i n v l o g i t ( c o e f ( fit.1 )[1] + c o e f ( fit.1 )[2] * 4) (Intercept) 0.4754819 > i n v l o g i t ( c o e f ( fit.1 )[1] + c o e f ( fit.1 )[2] * 4) + i n v l o g i t ( c o e f ( fit.1 )[1] + c o e f ( fit.1 )[2] * 3) (Intercept) 0.07995678 Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coefficients The Odds Scale Interpreting Logistic Regression Coefficients We can also interpret a logistic regression coefficient in terms of odds Since the coefficient β is linear in the log odds, e β functions multiplicatively on odds That is, around the mean value of 3.1, a unit increase in income should correspond to an e .326 increase in odds Let’s check out how that works by doing the calculations Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Interpreting Logistic Regression Coefficients Example (Interpreting Logistic Regression Coefficients) We saw in the preceding example that, at a mean income of 3, the predicted probability of supporting Bush is 0.3955251, which is an odds value of > odds.3 = .3955251 /(1 -.3955251 ) > odds.3 [1] 0.6543284 At an income level of 4, the predicted probability of supporting Bush is 0.4754819, which is an odds value of > odds.4 = 0 .4754819 /(1 -0.4754819 ) > odds.4 [1] 0.9065119 The ratio of the odds is the same as e β . > odds.4 / odds.3 [1] 1.385408 > exp( .3259947 ) [1] 1.385408 Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Crabs and Their Satellites Agresti (2002, p. 126) introduces an example based on a study in Ethology Each female horseshoe crab has a male crab resident in her nest The study investigated factors associated with whether the fameale crab had any other males, called satellites, residing nearby Potential predictors include the female’s color, spine condition, weight, and carapace width Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Predicting a Satellite The crab data has information on the number of satellites Suppose we reduce these data to binary form, i.e., Y = 1 if the female has a satellite, and Y = 0 if she does not Suppose further that we use logistic regression to form a model predicting Y from a single predictor X , carapace width Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Entering the Data Entering the Data The raw data are in a text file called Crab.txt. We can read them in and attach them using the command > crab.data ← r e a d . t a b l e ( " Crab.txt " , header = TRUE ) > attach ( crab.data ) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Setting Up the Data Next, we create a binary variable corresponding to whether or not the female has at least one satellite. > has.satellite ← i f e l s e ( Sa > 0 ,1 ,0) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Fitting the Model with R We now fit the logistic model using R’s GLM module, then display the results > fit.logit ← glm( has.satellite ˜ W , + family = binomial ) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Fitting the Model with R (Intercept) W Estimate −12.3508 0.4972 Std. Error 2.6287 0.1017 Multilevel z value −4.70 4.89 Logistic Regression Pr(>|z|) 0.0000 0.0000 Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Interpreting the Results Interpreting the Results Note that the slope parameter b1 = 0.4972 is significant From our β/4 rule, this indicates that 1 additional unit of carapace width around the mean value of the latter will increase the probability of a satellite by about 0.4972/4 = 0.1243 Alternatively, one additional unit of carapace width is associated with a log-odds multiple of e 0.4972 = 1.6441 This corresponds to a 64.41% increase in the odds Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Interpreting the Results Here is a plot of predicted probability of a satellite vs. width of the carapace. 0.6 0.4 0.2 Pr(Has.Satellite) 0.8 1.0 > curve ( i n v l o g i t ( b1 * x + b0 ) , 20 ,35 , xlab = " Width " , ylab = " Pr ( Has.Satel 20 25 30 35 Width Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Including Color as Predictor Dichotomizing Color The crab data also include data on color, and use it as an additional (categorical) predictor In this example, we shall dichotomize this variable, scoring crabs who are dark 0, those that are not dark 1 with the following command: > is.not.dark ← i f e l s e (C == 5 ,0 ,1) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Specifying the Model(s) The Additive Two-Variable Model The additive model states that logit(pi ) = b0 + b1 W + b2 C Let’s fit the original model that includes only width(W), then fit the model with width(W) and the dichotomized color(is.not.dark) > > + > + fit.null ← glm( has.satellite ˜ 1 , family = binomial ) fit.W ← glm( has.satellite ˜ W , family = binomial ) fit.WC ← glm( has.satellite ˜ W + is.not.dark , family = binomial ) Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Results Results for the null model: (Intercept) Estimate 0.5824 Std. Error 0.1585 z value 3.67 Pr(>|z|) 0.0002 Results for the simple model with only W: (Intercept) W Estimate −12.3508 0.4972 Std. Error 2.6287 0.1017 z value −4.70 4.89 Pr(>|z|) 0.0000 0.0000 Results for the additive model with W and C: (Intercept) W is.not.dark Estimate −12.9795 0.4782 1.3005 Std. Error 2.7272 0.1041 0.5259 Multilevel z value −4.76 4.59 2.47 Pr(>|z|) 0.0000 0.0000 0.0134 Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Comparing Models We can compare models with the anova() function > anova( fit.null , fit.W , fit.WC , test = " Chisq " ) 1 2 3 Resid. Df 172 171 170 Resid. Dev 225.76 194.45 187.96 Multilevel Df Deviance P(>|Chi|) 1 1 31.31 6.49 0.0000 0.0108 Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Plotting the W + C Model b0 ← c o e f ( fit.WC )[1] b1 ← c o e f ( fit.WC )[2] b2 ← c o e f ( fit.WC )[3] curve ( i n v l o g i t ( b1 * x + b0 + b2 ) , 20 ,35 , xlab = " Width " , ylab = " Pr ( Has.Satellite ) " , c o l = " red " ) curve ( i n v l o g i t ( b1 * x + b0 ) , 20 ,35 , lty =2 , c o l = " blue " ,add= TRUE ) legend (21 ,0 .9 , legend = c ( " light crabs " ," dark crabs " ) , lty = c (1 ,2) , c o l = c ( " red " ," blue " )) 1.0 > > > > + > > + 0.6 0.4 0.2 Pr(Has.Satellite) 0.8 light crabs dark crabs 20 25 30 35 Width Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Crab Data Example The Multivariate Crab Data Example A Crabby Interaction Specifying the Model(s) The additive model states that logit(pi ) = b0 + b1 W + b2 C Let’s add an interaction effect. > fit.WCi ← glm( has.satellite ˜ W + is.not.dark + + W : is.not.dark , + family = binomial ) The result is not significant. (Intercept) W is.not.dark W:is.not.dark Estimate −5.8538 0.2004 −6.9578 0.3217 Std. Error 6.6939 0.2617 7.3182 0.2857 Multilevel z value −0.87 0.77 −0.95 1.13 Pr(>|z|) 0.3818 0.4437 0.3417 0.2600 Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies An Important Application — Case Control Studies An important application of logistic regression is the case control study, in which people are sampled from “case” and “control” categories and then analyzed (often through their recollections) for their status on potential predictors. For example, samples of patients with or without lung cancer can be sampled, then asked about their smoking behavior. Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Relative Risk With binary outcomes, there are several kinds of effects we can assess. Two of the most important are relative risk and the odds ratio. Consider a situation where middle aged men either smoke (X = 1) or do not (X = 0) and either get lung cancer (Y = 1) or do not (Y = 0). Often the effect we would like to estimate in epidemiological studies is the relative risk, Pr(Y = 1|X = 1) Pr(Y = 1|X = 0) Multilevel Logistic Regression (9) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Retrospective Studies In retrospective studies we ask people in various criterion groups to “look back” and indicate whether or not they engaged in various behaviors. For example, we can take a sample of lung cancer patients and ask them if they ever smoked, then take a matched sample of patients without lung cancer and ask them if they smoked. After gathering the data, we would then have estimates of Pr(X = 1|Y = 1), Pr(X = 0|Y = 1) Pr(X = 1|Y = 0),and Pr(X = 1|Y = 0). Notice that these are not the conditional probabilities we need to estimate relative risk! Multilevel Logistic Regression Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies The Odds Ratio An alternative way of expressing the impact of smoking is the odds ratio, the ratio of the odds of cancer for smokers and nonsmokers. This is given by Pr(Y = 1|X = 1)/1 − Pr(Y = 1|X = 1) Pr(Y = 1|X = 0)/1 − Pr(Y = 1|X = 0) Multilevel Logistic Regression (10) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Key Identities By repeatedly employing 1 2 The definition of conditional probability, i.e., Pr(A|B ) = Pr(A ∩ B ) Pr(B ) = Pr(B ∩ A) Pr(B ) The fact that A ∩ B = B ∩ A it is easy to show that Pr(Y = 1|X = 1)/(1 − Pr(Y = 1|X = 1)) Pr(Y = 1|X = 0)/(1 − Pr(Y = 1|X = 0)) Pr(X = 1|Y = 1)/(1 − Pr(X = 1|Y = 1)) = Pr(X = 1|Y = 0)/(1 − Pr(X = 1|Y = 0)) Multilevel Logistic Regression (11) Introduction The Logistic Regression Model Binary Logistic Regression Binomial Logistic Regression Interpreting Logistic Regression Parameters Examples Logistic Regression and Retrospective Studies Some Key Identities Equation 11 demonstrates that the information about odds ratios is available in retrospective studies with representative sampling. Furthermore, suppose that an outcome variable Y fits a logistic regression model logit(Y ) = β1 X + β0 . As Agresti (2002, p. 170–171) demonstrates, it is possible to correctly estimate β1 in a retrospective case-control study where Y is fixed and X is random. The resulting fit will have a modified intercept β0∗ = log(p1 /p0 ) + β0 , where p1 and p0 are the respective sampling probabilities for Y = 1 cases and Y = 0 controls. Multilevel Logistic Regression