Logistic Regression - Statistics Assignment - Canada

Statistics Assignment Help

Question: In logistic regression, our goal is to learn a set of parameters ?T =(?0,?1,...,?k) by maximizing the conditional log likelihood of the training data. (a) [5 Points] Assuming you are given a dataset with n training examples and k features, write down a formula for the conditional log likelihood l(?) of the training data in terms of the class labels yi , the features X 1 (i), . . . , X k (i), and the parameters ?0, ?1, . . . , ?k, where the superscript (i) denotes the sample index. This will be your objective function (or cost function) for gradient ascent. (b) [5 Points] Compute the partial derivative of the objective function with respect to ?j , i.e. derive ?l/??j , for 0 ? j ? k where l is the objective function that you provided above. (c) [10 Points] Show particularly that (d) [5 Points] To maximize the log-likelihood, we set the partial derivatives (1) to zero. Show that when j = 0, the score equation is equivalent to Does the equation have any statistical interpretation? (e) [25 Points] Implement a logistic regression classifier using gradient ascent (see algo- rithm given in the slides) in the simple case where there is only one feature variable x1. For this part of the assignment, you have to submit R code where the following procedures are clearly identified: • Normalize the feature variable x1 (see item (f)). • Calculate the value of the objective function l(?0,?1). • Choose a value of the learning rate ? (you should try different values). • Initialize the parameter value and calculate the gradient ?l(?0,?1). • Update the parameter value. • Check whether gradient ascent has converged. Here, it is much better to look at the convergence of the values of l(?0,?1) than the convergence of the parameters themselves. • Complete the implementation of gradient ascent. • Predict the labels for a set of test examples. I am including the data set SAheart.data in which you should predict the value of the variable chd (response, coronary heart disease diagnosis) from the feature value ldl (low density lipoprotein cholesterol). You may use the first 100 rows for training and any values in the remaining rows for testing. (f) In machine learning, it is a standard routine to normalize or scale the feature variables to speed up the convergence of the learning algorithms and to ensure that the features contribute equally to the learning task. One way to achieve the normalization is by making the values of each feature in the data have zero mean (when subtracting the mean in the numerator) and unit variance, i.e., we replace the variable x1 with x1?? / ? , where ? and ? are respectively the mean and the standard deviation of the values of x1 in the training data. 2. This problem is a follow up on problem 1 of assignment 5 about the regression through the origin model of the form Y =?1x+?, where ? is the standard normal distribution. (a) First, generate a predictor x as a random vector of length n = 100 from the standard normal and a response y as follows. y = 2x + ? where ? is also a random vector of length n = 100 from the standard normal that takes exactly the same values as x. (b) Perform a simple linear regression of y onto x, without an intercept. Report the coefficient estimate ?ˆ 1, the standard error of this coefficient estimate, and the t-statistic and p-value associated with the null hypothesis H0 : ?1 = 0. Comment on these results. (You can perform regression without an intercept using the command lm(y? x + 0).) (c) Now perform a simple linear regression of x onto y without an intercept. Report the coefficient estimate ?ˆ 1, the standard error of this coefficient estimate, and the t-statistic and p-value associated with the null hypothesis H0 : ?1 = 0. Comment on these results. (d) What are the relationships between (a), the results obtained in (b) and the ones in (c)? (e) For the regression of Y onto X without an intercept, the t-statistic for H0 : (f) Show algebraically (to receive bonus points), and confirm numerically in R, that the t-statistic above can be written as (values) (g) Using the results from (d), argue that the t-statistic for the regression of y onto x is the same as the t-statistic for the regression of x onto y. (h) In R, show that when regression is performed with an intercept, the t- statistic forH0 :?1 =0 is the same for the regression of y onto x as it is for the regression of x onto y. 4. The data set low bwt.txt contains information for a sample of 100 low birth weight infants. The variables are sbp : maternal systolic blood pressure sex : gender of the baby toxemia : toxemia during pregnancy (yes or no) germ.hem : germinal matrix hemorrhage (yes or no) gest.age : gestational age in weeks apgar5 : five-minute APGAR score (a) Using germinal matrix hemorrhage as the response, fit a logistic regression model where the predictor variable x1 is the 5-minute APGAR score. Write the equation and interpret ?1, the estimated coefficient of Apgar score. (b) What is the estimate and 95% confidence interval for the slope (coefficient for apgar5) in the odds ratio scale? Interpret the estimate (what does the odds ratio mean?). (c) At the 0.05 level of significance, test the null hypothesis: H0 : ?1 = 0 where ?1 is the coefficient for apgar5. (d) If a new infant from the population has an APGAR score of 3 what is the predicted probability that this child will experience a brain hemorrhage? What is the probability if the child’s score is 7?