---
title: "Hwk #2: Classification methods and Penalization"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)

library(MASS)
```

For this homework we will use NHANES data that exists in a package for R.

NHANES consists of survey data collected by the US National Center for Health Statistics (NCHS) which has conducted a series of health and nutrition surveys since the early 1960's. Since 1999 approximately 5,000 individuals of all ages are interviewed in their homes every year and complete the health examination component of the survey. The health examination is conducted in a mobile examination center (MEC).

Note that there is the following warning on the NHANES website:
“For NHANES datasets, the use of sampling weights and sample design variables is recommended for all analyses because the sample design is a clustered design and incorporates differential probabilities of selection. If you fail to account for the sampling parameters, you may obtain biased estimates and overstate significance levels.”

For this homework, please ignore this warning and just apply our analyses to the data as if they were randomly sampled! We will be using the data called `NHANESraw`.

For questions that ask for your comments, it suffices to answer with one or two sentences in each case.

## Data Preparation

1. Install the package `NHANES` into R, load the `NHANES` package, and then run the command `data(NHANES)` which will load the NHANES data. Type `?NHANES` and read about the dataset.
```{r}
```

2. Make an object `nhanes` that is a subset version `NHANESraw` that does not include any missing data for `Diabetes`, `BPSysAve`, `BPDiaAve`, or `Age`.
```{r}
```

3. (1 point) Further subset the data such the observations with `BPDiaAve` equal to zero are removed.
```{r}
```

4. (1 point)
Make an object `nhanes09` that is a subset of `nhanes` to only the 2009_10 data. This will be your training dataset. Also make an object `nhanes11` that is a subset of `nhanes` to only the 2011_12 data. This will be your test dataset.
```{r}
```

## Logistic regression

5. (2 point) Fit a logistic regression model (call it `glm1`) using the `nhanes09` dataset. Use `Diabetes` as the outcome and averaged systolic blood pressure (`BPSysAve`) as a single predictor. Use the summary command to examine the fitted model. Generate the 95% confidence intervals for the `BPSysAve` coefficient.
```{r}
```

6. (1 point) Generate the estimate and 95% confidence interval for the odds-ratio associated with BPSysAve. Summarize the result.
```{r}
```

7. (1 point) Predict the probabilities of diabetes associated with each of the training observations of `BPSysAve`. Make a vector of predictions for diabetes based on whether the predictions are above or below 0.5.
```{r}
```

8. (1 point) Generate a confusion matrix that shows the number of false positives, false negatives, true positives, and true negatives in the training data. The rows should correspond to the true diabetes status and the columns should correspond to the predicted values.
```{r}
```

9. (1 point) Find the proportion of correctly classified observations in the training data.
```{r}
```

10. (2 points) Now repeat questions 7 to 9 but for predicting the test dataset.
```{r}
```

11. (1 point) Comment on the difference in results between the training and test prediction tables and classification accuracies.
```{r}
```

12. (1 point) Manually calculate the sensitivity and specificity estimates for the test dataset based on the 0.5 threshold.
```{r}
```

13. (2 points) Generate an ROC curve using the test data. What is the AUC and its 95% confidence interval?
```{r}
```

14. What value can you use to threshold the predicted probability to achieve a sensitivity of at least 0.6 and a specificity of at least 0.3?
```{r}
```

15. (2 points) Comment on the results of the analyses for the different thresholds in terms of the tables, classification accuracies, and sensitivity and specificity. Under what circumstances might you prefer each of the thresholds?

```{r}
```

16. (2 points) Fit a multiple predictor logistic regression (call it `glm2`) with `Diabetes` as outcome and predictors: `BPSysAve`, `BPDiaAve`, and `Age`. Use the `summary` command to examine the fitted model and determine the estimated coefficients, odds-ratios, and 95% confidence intervals thereof.
```{r}
```

17. (2 points) Generate an ROC curve for the `glm2` model using the test data. What is the AUC and its 95% confidence interval?
```{r}
```

18. (1 point) What is the maximum sensitivity level you can achieve if we require the specificity to be at least 0.3?
```{r}
```

19. (1 point) Would you prefer the single predictor or multiple predictor model if your objective was to maximize classification accuracy, and which threshold level would you choose? Comment on the reason for your choices.

```{r}
```

## Linear discriminant analysis

20. (2 points) Fit a linear discriminant analysis (`lda1`) with `Diabetes` as outcome and predictors of `BPSysAve`, `BPDiaAve`, and `Age` in the training dataset. Examine the fit by typing `lda1`.
```{r}
```

21. (2 points) Generate the confusion matrix for `lda1` using the test set. Compute the classification accuracy, sensitivity, and specificity.
```{r}
```

22. How do these measures compare with that of the logistic regression model with these predictors and 0.5 threshold?
```{r}
```


23. (3 points) Redo question 21 but with prior probabilities set to 0.5 for diabetes.
```{r}
```

24. (2 points) Comment on how LDA's performance changed when we changed the prior probabilities.
```{r}
```


## Penalized regression

26. Read in the dementia data "dementia2.csv" into a data frame called `dementia_dat`. This dataset contains measurements obtained from MRI brain scans and whether or not the patient has dementia. We'll try to build a prediction model for diagnosing dementia based on these derived measurements. How many observations are in this dataset? How many predictors are in this dataset?
```{r}
```

27. Load the `glmnet` and `caret` packages.
```{r}
```

28. (1 point) Set the random seed to 4 and then split the data into 2 sets (400 train, and 260 test) 
```{r}
```

29. (4 points) Perform cross-validated lasso in the training data to select the optimal penalty parameter lambda. Use 5 folds and search over the range $\lambda = 10^{3}$ to $\lambda = 10^{-3}$. Set `Dementia` as outcome with all other variables as predictors except `MacCohort_kr`. Use the `caret` package to do CV.
```{r}
```

30. (1 point) What is the optimal value of lambda?
```{r}
```


31. (1 point) Generate the confusion matrix for this final model.
```{r}
```

32. (1 point) How many non-zero coefficients are in the final model?
```{r}
```