---
title: 'Homework #4: Boosting, Dimension reduction, Clustering'
output: html_document
---

```{r setup, echo=FALSE}
set.seed(7)
```

We will analyze data from the Investigation of Serial Studies to Predict Your Therapeutic Response with Imaging and moLecular Analysis (I-SPY TRIAL) breast cancer trial. This dataset contains patients with measurements derived from MRI scans. The outcome of interest is to predict the final diameter of the tumor as measured through the MRI, which is stored in the column `MRI_LD_Tfinal`. We have longitudinal measurements from each patient, collected at timepoints T0, T1, T2, and Tfinal. T0 is pre-treatment. T1 is early treatment. T2 is inter-regimen. Tfinal is the final measurement.

1. Load the iSPY1 dataset by running the following. Notice that there is a `fakeSite` column with values ranging from 1-8. We've added this column to simulate combining data from 8 different sites.
```{r data}
ispy_dat <- read.csv("ispy1doctored_site.csv")
ispy_dat$HR_HER2status <- as.factor(ispy_dat$HR_HER2status)
```


## Boosting

We will create and evaluate a gradient boosted model that predicts `MRI_LD_Tfinal` using all the predictors measured before Tfinal.

2. (1 point) Hold out sites 7 and 8 for testing. Store the training data in `datTrain` and the test data in `datTest`.
```{r}
```

3. (2 points) In this homework, we will manually implement site-wise 3-fold cross-validation (i.e. two sites per fold) rather than using the `caret` package. Create vector with True and False values to split the rows in `datTrain` into 3 folds, where sites 1 and 2 are in the first fold, sites 3 and 4 are in the second fold, and sites 5 and 6 are in the third fold. Create a list with these three True/False vectors. It may be helpful to reference this list later in this homework.
```{r}
```

4. Load the `gbm` package.
```{r}
```

5. (3 points) We will select the hyperparameters for a gradient boosted model using site-wise three-fold CV. Create a function named `fit_fold` that takes as input the fold number `fold_idx`, number of trees, and interaction depth. The function will fit a gradient boosted model using `gbm` using the aforementioned predictors. Also exclude `fakeSite`. Train on all the folds except for the `fold_idx`-th one. The function should output the mean squared error of the fitted model on the held out fold.  Use the folds you made in question 3. Fix the shrinkage hyperparameter in `gbm` as 0.01.
```{r}
```

6. (4 points) Using the function you made in question 5, tune the number of trees and interaction depth using site-wise three-fold CV. Search over the values `n.trees=100, 200, 400, 800, 1600` and `interaction.depth=1, 2`. Which hyperparameter values minimize the cross-validated mean squared error?
```{r}
```

7. (2 points) Plot the cross-validated error with respect to `n.trees` for the interaction depth that attained the lowest CV error.

```{r}
```

8. (1 point) Refit the gradient boosted model on all the training data (`datTrain`) using the hyperparameters that minimized the CV error.
```{r}
```

9. (1 point) Evaluate the MSE of the fitted model on the test data. How much of the variance have we explained using the GBM?
```{r}
```


## Kmeans

Let's perform K-means on the iSPY data.

10. (1 point) Create a new data frame named `ispy_subdat` that only contains the continuous variables measured before Tfinal.
```{r}
```

11. (1 point) Before we run K-means, center and scale all the variables so that they have mean 0 and variance 1 (hint: use the `scale` command).
```{r}
```


12. (3 points) Tune the number of clusters used in K-means. To do this, use the function `fviz_nbclust` from the `factoextra` library. The function `fviz_nbclust` determines and visualizes the optimal number of clusters using different methods (within cluster sums of squares, average silhouette and gap statistics). Plot the average silhouette with respect to the number of clusters by passing in the argument `method="silhouette"`. What is the optimal number of clusters according to the silhouette statistic?
```{r}
```

13. (3 points) Refit k-means using the optimal number of clusters with 15 random initializations. Use the function `fviz_cluster()` to plot the clusters from K-means. Observations are represented by points in the plot, using principal components if $p > 2$. An ellipse is drawn around each cluster.

```{r}
```

13. (1 point) Plot the distribution of `MRI_LD_Tfinal` for each cluster created by K-means. How do they differ across the clusters?
```{r}
```