---
title: 'Lab #4:  Support Vector Machines'
output:
  html_document:
    df_print: paged
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
set.seed(24)

library(e1071)  ### library containing svm function
library(ggplot2)
library(ROCit)

svmPlot <- function(dt, svmModel, n=100, xname1="HeartRate", xname2="Fitness", outcome="ageGrp", ...) {
  grange <- apply(dt[ ,1:2], 2, range) ## determine range for both predictors
  x1 <- seq(from=grange[1,1], to=grange[2,1], length=n)
  x2 <- seq(from=grange[1,2], to=grange[2,2], length=n)
  xgrid <- expand.grid(X1=x1, X2=x2) ## makes a grid with every 
  ## combination of X1 and X2
  names(xgrid) <- c(xname1,xname2)
  ygrid <- predict(svmModel, xgrid)
  plot(xgrid, col=c("red","blue")[as.numeric(ygrid)], pch=20, cex=0.1, ...)
  points(as.matrix(dt[dt[ ,outcome]=="-1",1:2]), col="red", pch=19, cex=1.5)
  points(as.matrix(dt[dt[ ,outcome]=="1",1:2]), col="blue", pch=19, cex=1.5)
  points(dt[svmModel$index, 1:2], pch = 5, cex = 2.5, col = "black") ## index in an svm fit
  ## gives the index of the support vectors
  beta <- drop(t(svmModel$coefs)%*% as.matrix(dt[svmModel$index, 1:2]))
  ## above line extracts beta1 and beta2 from svmModel fit -- 
  ## "%*%" means matrix multiplication
  beta0 <- svmModel$rho ## and this one gives beta0
  abline(beta0/beta[2], -beta[1]/beta[2], lwd=3)
  abline((beta0-1)/beta[2], -beta[1]/beta[2], lty=2, lwd=3, col="blue")
  abline((beta0+1)/beta[2], -beta[1]/beta[2], lty=2, lwd=3, col="red")
}

```

1. Simulate some data using the following code.
```{r sim}
create_data <- function(nsamp1, nsamp2) {
  nsamp <- nsamp1 + nsamp2
  x <- matrix(rnorm(nsamp*2), ncol=2)
  y <- c(rep(-1,nsamp1), rep(1,nsamp2))
  x[y==1,] <- x[y==1,] + 1
  dat <- data.frame(x=x, y=as.factor(y))  
}

nsamp1 <- 10
nsamp2 <- 10
dat <- create_data(nsamp1, nsamp2)
```

2. Plot the data using `qplot` from the `ggplot2` library. Is the data perfectly separable? Where do you think the decision boundary should be?
```{r}
```

## Support Vector Classifier

3. Fit a support vector classifier on the data using the `linear` kernel. Set the `cost` parameter (the cost of violating the decision boundary) to 10. Name the model `svmFit`. Plot the fitted model.
```{r}
```

4. Use the in-build SVM `plot` function to visualize the fitted model with the training data.
```{r}
```

5. Try out the `svmPlot` function we provided (again plotting with the training data). Notice this plots the margin as well.
```{r}
```

6. What happens if you decrease the cost to 0.1 when fitting the support vector classifier? Refit the model and plot it. How do the number of support vectors and margin change?
```{r}
```

## Support Vector Machine

7. Let's consider a `radial` basis kernel now. Let's set `cost=10` and `gamma=0.1`. The `gamma` parameter controls how "wiggly" our decision boundary is. A smaller gamma means you are fitting a less wiggly decision boundary. Plot the fitted model.
```{r}
```

8. Now let's use the `radial` basis kernel with the same cost but set `gamma=100`. Compare the refitted model to the one above. Why do you think most of the plotted region is now classified to one class?
```{r}
```

## ROC curves for SVM

9. Create a test dataset `dat_test` with 100 samples from both classes.
```{r}
```

10. To get an ROC curve, we need the decision values from a model. Classify the points from `dat_test` using the support vector classifier `svmFit` by running `predict` and pass in the argument `decision.values=T`.
```{r}
```

11. The decision values from `svmFit` are stored in the attribute `decision.values`. Use this to plot an ROC curve.
```{r}
```

## SVM with Multiple Classes

12. Use the following code to create a third class. We'll append this new class to the previously simulated data.
```{r sim2}
set.seed(1)
nsamp3 <- 30
x <- matrix(rnorm(nsamp3*2), ncol=2)
y <- rep(0,nsamp3)
x[,2] <- x[,2] + 2
dat_class0 <- data.frame(x=x, y=as.factor(y))
dat_3class <- rbind(dat, dat_class0)
```

13. Plot the data using `qplot`.
```{r}
```

14. Fit a support vector classifier for the 3 classes with `cost=10`. Go through the documentation for `svm`. How is this 3-way classifier constructed?
```{r}
```

15. Fit a support vector machine for the 3 classes using the radial basis kernel with `cost=10` and `gamma=1`. Which kernel do you think is most appropriate for this data?
```{r}
```