1. Install and load the torch and torchvision packages.
library(torch)
library(torchvision)
  1. (1 point) Use the tiny_imagenet_dataset method to download a tiny version of the ImageNet dataset and name the dataset train_ds. Modify the template code as needed. Explain in your own words what the transform argument does. (When you finally knit this Rmd file, you may need to set download=FALSE rather than download=TRUE)
train_ds <- tiny_imagenet_dataset(
  "~/Downloads",
  # set this to download = TRUE the first time you run this code
  download = FALSE,
  split="train",
  transform = function(x) {
    x %>%
      transform_to_tensor() %>%
      transform_resize(c(224, 224))
  }
)
  1. (1 point) Create a vector train_subset_ind containing the index of all observations in train_ds with class labels 1, 2, or 3. The class labels are stored in the targets attribute.
NUM_CLASSES <- 3
train_subset_ind <- as.numeric(which(train_ds$targets <= NUM_CLASSES))
  1. (1 point) Create a subset of the original dataset composed of only observations with labels 1, 2, or 3 using the dataset_subset method and the vector train_subset_ind from question 3.
train_small <- dataset_subset(train_ds, indices=train_subset_ind)
  1. (2 points) Randomly select 20 images from the dataset and visualize them. To do this, create a data loader with batch size 20 with shuffle=TRUE and name it train_dl_shuffle. Read the first batch of data from train_dl_shuffle and use the plot_images function to plot the images.
train_dl_temp <- dataloader(train_small, batch_size=20, shuffle = T)
batch <- train_dl_temp$.iter()$.next()

plot_images(batch)

  1. (2 points) Create a new data loader for train_small that does not shuffle the data. Set the batch size to the maximum batch size. Name this new data loader train_dl.
ntrain <- length(train_subset_ind)
train_dl <- dataloader(train_small, batch_size=ntrain, shuffle = F)
  1. (3 points) Load a pretrained ResNet-18 model using the function model_resnet18 with pretrained=TRUE. Name the pretrained model pretrained_resnet. List the types of layers used in ResNet-18.
pretrained_resnet <- model_resnet18(pretrained = TRUE)
# This lists the major modules in Resnet-18. Note that `layer1` to `layer4` contain many layers within them.
pretrained_resnet
## An `nn_module` containing 11,689,512 parameters.
## 
## ── Modules ─────────────────────────────────────────────────────────────────────────────────────────
## ● conv1: <nn_conv2d> #9,408 parameters
## ● bn1: <nn_batch_norm2d> #128 parameters
## ● relu: <nn_relu> #0 parameters
## ● maxpool: <nn_max_pool2d> #0 parameters
## ● layer1: <nn_sequential> #147,968 parameters
## ● layer2: <nn_sequential> #525,568 parameters
## ● layer3: <nn_sequential> #2,099,712 parameters
## ● layer4: <nn_sequential> #8,393,728 parameters
## ● avgpool: <nn_adaptive_avg_pool2d> #0 parameters
## ● fc: <nn_linear> #513,000 parameters
# To see what layers are used in `layer1`, run the following:
pretrained_resnet$layer1$`0`
## An `nn_module` containing 73,984 parameters.
## 
## ── Modules ─────────────────────────────────────────────────────────────────────────────────────────
## ● conv1: <nn_conv2d> #36,864 parameters
## ● bn1: <nn_batch_norm2d> #128 parameters
## ● relu: <nn_relu> #0 parameters
## ● conv2: <nn_conv2d> #36,864 parameters
## ● bn2: <nn_batch_norm2d> #128 parameters
pretrained_resnet$layer1$`1`
## An `nn_module` containing 73,984 parameters.
## 
## ── Modules ─────────────────────────────────────────────────────────────────────────────────────────
## ● conv1: <nn_conv2d> #36,864 parameters
## ● bn1: <nn_batch_norm2d> #128 parameters
## ● relu: <nn_relu> #0 parameters
## ● conv2: <nn_conv2d> #36,864 parameters
## ● bn2: <nn_batch_norm2d> #128 parameters

Extracting features using a pretrained neural network

  1. (1 point) Our goal is to train a new model using the extracted features by ResNet-18. We’ll use the penultimate layer as the extracted features. To get extracted features, replace the last layer in pretrained_resnet with an identity module. That is, set the fc attribute in pretrained_resnet to a newly created identity module using the nn_identity function.
pretrained_resnet$fc <- nn_identity()
  1. (1 point) Set the pretrained ResNet-18 model to evaluation mode by calling the eval method of the pretrained_resnet object.
pretrained_resnet$eval()
  1. (1 point) Read the first batch of data from the data loader.
my_data_obs <- train_dl$.iter()$.next()
  1. (2 points) Feed this batch of data through our (modified) pretrained network pretrained_resnet to get the values of the penultimate layer. What is the dimension of this output? Print the embedding for the first observation.
penultimate_dat_obs <- pretrained_resnet(my_data_obs[[1]])
# Dimension of the embedding layer
penultimate_dat_obs$size()[2]
## [1] 512
# Embedding for the first observation
as_array(penultimate_dat_obs[1,])
##   [1] 1.207113504 0.200944901 0.044803616 1.157499075 0.264290720 0.925264001
##   [7] 0.155249625 0.676184356 0.895647764 0.444298774 0.379203558 0.150691241
##  [13] 1.917141318 1.266770005 1.183861256 0.143611223 0.094730228 1.755309582
##  [19] 0.016952451 0.183919594 1.256701589 0.177762717 1.355789661 0.387694389
##  [25] 0.058532871 0.801276982 0.331039667 0.717243791 0.173170999 1.758649111
##  [31] 0.905976415 0.856370509 0.461087435 0.521119535 0.159632280 0.707652450
##  [37] 0.697519422 0.056117725 1.537179589 0.320396394 1.067293763 1.928139210
##  [43] 0.371132553 0.557477534 0.839438617 1.422636867 1.025181174 0.163533986
##  [49] 2.104895592 1.661920547 3.876588821 0.124304146 0.712479234 0.154374957
##  [55] 0.120318592 0.620313585 0.439129412 1.612254739 1.323267579 0.600683451
##  [61] 0.424191892 0.115376689 1.534685373 1.085263133 3.780719995 0.515107334
##  [67] 0.724634290 0.308559030 0.170182675 0.301208258 0.006064893 0.488726854
##  [73] 1.401733994 2.658017159 1.061187267 0.041864358 0.147781998 0.342477918
##  [79] 0.000000000 0.001426696 0.712384224 0.227184549 1.179816365 0.379797876
##  [85] 0.096033983 0.711589873 0.798493207 1.351190805 0.202458784 0.317310691
##  [91] 0.009331440 1.706429601 1.805084825 0.865787804 1.595819592 1.861429214
##  [97] 0.828362107 0.680690706 1.102154493 0.390192956 0.000000000 0.143590778
## [103] 1.387220383 0.040990118 1.894780040 0.613120198 0.031011241 0.912258387
## [109] 1.738322973 0.104879163 0.000000000 0.243891403 2.818093538 0.016275909
## [115] 1.808280587 0.780575871 1.405580759 6.331046581 2.105104923 0.374919415
## [121] 0.340570420 0.619356990 0.992868125 1.133817077 0.083604783 0.119101360
## [127] 0.857477784 1.038056135 0.237387061 0.242196828 0.164856210 0.573873222
## [133] 0.172034562 1.231678247 0.474237353 0.215788007 0.214028031 0.358519495
## [139] 0.589977503 1.614218831 1.009909511 0.420256823 3.526296616 0.666777313
## [145] 0.105100639 1.361075878 0.796301603 0.000000000 0.288811356 0.301415145
## [151] 0.878433466 0.135410830 0.481582552 0.319009781 0.386068493 0.788496673
## [157] 0.652203858 1.577992916 1.908385873 0.235194951 0.355751127 0.237045437
## [163] 0.004344404 6.186452389 0.603673100 1.505599499 0.216010615 2.688925505
## [169] 1.343311667 1.238091707 0.000000000 0.625718713 1.974732041 0.499797106
## [175] 0.099074073 2.275650501 0.350358695 0.852393150 0.667687654 0.730784357
## [181] 0.224081874 0.982089221 0.420363486 0.139916465 0.679081440 0.166521266
## [187] 1.184725523 0.062958136 0.630958736 0.160881504 0.420637786 0.230536699
## [193] 1.308539748 1.083720088 1.586362243 1.269914150 2.210751772 0.055823568
## [199] 1.096650004 1.284518838 0.646672487 0.459666848 0.958084583 0.116700903
## [205] 0.114086218 0.059139982 1.332849622 1.910431504 0.540271878 2.547956467
## [211] 0.113594271 0.342790484 0.027598804 1.640129447 1.180102944 1.040226817
## [217] 0.056807470 1.417256355 1.092621803 1.277959108 1.365562439 1.484580994
## [223] 0.001461982 1.234201670 0.020065404 2.637241840 1.515387654 3.130200386
## [229] 1.426442623 0.782579601 1.656461596 1.269292116 0.110555179 0.236998260
## [235] 1.313513637 0.026646476 0.668902814 1.208230019 1.529349327 1.040558934
## [241] 0.233250931 0.263789088 0.482949018 1.139902830 1.188870311 1.529362679
## [247] 0.441988260 0.140828013 0.490846276 0.186077580 0.201866329 1.999679208
## [253] 0.269504130 0.028083131 0.570430934 1.007698894 0.887675226 1.082344174
## [259] 0.311956972 0.950958729 0.072719075 0.456972927 0.482843548 1.638813615
## [265] 0.972972929 0.003691797 0.857447863 0.832604885 0.854879260 0.281067491
## [271] 0.730854750 0.000000000 0.085614577 0.572932184 0.466144949 0.319392949
## [277] 2.391983271 1.145602345 1.917808414 0.236825675 1.782173991 0.206201926
## [283] 2.538312435 1.000379562 0.007711037 0.021524269 0.561011553 0.000000000
## [289] 0.766650438 0.905918300 0.011929247 0.913836718 0.604127049 1.094803691
## [295] 0.185918227 0.071334012 1.431621194 0.611909389 0.719535112 0.309552222
## [301] 4.243544579 0.147905290 1.060075283 0.095490299 0.664058626 0.431877911
## [307] 0.131425455 0.502890229 1.475465059 1.114265561 1.314778686 1.870570302
## [313] 0.103622310 1.570267081 1.574959159 0.967263520 0.011777486 1.398899674
## [319] 1.450793982 0.340451330 0.904479682 0.652566195 1.000543356 0.542700469
## [325] 0.215248182 0.768615127 0.217463955 0.075772867 0.023043998 0.671971619
## [331] 0.475603819 0.034750022 0.465231597 0.109936558 0.425637692 0.101555213
## [337] 0.862489879 0.577298284 0.706495225 0.358127832 0.224747404 0.262512356
## [343] 0.148166299 1.813800097 0.308938831 4.594698429 0.455276519 0.885049164
## [349] 1.763921022 0.375325799 1.594919443 0.063069351 0.200221002 0.435480922
## [355] 0.436478525 0.061529540 0.530365407 1.096576333 0.170133129 0.000000000
## [361] 1.722611547 0.237302125 0.569226623 0.816286504 1.394240737 0.899396598
## [367] 0.575550914 0.364930809 1.151639104 0.005917147 0.034550797 0.602012455
## [373] 0.162936166 0.842985630 0.186338350 0.499823987 0.279198676 0.021558650
## [379] 0.025879854 0.136595696 1.101934910 0.641062677 0.285797894 0.004803643
## [385] 0.225287750 1.199215055 1.501320958 0.001829864 0.692721844 0.146505669
## [391] 0.964543998 2.562161207 0.291156679 0.608215749 1.791354418 0.587162197
## [397] 0.128447577 1.437805772 1.937972426 1.270518303 0.464291245 0.771103084
## [403] 0.469809771 0.000000000 3.315223455 0.087316692 2.935552120 0.158951908
## [409] 0.104478493 0.510417640 1.487749100 1.325204968 0.695640504 0.059846651
## [415] 0.196692899 0.943257928 0.204020113 1.825250149 0.461391062 1.006837249
## [421] 1.328020692 0.036536008 0.116215028 0.015539750 2.716717720 0.435529292
## [427] 1.754887581 0.392524809 3.478302002 1.202320814 0.031112006 1.639729857
## [433] 0.642176092 0.258908570 0.436735153 0.334628165 0.500752330 0.581827760
## [439] 0.651061356 0.860352039 0.742652118 0.763543427 0.817550063 0.664818287
## [445] 0.000000000 0.012205428 0.551300347 1.188073277 0.138549060 0.387653261
## [451] 1.835436940 1.434352279 0.772637367 1.724887729 0.008789166 0.785233319
## [457] 0.035536282 0.195654288 1.659077048 0.533929229 3.782554150 0.032737080
## [463] 0.605919600 0.778325438 1.599451661 0.088696375 0.226509422 0.023592427
## [469] 0.993101537 0.747188747 0.245160863 1.527272940 0.079864062 0.523966908
## [475] 0.303521395 0.062280890 0.918358028 1.804421902 0.005205288 0.984873712
## [481] 0.536026418 0.007660855 1.638919592 1.148744226 0.693559349 0.042872496
## [487] 0.607430458 0.127917334 0.742073655 0.873417616 0.724297881 0.000000000
## [493] 0.870360136 1.120459318 0.278388232 0.026603756 0.731751680 0.118591346
## [499] 1.799911141 0.164419800 1.105380654 0.702032149 1.180765152 0.293059587
## [505] 0.958281398 1.841425419 0.277966142 1.034368277 0.385646433 2.211461067
## [511] 0.957582891 0.407551795

Training a model using extracted features

  1. (2 points) Create a data frame with the embeddings from ResNet-18 and their corresponding class labels.
my_df <- data.frame(as_array(penultimate_dat_obs), y = as.numeric(train_ds$targets[train_subset_ind]))
  1. (2 points) Split this data so that 4/5 of the data is used for training and 1/5 of it is used for testing.
my_train_idxs <- sample(nrow(my_df), size=nrow(my_df) * 4/5, replace=F)
my_df_train <- my_df[my_train_idxs,]
my_df_test <- my_df[-my_train_idxs,]
  1. (2 points) Fit a random forest for this multiclassification problem using the training data. Use the default hyperparameter settings.
library(randomForest)
## randomForest 4.6-14
## Type rfNews() to see new features/changes/bug fixes.
rf <- randomForest(as.factor(y) ~ ., data=my_df_train)
rf
## 
## Call:
##  randomForest(formula = as.factor(y) ~ ., data = my_df_train) 
##                Type of random forest: classification
##                      Number of trees: 500
## No. of variables tried at each split: 22
## 
##         OOB estimate of  error rate: 14.17%
## Confusion matrix:
##     1   2   3 class.error
## 1 359   7  26  0.08418367
## 2   9 343  49  0.14463840
## 3  15  64 328  0.19410319
  1. (1 point) Evaluate the random forest model on the test data. What is the accuracy of this model?
preds <- predict(rf, newdata = my_df_test)
mean(preds == as.factor(my_df_test$y))
## [1] 0.8833333