**Setup 

cd "/Users/kristenaiemjoy/Box Sync/_Teaching/Biostat 212/Biostat212.2017/Assignments/Assignment 2 cleaning" //modify filepath to your directory

capture log close //close any open log files

set linesize 120 //increase the linesize (width of the page) helps for formatting the Markdown document

quietly log using biostat212_assignment2_name.smcl, replace //using "quietly" will stop print out in your smcl file making a nicer looking .pdf when you use markdown



/***
#Biostat 212: 2017
#Assignment 2: Data cleaning
## Your Name

---

_Instructions_   
In this assignment we will be cleaning a subset of the NHANES data set "nhanes_raw,csv"
We will import the data from a .csv file and export a clean data set. 
Answer questions 1-10 in the .do file.
The commands you will need to answer these questions are already included in the
.do file. You should be able to run the entire .do file at once and create a PDF 
with you answers and the code using the MarkDoc package. The syntax for calling the
MarkDoc package is already included in this .do file. 
To get the marcdoc command to work, you need to install 3 packages "marcdoc" "weaver" and "statax".
To install these packages open the "MarkDoc_Installation.do" file on the course website and run all 
three commands.  

Turn in the PDF you create using MarkDoc, which should include your name and answers 
to questions 1-10


The full .do file may take some time to run. 


***/

********************************************************************************
/***
##IMPORT 
Imprt the the raw .CSV file
***/
import delimited "nhanes_raw.csv", clear


/***
Browse data to make sure it worked. 
Use the exploring commands we learned last week to answer the following quesitons 

####1. How many observations are there? [_Answer here_]

####2. How many variables are there? [_Answer here_]

***/
br
count
describe
summarize

********************************************************************************
/***
##FORMAT VARIABLES
Convert _Age_ to a numeric variable

####3. What format is age currently in? [_Answer here_]   
_Hint: looks at "storage type" in the describe command_
***/
sum age //we currently cannot sum age becuase there it is not stored as a numeric variable
describe age
tab age //one person's age is stored as "sixty one" rather than "61" 
replace age = "61" if id=="NC0003" //relace "sixty one" with 61
destring age, generate(age_n)  //convert age into a numeric variable called age_n
list age age_n in 1/10 //compare old age variable and new age variable
summarize age age_n
describe age age_n
drop age //drop old age variable
rename age_n age //rename new age variable (age_n) to age 
label variable age "Age (years)" //label the age variable 

/***
####4. What is the highest age in this dataset? [_Answer here_]
***/
inspect age //looks like there is an outlier (unrealistic high value)
spikeplot age //you can also visualize the outlier with a spikeplot
graph export spikeplot.png, replace width(300) 
txt "![_Figure 1_. Spikeplot of __age__ outliers](spikeplot.png)" _n   //this command saves the spikeplot image into our markdown file

********************************************************************************
/***
##REMOVE OUTLIERS
remove outliers for age for age variable
***/
list id age if age > 100 //what is the id number for the outlier
replace age = 52 if id=="NC0020"
list id age if id=="NC0020"

********************************************************************************
/***
##MISSING DATA
You probably noticed that that there were some '-99' values for age and other variables.
 
We need to convert these to the Stata format for missing data (a period ".")

age, height and weight all have "-99" values

####5. How many missing values were coded as -99 for age? [_Answer here_]
***/
mvdecode age height weight, mv(-99) //this command recodes -99 to missing "."


/***
Now that we have removed outliers and corrected missing values of age we can summarize it.   
 
####6. What is the mean age? [_Answer here_]
***/
sum age 

********************************************************************************
/***
##LABELS
Lets create value lables for the variable sex. 0=female 1=male

####7. How many males in this sample? [_Answer here_]
***/
tabulate sex
label define SexLabel 0 "Female" 1 "Male"
label values sex SexLabel
tabulate sex

/***
####8. How many males in this sample? [_Answer here_]
***/

********************************************************************************
/*** 
##GENERATING NEW VARIABLES
Convert race from a string variable to a labeled numeric variable
***/
tabulate race
describe race
encode race, generate(race_n) //create a new categorical variable for race with labels
summarize race race_n
describe race race_n
tabulate race race_n
drop race //drop original race variable
rename race_n race //rename new race variable with name "race"
order id age sex race //change the order of the variable, but race after sex


/***
####9. What percentage of the population is white? [_Answer here_]
***/
tab race


/*** 
create variable for bmi, from weight and height
***/
generate bmi = weight / ((height/100)^2)
label variable bmi "Body Mass Index (kg/m^2)" //label bmi
format %5.1f bmi //format bmi to one decimal place 

/***
####8. What is the median bmi? [_Answer here_]
***/
sum bmi, detail 


/*** 
create a categorical varibale for blood pressure
***/
describe sbp
summarize sbp
recode sbp (0/119 = 0 "Normal") (120/max = 1 "Hypertension"), generate(HighBP)
label variable HighBP "Hypertension"


/***
####9. How many people have hypertension? [_Answer here_]
***/
tabulate HighBP 

********************************************************************************
/*** 
##DATES
reformat "dob" as to a date format

watch the data browser as you make these transformations
***/
br dob
generate dob_n = date(dob,"DM19Y")
format %tdMonth_DD,_CCYY dob_n
drop dob
rename dob_n dob
label var dob "Date of Birth"


********************************************************************************
/*** 
##DUPLICATES
find and remove duplicates
***/
duplicates report 
duplicates list //list duplicate observations


/***
####10. What are the ID numbers for the 2 duplicate observations? [_Answer here_]
***/
duplicates drop //drop duplicates
duplicates list 



********************************************************************************
/*** 
##SUMMARY TABLE
summarize bmi, cholesterol and sbp.
Just run this code and see results in your MarkDown PDF:
***/

//OFF
foreach var of varlist bmi cholesterol sbp  {
      summarize `var'
	  local `var'_n : display r(N)
      local `var'_mean : display %9.2f r(mean)
      local `var'_sd   : display %9.2f r(sd)
}

//ON
tbl ("Variable Name", "N", "Mean", "SD" 										\ 			///
"_BMI_",`bmi_n',`bmi_mean', `bmi_sd' 											\   		///
"_Cholesterol_", `cholesterol_n',`cholesterol_mean', `cholesterol_sd' 			\ 			///
"_Systolic Blood Pressure_",`sbp_n',`sbp_mean', `sbp_sd')      								///
, title("Table 1. Summary of _BMI_, _Cholesterol_, and _Systolic Blood Pressure_")			
	   
	   
	
********************************************************************************
/*** 
##SAVE THE CLEAN DATA SET
***/
save nhanes_clean.dta, replace //Save clean data




qui log close //close log (quietly so it wont appear in markdown document



**The following command creates a PDF or word document from your log file. Make sure the name of the file after the word markdown matches the name of your log file
markdoc biostat212_assignment2_name, replace export(pdf)  //you can also export as a docx or an html, just change the file type in the () after export