diff --git a/Assignment6.Rmd b/Assignment6.Rmd index 8e65135..0153f8b 100644 --- a/Assignment6.Rmd +++ b/Assignment6.Rmd @@ -1,6 +1,6 @@ --- title: "Assignment 6" -author: "Charles Lang" +author: "Yi Yang" date: "11/16/2016" output: html_document --- @@ -25,7 +25,7 @@ library(rpart) #Upload the data sets MOOC1.csv and MOOC2.csv M1 <- read.csv("MOOC1.csv", header = TRUE) -M2 <- +M2 <- read.csv("MOOC2.csv", header = TRUE) ``` @@ -33,10 +33,10 @@ M2 <- ```{r} #Using the rpart package generate a classification tree predicting certified from the other variables in the M1 data frame. Which variables should you use? -c.tree1 <- +c.tree1 <- rpart(certified ~ grade + assignment, method="class", data=M1) #Check the results from the classifcation tree using the printcp() command - +printcp(c.tree1) #Plot your tree @@ -45,6 +45,9 @@ post(c.tree1, file = "tree1.ps", title = "MOOC") #This creates a pdf image of th ``` +### I chose the grade variable and the assignment variable. According to the plot, if a student's average grade for the course is less than 12.5, or the average grade for the course assignments is less than 7.5, he or she is more likely to be uncertified (unpaid for the course). + + ##Part II #The heading "xerror" in the printcp table stands for "cross validation error", it is the error rate of assigning students to certified/uncertified of the model averaged over 10-fold cross validation. CP stands for "Complexity Parameter" and represents the cost to error for adding a node to the tree. Notice it decreases as we add more nodes to the tree which implies that more nodes make better predictions. However, more nodes also mean that we may be making the model less generalizable, this is known as "overfitting". @@ -52,7 +55,7 @@ post(c.tree1, file = "tree1.ps", title = "MOOC") #This creates a pdf image of th #If we are worried about overfitting we can remove nodes form our tree using the prune() command, setting cp to the CP value from the table that corresponds to the number of nodes we want the tree to terminate at. Let's set it to two nodes. ```{r} -c.tree2 <- prune(c.tree1, cp = )#Set cp to the level at which you want the tree to end +c.tree2 <- prune(c.tree1, cp = 0.058182)#Set cp to the level at which you want the tree to end #Visualize this tree and compare it to the one you generated earlier @@ -70,16 +73,67 @@ table(M2$certified, M2$predict1) table(M2$certified, M2$predict2) +# error rate +mean(M2$certified!=M2$predict1) +mean(M2$certified!=M2$predict2) ``` +### According to the table, the pruned tree does a better job in making predictions about the the students in the second data set than the original tree. The pruned tree has a lower error rate (0.4637). However, both of the trees have high error rates thus the model is not ideal for our data. Thus we might need to modify the model by including other variables. + ##Part III Choose a data file from the (University of Michigan Open Data Set)[https://github.com/bkoester/PLA/tree/master/data]. Choose an outcome variable that you would like to predict. Build two models that predict that outcome from the other variables. The first model should use raw variables, the second should feature select or feature extract variables from the data. Which model is better according to the cross validation metrics? ```{r} +record <- read.csv("student.record.csv", header = TRUE) + +## Model 1 (raw variable: SAT total and high school GPA) +D1 <- dplyr::select(record, LAST_SATI_TOTAL_SCORE, HSGPA, SEX) + +D1[D1 == 0.00] <- NA +D1 <- na.omit(D1) + +D1sample <- dplyr::sample_n(D1, 5000, replace = TRUE) + +c.treeM1 <- rpart(SEX ~ HSGPA + LAST_SATI_TOTAL_SCORE, method="class", data=D1sample) + +printcp(c.treeM1) + +post(c.treeM1, file = "treeM1.ps", title = "Model 1") + +D1$predict <- predict(c.treeM1, D1, type = "class") + +table(D1$SEX, D1$predict) +## Model 2 (extract variable: ACT total) +D2 <- dplyr::mutate(record, LAST_ACT_TOTAL_SCORE = LAST_ACT_ENGL_SCORE + LAST_ACT_MATH_SCORE + LAST_ACT_READ_SCORE + LAST_ACT_SCIRE_SCORE + LAST_ACT_COMP_SCORE) + +D2 <- dplyr::select(D2, HSGPA, SEX, LAST_ACT_TOTAL_SCORE) + +D2[D2 == 0.00] <- NA +D2 <- na.omit(D2) + +D2sample <- dplyr::sample_n(D2, 5000, replace = TRUE) + +c.treeM2 <- rpart(SEX ~ HSGPA + LAST_ACT_TOTAL_SCORE, method="class", data=D2sample) + +printcp(c.treeM2) + +post(c.treeM2, file = "treeM2.ps", title = "Model 2") + +D2$predict <- predict(c.treeM2, D2, type = "class") + +table(D2$SEX, D2$predict) + + +# error rate +mean(D1$SEX!=D1$predict) +mean(D2$SEX!=D2$predict) ``` +### The first model uses raw variables (SAT total score and high school GPA) to predict the outcome variable (gender), while the second model features an extract variable from the data (ACT total score) instead of SAT. + +### Model 1 has a slightly lower error rate comparing to Model 2, which indicates that SAT total score has higher accuracy of predicting the student's gender than ACT total score. However, both of the trees have high error rates thus the model is not ideal for our data. Thus we might need to modify the model by taking other variables into consideration. ### To Submit Your Assignment diff --git a/Assignment6.html b/Assignment6.html new file mode 100644 index 0000000..088f271 --- /dev/null +++ b/Assignment6.html @@ -0,0 +1,348 @@ + + + + + + + + + + + + + + + +Assignment 6 + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+ + + + + + + + + + + + + + +
+

Addignment 6

+

In this assignment you will be looking at data from a MOOC. It contains the following per-student variables:

+

certified (yes/no) - Whether or not a student paid for the course
+forum.posts (numeric) - How many forum posts a student made throughout the course
+grade (numeric) - A student’s average grade for the course exam
+assignment (numeric) - A student’s average grade for the course assignments

+
+

Part I

+
+
+
+

Packages

+
library(rpart)
+
## Warning: package 'rpart' was built under R version 3.5.2
+
+
+

Data

+
#Upload the data sets MOOC1.csv and MOOC2.csv
+M1 <- read.csv("MOOC1.csv", header = TRUE)
+
+M2 <- read.csv("MOOC2.csv", header = TRUE)
+
+
+

Decision tree

+
#Using the rpart package generate a classification tree predicting certified from the other variables in the M1 data frame. Which variables should you use?
+
+c.tree1 <- rpart(certified ~ grade + assignment, method="class", data=M1)
+
+#Check the results from the classifcation tree using the printcp() command
+printcp(c.tree1)
+
## 
+## Classification tree:
+## rpart(formula = certified ~ grade + assignment, data = M1, method = "class")
+## 
+## Variables actually used in tree construction:
+## [1] assignment grade     
+## 
+## Root node error: 275/1000 = 0.275
+## 
+## n= 1000 
+## 
+##         CP nsplit rel error   xerror      xstd
+## 1 0.923636      0  1.000000 1.000000 0.0513455
+## 2 0.058182      1  0.076364 0.076364 0.0164880
+## 3 0.010000      2  0.018182 0.018182 0.0081108
+
#Plot your tree
+
+post(c.tree1, file = "tree1.ps", title = "MOOC") #This creates a pdf image of the tree
+
+

I chose the grade variable and the assignment variable. According to the plot, if a student’s average grade for the course is less than 12.5, or the average grade for the course assignments is less than 7.5, he or she is more likely to be uncertified (unpaid for the course).

+
+
+

Part II

+
+
+
+

The heading “xerror” in the printcp table stands for “cross validation error”, it is the error rate of assigning students to certified/uncertified of the model averaged over 10-fold cross validation. CP stands for “Complexity Parameter” and represents the cost to error for adding a node to the tree. Notice it decreases as we add more nodes to the tree which implies that more nodes make better predictions. However, more nodes also mean that we may be making the model less generalizable, this is known as “overfitting”.

+
+
+

If we are worried about overfitting we can remove nodes form our tree using the prune() command, setting cp to the CP value from the table that corresponds to the number of nodes we want the tree to terminate at. Let’s set it to two nodes.

+
c.tree2 <- prune(c.tree1, cp = 0.058182)#Set cp to the level at which you want the tree to end
+
+#Visualize this tree and compare it to the one you generated earlier
+
+post(c.tree2, file = "tree2.ps", title = "MOOC") #This creates a pdf image of the tree
+
+
+

Now use both the original tree and the pruned tree to make predictions about the the students in the second data set. Which tree has a lower error rate?

+
M2$predict1 <- predict(c.tree1, M2, type = "class")
+
+M2$predict2 <- predict(c.tree2, M2, type = "class")
+
+table(M2$certified, M2$predict1)
+
##      
+##         no  yes
+##   no  2056   24
+##   yes 7790  130
+
table(M2$certified, M2$predict2)
+
##      
+##         no  yes
+##   no   896 1184
+##   yes 3453 4467
+
# error rate
+mean(M2$certified!=M2$predict1)
+
## [1] 0.7814
+
mean(M2$certified!=M2$predict2)
+
## [1] 0.4637
+
+

According to the table, the pruned tree does a better job in making predictions about the the students in the second data set than the original tree. The pruned tree has a lower error rate (0.4637). However, both of the trees have high error rates thus the model is not ideal for our data. Thus we might need to modify the model by including other variables.

+
+
+

Part III

+

Choose a data file from the (University of Michigan Open Data Set)[https://github.com/bkoester/PLA/tree/master/data]. Choose an outcome variable that you would like to predict. Build two models that predict that outcome from the other variables. The first model should use raw variables, the second should feature select or feature extract variables from the data. Which model is better according to the cross validation metrics?

+
record <- read.csv("student.record.csv", header = TRUE)
+
+## Model 1 (raw variable: SAT total and high school GPA)
+D1 <- dplyr::select(record, LAST_SATI_TOTAL_SCORE, HSGPA, SEX)
+
+D1[D1 == 0.00] <- NA
+D1 <- na.omit(D1)
+
+D1sample <- dplyr::sample_n(D1, 5000, replace = TRUE) 
+
+c.treeM1 <- rpart(SEX ~ HSGPA + LAST_SATI_TOTAL_SCORE, method="class", data=D1sample)
+
+printcp(c.treeM1)
+
## 
+## Classification tree:
+## rpart(formula = SEX ~ HSGPA + LAST_SATI_TOTAL_SCORE, data = D1sample, 
+##     method = "class")
+## 
+## Variables actually used in tree construction:
+## [1] HSGPA                 LAST_SATI_TOTAL_SCORE
+## 
+## Root node error: 2367/5000 = 0.4734
+## 
+## n= 5000 
+## 
+##         CP nsplit rel error  xerror     xstd
+## 1 0.125053      0   1.00000 1.00000 0.014916
+## 2 0.019011      1   0.87495 0.88255 0.014734
+## 3 0.010000      3   0.83692 0.87621 0.014718
+
post(c.treeM1, file = "treeM1.ps", title = "Model 1") 
+
+D1$predict <- predict(c.treeM1, D1, type = "class")
+
+table(D1$SEX, D1$predict)
+
##    
+##         F     M
+##   F  9001  6514
+##   M  6857 10642
+
## Model 2 (extract variable: ACT total)
+D2 <- dplyr::mutate(record, LAST_ACT_TOTAL_SCORE = LAST_ACT_ENGL_SCORE + LAST_ACT_MATH_SCORE + LAST_ACT_READ_SCORE + LAST_ACT_SCIRE_SCORE + LAST_ACT_COMP_SCORE)
+
+D2 <- dplyr::select(D2, HSGPA, SEX, LAST_ACT_TOTAL_SCORE)
+
+D2[D2 == 0.00] <- NA
+D2 <- na.omit(D2)
+
+D2sample <- dplyr::sample_n(D2, 5000, replace = TRUE) 
+
+c.treeM2 <- rpart(SEX ~ HSGPA + LAST_ACT_TOTAL_SCORE, method="class", data=D2sample)
+
+printcp(c.treeM2)
+
## 
+## Classification tree:
+## rpart(formula = SEX ~ HSGPA + LAST_ACT_TOTAL_SCORE, data = D2sample, 
+##     method = "class")
+## 
+## Variables actually used in tree construction:
+## [1] LAST_ACT_TOTAL_SCORE
+## 
+## Root node error: 2346/5000 = 0.4692
+## 
+## n= 5000 
+## 
+##        CP nsplit rel error  xerror     xstd
+## 1 0.10401      0   1.00000 1.00000 0.015042
+## 2 0.01000      1   0.89599 0.89599 0.014878
+
post(c.treeM2, file = "treeM2.ps", title = "Model 2") 
+
+D2$predict <- predict(c.treeM2, D2, type = "class")
+
+table(D2$SEX, D2$predict)
+
##    
+##         F     M
+##   F 23783  5383
+##   M 18944  8220
+
# error rate
+mean(D1$SEX!=D1$predict)
+
## [1] 0.40501
+
mean(D2$SEX!=D2$predict)
+
## [1] 0.4318658
+
+

The first model uses raw variables (SAT total score and high school GPA) to predict the outcome variable (gender), while the second model features an extract variable from the data (ACT total score) instead of SAT.

+
+
+

Model 1 has a slightly lower error rate comparing to Model 2, which indicates that SAT total score has higher accuracy of predicting the student’s gender than ACT total score. However, both of the trees have high error rates thus the model is not ideal for our data. Thus we might need to modify the model by taking other variables into consideration.

+
+
+

To Submit Your Assignment

+

Please submit your assignment by first “knitting” your RMarkdown document into an html file and then commit, push and pull request both the RMarkdown file and the html file.

+
+
+
+ + + + +
+ + + + + + + + diff --git a/tree1.ps b/tree1.ps new file mode 100644 index 0000000..e71d1b5 Binary files /dev/null and b/tree1.ps differ diff --git a/tree2.ps b/tree2.ps new file mode 100644 index 0000000..a6a7043 Binary files /dev/null and b/tree2.ps differ diff --git a/treeM1.ps b/treeM1.ps new file mode 100644 index 0000000..97c0d40 Binary files /dev/null and b/treeM1.ps differ diff --git a/treeM2.ps b/treeM2.ps new file mode 100644 index 0000000..eb74d20 Binary files /dev/null and b/treeM2.ps differ