Asif Dhanani and Seung Yeon Lee and Phitchaya Phothilimthana and Zachary Pardos

EECS Department, University of California, Berkeley

Technical Report No. UCB/EECS-2014-131

May 29, 2014

http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-131.pdf

In the knowledge-tracing model, error metrics are used to guide parameter estimation towards values that accurately represent students' dynamic cognitive state. We compare several metrics, including log likelihood (LL), root mean squared error (RMSE), and area under the receiver operating characteristic curve (AUC), to evaluate which metric is most suited for this purpose. LL is commonly used as an error metric in Expectation Maximization (EM) to perform parameter estimation. RMSE and AUC have been suggested but have not been explored in depth. In order to examine the effectiveness of using each metric, we measure the correlations between the values calculated by each and the distances from the corresponding points to the ground truth. Additionally, we examine how each metric compares to the others. Our findings show that RMSE is significantly better than LL and AUC. With more knowledge of effective error metrics for estimating parameters in the knowledge-tracing model, we hope that better parameter searching algorithms can be created.

---

BibTeX citation:

@techreport{Dhanani:EECS-2014-131,
    Author= {Dhanani, Asif and Lee, Seung Yeon and Phothilimthana, Phitchaya and Pardos, Zachary},
    Title= {A Comparison of Error Metrics for Learning Model Parameters in Bayesian Knowledge Tracing},
    Year= {2014},
    Month= {May},
    Url= {http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-131.html},
    Number= {UCB/EECS-2014-131},
    Abstract= {In the knowledge-tracing model, error metrics are used to guide parameter estimation towards values that accurately represent students' dynamic cognitive state. We compare several metrics, including log likelihood (LL), root mean squared error (RMSE), and area under the receiver operating characteristic curve (AUC), to evaluate which metric is most suited for this purpose. LL is commonly used as an error metric in Expectation Maximization (EM) to perform parameter estimation. RMSE and AUC have been suggested but have not been explored in depth. In order to examine the effectiveness of using each metric, we measure the correlations between the values calculated by each and the distances from the corresponding points to the ground truth. Additionally, we examine how each metric compares to the others. Our findings show that RMSE is significantly better than LL and AUC. With more knowledge of effective error metrics for estimating parameters in the knowledge-tracing model,  we hope that better parameter searching algorithms can be created.},
}

EndNote citation:

%0 Report
%A Dhanani, Asif 
%A Lee, Seung Yeon 
%A Phothilimthana, Phitchaya 
%A Pardos, Zachary 
%T A Comparison of Error Metrics for Learning Model Parameters in Bayesian Knowledge Tracing
%I EECS Department, University of California, Berkeley
%D 2014
%8 May 29
%@ UCB/EECS-2014-131
%U http://www2.eecs.berkeley.edu/Pubs/TechRpts/2014/EECS-2014-131.html
%F Dhanani:EECS-2014-131