A comprehensive, multi-modal evaluation of the assessment system of an undergraduate research methodology course: Translating theory into practice. | Pakistan Journal of Medical Sciences 2014

Author(s)

Hamza Mohammad Abdul-Ghani College of Medicine, King Saud University, P.O. Box 230155, Riyadh 11321, Kingdom of Saudi Arabia

Gominda G. Ponnamperuma College of Medicine, University of Colombo, Colombo, Sri Lanka

Farah Ahmad Department of Medical Education, College of Medicine, King Saud University, P.O. Box 230155, Riyadh 11321, Kingdom of Saudi Arabia

Zubair Amin Department of Pediatrics, College of Medicine, National University of Singapore, Singapore

Abstract

Objective: To evaluate assessment system of the ‘Research Methodology Course’ using utility criteria (i.e. validity, reliability, acceptability, educational impact, and cost-effectiveness). This study demonstrates comprehensive evaluation of assessment system and suggests a framework for similar courses. Methods: Qualitative and quantitative methods used for evaluation of the course assessment components (50 MCQ, 3 Short Answer Questions (SAQ) and research project) using the utility criteria. Results of multiple evaluation methods for all the assessment components were collected and interpreted together to arrive at holistic judgments, rather than judgments based on individual methods or individual assessment. Results: Face validity, evaluated using a self-administered questionnaire (response rate-88.7%) disclosed that the students perceived that there was an imbalance in the contents covered by the assessment. This was confirmed by the assessment blueprint. Construct validity was affected by the low correlation between MCQ and SAQ scores (r=0.326). There was a higher correlation between the project and MCQ (r=0.466)/SAQ (r=0.463) scores. Construct validity was also affected by the presence of recall type of MCQs (70%; 35/50), item construction flaws and non-functioning distractors. High discriminating indices ( > 0.35) were found in MCQs with moderate difficulty indices (0.3-0.7). Reliability of the MCQs was 0.75 which could be improved up to 0.8 by increasing the number of MCQs to at least 70. A positive educational impact was found in the form of the research project assessment driving students to present/publish their work in conferences/ peer reviewed journals. Cost per student to complete the course was US$164.50. Conclusions: The multi-modal evaluation of an assessment system is feasible and provides thorough and diagnostic information. Utility of the assessment system could be further improved by modifying the psychometrically inappropriate assessment items.

Publication Details

Page(s) 227-232

DOI DOI not available

Published Journal: Pakistan Journal of Medical Sciences, Volume: 30, Issue: 2, Year: 2014

Keywords

Keywords are not available for this article.

References

References are not available for this document.

Citations

Citations are not available for this document.