Stephen R. Poteau, Ph.D.
Gone are the days when accrediting bodies accepted indirect measures such as tests, papers, and student evaluations as viable outcome metrics of student learning. However, it may be short-sighted to write off any and all indirect measures of assessment of student learning as barren of any valuable information. Indirect measures of learning outcomes have been touted as viable and reasonably correlated with direct measures of actual performance (Holden, Anastas, & Meenaghan, 2003; Holden, Barker, Rosenberg, & Onghena, 2008), but have also been discredited as poor predictors of actual learning (Fortune, Lee, & Cavazos, 2005; Price & Randall, 2008). Many researchers question whether direct and indirect measures lie on a continuum or if each measure uniquely taps different constructs. If the former is true, indirect measures hold no value. Conversely, if the latter is true, certain types of indirect measures may have merit in outcomes assessment.
In a study assessing the degree to which students felt that course objectives, as outlined in the syllabus, were met on a 5-point Likert scale (ranging from 1 = this objective was met to small degree to 5 = this objective was met to a very great degree), Calderon (2013) found this indirect measure to be related to direct measures of student learning (i.e., ratings by field instructors and an objective standardized test). This suggests that students’ perceptions of achievement of objectives were related to their actual achievements/learning. This finding, however, was not replicated with a different cohort in the same study. That is, there was a discrepancy between this cohort’s perceptions of achievement and actual achievement, as measured by indirect and direct instruments respectively. Specifically, their perceptions of achievement were higher than their actual levels of achievement. Further, the perceptions of their performance were higher than the perceptions of the cohort whose actual performance was superior. Essentially, they did not know what they did not know. To put it another way, the perceptions of achievement of the cohort whose actual performance was higher were not high enough.
These results lend credence to the notion that indirect measures tap a different construct than actual performance. Perceptions of achievement, as reflected in indirect measures, may be more related to student satisfaction than to actual learning or acquisition of competencies. Indirect measures, therefore, may hold valuable information regarding the educational experience, but not the acquisition of knowledge and skills. As Calderon (2013) suggests, future research should attempt to identify the factors involved in students’ learning experiences to better understand the relationships between such experiences and actual learning.
Calderon, O. (2013). Direct and indirect measures of learning outcomes in an MSW program: What do we actually measure?. Journal of Social Work Education, 49(3), 408-419.
Fortune, A.E., Lee, M., & Cavazos, A. (2005). Achievement motivation and outcome in social work field education. Journal of Social Work Education, 41, 115-129.
Holden, G., Anastas, I., & Meenaghan, T. (2003). Determining attainment of the EPAS foundation program objectives: Evidence for the use of self-efficacy as an outcome. Journal of Social Wo.-k Education, 39, 425-440.
Holden, G., Barker, K., Rosenberg, G., & Onghena, P. (2008). The Evaluation Df Self-Efficacy Scale for assessing progress toward CSWE accreditation related objectives: A replication. Research on Social Work Practice, 18, 42—46.
Price, B. A., & Randall, C. H. (2008). Assessing learning outcomes in quantitative courses: Using embedded questions for direct assessment. Journal of Education for Business, 83(5), 288-294.