Stephen R. Poteau, Ph.D.
Gone are the days when accrediting bodies accepted indirect
measures such as tests, papers, and student evaluations as viable outcome
metrics of student learning. However, it may be short-sighted to write off any
and all indirect measures of assessment of student learning as barren of any
valuable information. Indirect measures of learning outcomes have been touted as
viable and reasonably correlated with direct measures of actual performance
(Holden, Anastas, & Meenaghan, 2003; Holden, Barker, Rosenberg, & Onghena, 2008), but have also been discredited
as poor predictors of actual learning (Fortune, Lee, & Cavazos, 2005; Price
& Randall, 2008). Many researchers question whether direct and indirect
measures lie on a continuum or if each measure uniquely taps different
constructs. If the former is true, indirect measures hold no value. Conversely,
if the latter is true, certain types of indirect measures may have merit in
outcomes assessment.
In a study assessing the degree to which students felt that
course objectives, as outlined in the syllabus, were met on a 5-point Likert
scale (ranging from 1 = this objective was met to small degree to 5 = this
objective was met to a very great degree), Calderon (2013) found this indirect
measure to be related to direct measures of student learning (i.e., ratings by
field instructors and an objective standardized test). This suggests that
students’ perceptions of achievement
of objectives were related to their actual achievements/learning. This finding,
however, was not replicated with a different cohort in the same study. That is,
there was a discrepancy between this cohort’s perceptions of achievement and actual achievement, as measured by
indirect and direct instruments respectively. Specifically, their perceptions
of achievement were higher than their actual levels of achievement. Further,
the perceptions of their performance were higher than the perceptions of the
cohort whose actual performance was superior. Essentially, they did not know
what they did not know. To put it another way, the perceptions of achievement of
the cohort whose actual performance was higher were not high enough.
These results lend credence to the notion that indirect
measures tap a different construct than actual performance. Perceptions of
achievement, as reflected in indirect measures, may be more related to student satisfaction
than to actual learning or acquisition of competencies. Indirect measures,
therefore, may hold valuable information regarding the educational experience,
but not the acquisition of knowledge and skills. As Calderon (2013) suggests,
future research should attempt to identify the factors involved in students’
learning experiences to better understand the relationships between such
experiences and actual learning.
References
Calderon, O. (2013). Direct and indirect measures of
learning outcomes in an MSW program: What do we actually measure?. Journal of Social Work Education, 49(3),
408-419.
Fortune, A.E., Lee, M., & Cavazos, A. (2005).
Achievement motivation and outcome in social work field education. Journal
of Social Work Education, 41, 115-129.
Holden, G., Anastas, I., & Meenaghan, T. (2003).
Determining attainment of the EPAS foundation program objectives: Evidence for
the use of self-efficacy as an outcome. Journal of Social Wo.-k Education,
39, 425-440.
Holden, G., Barker, K., Rosenberg, G., & Onghena, P.
(2008). The Evaluation Df Self-Efficacy Scale for assessing progress toward
CSWE accreditation related objectives: A replication. Research on Social
Work Practice, 18, 42—46.
Price, B. A., & Randall, C. H. (2008). Assessing
learning outcomes in quantitative courses: Using embedded questions for direct
assessment. Journal of Education for Business, 83(5), 288-294.
