PCOM Assessment

PCOM Assessment

Tuesday, March 18, 2014

The Strengths & Shortcomings of Indirect Measures of Assessment



Stephen R. Poteau, Ph.D. 

Gone are the days when accrediting bodies accepted indirect measures such as tests, papers, and student evaluations as viable outcome metrics of student learning. However, it may be short-sighted to write off any and all indirect measures of assessment of student learning as barren of any valuable information. Indirect measures of learning outcomes have been touted as viable and reasonably correlated with direct measures of actual performance (Holden, Anastas, & Meenaghan, 2003; Holden, Barker, Rosenberg, &  Onghena, 2008), but have also been discredited as poor predictors of actual learning (Fortune, Lee, & Cavazos, 2005; Price & Randall, 2008). Many researchers question whether direct and indirect measures lie on a continuum or if each measure uniquely taps different constructs. If the former is true, indirect measures hold no value. Conversely, if the latter is true, certain types of indirect measures may have merit in outcomes assessment.

In a study assessing the degree to which students felt that course objectives, as outlined in the syllabus, were met on a 5-point Likert scale (ranging from 1 = this objective was met to small degree to 5 = this objective was met to a very great degree), Calderon (2013) found this indirect measure to be related to direct measures of student learning (i.e., ratings by field instructors and an objective standardized test). This suggests that students’ perceptions of achievement of objectives were related to their actual achievements/learning. This finding, however, was not replicated with a different cohort in the same study. That is, there was a discrepancy between this cohort’s perceptions of achievement and actual achievement, as measured by indirect and direct instruments respectively. Specifically, their perceptions of achievement were higher than their actual levels of achievement. Further, the perceptions of their performance were higher than the perceptions of the cohort whose actual performance was superior. Essentially, they did not know what they did not know. To put it another way, the perceptions of achievement of the cohort whose actual performance was higher were not high enough.

These results lend credence to the notion that indirect measures tap a different construct than actual performance. Perceptions of achievement, as reflected in indirect measures, may be more related to student satisfaction than to actual learning or acquisition of competencies. Indirect measures, therefore, may hold valuable information regarding the educational experience, but not the acquisition of knowledge and skills. As Calderon (2013) suggests, future research should attempt to identify the factors involved in students’ learning experiences to better understand the relationships between such experiences and actual learning.  

References
Calderon, O. (2013). Direct and indirect measures of learning outcomes in an MSW program: What do   we actually measure?.  Journal of Social Work Education, 49(3), 408-419.
Fortune, A.E., Lee, M., & Cavazos, A. (2005). Achievement motivation and outcome in social work field education. Journal of Social Work Education, 41, 115-129.
Holden, G., Anastas, I., & Meenaghan, T. (2003). Determining attainment of the EPAS foundation program objectives: Evidence for the use of self-efficacy as an outcome. Journal of Social Wo.-k Education, 39, 425-440.
Holden, G., Barker, K., Rosenberg, G., & Onghena, P. (2008). The Evaluation Df Self-Efficacy Scale for assessing progress toward CSWE accreditation related objectives: A replication. Research on Social Work Practice, 18, 42—46.
Price, B. A., & Randall, C. H. (2008). Assessing learning outcomes in quantitative courses: Using embedded questions for direct assessment. Journal of Education for Business, 83(5), 288-294.

3 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. I completely agree. As a future physician, I will be working hands on with patients regardless of the specialty I choose. I feel too much of school is focused on sitting and learning in front of a screen and then measured by multiple choice exams. All forms of education, but especially medical should be more focused on the hands on aspect of learning and less on the multiple choice aspect.

    For example, I want to be a surgeon and I feel the most helpful aspect of medical school so far was working in the anatomy lab because I was able to physically touch and see what I was dealing with. My performance in the lab outweighed my performance on multiple choice portions of the same exam every time. While multiple choice exams are part of school, they are not realistic in the actual day to day job of a physician. Once I graduate from medical school, I will never take a multiple choice exam to help heal a patient, but I will use the hands on skills I learned in the anatomy lab, OMM lab, and PCS practicals.

    ReplyDelete
  3. Effectiveness of teaching, learning,retaining and testing have always been very difficult to measure. Although new paradigms are needed, I have seen merit in the broad combination of multiple methods (indirect and actual each play a role). Learning can be fun and difficult at the same time......Learning is an ongoing, longitudinal process that each individual develops (personal method) as he/she matures.

    ReplyDelete