Tuesday, September 16, 2014 at 1:30 p.m.
Room 310, Neville Scarfe Building, 2125 Main Mall
Title: “Multilevel Validity: Assessing the Validity of Program-level Inferences Based on Aggregate Student Perception about Their General Learning.”
Supervisor: Dr. Kadriye Ercikan (MERM)
Supervisory Committee: Dr. Bruno Zumbo (MERM) & Dr. Charles Ungerleider (Educational Studies)
University Examiners: Dr. Amy Scott Metcalf (Educational Studies) & Dr. Sterrett H. Mercer (School Psychology)
External Examiner: Dr. Kurt Geisinger (The University of Nebraska – Lincoln)
ABSTRACT
Aggregate survey results are commonly used by universities in Canada to compare effective educational practices across program majors within a university and between equivalent majors across campuses. Despite this recurrent practice, many researchers neglect to examine the multilevel validity of inferences made from program-level responses. This study illustrates the importance of determining the multilevel validity of program-level inferences prior to making conclusions based on survey data.
Survey responses regarding student perceptions about their general learning outcomes and ratings about the learning environment were collected from the National Survey of Student Engagement (NSSE) and the Undergraduate Experience Survey (UES). The analytic procedures used in this study included two-level exploratory multilevel factor analyses (MFA) and three statistical approaches to determine the appropriateness of aggregation: analysis of variance (ANOVA), the within and between analysis (WABA), and the unconditional multilevel model. Multilevel regression models were applied to survey data to examine the relationships of program-level characteristics with perceived student learning outcomes.
The results led to four conclusions regarding the use of student survey results aggregated to the program level. First, results from the MFA revealed that the multilevel structure of items regarding perceived learning were consistent across the student and program levels for most samples, but the multilevel structure of items regarding the learning environment was not supported at the program level. Second, results from the ANOVA and unconditional multilevel models indicated that aggregation to the program level for perceived learning was statistically appropriate for three out of the four study samples; however, WABA results indicated that aggregation to a level lower than the program major was more suitable. Aggregation to the program level was not supported for any the learning environment scales across all three procedures. Third, aggregation was variable dependent as demonstrated by lower levels of within-program agreement on ratings of the learning environment, but larger levels of agreement with perceived learning outcomes. Finally, student-level perceptions about learning were partially influenced by student- and program-level characteristics; however, program means were not highly reliable and results did not support making program comparisons. Implications for educational research and recommendations for further research were discussed.