Faculty training to support empirically driven course design potentially represents one of the most scalable options available for improving student outcomes at an institution. One strategy to assess effectiveness related to institutionally supported training has been to consider Student Evaluation of Educational Quality (SEEQ) surveys in courses taught by faculty who participated in the training program. For this reason, the UMBC Report Exchange (REX) team has recently released the SEEQs and Other Course Metrics By Training report for the campus community to look further into these data on their own. For those who are unfamiliar with the data warehouse and would like to learn more, please consult this FAQ article to learn how to gain access.

There are two simultaneous advantages of effective training: 1) the better courses are designed, the greater the likelihood unit-level objectives align back to course-level goals and ultimately to institutional functional competencies, and therefore we would expect improved outcomes of various measures, and  2) the better enterprise tools are leveraged through intentional design, the more meaningful the signal – metadata – from activity within a course, and in turn, the more precise predictive modeling might be to inform behavioral nudging.

Similar to prior explorations, this current analysis shows PIVOT courses in terms of average SEEQs and student behaviors in courses as “signal” (i.e., tool interactions). Notably, this trend held when looking at individual instructors’ pre/post training. Overall, there is a statistically-significant SEEQ gain for courses instructed by a PIVOT participant (.09; p<.001), as illustrated above in Figure 1.

Figure 1: Mean Distribution of Bb Course SEEQs, by Term and PIVOT Participation

The difference in averages is even more pronounced when looking at the Fall 2020 and Spring 2021 terms separately, where we see an initial gain of approximately .06 in the Fall, followed by an even greater increase to .11 by the Spring. One way to interpret this increase is the faculty who participated in training began applying new techniques learned through PIVOT in the Fall and then were able to more fully apply these approaches as they honed their pedagogy in the Spring term.

Other treatments, including the Alternate Delivery Program (ADP), which informed PIVOT, as well as ongoing programming in the form of one-time training and webinars, do not appear to have the same positive, measurable impact on SEEQs. Including SEEQs in a model with PIVOT, ADP, and myUMBC training event data shows no significant relationship with SEEQs and either ADP or myUMBC-documented training. In other words, for non-PIVOT trainings, there’s not a similar discernable lift.

PIVOT and ADP courses, on the other hand, tend to also have higher DFW rates (an uncontrolled mean difference of 2% and 2.2%, respectively). However, analysis to address the role of course design using propensity score matching indicates the relationship between training and DFWs is not statistically significant and therefore the increased prevalence of this negative outcome appears to be a function of course design rather than training (i.e., how tools are used, rather than the actual treatment).

Although the methodologies do not control for instructor-level effects, like how one teaches or interacts with students, since both PIVOT and non-PIVOT trained faculty were exposed equally to the pandemic discontinuity – i.e., as a natural experiment there was simultaneous disruption – subsequent mean SEEQ differences are accurate measurements of treatment effects. In other words, maybe the training helped, or maybe PIVOT instructors, on average, contemporaneously and inexplicably earned higher mean SEEQ values due to another yet-to-be-determined variable. However, in the notable absence of such a separate, identifiable catalyst, the mean differences are attributable to the training and the relationship appears causal. Moving forward, DoIT plans to continue investigating PIVOT’s potential impacts on students' learning and engagement as well as faculty satisfaction and course design.

~ by Tom Penniston
]]>

Faculty training to support empirically driven course design potentially represents one of the most scalable options available for improving student outcomes at an institution. One strategy to...

https://doit.umbc.edu/post/111257/ https://beta.my.umbc.edu/api/v0/pixel/news/111257/guest@my.umbc.edu/f231dcdb04a97568b0cb18cb5a25bde0/api/pixel analytics online-learning penniston pivot seeq su2021 training Instructional Technology https://beta.my.umbc.edu/groups/instructional-technology https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xsmall.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/original.png?1446126703 https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xxlarge.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xlarge.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/large.png?1446126703 https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/medium.png?1446126703 https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/small.png?1446126703 https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xsmall.png?1446126703 https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xxsmall.png?1446126703 Instructional Technology https://assets3-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/xxlarge.jpg?1627309785 https://assets1-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/xlarge.jpg?1627309785 https://assets4-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/large.jpg?1627309785 https://assets3-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/medium.jpg?1627309785 https://assets3-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/small.jpg?1627309785 https://assets4-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/xsmall.jpg?1627309785 https://assets3-beta.my.umbc.edu/system/shared/thumbnails/news/000/111/257/1975e98dceaf38deee4cda7b0e2de977/xxsmall.jpg?1627309785 6 0 true Thu, 19 Aug 2021 15:58:34 -0400 Thu, 19 Aug 2021 16:47:27 -0400 Training Associated with Higher SEEQs & Course Interactions There is a statistically significant, positive relationship between a faculty member completing the Planning Instructional Variety in Online Teaching (PIVOT) training and elevated course-level average values on Student Evaluation of Educational Quality (SEEQ) surveys (p<.001). For Fall 2020, SEEQs increased by about .08 for those who completed the training when compared with classes taught by instructors who did not (4.354 on a scale of 1-5). Courses taught by PIVOT-trained instructors also have increased Bb interactions, which are both indicative of improved engagement, and can also be leveraged for more precise predictive modeling to inform student outreach. DFW rates for PIVOT-instructed courses did not increase when compared with the pre-pandemic baseline.

DoIT’s ’s Instructional Technology team launched the PIVOT initiative during the Spring 2020 semester to support the needs of student, faculty, and staff thrust into fully online learning due to COVID-19. There have been 650 total participants in the PIVOT training, with 144 individuals completing the Live track, and another 283 completing the more extensive PIVOT Plus programming. Past survey evaluation has indicated students are very content with courses taught by PIVOT-trained faculty. The current analysis is based on a sample of 6,768 Blackboard (Bb) courses, beginning Fall 2019 through Fall 2020.

By Tom Penniston

]]>

There is a statistically significant, positive relationship between a faculty member completing the Planning Instructional Variety in Online Teaching (PIVOT) training and elevated course-level...

https://doit.umbc.edu/news/?id=98871 https://beta.my.umbc.edu/api/v0/pixel/news/98871/guest@my.umbc.edu/43354770e7881d197431a2dad743bb8f/api/pixel analytics online-learning penniston pivot seeq training Instructional Technology https://beta.my.umbc.edu/groups/instructional-technology https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xsmall.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/original.png?1446126703 https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xxlarge.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xlarge.png?1446126703 https://assets1-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/large.png?1446126703 https://assets4-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/medium.png?1446126703 https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/small.png?1446126703 https://assets2-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xsmall.png?1446126703 https://assets3-beta.my.umbc.edu/system/shared/avatars/groups/000/001/164/dec3b026b81ee6d890a8f82f75c94a2e/xxsmall.png?1446126703 Instructional Technology https://assets1-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/xxlarge.jpg?1612200064 https://assets1-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/xlarge.jpg?1612200064 https://assets1-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/large.jpg?1612200064 https://assets3-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/medium.jpg?1612200064 https://assets2-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/small.jpg?1612200064 https://assets1-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/xsmall.jpg?1612200064 https://assets2-beta.my.umbc.edu/system/shared/thumbnails/news/000/098/871/f2adb5ad1aa4c170f31940d55e426e0d/xxsmall.jpg?1612200064 5 0 true Tue, 02 Feb 2021 14:02:41 -0500 Tue, 08 Mar 2022 10:07:10 -0500