Earlier this semester, Taylor & Francis published Data Analytics & Adaptive Learning: Research Perspectives (2023), which includes a UMBC case study about Dr. Tara Carpenter's use of "spaced practice" in her high-enrollment, introductory course in general chemistry. 

In "Banking on adaptive questions to nudge student responsibility for learning in general chemistry," Carpenter and DoIT staff co-authors John Fritz and Tom Penniston describe Carpenter's course redesign efforts using learning science, analytics, and adaptive learning technology to nudge students into deeper, long-term learning, application and retention instead of cramming and short-term memorization to prepare for her exams.

The campus community can view the chapter here (UMBC account login req'd) or view Carpenter's 2022 presentation of related findings as part of the Learning Analytics Community of Practice.

By John Fritz

DoIT has a new addition to our suite of data visualizations supporting course engagement/instruction and teaching: The Blackboard course "waterfall" report (UMBC vpn req'd if using off-campus). 

As illustrated below, the waterfall dashboard summarizes student interactions within our Blackboard Learning Management System (LMS) on a week-by-week basis. Each row represents a student and each column represents a week in the semester, with the density of an individual cell color representing the student's time spent in the Blackboard course that week. Each course waterfall is organized by the student's final course grade on the left.

To encourage, identify, and share effective practices in the use of data to inform teaching and learning, the Provost's Office has once again awarded learning analytics mini grants to six faculty and two staff for the 2023-24 academic year (the three new grants and three renewals are described below):

Katie Birger (Public Health): "Analyzing Factors Affecting Student Success in Public Health Courses." This new proposal aims to standardize PBHL 300: Research Methods in Health, to address observed inconsistencies that potentially impact students' success in the subsequent PBHL 420 course (Epidemiology). Birger plans to investigate these discrepancies by analyzing data from Blackboard sections of both courses, to identify variables such as instructor differences, textbook usage, attendance, homework submission, and access to Panopto recordings that might affect student success rates. The findings will help standardize effective learning activities and enhance the learning assessment plan. To do so, Birger aims to implement and test evidence-based curriculum changes in PBHL 300 and then monitor student performance in PBHL 420. The overall goal is to improve student knowledge and skills in public health, better preparing them for the workforce or graduate education and enhancing the continuous improvement of the Public Health curriculum.

Joby Taylorand Emily Passera (Shriver Center): "Knowing the student, knowing the staff: How a reciprocal relationship alongside data-informed insights improves student outcomes in community engaged learning."This new proposal will delve into the Shriver Center's student data to help improve practitioners' understanding of what interventions and environments best scaffold success. Additionally, the researchers will investigate the potential benefits of incentivization through micro-credentialing. Through these approaches, Taylor and Passera hope to support both student growth and improve their pathways to academic success.

Rebecca Williams(CS/EE): "Improving Student Visual Literacy & Critical Thinking in CMSC 436/636 (Data Visualization)."This new proposal focusing on course redesign as an intervention to address student misconceptions about visualizations, such as the misunderstanding that simple, easy-to-read, charts are always effective and the belief that data visualizations are entirely objective. Williams' proposed interventions include having students search for, annotate, and critique visualization examples, to supplement those she provides (and models) herself. As such, students are more likely to develop these skills and the process is likely to lead to the long-term retention of these same skills.

Sarah Bass and Tara Carpenter(Chemistry): "Nudging CHEM 101 Students with myUMBC Personal Posts."This is a renewal proposal building off of Carpenter's prior efforts to advance student metacognition and responsibility for learning as well as conducting focus groups about student motivation in moving from general to organic chemistry. Specifically, Bass and Carpenter will use myUMBC's "personal post" functionality to nudge students in their initial exposure to and use of course resources for practice and exam preparation in CHEM 101 "Principles of Chemistry I," which they are co-teaching this semester. DoIT staff have also worked with Math and Psychology faculty to use myUMBC personal posts in this way to scale course-based nudging of students. As such, Bass and Carpenter will customize messaging focused on nudging students about their engagement and preparation for key course milestones and assessments.

Karen Chen(Information Systems): "Learning Analytics by Students for Students." In this renewal proposal building on her 9/30/22 LA community of practice presentation of the same name, Chen intends to further explore student agency and literacy in the development and use of learning analytics. She's particularly interested in leveraging de-identified UMBC data to inform student analytics projects in two data science courses, IS 428 "Data Mining Techniques and Applications" and IS 733 "Data Mining" in Fall 2023. These include the use of "Spaced Practice" in General Chemistry courses as well as student use of and anonymous feedback about the myUMBC Check My Activity (CMA) feedback tool for students.

Cody Goolsby-Cole(Physics): "How Does Student Performance on Ungraded Practice Questions Correlate to Performance on Exams?"In this renewal proposal, Goolsby-Cole seeks to increase student use of a growing CNMS practice of using online questions banks to prepare for high-stakes exams. Specifically, he has used ungraded practice questions, in terms of points contributing to a final grade, but including correct and incorrect answers displayed to students. Students in his PHYS 122 "Introductory Physics II," who use the practice questions clearly perform better on unit, midterm and final exams, but he'd like to see more students do so. He plans to continue development of his practice questions, but also do more proactive nudging with myUMBC personal posts (similar to Carpenter and Bass in Chemistry described above) to see if and how more students might use and benefit from the practice question bank.

"On behalf of Interim Provost Dauwalder, we're pleased to continue offering these learning analytics mini grants to help faculty use institutional data that might inform their course designs and instruction to the benefit of our students," says Robert Carpenter, associate provost and deputy CIO. "We are excited by these interesting projects and look forward to what the faculty may find."

These one-year learning analytics awards are renewable pending receipt of a final report, paper submitted for publication or conference presentation. In addition to use of UMBC's Report Exchange (REX) data warehouse, and a Tableau "viewer" license, faculty recipients can consult with staff from Analytics and Business Intelligence, Instructional Technology and Institutional Research and Decision support (IRADS). 

For more information about this year's workshops, speakers and another learning analytics mini-grant call for proposals to be announced in Spring 24, please visit doit.umbc.edu/analytics/community.

By John Fritz

• Image "Alt" tag: "2023-24 UMBC Learning Analytics Mini Grant Recipients Katie Birger, Emily Passera, Joby Taylor, Rebecca Williams, Sarah Bass, Tara Carpenter, Cody Goolsby-Cole, and Karen Chen"

To encourage, identify, and exchange effective practice in the use of data to inform teaching and learning, the Provost's Office has once again awarded learning analytics mini grants to three faculty for the 2022-23 academic year (the three faculty and their proposed topics are included below):

Tara Carpenter (Chemistry): "Modeling Effective Learning Strategies in Introductory Chemistry." This is a renewal of Carpenter's 21-22 mini grant, focused on students' understanding and use of "spaced practice" as an alternative to cramming for exams in her CHEM 102 "Principles of Chemistry II" course. Specifically, she is focused on how students can take their spaced practice "lessons learned" with them to CHEM 351 "Organic Chemistry," which requires CHEM 102. Carpenter presented her initial findings last spring, and has already offered a "spaced practice" workshop this fall for her Spring 22 CHEM 102 students currently enrolled in CHEM 351. She is also working with the FDC and DoIT to conduct a series of focus groups to better understand the challenges students face in setting up spaced practice on their own.

Karen Chen (Information Systems): "Bumpy Journey: Exploring Gateway Courses Failures and Major Switch - a Case Study with Information Systems Majors." This is a renewal of Chen's 21-22 mini grant focused on what is sometimes referred to "curricular analytics," particularly in CoEIT. She presented her initial findings at Indiana University's 2022 Learning Analytics Summit in May, but also leveraged this work to look at how she teaches data science by engaging students as creators, not just consumers, of learning analytics insights and interventions. Her presentation, "Analytics by Students for Students," was the first in UMBC's Fall 2022 data science and learning analytics series.

Cody Goolsby-Cole (Physics): "How Does Student Performance on Ungraded Practice Questions Correlate to Performance on Exams?"This is a new proposal emerging from Goolsby-Cole's participation in a grant to promote "Equity & Digital Education" by the Every Learner Everywhere Network. Building on a growing CNMS practice of using question banks and calculated question types for online exams in Blackboard, Goolsby-Cole also created practice questions that were ungraded, in terms of points contributing to a final grade, but including correct and incorrect answers displayed to students. During the first six units of his Spring 22 course, PHYS 122 "Introductory Physics II," he found students earning a 70% on the practice questions earned an average of 92% on the exam that followed. Students who did not use practice questions earned an average exam grade of 77%. He plans to build out and assess this practice environment further, including practice exams, to see if and how more students might use and benefit from it.

"On behalf of the Provost, we're pleased to continue offering these learning analytics mini grants to help faculty use institutional data that might inform their course designs and instruction to the benefit of our students," says Robert Carpenter, associate provost and deputy CIO. "We are excited by these interesting projects and look forward to what the faculty may find."

These one-year learning analytics awards are renewable pending receipt of a final report, paper submitted for publication or conference presentation. In addition to use of UMBC's Report Exchange (REX) data warehouse, and a Tableau "viewer" license, faculty recipients can consult with staff from Analytics and Business Intelligence, Instructional Technology and Institutional Research and Decision support (IRADS). 

For more information about this year's workshops, speakers and another learning analytics mini-grant call for proposals to be announced in Spring 23, please visit doit.umbc.edu/analytics/community. 

by John Fritz

For the Fall 2022 semester, DoIT's Analytics & Instructional Technology groups will continue to offer student success and learning analytics-related workshops for the UMBC community. 

All workshops will start at noon, and be available via video conference (the virtual meeting URL be announced on each myUMBC event registration site the morning of the event). To register and view more information, check the links below:

"Learning Analytics By Students For Students?" (9/30, Noon). In this session, Dr. Karen Chen, assistant professor of Information Systems, will make the case for why and how students can and should play a more active role in the design and development of learning analytics to improve student success. 

"Using Analytics to Predict & Improve Sophomore Retention" (10/14, Noon). In this session, student success colleagues Robert Carpenter, Delana Gregg and Len Mancini, will share how UMBC has worked with HelioCampus to build very accurate models that predict second year retention at the beginning of their third semester. In addition to explaining how the models work, key inputs into them, and where to find the results, they will share how we have begun to act on them. 

"Tips & Tricks for Telling Your Story with Data" (11/10, Noon). In this session, Mike Sharkey, an analytics thought leader, teacher, consultant and long-time friend of UMBC, will share his lessons learned using data, narrative and visualizations (including Tableau) to raise awareness and inspire change. 

"Mapping Student Pathways Through the Curriculum" (12/2, Noon). As institutions continue to identify barriers to student success one area to explore is how we engage faculty and departments in using learning analytics to take a fresh look, along with a greater understanding, of student pathways toward graduation. In this talk, Dr. Martha Oakley, Assoc. Vice Provost for Undergraduate Education and professor of chemistry at Indiana University Bloomington, will highlight recent developments by IU's Bloomington Assessment and Research (BAR) team to provide faculty and administrators with process maps that show how students actually navigate through curricula. 

To view prior data science and learning analytics meetings (and recordings), please visit doit.umbc.edu/analytics/community.

~ By John Fritz

Before the Spring 22 semester started, four Biological Sciences professors -- David Eisenmann, Hua Lu, Jeffrey Leips, and Kevin Omland -- reached out to DoIT to see if they could (and should) try an experiment: proctor their first semester exam in the University Center (UC) Ballroom and online (via Blackboard). 

All four have been teaching in the UC as part of the pandemic-driven retrofits for teaching designed by DoIT and managed by Events & Conference Services (ECS). Also, after teaching remotely for nearly two years, they'd come to like the benefits of online exams other faculty have discovered with Blackboard exam question pools, including randomization, item analysis and quicker student feedback. So, as more classes were offered on campus, the faculty wondered if they could leverage student laptops and mobile devices that made UMBC's pandemic pivot to online learning possible. 

"Initially, our biggest concern was power for student laptops to make it through a 50-minute class period and exam," says Omland, who reached out on behalf of the group, and has co-taught BIOL 142 "Foundations of Biology: Ecology and Evolution" with his colleague Leips for 20 years. "Then, we wondered if the campus WiFI would melt, so we thought maybe we should check with DoIT first." They were joined by colleagues Eisenmann and Lu, who co-teach BIOL 302 "Molecular and General Genetics."

After several emails exploring potential issues -- and two RT tickets to schedule DoIT support on their first exam dates -- the faculty briefed their students and conducted a "dry run" test of the UC's power and WiFi 1-2 weeks before the exams. "We appreciated that step very much," says DoIT's David Toothe, Director of Enterprise Infrastructure Solutions, who leads the AV and Desktop Support units that collaborated with the faculty. It also helped that DoIT's Networks group had just upgraded the UC WiFi as part of a larger campus wireless initiative in January.

Interestingly, each course had similarities and differences in their approach to online exams proctored in the UC:

• In addition to using Bb exam question banks, both courses enroll over 200 students, and planned to use adjacent rooms (301, 310 and 312) for overflow or so students could plug into extra power outlets provided by DoIT, if needed. They also had paper back ups of the exam, in case there were issues that could not be fixed during the test.
• BIOL 302 required the use of Respondus Lockdown Browser (not Respondus Monitor) to access and take the test, and asked students to update their browsers/software before hand. It also uses Bb Original. However, BIOL 142 used an "open note" and "open web" (but not "open neighbor") approach, assigned students to specific UC rooms, and uses Bb Ultra.

Results

During the last week of February, each course administered its first exam of the Spring 22 term. Results included the following (excerpted with permission from emails provided by Eisenmann and Omland):

BIOL 302 "Genetics" (1st SP22 exam on 2/24/22)

• No students reported Wifi connectivity issues to [instructors]
• Several students had problems with LDB but went away upon hard restart.
• ~6 students (out of 200) needed to use outlets, the rest were fine.
• The calculator in the menu bar does not appear for students using LDB on a Chromebook [a known issue DoIT has reported and documented in a related FAQ]. Otherwise no Chromebook or iPad issues.
• 2 students had problems that could not be fixed and took paper copies of a similar exam.
• For a class of [302's] size we would have used 2500 pieces of paper to print this one exam. This way we printed 10% of that number (back up paper copies and scratch paper to give out).
  
  

BIOL 142 "Foundations of Biology" (1st SP22 exam on 2/25/22)

• 4 students needed paper copies of the exam
• One student said he always has trouble connecting in rm 301, so we moved him to 312 where he seemed to have no problem.  
• 5 students used outlets in rm 312.
• The only fairly annoying problem was that one student was unable to submit her exam at the end. [Took photos of each exam page, No idea what happened]
  
  

"After this experience I say we SO need a testing center," says Eisenmann, a sentiment echoed by others on the department's faculty email group after he and others shared their results. "This is the first time I have ever given an exam to 200+ people in 20+ years, online or in-person, and not really worried about cheating. And so much more pleasant for them and us than stadium seating."

Added Omland: "I spoke with about 10 students afterward - 8 said the format was fine, but 2 said they preferred paper exams so they could cross out answers, write on the side. We really appreciated the ability to have students spread out from each other . . . and walk to any part of the room to answer student questions.  Also, two students reported that they were home with covid. Since it is an online Bb exam with open book, open web etc, we just had them take the exam at home at the same time as the other students.  Can't do that with a paper exam!"

Lu also spoke briefly with about 8 students after the 302 exam. "Only one expressed that she was not used to clicking on the screen for an exam but she did not seem to be bothered by it. All students seemed to appreciate the benefit of the format - reducing the use of paper, reducing grading labor (their own words). [We] were also very happy to see that the cheating problems could be largely eliminated/reduced with this format. As I walked around the room, I rarely saw that students next to each other had the same screen. Of course, this will depend on the pool size of a given exam."

DoIT briefly presented the Biology faculty's "experiment" at the February 28 Faculty Senate Computer Policy Committee (CPC), which had been exploring the idea of a UMBC testing center before the pandemic. The biology faculty have also been invited to share their experience at an upcoming Classroom Committee meeting this semester.

By John Fritz