OLI Supporter
Project Overview
This project details the challenges and best practices of designing courses for online learning through findings from both primary and secondary research. Our primary research focused on pedagogy, implementation, and instructional design. The team also worked closely with scholars at Carnegie Mellon to keep up with various technical updates involved in the ever-changing research space. From there, our secondary literature-based research gave us an overview of what features current online platforms are pursuing and the idealized learning outcomes students can achieve. Our synthesis pointed to instructors as our main stakeholders alongside learning engineers. In the end, both research streams proved valuable in extracting insights and ideation.
My role
UX Researcher (Team project)
Timeline
Feb.2021 - Aug 2021
Project Type
Product Design
Design methods
Double diamond method, user interviews,
affinity diagrams, personas, empathy maps, storyboards
Research Findings
Our project followed the double-diamond research method, which comprised four phases as depicted below. Research methods were described under each phase. We utilized the double diamond method to find innovation opportunities.
What are the current online learning trends?
Open Learning Initiative was chosen as the target learning platform
Free education at scale is popular
Most of the learning platforms are not built around learning science principles
Systems that track and adapt to student performance are in demand
Data analysis is growing in popularity, but currently, data collection is limited to whichever dashboard is integrated into a system
After casting a wide net of learning tools, open learning initiative (OLI) was chosen as our target learning platform due to its grounded research base and strong data collection and analysis skills. OLI uses scientifically tested pedagogy to build online learning experiences which were not found on other platforms. At the same time, evidence showed that the unique features embedded in the OLI system can generate better outcomes for learners.
How to identify our users and their needs?
Focusing on pedagogy, implementation, and instructional design, we conducted interviews and focus groups with 20 learning engineers and instructor users of OLI in total. We took contextualized approaches to provoke discussion during these sessions, and the results were promising.
Our synthesis process is summarized on the left side, where we found patterns in sentiments from instructors and builders of online learning experiences from the hundreds of data points collected in the interviews, focus groups, and think-aloud. We then delved deeper and generated meaningful insights with “how might we” statements. Human-centered solutions were created based on the pain points we interpreted from the OLI users, and are presented as pitches in the next section of the report.
Who are our users?
Passionate Dan, professor
Goals:
Check students formative performance
Give critique or enhance critical thinking
Gather data on how students interact with the program
Challenges:
Learning to understand OLI data is hard and takes too much time.
Not sure if the redesign would work better than the original version.Frequent errors in OLI that lead to misunderstanding
Doesn't get instant feedback from students through the formative assessment with an online platformCan't change the content in the course once it has been deployed.
Efficiency Cynthia, first-year professor
Goals:
Get a direct report from learning engineers to improve her course
Spend the least time exploring OLI data personally
Challenges:
Communication with learning engineers
4 subject matter experts to write a course
No time to redesign
The alignment sheet includes too much information (even page numbers in OLI course) that drives me crazy, easily making mistakes
Busy Bee Jim, learning engineer
Goals:
Support instructors design contents
Support instructors collect data
Support instructors make improvements
Challenges:
Spends too much time moving things from google docs to OLI
Need subject-matter experts with the revision of contents
Hard to schedule meetings with the professor who are busy
The user flow on the learning engineer side is troublesome.
What troubles our users in the OLI platform?
7 pain points were synthesized based on the interviews, focus groups, and think-aloud.
Data Visualization: Instructors want more easy-to-use and customizable data visualizations on student performance
Time Limitations: The amount of time that instructors spend on redesigning courses does not allow them to make the updates that are needed, using both quantitative and qualitative feedback from students
Course Refinement: There is a gap between research and practice when it comes to best practices for writing and refining courses
Coordination and Workflow Management:The communication/work flow between learning engineers and instructors is not smooth enough
Human Feedback: Instructors want to integrate more human feedback in online learning
KC Management: KC structures need occasional updates and refinement
Data Collection: Imprecise data collection can lead to skewed hypotheses about students’ learning
Ideation
We generated 5 ideas using storyboards and conducted speed dating testing of each of the idea with our users.
Idea 1: Dash +
An easy-to-use and customizable learning dashboard for instructors to better master students’ knowledge states and formative outcomes.
Evaluation: Dash+ is powerful because it meets the needs of every stakeholder. It provides both low-stakes and high-stakes assessment information. It is unique because it presents learner data in an easily digestible and customizable way. However, to implement this idea, we would need to access qualitative and formative assessment data, which could be hard to get.
Idea 2: OLI Suggest
A tool that searches previous existing Learning Objectives or Knowledge Components for instructors who are making new courses.
Evaluation: The use of OLI suggest could benefit new instructors who have difficulty generating well-designed learning objectives by showing them references to existing courses. It would also be powerful if it can automatically link the learning objectives with corresponding skills. However, it may be difficult to implement as we will include natural language processing algorithms to make it successful. Also, collecting and refining resources would require access to a variety of online learning platforms beyond our current reach.
Idea 3: MATCHIT
A collaborative course authoring alignment tool for course objectives and contents.
Evaluation: MatchIT is powerful because it leverages the learning science principle of alignment within the backward design. It is novel because it combines a modern UI with time-tested principles, allowing instructors to easily locate assessments and activities and how they tie together. If not built carefully, this could end up being a horizontal shift between spreadsheets and our UI, so we would be careful to add novel features that enhance the user experience.
Idea 4: REVIEW AL
A tool to help instructors iterate on instructional methods and Knowledge Component structures.
Evaluation: Review AL is a powerful tool since it enables instructors to verify the effectiveness of different instructional implements automatically through machine learning. It utilizes learning science by inducing procedural production rules based on the ACT-R framework. This would automate an otherwise ambiguous science of course improvement and give instructors the confidence to redefine their instructional modules. The disadvantage is that it would need to work in an interface understood by AL, which can be distinct from the interface the instructor already has.
Idea 5: MATCHIT AI
A tool for instructors to automatically check alignment between Knowledge Components and Learning Objectives.
Evaluation: AI is the best of both worlds when it comes to backward design. One major advantage we have is that the AI has already been built by Apprentice Learner, but it is not currently being used in any commercial products. This lets us leverage educational research techniques, more quickly and without doing any harm to students. It also pushes the envelope on how fast educational research can be conducted. One possible roadblock is the current limitations in problem types that are accepted by AL.