Improving Sheridan’s student chatbot
Improving Sheridan’s student chatbot
Role
UX Researcher
Content Designer
Platform Associate
Duration
4 months
Tools
Figma
ServiceNow
Comm100
Team
Me!
Farwa Babar
Christopher Kovacs
Project Overview
Project Overview
Chatbot Improvement
Chatbot Improvement
Chatbot Improvement
Every semester, thousands of students rely on the student chatbot for important school information. We focused on making answers clearer, more accurate, and easier to find.
Every semester, thousands of students rely on the student chatbot for important school information. We focused on making answers clearer, more accurate, and easier to find.
PROBLEM SPACE
PROBLEM SPACE
Questions Were Left Unanswered
Questions Were Left Unanswered
By reviewing past student logs and testing, we identified 5 core areas for improvement, based on common student frustrations and frequently unanswered high-traffic questions.
By reviewing past student logs and testing, we identified 5 core areas for improvement, based on common student frustrations and frequently unanswered high-traffic questions.
By reviewing past student logs and testing, we identified 5 core areas for improvement, based on common student frustrations and frequently unanswered high-traffic questions.
Student fails to receive an answer from the previous
comm100 chatbot interface.
Student fails to receive an answer from the previous
comm100 chatbot interface.
Student fails to receive an answer from the previous comm100 chatbot interface.
Main problem areas
Main problem areas
Main problem areas
01
01
Information gaps in topic coverage causing important prompts being left unanswered.
Information gaps in topic coverage causing important prompts being left unanswered.
"I lost my semester fees through an online scam and I need a financial aid” (left unanswered).
"I lost my semester fees through an online scam and I need a financial aid” (left unanswered).
— Student query log
— Student query log
02
02
Some existing topics overlapped, competing for the same questions.
Some existing topics overlapped,
competing for the same questions.
Mental health support topics were combined as one answer even though international, domestic and general counselling were separate resources.
Mental health support topics were combined as one answer even though international, domestic and general counselling were separate resources.
03
03
Some topics weren't being triggered properly by student phrasing.
Some topics weren't being triggered properly by student phrasing.
Many students were asking for graduation letters but phrased it as a completion letter and were left unanswered.
Many students were asking for graduation letters but phrased it as a completion letter and were left unanswered.
— Student query logs
— Student query logs
04
04
There is no fallback message or response for time sensitive related topics.
There is no fallback message or response for time sensitive related topics.
“I never got my class schedule and no emails, no communication at all” (left unanswered).
“I never got my class schedule and no emails, no communication at all” (left unanswered).
— Student query log
— Student query log
05
05
There is a general misunderstanding among students regarding the chatbot’s functionality which can lead to more frustration.
There is a general misunderstanding among students regarding the chatbot’s functionality which can lead to more frustration.
Current chatbot provided no context on how to prompt.
Current chatbot provided no context on how to prompt.
Key Improvements
Key Improvements
Key Improvements
Greeting Message
Greeting Message
Before: The sign-in form was a pain point for students because it was tedious, and they didn’t understand how their information would be used.
After: Replaced manual input with single sign-in (SSO) and created a linked chatbot guide to guide students on prompting.
Before: The sign-in form was a pain point for students because it was tedious, and they didn’t understand how their information would be used.
After: Replaced manual input with single sign-in (SSO) and created a linked chatbot guide to guide students on prompting.
Before
Before
After
After
Conversation Matching
Conversation Matching
Added more diverse phrasing (utterances) based on actual student quotes. This triggered more relevant conversation topics based on how students actually phrase their queries.
Added more diverse phrasing (utterances) based on actual student quotes. This triggered more relevant conversation topics based on how students actually phrase their queries.
Ending Message
Ending Message
Before: Mandatory input fields prevented students from giving feedback.
After: Shortened ending message is now tailored for prospective students, enrolled students, and staff.
Before: Mandatory input fields prevented students from giving feedback.
After: Shortened ending message is now tailored for prospective students, enrolled students, and staff.
Before
Before
After
After
Added New Conversation Topics (Intents)
Added New Conversation Topics (Intents)
We identified 8 topics that needed to be added to the chatbot based on
high-traffic conversations that didn't receive an answer.
We identified 8 topics that needed to be added to the chatbot based on
high-traffic conversations that didn't receive an answer.
Obtaining
Log In Details
Obtaining
Log In Details
Leave of
Absence
Leave of
Absence
Emergency
Fund
Emergency
Fund
How do I receive my
scholarship
How do I receive my
scholarship
Receiving
Diploma
Receiving
Diploma
Failed
Course
Failed
Course
Application
Assistance
Application
Assistance
Student
Advisement
Student
Advisement
Our process
Our process
Learning About Conversational Design
Learning About Conversational Design
This was my first time working with a chatbot, and I wanted to understand the space and existing industry standards before starting our own process.
This was my first time working with a chatbot, and I wanted to understand the space and existing industry standards before starting our own process.
Competitive Analysis
Competitive Analysis
We went through chatbots across various industries (including school platforms) to understand common patterns and potential pain points.
Insight: Helped us understand real time chatbot interactions and gave us a benchmark before designing.
We went through chatbots across various industries (including school platforms) to understand common patterns and potential pain points.
Insight: Helped us understand real time chatbot interactions and gave us a benchmark before designing.
Google Certificate
Google Certificate
Gained a foundational understanding of industry standard chatbot principles things like persona, tone, error handling, and conversational flow structure.
Insight: This gave our team a shared vocabulary and framework to
design from.
Gained a foundational understanding of industry standard chatbot principles things like persona, tone, error handling, and conversational flow structure.
Insight: This gave our team a shared vocabulary and framework to
design from.
ServiceNow for chatbots
ServiceNow for chatbots
During the time, the chatbot was transitioning to a new software. Our team learned about specific technicalities of the virtual agent on the platform.
Insight: Helped us connect the design principles we'd learned to what was actually possible to build on the platform.
During the time, the chatbot was transitioning to a new software. Our team learned about specific technicalities of the virtual agent on the platform.
Insight: Helped us connect the design principles we'd learned to what was actually possible to build on the platform.
Chatbot Testing
Chatbot Testing
We focused our chatbot testing on conversation flow by looking through real student queries from chatbot logs especially those in trending and high-traffic topics.
We tested based on:
We focused our chatbot testing on conversation flow by looking through real student queries from chatbot logs especially those in trending and high-traffic topics.
We tested based on:
Prompting with informal language and with different student phrasing
Prompting with informal language and with different student phrasing
Accuracy of responses to common and high traffic student questions
Accuracy of responses to common and high traffic student questions
Tone and clarity of messages
Tone and clarity of messages
How well student intents were detected and routed
How well student intents were detected and routed
Identifying Drop-off Points
Using real student query logs, we identified the main stages of the chatbot user flow. One of the most common friction points was the chatbot providing incorrect answers, which caused students to exit the chat before finding a resolution.
User Flow Analysis Insights
User Flow Analysis Insights
Greeting
Conversation
Ending
Identifying Drop-off Points
Using real student query logs, we identified the main stages of the chatbot user flow. One of the most common friction points was the chatbot providing incorrect answers, which caused students to exit the chat before finding a resolution.
Lessons Learned
Lessons Learned
1
1
Conversational design: Learned how to write clearer, more natural chatbot responses that feel human and easy to follow.
Conversational design: Learned how to write clearer, more natural chatbot responses that feel human and easy to follow.
2
2
Working with stakeholders: Learned how to handle platform limitations, time constraints, and different team priorities. Adjusted our goals based on what was realistically possible.
Working with stakeholders: Learned how to handle platform limitations, time constraints, and different team priorities. Adjusted our goals based on what was realistically possible.
Next STEPS
Next STEPS
I’d measure the impact of the improvements and adjust based on looking at several chatbot KPIs that are already available in the platform’s analytics such as:
I’d measure the impact of the improvements and adjust based on looking at several chatbot KPIs that are already available in the platform’s analytics such as:
Chatbot resolution rate
Chatbot resolution rate
How often the chatbot answers questions without needing
a live agent
How often the chatbot answers questions without needing
a live agent
Conversation drop-off rate
Conversation drop-off rate
Where students leave the conversation
Where students leave the conversation
Conversation completion rate
Conversation completion rate
Whether students reach the end of a chat flow
Whether students reach the end of a chat flow
Topic usage
Topic usage
Which topics students use most and least
Which topics students use most and least
