Improving Sheridan’s student chatbot
Improving Sheridan’s student chatbot
Role
UX Researcher
Content Designer
Platform Associate
Duration
4 months
Tools
Figma
ServiceNow
Comm100
Team
Me!
Farwa Babar
Christopher Kovacs
Overview
Overview
Chatbot Improvement for easier student information access
Chatbot Improvement for easier student information access
Every semester, thousands of students rely on the student chatbot for important school information. We focused on making answers clearer, more accurate, and easier to find for the upcoming school semester.
Every semester, thousands of students rely on the student chatbot for important school information. We focused on making answers clearer, more accurate, and easier to find for the upcoming school semester.
PROBLEM
PROBLEM
Student questions were left unanswered
Student questions were left unanswered
By reviewing past student logs and testing, my teammate and I identified 5 core areas for improvement, based on common student frustrations and frequently unanswered high-traffic questions.
By reviewing past student logs and testing, my teammate and I identified 5 core areas for improvement, based on common student frustrations and frequently unanswered high-traffic questions.
Student fails to receive an answer from the previous
comm100 chatbot interface.
Student fails to receive an answer from the previous
comm100 chatbot interface.
Student fails to receive an answer from the previous comm100 chatbot interface.
Main research insights: 5 key pain point themes
Main research insights: 5 Key pain point themes
01
01
Information gaps in topic coverage causing important prompts being left unanswered.
Information gaps in topic coverage causing important prompts being left unanswered.
"I lost my semester fees through an online scam and I need a financial aid” (left unanswered).
"I lost my semester fees through an online scam and I need a financial aid” (left unanswered).
— Student query log
— Student query log
02
02
Some existing topics overlapped, competing for the same questions.
Some existing topics overlapped,
competing for the same questions.
Mental health support topics were combined as one answer even though international, domestic and general counselling were separate resources.
Mental health support topics were combined as one answer even though international, domestic and general counselling were separate resources.
03
03
Some topics weren't being triggered properly by student phrasing.
Some topics weren't being triggered properly by student phrasing.
Many students were asking for graduation letters but phrased it as a completion letter and were left unanswered.
Many students were asking for graduation letters but phrased it as a completion letter and were left unanswered.
— Student query logs
— Student query logs
04
04
There is no fallback message or response for time sensitive related topics.
There is no fallback message or response for time sensitive related topics.
“I never got my class schedule and no emails, no communication at all” (left unanswered).
“I never got my class schedule and no emails, no communication at all” (left unanswered).
— Student query log
— Student query log
05
05
There is a general misunderstanding among students regarding the chatbot’s functionality which can lead to more frustration.
There is a general misunderstanding among students regarding the chatbot’s functionality which can lead to more frustration.
Current chatbot provided no context on how to prompt.
Current chatbot provided no context on how to prompt.
Solution
Shortened sign-in process
Before: The sign-in form was a pain point for students because it was tedious, and they didn’t understand how their information would be used.
After: Replaced manual input with single sign-on (SSO). Additionally, I wrote and proposed a linked chatbot guide to guide students on prompting.
Before: The sign-in form was a pain point for students because it was tedious, and they didn’t understand how their information would be used.
After: Replaced manual input with single sign-on (SSO). Additionally, I wrote and proposed a linked chatbot guide to guide students on prompting.
Before
Before
After
After
Increasing conversation matches with student phrasing
Added more diverse phrasing (utterances) based on actual student quotes. This triggered more relevant conversation topics based on how students actually phrase their queries.
Added more diverse phrasing (utterances) based on actual student quotes. This triggered more relevant conversation topics based on how students actually phrase their queries.
Shortened ending message
Before: Mandatory input fields prevented students from giving feedback.
After: Shortened ending message is now tailored for prospective students, enrolled students, and staff. (The new feedback system wasn't fully implemented).
Before: Mandatory input fields prevented students from giving feedback.
After: Shortened ending message is now tailored for prospective students, enrolled students, and staff. (The new feedback system wasn't fully implemented).
Before
Before
After
After
Added New Conversation Topics (Intents)
Added New Conversation Topics (Intents)
My teamate and I identified 8 topics that needed to be added to the chatbot based on high-traffic conversations that didn't receive an answer.
My teamate and I identified 8 topics that needed to be added to the chatbot based on high-traffic conversations that didn't receive an answer.
Obtaining
Log In Details
Obtaining
Log In Details
Leave of
Absence
Leave of
Absence
Emergency
Fund
Emergency
Fund
How do I receive my
scholarship
How do I receive my
scholarship
Receiving
Diploma
Receiving
Diploma
Failed
Course
Failed
Course
Application
Assistance
Application
Assistance
Student
Advisement
Student
Advisement
process
process
Learning about conversational design
Learning about conversational design
This was my first time working with a chatbot, and I wanted to understand the space and existing industry standards before starting our own process.
This was my first time working with a chatbot, and I wanted to understand the space and existing industry standards before starting our own process.
Competitive Analysis
Competitive Analysis
We went through chatbots across various industries (including school platforms) to understand common patterns and potential pain points.
Insight: Helped us understand real time chatbot interactions and gave us a benchmark before designing.
We went through chatbots across various industries (including school platforms) to understand common patterns and potential pain points.
Insight: Helped us understand real time chatbot interactions and gave us a benchmark before designing.
Google Certificate
Google Certificate
Gained a foundational understanding of industry standard chatbot principles things like persona, tone, error handling, and conversational flow structure.
Insight: This gave our team a shared vocabulary and framework to
design from.
Gained a foundational understanding of industry standard chatbot principles things like persona, tone, error handling, and conversational flow structure.
Insight: This gave our team a shared vocabulary and framework to
design from.
ServiceNow for chatbots
ServiceNow for chatbots
During the time, the chatbot was transitioning to a new software. Our team learned about specific technicalities of the virtual agent on the platform.
Insight: Helped us connect the design principles we'd learned to what was actually possible to build on the platform.
During the time, the chatbot was transitioning to a new software. Our team learned about specific technicalities of the virtual agent on the platform.
Insight: Helped us connect the design principles we'd learned to what was actually possible to build on the platform.
Chatbot testing with real student queries
Chatbot testing with real student queries
I focused our chatbot testing on conversation flow by looking through real student queries from chatbot logs, especially those in trending and high-traffic topics. I tested based on these factors:
I focused our chatbot testing on conversation flow by looking through real student queries from chatbot logs, especially those in trending and high-traffic topics. I tested based on these factors:
Prompting with informal language and with different student phrasing
Prompting with informal language and with different student phrasing
Accuracy of responses to common and high traffic student questions
Accuracy of responses to common and high traffic student questions
Tone and clarity of messages
Tone and clarity of messages
How well student intents were detected and routed
How well student intents were detected and routed
Identifying student conversation drop-off points
Using real student query logs, we identified the main stages of the chatbot user flow. One of the most common friction points was the chatbot providing incorrect answers, which caused students to exit the chat before finding a resolution.
User Flow Analysis Insights
User Flow Analysis Insights
Greeting
Conversation
Ending
Identifying Drop-off Points
Using real student query logs, we identified the main stages of the chatbot user flow. One of the most common friction points was the chatbot providing incorrect answers, which caused students to exit the chat before finding a resolution.
Lessons Learned
Lessons Learned
1
1
Conversational design: It was my first time taking on a content-focused role, which helped me realize how important copy is within service design. I was able to focus on writing accessible copy that actually made sense and helped people.
Conversational design: It was my first time taking on a content-focused role, which helped me realize how important copy is within service design. I was able to focus on writing accessible copy that actually made sense and helped people.
Conversational design: It was my first time taking on a content-focused role, which helped me realize how important copy is within service design. I was able to focus on writing accessible copy that actually made sense and helped people.
2
2
Working with stakeholders: As my first real-world work experience, my co-op was a huge learning curve in dealing with platform limitations, tight timelines, and competing team priorities. When our initial plans weren't possible, it was important for us adapt quickly and reprioritize scope so we could still hit the main goals for everyone involved.
Working with stakeholders: As my first real-world work experience, my co-op was a huge learning curve in dealing with platform limitations, tight timelines, and competing team priorities. When our initial plans weren't possible, it was important for us adapt quickly and reprioritize scope so we could still hit the main goals for everyone involved.
Working with stakeholders: As my first real-world work experience, my co-op was a huge learning curve in dealing with platform limitations, tight timelines, and competing team priorities. When our initial plans weren't possible, I learned how to adapt and reprioritize so we could still hit the main goals for everyone involved.
3
Designing for my own demographic: As a student myself, I could draw on firsthand experiences to identify pain points and inform design decisions, while also being mindful to validate my assumptions with research and balance personal perspective
with fact.
Designing for my own demographic: As a student myself, I could draw on firsthand experiences to identify pain points and inform design decisions, while also being mindful to validate my assumptions with research and balance personal perspective
with fact.
Working with stakeholders: As my first real-world work experience, my co-op was a huge learning curve in dealing with platform limitations, tight timelines, and competing team priorities. When our initial plans weren't possible, I learned how to adapt and reprioritize so we could still hit the main goals for everyone involved.
Next STEPS
Next STEPS
I’d measure the impact of the improvements and adjust based on looking at several chatbot KPIs that are already available in the platform’s analytics such as:
I’d measure the impact of the improvements and adjust based on looking at several chatbot KPIs that are already available in the platform’s analytics such as:
I’d measure the impact of the improvements and adjust based on looking at several chatbot KPIs that are already available in the platform’s analytics
such as:
Chatbot resolution rate
Chatbot resolution rate
How often the chatbot answers questions without needing
a live agent
How often the chatbot answers questions without needing
a live agent
Conversation drop-off rate
Conversation drop-off rate
Where students leave the conversation
Where students leave the conversation
Conversation completion rate
Conversation completion rate
Whether students reach the end of a chat flow
Whether students reach the end of a chat flow
Topic usage
Topic usage
Which topics students use most and least
Which topics students use most and least
CLAUDE MONET
Thank you for reading!
Click The Card
FP
PAINTING BY
CLAUDE MONET
Thank you for reading!
Click The Card
FP
PAINTING BY