Machine Learning Peril Determination Redesign
Problem
Business Value Proposition: 10% of claims are refiled. Of those, 65% are due to users mis-triaging.
Customer Value Proposition: Current peril experience is long, complicated, and clunky. When on mobile, you can't even see all the options together.
Proposed Solution
Through a new UI experience and a peril determination ML model, we are testing to solve for these problems.
Testing Metrics:
- Refile rate - Mis triaging of a a peril leads to refile of a new claim which adds additional time and effort to complete a claim.
- Initiations
Granular metrics:
- Repair / replacement rates.
- How often users edit peril predicted by model.
- CLOE - customer level of effort. Improve customer experience by reducing the number of steps in determining perils.
- How often the model returns any prediction based on input.
- Users sometimes type nonsense, or things even humans can't decipher.
- Customer time to complete task, through big submit (SUS).
- Are we actually making it easier for users? Or just annoying them.
How
Peril Wizard utilizes Natural Language Processing and Machine Learning model aim to build a simpler and accurate peril determination process.
The model is trained on 50K past Sprint customers’ text descriptions. The model has been trained for single and combination of keywords to predict the peril type.
Flow
We keep a high threshold of confidence level to accept the prediction from the model else the experience falls back to the current experience.
UI Iterations
Prototype:
Future State Design Prototype: https://asurion.invisionapp.com/share/WNNRLAA7AB9
Step 1 Design Prototype: https://asurion.invisionapp.com/share/EXNMALECP8G
User Testings
We performed an in-person paper prototype user testing to gauge people’s perception of writing in the text box. We also did a Survey monkey impression test to find out people’s feelings toward humanized version with illustration. We finally tested the end-to-end experience to validate our questions and concerns.
Paper Prototype UT: https://paper.dropbox.com/doc/Smart-Peril-Paper-Prototype-User-Testing--ALWDjO_nGnA_YlhdW~eHhxS0Ag-mHbZ5xsYgDTWD3h2gfOYO
SurveyMonkey Impression Test: https://paper.dropbox.com/doc/Smart-Peril-SurveyMonkey-Impression-Test--ALUVbtW0sk1~sxWRekpSFkdHAg-wDIGIYx5yZM3fdHgQMaaY
Result
We initially ran the pilot test at 25% traffic on Sprint for two weeks (6/22-7/08) to monitor the data. After tweaking the model and UI for multiple iterations, all leading metrics are favorable and hence we will be ramping up the traffic to 50%. And later 100% for Sprint.
- CLOE (Customer Level of Effort): We are seeing significant drop in CLOE per interaction. Control is ~ 230 seconds and pilot is ~188 seconds. There is no overlap in 95% confidence intervals. That is about 45 seconds drop in CLOE per interaction.
- Web NPS: There isn’t a significant impact on NPS, just slightly higher for pilot.
- Web Initiations and Completions: We have 1.3% higher initiations with pilot, while similar % of completions. Referring to the second table, the ‘Predicted’ population has much higher initiation and completion rates as compared to control. This of course is debatable since the ‘Did Not Predict’ bucket that is visible through the pilot also lies in the control population. These are customers who have either something too complex with peril/may be they shouldn’t even be in the new claim flow path such as customers coming back from unsuccessful repair.
- Repair Rates: Overall repair rates are at par for both control and pilot, about 23%. But there is a slight improvement in repair completed rates, about 1.2%.
- Web NPS: There isn’t a significant impact on NPS, just slightly higher for pilot.
- Model’s Prediction Rate:
