top of page

Driving Gen AI Adoption for Internal EY Consultants
  Ernst & Young (EY) LLP- Lead UX Researcher



A foundational 4-week research program designed to uncover barriers to gen AI tool aka EYQ usage for EY consultants from Business Consulting service-line to inform adoption strategies.  This work led to a 45% increase in adoption in 2 months.
EYQ was created for consultants with collaboration with MS Co-pilot for EY specific context e.g., previous consulting works with clients or clients acquisition process. 
 

Disclaimer: Due to NDA, I will not be able to share any original documentations or designs tested and will be focusing on process, high-level findings and recommendations.

#1

My Role

Lead UXR who strategized, conducted this study 

#2

Duration

A 4-week study to discover barriers in the gen AI tool usage 

#3

Collaboration

​
​

  • UX Designers

  • UXRs 

  • Product Managers

  • Engineers

  • Leadership aka Partners

  • Legal and Compliance

#4

Key Impact

45% increment in adoption 

Overview

​       How might we identify and mitigate key barriers to adoption by consultants...if the usage among the users is dwindling?​

Key Metrics:​

​

  • Customer satisfaction (CSAT) Score

  • Adoption Metrics

  • Self- reported metrics of time saved/ week

  • Bug reported by users 

  • Feature requests 

​       A three-prong approach helped for dense qualitative data collection in short time. Along with1:1 interviews, a usage survey and text analysis of grievance tickets and bug reports (from consultants) helped to achieve the tactical insights we needed.

10 pilot users from service line participated in 1:1 remote interview. They were asked to perform common tasks with AI tool, share their experience, and explore their positive and negative interactions with the tool.

User Interviews

A survey was sent to 200 consultants with Likert-scale and open-ended questionnaire to quantify satisfaction with AI tool, collect specific feedback on challenges in usage, expectations, time saved, and understand tool's ability to handle and text and visual inputs and outputs

User Surveys

A text analysis of reported grievances and bugs was conducted to determine which type of texts/ questions led to issues and thus identify areas of improvement.

Text Analysis
TIMELINE

 -Project kick-off

 -Communication with admin teams for related service tickets and bugs

 -Sent recruitment invitations for 1:1 interview

 -Design and launch survey in Qualtrics

 -Create moderator's guide including a service-line specific task-list

Week #1

 - 10 moderated interviews conducted by a Sr. UXR

 - Survey response monitoring

 - Text analysis for existing tickets and bug reports

​

Week #2
Week #3

- Analysis from Interview data      

- Data analytics covered the qualitative insights

- Qualtrics' Text iQ for qualitative insights

-Analyse the common themes

Week #4

- A List of  final enhancements shared with product teams & integrated with roadmap

​

MY ROLE & CROSS-FUNCTIONAL COMMUNICATION

​       A Collaborating daily-mainly through MS Teams meetings and messages, Comments via MS Word comments 

Cross-functional teams involved:

  • Adoption team

  • AI Labs team- Product Managers, Engineering

  • UX Designers

  • Data Analytics

  • Leadership aka Partners

  • Legal and Compliance

As a Lead UX Researcher I...​

  • Led the kick-off meeting to understand what the key problems were faced by adoption team

  • Study design and planning

  • Created discussion guides

  • 1:1 Interview moderation

  • Data collection and analysis

  • Socialised project status updates and report

  • Post-study collaborations (Scoping feasibility, refining actionable insights)

Analysis
Analysis.png

As a Lead UX Researcher I...​

  • I tackled the qualitative thematic analysis of data from user interviews, Survey, and grievances tickets/bug reports' text analysis

  • Created an excel document report enlisting all the potential bugs reported by user

  • Collaborated with data analytics team for quantitative segment of the survey resulting in a CSAT score

Insights and Recommendations

The main deliverable of this study was a list of feature enhancements which was shared with product and engineering leads.

High Impact Findings #1
(From Interviews, Text analysis from survey & service and Bug tickets)

​​​

  • EYQ's incompatibility with Excel is major reason in adoption drop

  • Highly requested features like ability to-

    • download content in pdf and word format

    • paste directly in Outlook ​

Week #1

Incorporate MS Excel integration on product roadmap with very high priority.

 Established as main acceptance criteria for upcoming release

​

  • Also, evaluate the feasibility for other top requested features and priorities them for upcoming release 

Actionable Recommendations 
Included in Updated roadmap
​Behavioural Insights

MS Excel is the most important tool used on daily-basis by all the consultants for various reasons. Hence, it is crucial for EYQ to help to crunch the data from excel and allowing the data to be exported in an excel. Participants felt their main need was overlooked from the use cases for this tool.

​

High Impact Findings #2 
(From Interviews, Text analysis from survey & service and Bug tickets)

​​​​​

  • 70% expected to have a private mode for EYQ to ask questions related to groundwork about the client which otherwise may perceived as ''one at least should know this about a client"

Actionable Recommendations 

 Implement a toggle with "Private mode" with a tool-tip indicating the purpose.

​

​​​

Implemented
​Behavioural Insights

EYQ was mainly built for consultants to be used for their research, document creation at various stages of client interaction in the project. The cohort of particularly junior to mid-level senior, and senior consultants feel the strongest imposter syndrome.

They did not wish to see their "silly questions about the project or the client" and felt the pressure to always appear well-informed and confident.

High Impact Findings #3
(From Interviews, Text analysis from survey & service and Bug tickets)

​​​​​

  • EYQ "breaks" or produces no response on large queries particularly which involves nested actions (one action within another)

Actionable Recommendations 

  •  Provide feedback to EYQ users conveying the delay in response or "cannot complete the task" message.​​​​

​

 

  • For upcoming version, a technical introspection to is advised to for handling large  queries and their potential impact on the model.​​​​

Implemented
Included in Updated roadmap
​Behavioural Insights

EYQ failed to inform users about what's going on within a reasonable timeframe (A heuristic violation) leading to users' confusion and frustration.

​       â€‹ Post-Study Impact 

The immediate impact of the study was traced over 2 months span with Adobe analytics, and a new CSAT survey after 2 months.

Increase in adoption tracked with Adobe analytics (measured with increase in unique user sign-ups, daily usage time )

45%

A staggering increase in CSAT score from the previous study survey

(Previous CSAT Score-48%)

*(Qualifying industry wide accepted 'Good' customer satisfaction' score)

​

​

70%

Time saved in overall document drafting process due to document download, easy to copy to outlook features 

(Reported by Client-acquisition teams)

​

15%

​       â€‹

For long term impact, given ever evolving nature of Gen-AI, EYQ's impact was set to be measured with quarterly CSAT surveys, dedicated UX Research activities, as part of product development.

​       â€‹ Challenges & Reflection 

​

 

#1 Recommendation for 'Private Mode' toggle conflicted with tool's collaborative usage purpose

​

​

For leadership to have a buy in for 'Private mode' implementation the framing of the issue was important.

Setting the context from increment in adoption perspective played a crucial role

​

Form, initial prioritisation sessions with leadership, engineering, a requirement to add a toggle for 'Private mode' was added to backlog with keeping the collaborative purpose intact i.e., public as default mode.

​

(Post implementation the following UX Research and analytics showed users preferred a 'Private mode' frequently, and expected that as default mode)

​

 

#2 Encouraging a collaboration among teams to for prioritisation based on stakeholder requirements and user research recommendations

​

​To utilise product requirements coming from different sources gauging engineering team to learn​

#1. The roots of the problem

#2. prioritise the user needs based on impact and implantation feasibility

​

This fully collaborative approach led to a long-term investment for UX Research efforts incorporating primary and secondary research

​

​

​

​

Next
bottom of page