Measuring the Impact of EY's Cyber Security & Compliance Audit Tool
Disclaimer: Due to NDA, I will not be able to share any images of the original designs tested and will be focusing on process, high-level findings and recommendations.
What's a Cyber Security Audit Tool?
A cybersecurity audit and compliance tool is a software solution designed to assess an organization's security posture and ensure adherence to regulatory standards. These tools automate the process of evaluating security policies, identifying vulnerabilities, and generating reports to help businesses meet industry-specific compliance requirements, such as GDPR, HIPAA, or ISO 27001. They often include features like risk assessments, policy management, real-time monitoring, and automated remediation recommendations, making it easier for organizations to maintain strong security measures and reduce the risk of cyber threats.
PRODUCT LIFECYCLE
DISCOVER
DEFINE
IDEATE
PROTOTYPE
EVALUATE
We are here
The business goal was to understand the perceived value & potential adoption challenges surrounding the audit tool to build an industry-leading tool.
​
​
-
How can we deliver high user value in the July 2022 release of Cyber Security Audit and compliance tool?
-
What input on user behaviour for learning a new tool can be gathered to inform our beta product in and beyond launch?
The user goal revolved around finding the middle ground between current (without a dedicated audit tool) and updated practice while using an audit tool for streamlined processes.
​
-
The research aimed to gather insights into user perceptions, pain points & safety/security concerns ensuring that the first launch delivered a valuable experience that would streamline daily user experiences.
-
Understanding participants' likelihood to integrate the new tool into their daily routines, probing their initial reactions, perceived value, and potential barriers to consistent use.
1
2
3
4
5
6
Kick-Off
Methodology & Screener
Conduct Research
Synthesis & Recommendations
Share Findings Widely
Measure Impact
2
Phases
Researchers
2
11
Cross-Functional Teams Involved
35
60-Minute
Moderated-Remote
Interviews
CROSS-FUNCTIONAL COMMUNICATION & MY ROLE
Cross-functional teams involved:
-
Product Management
-
UX Design
-
Engineering
-
Data Analytics
-
Branding and Marketing
-
Customer Support/ Admin
-
Content Management
-
Sales
-
Legal and Compliance
-
Business Development
-
Partners (Sponsors)
We communicated in a variety of ways:
-
Regular Meetings
-
Workshop and Brainstorming Sessions
-
Stakeholders were invited to observe interviews
-
Presentation read-outs
-
Chats via MS Teams and asynchronous collaboration tools - MIRO, MS Word,
-
JIRA
-
Journey Mapping sessions- (Miro)
-
Content Management
-
Weekly/Bi-weekly Research update newsletters
-
Legal and Compliance
-
1:1 conversation
My role and responsibilities as a leading researcher​
-
Study design and Planning: Led the kick-off meetings with cross- functional teams, designed the research study, established clear objectives, research questions, and methodologies tailored to evaluation
-
Participant Recruitment: Identified and recruited a diverse group of participants representative of Cyber Security & Transformation auditors from geographically distributed teams, ensuring a variety of perspectives and use cases were considered. It included a group of new joiners to firm partners
-
Interview Moderation: Conducted in-depth, 1:1 moderated and remote interviews. This involved presenting new features, observing interactions, and probing for detailed feedback on concerns, functionality, and overall user experience
-
Data Collection & Analysis: Recorded qualitative and quantitative data (Benchmark score SUS). I then synthesised the data to identify common themes for user pain points, concerns, likings. Involving insights from the users, and areas of improvement.
-
Synthesis, Reporting and Read-outs: Synthesised the findings into comprehensive reports and presentations, highlighting key insights and actionable recommendations. I communicated these results to stakeholders from cross- functional teams.
-
Pre and Post study collaborations: Worked closely with product team to understand product/feature for better questions, aligning study with business and user goals. Post study collaborations with stakeholders evolving into continuous improvement to ensure the final launch met user needs and alignment with business goals
-
Establish and Lead ResearchOps: Created and managed a participants' pool for upcoming research efforts (Screening, Scheduling)
Participant Screener: Based on intended services and previous research, four main participant groups were recruited to take part in this research study.
​​
1. New joiners
2. Senior
3. Leads and Managers 4. Firm Partners (For Cyber Security)
​​
Phase 1: Pre-Design Lockdown
​
Given the constantly changing environment and plans, the goal of this phase was to find out Cyber security and Compliance Audit Tool's perceptions and answer the fundamental questions for early direction for different features.
​
We focused mainly on the formative testing.
​​
​
​​
Phase 2: Post-Design Lockdown- Evaluation​
​
The goal of this phase was to measure the impact of the all the design and feature changes in the big-picture ideas and target research questions regarding the end-to-end flows, design components that make up the audit tool.
Also, obtain a UX benchmark score- System Usability Score (SUS)
​
We focused mainly on understanding overall design performance aka summative testing with obtaining a benchmark score
​​
​​​
Phase 1: Pre-Design Lockdown
​
​
​​
Phase 2: Post-Design Lockdown
​
​
System Usability Score (SUS): 78 (Good)
​​
High-Level recommendations for overall features.
​​
Sharing widely- The final report was a 45-slide PowerPoint presentation and had an audience of 100+ eager developers, designers, researchers, product managers & Partners.
Due to the importance of this subject for so many teams, there was a lot of communication before the official share-out. We announced the project broadly over 200 employees, then sent out loose preliminary findings followed by solidified findings a week later. This report was archived in EY's internal system to be read as a document, along with the creation of the formal deck.
Highlight the most appreciated features of Cyber Security and Compliance audit tool, such as the ability to upload Standard questionnaire templates, ability add ad-hoc question, and post-audit report generation.
Prioritize detailed help sections explaining industry standards for users avoid confusion and provide easy access to information.
Prioritize microapps that help users minimize app switching, increase efficiency.
Create transparency surrounding information sharing & data storage for confidential documents shared by clients.
​
​
Let's talk about the impact of this research
Research was heavily involved in this development process for this tool. As we solidified our findings, the design, engineering and product management teams were already excited to understand the roots of the recommendations and make changes. Here are a few of the impactful decisions that arose from this work.
Revised product roadmap
​
Of the 5 main features that we tested, we learned that some perceived better than
others when it came to perceptions of value, anticipated frequency of use.
​
One of the proposed features, "Surveys" was fully deprioritised based on this research as it did not align with mental models and contradicted the current user behavior of using surveys in the process.
Content Management
​
This study led to a content audit to identify what is missing from the available help sections.
Also, giving users clear information about data sharing, storage related information for better communication with clients
​Leading to implementation of sophisticated CMS solution to have detailed help sections explaining industry standards, GDPR policies informing users about latest updates in policies and tool
​
Branding & Marketing
​
Helping branding and management team to build strategy to improve tool adoption globally​
Leading into monitoring adoption with Adobe Analytics, expanding business opportunities with other services-lines with highly praised features
​
1200+
Consultants globally signed up for the tool in month after release
9-45Hrs/ Engagement
Time saved for reporting & assessment recommendations per engagement reported
(over a span of 1year)
400+
Consultants signed up for a consistent improvement programme aka ResearchOps
Extended Collaboration​
Over the next fiscal year, collaboration among stakeholders and UX team continued delivering incremental improvement by empowering product roadmap including concept testing, surveys, Personae, updated journey maps, etc.
​
​
Emphasis on Data- driven Decisions
Quarterly planning based on insights from UX studies, leading a user wants and need analysis workshops for requirements prioritisation​
​