Cyber Security Audit Tool

Measuring the Impact of EY's Cyber Security & Compliance Audit Tool

Disclaimer: Due to NDA, I will not be able to share any images of the original designs tested and will be focusing on process, high-level findings and recommendations.

What's a Cyber Security Audit Tool?

A cybersecurity audit and compliance tool is a software solution designed to assess an organization's security posture and ensure adherence to regulatory standards. These tools automate the process of evaluating security policies, identifying vulnerabilities, and generating reports to help businesses meet industry-specific compliance requirements, such as GDPR, HIPAA, or ISO 27001. They often include features like risk assessments, policy management, real-time monitoring, and automated remediation recommendations, making it easier for organizations to maintain strong security measures and reduce the risk of cyber threats.

PRODUCT LIFECYCLE

DISCOVER

DEFINE

IDEATE

PROTOTYPE

EVALUATE

We are here

The business goal was to understand the perceived value & potential adoption challenges surrounding the audit tool to build an industry-leading tool.

How can we deliver high user value in the July 2022 release of Cyber Security Audit and compliance tool?
What input on user behaviour for learning a new tool can be gathered to inform our beta product in and beyond launch?

The user goal revolved around finding the middle ground between current (without a dedicated audit tool) and updated practice while using an audit tool for streamlined processes.

The research aimed to gather insights into user perceptions, pain points & safety/security concerns ensuring that the first launch delivered a valuable experience that would streamline daily user experiences.
Understanding participants' likelihood to integrate the new tool into their daily routines, probing their initial reactions, perceived value, and potential barriers to consistent use.

CROSS-FUNCTIONAL COMMUNICATION & MY ROLE

Cross-functional teams involved:

Product Management
UX Design
Engineering
Data Analytics
Branding and Marketing
Customer Support/ Admin
Content Management
Sales
Legal and Compliance
Business Development
Partners (Sponsors)

We communicated in a variety of ways:

Regular Meetings
Workshop and Brainstorming Sessions
Stakeholders were invited to observe interviews
Presentation read-outs
Chats via MS Teams and asynchronous collaboration tools - MIRO, MS Word,
JIRA
Journey Mapping sessions- (Miro)
Weekly/Bi-weekly Research update newsletters
1:1 conversation

My role and responsibilities as a leading researcher

Study design and Planning: Led the kick-off meetings with cross- functional teams, designed the research study, established clear objectives, research questions, and methodologies tailored to evaluation
Participant Recruitment: Identified and recruited a diverse group of participants representative of Cyber Security & Transformation auditors from geographically distributed teams, ensuring a variety of perspectives and use cases were considered. It included a group of new joiners to firm partners
Interview Moderation: Conducted in-depth, 1:1 moderated and remote interviews. This involved presenting new features, observing interactions, and probing for detailed feedback on concerns, functionality, and overall user experience
Data Collection & Analysis: Recorded qualitative and quantitative data (Benchmark score SUS). I then synthesised the data to identify common themes for user pain points, concerns, likings. Involving insights from the users, and areas of improvement.
Synthesis, Reporting and Read-outs: Synthesised the findings into comprehensive reports and presentations, highlighting key insights and actionable recommendations. I communicated these results to stakeholders from cross- functional teams.
Pre and Post study collaborations: Worked closely with product team to understand product/feature for better questions, aligning study with business and user goals. Post study collaborations with stakeholders evolving into continuous improvement to ensure the final launch met user needs and alignment with business goals
Establish and Lead ResearchOps: Created and managed a participants' pool for upcoming research efforts (Screening, Scheduling)

1. New joiners
2. Senior
3. Leads and Managers 4. Firm Partners (For Cyber Security)

Participant Screener:

Based on intended services and previous research, four main participant groups were recruited to take part in this research study.

Phase 1: Pre-Design Lockdown

Given the constantly changing environment and plans, the goal of this phase was to find out Cyber security and Compliance Audit Tool's perceptions and answer the fundamental questions for early direction for different features.

We focused mainly on the formative testing.

Phase 2: Post-Design Lockdown- Evaluation

The goal of this phase was to measure the impact of the all the design and feature changes in the big-picture ideas and target research questions regarding the end-to-end flows, design components that make up the audit tool.

Also, obtain a UX benchmark score- System Usability Score (SUS)

We focused mainly on understanding overall design performance aka summative testing with obtaining a benchmark score

System Usability Score (SUS): 78 (Good)

High-Level recommendations for overall features.

Highlight the most appreciated features of Cyber Security and Compliance audit tool, such as the ability to upload Standard questionnaire templates, ability add ad-hoc question, and post-audit report generation.

Prioritize detailed help sections explaining industry standards for users avoid confusion and provide easy access to information.

Prioritize microapps that help users minimize app switching, increase efficiency.

Create transparency surrounding information sharing & data storage for confidential documents shared by clients.

Sharing widely- The final report was a 45-slide PowerPoint presentation and had an audience of 100+ eager developers, designers, researchers, product managers & Partners.

Due to the importance of this subject for so many teams, there was a lot of communication before the official share-out. We announced the project broadly over 200 employees, then sent out loose preliminary findings followed by solidified findings a week later. This report was archived in EY's internal system to be read as a document, along with the creation of the formal deck.

Let's talk about the impact of this research

Research was heavily involved in this development process for this tool. As we solidified our findings, the design, engineering and product management teams were already excited to understand the roots of the recommendations and make changes. Here are a few of the impactful decisions that arose from this work.

Revised product roadmap

Of the 5 main features that we tested, we learned that some perceived better than

others when it came to perceptions of value, anticipated frequency of use.

One of the proposed features, "Surveys" was fully deprioritised based on this research as it did not align with mental models and contradicted the current user behavior of using surveys in the process.

Content Management

This study led to a content audit to identify what is missing from the available help sections.

Also, giving users clear information about data sharing, storage related information for better communication with clients

Leading to implementation of sophisticated CMS solution to have detailed help sections explaining industry standards, GDPR policies informing users about latest updates in policies and tool

Branding & Marketing

Helping branding and management team to build strategy to improve tool adoption globally

Leading into monitoring adoption with Adobe Analytics, expanding business opportunities with other services-lines with highly praised features

1200+

Consultants globally signed up for the tool in month after release

9-45Hrs/ Engagement

Time saved for reporting & assessment recommendations per engagement reported
(over a span of 1year)

400+

Consultants signed up for a consistent improvement programme aka ResearchOps

Extended Collaboration

Over the next fiscal year, collaboration among stakeholders and UX team continued delivering incremental improvement by empowering product roadmap including concept testing, surveys, Personae, updated journey maps, etc.

Emphasis on Data- driven Decisions

Quarterly planning based on insights from UX studies, leading a user wants and need analysis workshops for requirements prioritisation