Contact

Chameleon

UX Research + Design • 2020

Our internal teams have had a notoriously rocky relationship with Chameleon, our ad configuration and multivariate testing tool. To allow our users to work more scalably as more customers opted into our media applications, I led the complete redesign and worked closely with stakeholders and engineers to modernize the system.

My Role

Lead UX Designer + Researcher

Created for

Internal teams

Project type

Internal tool design

User Research

To kick off discovery for this long-awaited redesign, I interviewed 12 key users of Chameleon: partner integration engineers, PMs, dev teams, and optimization managers - each with their own level of expertise in launching and monitoring multivariate tests (as well as navigating the tool’s shortcomings). In general their test-launching process is as follows: once a partner opts into our machine-learning platform, our internal teams launch split tests from Chameleon to help partners maximize ad revenue with catered end user experiences.

After learning their processes and shadowing their daily workflows, I imported all raw interview notes into a research repository and developed a qualitative data tagging system to spot trends.

Dovetail, a UX Research repository tool I used to spot trends in pain points such as “inefficiency” and “dependency.”


Two of the main insights we synthesized to guide our feature recommendations were:
  1. The Search functionality needed to be more robust to return results that users actually care about.
  2. 22% of user pain points had to do with the Create Configuration flow, particularly the inefficiencies around creating test variants.

The Problem

With the increasing volume of multivariate tests that we run for our partners, our internal teams struggled to work scalably due to Chameleon’s shortcomings when it came to their test-launching workflows, specifically around Search and Create Configurtion.

Our goal was to redesign Chameleon to help remove pain points and dependencies preventing our teams from efficiently launching and maintaining thousands of ongoing, multivariate tests.

Designing The Experience

Redesigning Chameleon involved rebuilding a lot of the functionality that exists today, so to get buy-in from our stakeholders (i.e. the tech leads and PM) I mapped out the primary user flows along with feature recommendations informed by our user insights above. I used Figma to share the recommendations in the context of the user flow. By breaking this huge project up into smaller chunks, the team had an easier time managing priorities and timelines together.

Complete user flow + feature update recommendations.


The primary user flow we will be solving for our MVP.

User Flow 1: Search Configuration

Our research showed that the search results page as it stands today slowed our users down when they were searching for specific configuration.

Hypothesis: If we update the filters and display list results in a table format, users will find their configurations faster because users need more granular matches and an easier way to grasp contextual information.
Original Search Page

From user interviews, we gathered requently mentioned user pain points around the Search functionality (n = 12):

  1. 50% of users expressed the need for more sort or filters, many of whom were frustrated with the lack of text matches at a more granular level.
  2. Difficulty “hunting down a config” was a common topic. For 33% of users, the learning curve for certain sort metrics also created dependencies.
Redesigned Search Page
A couple tests and iterations later, the feedback for our final design for Search is as follows (n = 4):
  1. We added exclusions, text matches within strings, and ability to add filters. 100% of users found new default filters + the ability to add more as needed very helpful.
  2. All users found the new table format easier for scanning for information. In addition, new sort-by features and tooltips on hover for context removed dependencies.

Building + Testing

To build our MVP our engineer and I used the Vuetify framework with ready-made Material Design components. Soon we were able to ship a functioning Search MVP for Chameleon 2.0 and test for more user feedback.

Chameleon 2.0 - Search



So far the new Search’s positive reception has been validating the new design. One partner integration engineer says, “It’s definitely faster,” while another tells us that using the new beta product “has been a pretty painless transition” so far!

While we await additional user feedback for User Flow #1: Search, now a much more robust feature handling thousands of configurations serving custom treatments for our partners sites, we move on to solving for the 2nd user flow we prioritized: Create Configuration.

User Flow 2: Create Configuration

The main insight from user research informing the solution for this portion of the tool was that we need to reduce human error and inefficiencies caused by lack of UI feedback. One major UX improvement around configuration creation we wanted to focus on was that of assigning variants. This is not simply an A/B testing tool we’re building. Often we test 3+ variants for our partners, so theoretically, the UI must accommodate any number of variants.

To illustrate the user flow, suppose she is setting up the below 3-way test with the following configuration parameters:

Researching The Problem

The current flow for setting up the 3-way test variants is below. As observed during user research, there’s a ton of back-tracking and high cognitive load in the steps taken to reach the user goal: setting up test variants (indicated in black).

Original user flow for Create Configuration



Original Create Configuration

Here is a consolidation of user pain points around creating test variants (n = 12):

  1. 42% of users commented on the steep learning curve and high cognitive load in this unintuitive attribute-first approach.
  2. 66% disliked the amount of page leaves required for certain steps, e.g. adding new attributes/values and reviewing variants on the next page.
  3. Once the matrix of all possible variants were generated, 83% were frustrated with the process of finalizing variants (“You have so many options and it’s really overwhelming.” -PM).

The Solution: Create Configuration

Our proposed solution for the redesign reduces the number of steps and eliminates the need to leave the configuration page:

Hypothesis: If we switch the user flow to a “variants-first” approach with sufficient UI feedback, then users are able to achieve the user goal of setting up test variants in an intuitive way because we help reduce the cognitive load as well as redundancies experienced today.

Redesigned user flow for Create Configuration: We reduce the number of steps and cognitive load by leading with test variant set-up (indicated in black) and achieving the user goal through subsequent steps in an intuitive way.

Prototype V1: Create Configuration

We led a usability test with n = 4 users and received feedback around a reimagined flow for test variants:

  1. Despite the major UX change, 100% of users found the new flow more intuitive and “way easier to review” -partner integration engineer.
  2. 50% of users scrolled up and down repeatedly. One PM commented on liking “the simplicity…[but I] wish it was more compact.”
  3. A couple users were uncertain what to expect if they deleted an attribute.
Prototype V2: Create Configuration

We made some layout changes based on feedback and additional analyses around creating test variants. For example, the database showed us that >90% of configurations are created with 4 or less variants. Here's the feedback for our final design (n = 4):

  1. Once again, 100% of users found the new flow intuitive (“this [UI feedback] helps” -partner integration engineer).
  2. A compact UI fits 4 variants in view at any time. 3 out of 4 users understood how this table was building itself out.
  3. When tasked with deleting an attribute, all users were able to do so.

Next Steps

We’re in the midst of iterating on several new user flows to reach our objective in simplifying the overall process of setting up a multivariate test. So far, the user feedback for Use Flow #1: Search and User Flow #2: Create Configurations have been a combination of positive as well as requests for even more capabilities now that the users have had time to experiement and get a sense of what's possible with Chameleon 2.0!

The team and I will continue to follow this cyclical design process for addressing all the new feature recommendations we've broken up into manageable pieces together as a team. We're proud of the progress we've made thus far, and I’m looking forward to collaborating with the engineers to componentize, share feedback, and analyze the ROI as the remainder of features get released. Back to projects

© 2023 Brenda Hu