Chapter 10: Apply the Changes, Analyze the Results, and Repeat

CHAPTER 10: Apply the Changes, Analyze the Results, and RepeatI have no special talent. I am only passionately curious.

- Albert Einstein
A few years ago, the designer and engineer Peter Skillman held a competition to build the tallest possible structure in 20-minutes with the following items: 

• 20 pieces of uncooked spaghetti
• One yard of transparent tape
• One yard of string
• One standard-size marshmallow

The marshmallow had to be placed on top to win the competition, and the entire structure had to remain motionless for three seconds. 
After running this challenge with more than 2,000 people from various backgrounds (from some of the top universities like Stanford, the University of California, and more to boot), Skillman was surprised by the results. 
Business school students performed the worst. What was even more shocking was that the best performing group were not engineering prodigies or physics majors. They were kindergarteners. 
What happened?
The business students were talking and thinking strategically. They examined the materials. They brainstormed ideas, debated the best options, asked savvy questions, and honed in on the most promising ideas. It was professional, rational, and intelligent. The process resulted in a shared decision to pursue one strategy. Then, with five minutes left, they divided up the tasks and started to build. 
The kindergartners took a different approach. Instead of strategizing, analyzing hypotheticals, debating, proposing, and asking questions, they simply start building. They rarely spoke to each other, and when they did, it was in short bursts, “Here! Now, here!” 
By the time the 20 minutes were up, they had experimented with 8 to 10 different structures before landing on the most stable one. 

Figure 10.1 The kindergartners in Skillman’s design challenge
In dozens of trials, kids built structures up to 26 inches tall, while business school students built structures less than 10 inches (on average). Kindergartners also beat out teams of lawyers (who built towers that averaged 15 inches), as well as teams of CEOs (22 inches).
The lesson here: multiple iterations usually beat a commitment to the first idea and making it work. It’s best to learn by doing.
This is an important principle to apply to user onboarding. From my experience, most teams do big launches to improve their user onboarding. Then, they iterate just once or twice a year. It’s much better to analyze the results, reiterate and implement changes quickly than to overthink and strategize about it for months. 
We’re now at the final step of the EUREKA framework. It’s time to apply changes to your onboarding experience. In this chapter, we’ll discuss how to run rapid-fire experiments that will produce compounding wins.
The Growth Process

Companies that grow the fastest are also the ones that learn the quickest. They use this as a massive competitive advantage. 
That’s where our “Triple A” sprint comes in. It focuses on rapidly identifying problems, building solutions, and measuring impact. The process follows a one-month sprint cycle to identify and deliver an improvement to a product. It consists of three A’s:

1. Analyze: Identify inputs that drive the outcomes you want for your business
2. Ask: Answer these three questions:
   1. 
      Where do you want to go?
   2. 
      Which levers can you pull to achieve the desired outcome?
   3. 
      Which inputs should you invest in?
3. Act: Choose an input and implement it. 

Then, you go through the Triple A cycle again, considering any learnings from the previous iterations to future ones.

I use this process when working with ProductLed clients. But you can use any iterative growth process such as Sean Ellis’ high-tempo four-step process, Brian Balfour’s six-step growth machine process or Growth Tribe’s G.R.O.W.S. process. You’ll notice that most of the work you’ve completed for the EUREKA framework contributes to the Analyze and Ask steps of the Triple A process.
By now, the majority of the ideas to improve user onboarding should have emerged from the previous steps in the EUREKA framework. Those steps are focused on holistically analyzing your data and gathering insights as a cross-functional team. 
Prioritizing Onboarding Improvements

From my experience, the problem in the growth process doesn’t lie with Ideation but in the Prioritization of growth ideas. For that, I use the Action Priority Matrix:
Here’s what this means:

• High-impact, low-effort are low-hanging fruits and quick wins. Start with these ideas.
• High-impact, high-effort could be big swings and transformational. These are major projects that require assessment and planning. 
• Low-impact, low-effort are momentum builders. These ideas are good as morale boosters but result in incremental improvements.
• Low-impact and high-effort ideas are ones to avoid.

Split Testing Growth Ideas

Ideally, you want to split test any onboarding changes. Let’s say you want to validate that your Straight-Line Onboarding performs better than your current onboarding experience. You can send half of your new users through the new onboarding and the other half through your current process. After a few weeks, compare the week-to-week retention rate to determine if the retention curve has improved. 
At this stage, it’s critical to iterate quickly. If you’re stuck at this step due to engineering bandwidth, consider tools like Appcues, Pendo, and UserPilot to implement Product Bumpers. Userlist, Intercom or Drift can enact conversational bumpers.
Socializing Learnings

Another step that teams often miss is ensuring the learnings from each growth experiment are shared across the whole organization. Since changes in the user onboarding impact so many teams, it’s really important to share the process, progress, and learnings with them.
Sharing with teams needs to become a habit. 
Doing this will accomplish a few things. First, it excites your teammates (and the executive team) about onboarding. It’s also a great way to create teamwide empathy to remind everyone that their work has a real impact on everyday lives. Finally, sharing the successes and impact of onboarding will ensure that you can continue to dedicate company resources to the project.
How you communicate this is entirely up to you, but here are a few things to try:

• An “Onboarding Wins” email that’s sent out to the onboarding team and other coworkers who have expressed interest in the process
• A dedicated Slack channel where you share test results and discuss upcoming tests 
• A slide deck in a shared folder (or in your company dashboards)
• An ongoing lunch-and-learn program
• Monthly or quarterly meetings

Beyond The Initial Onboarding

At this point, I want to stress once again that improving user onboarding is a continuous process. As your product evolves to the market it serves, the user onboarding experience must adapt to those product changes. 
To that point, the process of onboarding users never ends.
As new users complete the initial onboarding and become regular users, your onboarding team can (and should) help them adopt new capabilities and use cases of the product. You can apply the same EUREKA process to various steps in the user journey. Here are some common places to consider:

• Different entry points to your product will require a different onboarding experience. The journey and knowledge of someone who signed up from an ad will likely be completely different from someone who signed up through a Google search or a webinar campaign.
• Different Customer Jobs will also require personalized onboarding. 
• Invited users should be treated differently. You’ll need to give them more information about why they’re being invited and why they should use your product.
• New UI and product changes for existing users should be communicated thoughtfully. You’ll find the most engaged users will be the most resistant to change. How can you implement the principles you’ve learned in the EUREKA framework to help them adopt those new changes?

Generally speaking, after the initial onboarding is completed, you can onboard users to go deeper or wider with your product.
Going Deeper

First, go deeper by educating new customers about advanced configurations and features to help them do a Customer Job more effectively. 
For example, for sales executives who signed up for Calendly to meetings with prospects, the First Strike occurs when a prospect schedules a meeting using their Calendly link. For the next onboarding iteration, you could inform them about advanced features that would help them close more deals, such as:

• Embedding Calendly on a landing page
• Sending SMS notifications to prospects so they don’t miss meetings
• Automatically distributing meetings to their team based on availability, priority, or equity
• Integrating Facebook pixel code to the Calendly page to launch retargeting campaigns for those who viewed the Calendly page but didn’t book a meeting

Going Wider 
The other option is to go wider by introducing solutions to different problems that your customers face. 
For example, there are many reasons why people sign up for Hubspot:

• To generate leads with landing pages
• Nurture and close leads with email workflows
• Set up a company’s website 
• Manage social media
• Track the ROI of Facebook, Instagram, LinkedIn, and Google Ad campaigns with precision
• Create and share links to book meetings with leads
• Make, log, and record calls
• Manage the sales team
• Live chat with website visitors with a chatbot

Let’s say you know that a new customer signed up for Hubspot specifically to generate leads with landing pages. After achieving that objective, you can educate them on using social media to drive traffic to that landing page. If they have a sales team, you could also let them know about the benefits of subscribing to Hubspot Sales along with Hubspot Marketing.
What’s Next?
Before we wrap up, some final thoughts. So far, I’ve covered the steps to improve your user onboarding without any assistance from anyone in your team. But sometimes it’s best to have some help. Next, we’ll look at how sales and high-touch support fit into the user onboarding experience.