Inspired by Jake’s Blog
Twyla’s product team started our Design Sprint journey after reading some of Jake Knapp’s early blog posts. After having had a few failed attempts to build a customer research discipline, I was intrigued and hopeful that Design Sprints could help us focus our research efforts. We quickly set out to run our first sprint.
Our first sprint was mostly a failure due to poor attendee selection, omitting exercises, and lack of an experienced facilitator. The attendees were limited to the product team and did not include stakeholders or business owners. We omitted exercises and compressed full days into hours in a misguided attempt to reduce the time commitment. The prototype we generated was low fidelity. We didn’t develop a solid research plan and we didn’t follow through on interviews that targeted our questions. Therefore, we didn’t garner deep insights and we weren’t able to share them broadly. We did benefit from raising awareness of sprints and we made our first steps towards creating a culture of customer development.
Feeding off the Book
After a few months, the Sprint book was released. Reading the book several times reinvigorated us with the confidence to improve upon our previous sprints. We now had clear examples of how to run the exercises, checklists, and guidance on how to approach the interviews. We set out to run Twyla Sprint #2. This time, we included almost the entire company in an effort to embrace inclusion and ensure all stakeholders had a voice. In effect, we went to the opposite extreme. We still lacked the undivided attention that is essential for a successful Sprint. Stakeholders filtered in and out of the room at random times. They were not truly bought into the process or the problem we were trying to solve. We didn’t spend the time to educate attendees on the process and why we were doing this crazy thing in the first place. On the positive side, we did gain more experience facilitating and starting to harness the power of individual work.
Google Ventures Invests
In August of 2016, Google Ventures (GV) led the Twyla Series A financing. It’s hard to say whether the product team was more excited about Twyla closing our Series A or the potential of getting to work with the GV Design Sprint team.
The GV Design team is one of the resources that GV generously provides for its portfolio companies. Not only is it a free service, but they highly encourage portfolio companies to use it. They didn’t have to twist our arms! We found the design team a pleasure to work with. We had a few calls with them to better understand our competitive landscape and problem space. They peppered us with questions to help them understand our company and our customers. It was helpful to get a third party perspective that was inquisitive and non-judgmental.
We conducted a few remote research sessions with Michael Margolis, GV’s research expert. He coached us on how to build a good recruiting screener, helped us source and filter candidates, and provided tips on better customer interviews. We interviewed customers interacting with our current site as well as a few other online art stores that most closely resembled our brand. GV insisted that the Twyla team all watch remotely in real time, while Michael conducted the interviews as they watched from California. Watching in real time ensures that everyone sees the entire session. This increases empathy, as they don’t rely on a synopsis from the team. This also means everyone participates in the debrief at the end of the day when we synthesize the learnings. GV also encouraged us to each bring our own headphones, which helped us all hear the interviews more clearly and avoid temptation of distracting dialog in the room.
The feedback and reactions from our sessions were a bit harsh, so the team’s knee-jerk reaction was to blame our recruit. This is a typical pitfall and GV warned us that it is tempting to blame your recruiting process when you don’t hear what you’d like to hear. We insisted that it must be a bad recruit, so they happily ran another set of interviews. The results were the same. At this point, it was clear that we would benefit from having the GV Design team run a Sprint with us in Austin. First we set about some homework, mainly to better define the customer and frame the problem space.
It is tempting to blame your recruiting process when you don’t hear what you’d like to hear.
Sprinting with GV
About 2 months later, on a brisk December morning, Jake, JZ, and Braden joined Twyla at our offices to kick off our 3rd Design Sprint. This sprint was markedly different. Their preparation included careful attendee selection, ordering sprint supplies and snacks, as well as training us on the process. They guided us on how to prepare other attendees to set proper expectations and even assigned homework for our experts.
Day 1 started off with a video overview of the Sprint process, an overview of each day and a detailed checklist for Day 1’s agenda. This attention to planning and prep work ensured that everyone was bought into the process and had a basic understanding of the participatory process they were about to engage in for the next 5 days.
Day 2 started off a bit shaky. We had moved into our new offices a month prior and other parts of the building were still under construction. As I approached the building, I could see firemen and our neighbor’s employees congregating on the streets. Apparently construction dust had set off the alarms and the fire department couldn’t turn them off. Since we couldn’t re-enter the building, I quickly scheduled a conference room at Capital Factory. While it is important to stick to the schedule and adhere to the process, it is also important to make good use of the time you have reserved. This is just one example during the week of pivoting to accommodate the environment or the things unfolding in the room. I was really impressed by how Jake, John, and Braden were so nimble and adaptive.
Using a fake brand allowed us to test our new prototypes alongside our current site, so that we were able to have a control in our test.
One of my key takeaways from prototyping with the GV team is the use of the fake brand. Using a fake brand allowed us to test our new prototypes alongside our current site, so that we were able to have a control in our test. I also found it beneficial to watch how the GV team approached prototyping. Everything from adjusting the level of fidelity, to delegating tasks came together quickly and efficiently. They are a well-oiled machine.
Observing Michael’s research continued to help us hone our interviewing and research skills. He prepares an interview plan to ensure that he addresses each of the Sprint questions and in a manner that flows organically. The plan always begins with a warm introduction and small talk. Since Michael reviewed their screener, he has a good understanding of their background and can easily lead them toward his interview plan without it seeming forced or unnatural. He also introduced himself as a Google Venture researcher who is studying how people buy art online. That prevented the user from knowing that he has an interest in Twyla.
During the Sprint we were all jazzed about pricing transparency and prototyped a solution that gave the user a breakdown of pricing structure. It highlighted the fact that we pay the artist royalty and the cost to manufacture, frame, ship etc. All of the users had a strong negative reaction to this part of the prototype. Some even complained about it before we even steered them toward it. I was excited to see such clear and actionable results. One of our sprinters said to me: “If that idea hadn’t come up in the sprint, we woulda built it and god knows how long it would have sat there on the site turning people off.”
Having an unbiased 3rd party both facilitate and contribute to synthesizing the learnings helped us embrace the true learnings without glossing over the more harsh realties.
Again, we gathered together to observe the interviews in real time so that we could all take notes individually, then immediately reviewed together before the next interview. At the end of the day, we compiled and reviewed all the notes, then scheduled a retrospective early in the following week to complete a final review. Braden, John, and Jake followed up later with a summary of their findings and suggestions. This document was honest and accurate, which is something I’m not sure we could have done ourselves at that time. Having an unbiased 3rd party both facilitate and contribute to synthesizing the learnings helped us embrace the true learnings without glossing over the more harsh realties. Now that we’ve experienced this first hand; our goal is to internally embrace and encourage this level of candor and honesty.
“I never would have had a platform for pitching that idea to execs outside of the sprint”, Shane O’Donnell, product designer @ Twyla.
The biggest success in our sprint was our $30 trial. For only $30, a user can try a piece at home for 30 days before the remainder of the balance is charged to their card. While the financial impact to the business is similar to our 30-day return policy, users appreciated this offer more than our standard return policy and it provided them more confidence to buy art site-unseen. We implemented the $30 shortly after the Sprint and conversion increased dramatically and it still accounts for the majority of our orders. “I never would have had a platform for pitching that idea to execs outside of the sprint”, Shane O’Donnell, product designer @ Twyla.
Twyla on our Own
Using our newly acquired skills, two months later, we ran a Sprint all on our own. While there was definitely room for improvement, the results were exactly what we had hoped for. We did a much better job of attendee selection as well as preparation this go-round. Education was a bit lacking so we had to spend a bit more time initially explaining the process to some new Sprinters. We should have taken the opportunity to bring them up to speed beforehand.
This Sprint we were all convinced that a tiered benefits structure for our trade program would be a slam dunk and we had grand plans for creative ways to incentivize designers to move up the tiers. Designers disliked this without condition. They all had a distaste for seeing a higher discount than they currently received on purchases. After the sprint, one of our designers told me: “Think about how much work that would have been to build! It was weird, because I was so proud to have that idea crushed. It showed the power of the sprints to shortcut effort.”
“Think about how much work that would have been to build! It was weird, because I was so proud to have that idea crushed. It showed the power of the sprints to shortcut effort.”
During the interviews, we validated the acceptable discount ranges for our trade program and we settled on 20%. We also validated that designers appreciate a personal connection with the brand. Prominent placement of their account representative, along with a personal photo and contact details afforded us extra confidence with the brand. They also got excited about premium services such as custom renderings and art advisor services. They like knowing they could lean on our staff of experts to make them more effective for their clients.
While there is always room to improve, we’ve learned a ton over the past 16 months and we look forward to what there is to learn from future sprints. If you have any questions or would like to swap stories, please look me up on Medium, Twitter or LinkedIn.
I’ll be covering these topics and more as I tell the Twyla Design Sprint story at the first Austin Design Sprint meetup on July 12 6:30 @ Capital Factory.
Hope to see you there!
