Fail Forward: The Secret to Succeeding with Weekly Releases Releasing on a weekly basis no matter what is not an easy adjustment for most organizations to make. Even very small startups can become blocked and then stretch out their cycles to weeks or months. In Testlio’s recent fireside chat, Kevin Henrikson, co-founder of Acompli (now Outlook Mobile) and Director Partner of Engineering at Microsoft spoke with Testlio CEO Kristel Kruustük about how the Acompli team first committed to weekly releases, the positive effects that had on the product, and how Henrikson was able to bring some of those benefits to another Microsoft product a couple years after the acquisition. The story behind Outlook Mobile’s short releases Initially, the team of 19 developers released on two-week sprints, which is common in agile. Kevin noticed that for eight or nine working days, no one was checking anything in. “Then on the last Thursday, all the code would come in,” says Henrikson. “Every two weeks I’d see this, the second Thursday, all the code would come in and on Friday we’d furiously try to sort it out.” He decided to experiment with one-week sprints, and sure enough, all of the code would come in on Thursday, but now it was at least every Thursday, which helped to smooth out the delivery process. The shorter the development cycles, the easier it became to fix any issues found during weekend testing cycles with Testlio. Henrikson explains: I just realized that it worked. How can I convince the developers to fix [an issue] in four hours? And I thought well you had a week of work and you fixed it and broke it in a week, so now you have four hours, either take it off or put it on. Either fix it or go backwards. And that became an incredibly powerful way to have that conversation. Kevin Henrikson How does failing forward apply to software? So many companies are completely resistant to the idea of weekly releases. There’s always some excuse or block that seems impossible to resolve. The Acompli team was able to get fully on board when they realized that there was actually less fear and pressure with shorter releases. You want just the minimal amount of software in flow, so it can get through the pipeline and get to customers and you can get feedback on it, and that feedback powers everything else. Kevin Henrikson In general, failing forward is about removing the “innovation-limiting shackles of perfection.” With starting a new business, it can mean starting now with what you have. With software, it means developing in short cycles, shipping fast and fixing issues as they are discovered—instead of waiting to ship perfection. “We ship lots of bugs but we fix them before most people notice. Our servers always go down, we have VM problems…but if we can detect it faster than most users notice, it’s the same as not creating it,” says Henrikson. “And that is a much more fun way to work. You’re working on the thing that matters. Put the fix in and keep moving.” The benefits of working in small batches There are a few massive benefits to short releases: Fix known issues faster Build in quality from the start Lessen tension between product managers and developers Make better use of customer feedback Henrikson details the effect that weekly releases had on the Acompli team: Once we started to get into that mode, developers started to self-censor their work. So if you have a really risky thing coming in and you’re like oh it’s sort of jank, in the traditional enterprise world you’re like check it in and let QA find the bugs. But that doesn’t work when you ship it on Monday because now your Monday is screwed because you checked in crappy code. The magic of that trick was, if this is kind of questionable code, wait a week. You’re only waiting a week, you’re not like waiting a month or six weeks or whatever your traditional cycle is. If you’re really confident and you think it’s good then you’ll push it in. Now it works even at the scale of a hundred plus developers. It’s a smooth way. By shortening the amount of code that had been built but was not yet in the hands of customers, the Outlook Mobile team has been able to quickly fix bugs and move on to new projects. Traditionally, PMs and engineering can experience a lot of tension regarding when to build new features versus when to fix existing ones, but with shorter cycles, it’s faster to fix the bugs and move on to something new. The 2016 State of DevOps Report identifies the reduction of unplanned work as paramount to becoming a top performing organization. Less time spent fixing code means more time for building. Bringing short dev cycles to Outlook for Mac As Henrikson points out, “Nobody does agile the same.” To shorten release cycles, teams have to pull off the bandaid, get started and then smooth out the process along the way. Initially, at Microsoft, no one was all that impressed by weekly releases with a two-year old mobile app, considering that other products have massive 30-year-old code bases. The team that works on Outlook Mac joined up under Henrikson’s guidance, and he proposed that they start shipping every week to real users and take support tickets directly. The concern from the team was that the build times were 8 hours because the code base was so huge. Here’s how Henrikson approached the issue: “If it takes that long to build and you need to release on Friday, start on Wednesday. Code for two days and then ship for three, and that’s what they did to start with.” Once they had the system, they were able to continuously improve upon it. The results have been incredible. “Their crash rates are lower than they’ve been in 10 years,” says Henrikson. Why integrated support is essential Failing forward and releasing quickly requires direct feedback from the customer to engineering. Traditionally in large organizations, developers might not find about customer issues for multiple months. Support teams would instruct customers to try different fixes while developers remained oblivious to customer concerns. Integrating support and development is essential to agile and DevOps methodologies, because developers need to immediately know their true priorities. With constant feedback, engineering can pull a build back and revert it if absolutely necessary. “If you know right away, you fix it right away, and you reduce the pain,” says Henrikson. “The actual pain and suffering of a customer is not the pain, it’s the pain over the duration.” Fast fixes and continuous feedback allow for development to be freed up from spending too much time on the backlog. Weekly releases can be completely transformative for your customers, your product, and your team. For QA that’s equally transformative and keeps up with your release cycles no matter what, reach out to us for a demo.