4 lessons QA teams can learn from the Iowa Caucus App failure

Scrutiny of the IowaReporter – the app that caused chaos during the Iowa Democratic caucuses on February 3rd – continues to swirl as officials and the media try to understand what caused the app’s issues. While much of the media coverage focuses on what went wrong, it also uncovers some lessons for Quality Assurance teams on how to deliver a successful live event and app experience. Below, we’ll focus on what QA teams can do differently to succeed with mobile app testing and launches.

What Went Wrong with the IowaReporter App

However, in a broader context, it appears the app’s technical issues were the result of hasty deadlines, rushed development and bypassing normal app testing procedures — issues we’ll explore in the lessons for QA teams below.

1. Test Early and Continuously 

“From a testing perspective, test case development should have begun and evolved in parallel with the application development,” says Testlio app tester Natalie Ball. “The application should have gone through a full-range of functional and UI/UX testing cycles to resolve the data integrity and security problems. Hiring a QA partner to develop a thorough test plan and conduct end-to-end testing throughout the development cycle would have eliminated many of the issues with the application.”

2. Conduct a Rehearsal Test

The Shadow team likely didn’t foresee the app’s defects during caucus night because an app’s usability and functionality on a small-scale is often very different than in a live event. “Sure, the app might run great when a few internal members are online with the app, but what happens to the servers when they load hundreds of thousands of users to the system?” says Brandon Blewett, one of Testlio’s functionality testing leads. “By running functionality, load testing, and usability testing, many of the issues with the Iowa Caucus app likely would have been averted.”

To help fully prepare for live-events, QA teams can conduct rehearsal tests using a network of app testers, who will test the app simultaneously on a range of devices.

3. Test with Real Users

Even with functional testing, security testing, and load testing, some defects can only be exposed when tested with real people. For example, the IowaReporter was not released in app stores, but instead could only be downloaded through TestFlight and TestFairy, mobile testing apps normally used by developers and testers, but not well known to the public. (In fact, even professional testers getting into mobile testing often need training on how to use these apps.) Thus, app’s actual users — the precinct heads in Iowa — naturally had major issues just downloading and logging into the app.

Testing through multiple levels and many real testers would have surely brought to light the flaws of the app before reaching the end-users,” says testing lead Brandon Blewett. Many apps fail before gaining any traction due to a rushed release that does not consider the human element.”

4. Don’t Rush Launches

It’s important to note that the technical issues described above (the code defects and lack of testing) were part of a larger management decision to develop and launch the IowaReporter app on a comparatively short timeline. 

Instead of a couple of months, a more typical timeline for development and testing this type of app would have been a year, according to Testlio’s QA experts. “Testing and bug fixes should have been completed 6 months before first-use to accommodate end-user training,” says tester Natalie Ball.   

Delivering a seamless live event and a successful app experience depend on common sense development decisions, but also a thorough testing process that integrates testing on real users.  

Helpful resources:

Jumbotron image