4 lessons QA teams can learn from the Iowa Caucus App failure

, Employer Brand Manager based in Tallinn, Estonia.

Scrutiny of the IowaReporter – the app that caused chaos during the Iowa Democratic caucuses on February 3rd – continues to swirl as officials and the media try to understand what caused the app’s issues. While much of the media coverage focuses on what went wrong, it also uncovers some lessons for Quality Assurance teams on how to deliver a successful live event and app experience. Below, we’ll focus on what QA teams can do differently to succeed with mobile app testing and launches.

What Went Wrong with the IowaReporter App

The IowaReporter app, which was built on React Native for Android and iOS, suffered from a number of software defects as well as larger usability issues. The crux of the software issues appears to be in its data reporting functionality. When the precinct heads tried to report the voting results to a verification system, a data-formatting error prevented the results from being accepted. This, in turn, created a standstill and mass-confusion in the reporting process. (More on the specific software issues can be found in this Vice article). 

However, in a broader context, it appears the app’s technical issues were the result of hasty deadlines, rushed development and bypassing normal app testing procedures — issues we’ll explore in the lessons for QA teams below.

1. Test Early and Continuously 

The app’s software defects could have been addressed if they were identified through a standard testing process. However, due to a rushed timeline, it appears the biggest mishap with the IowaReporter was the lack of an end-to-end mobile app testing process. Though the company says it did hire an external auditor for security testing, it seems to have conducted limited functionality testing and usability testing. 

“From a testing perspective, test case development should have begun and evolved in parallel with the application development,” says Testlio app tester Natalie Ball. “The application should have gone through a full-range of functional and UI/UX testing cycles to resolve the data integrity and security problems. Hiring a QA partner to develop a thorough test plan and conduct end-to-end testing throughout the development cycle would have eliminated many of the issues with the application.”

2. Conduct a Rehearsal Test

In an interview with Motherboard, Shadow’s CEO Gerard Niemira reflected on the app’s performance prior to the caucuses: “The app was sound, the data that came out of the app was sound, the math that was done on the app was sound…All the results we collected on the app were sound and have been verified as such.” 

The Shadow team likely didn’t foresee the app’s defects during caucus night because an app’s usability and functionality on a small-scale is often very different than in a live event. “Sure, the app might run great when a few internal members are online with the app, but what happens to the servers when they load hundreds of thousands of users to the system?” says Brandon Blewett, one of Testlio’s functionality testing leads. “By running functionality, load testing, and usability testing, many of the issues with the Iowa Caucus app likely would have been averted.”

To help fully prepare for live-events, QA teams can conduct rehearsal tests using a network of app testers, who will test the app simultaneously on a range of devices.

“At Testlio, we use a framework for building a test strategy around live events which includes a full-scale rehearsal test that usually occurs several weeks before the event,” says Testlio’s QA Project Manager, John Ruberto in an interview with TechTarget. “For event-based software, you have to have time to prepare and rehearse, to work out the kinks before the event, then manage the configuration & prevent regression errors from sneaking in.”

3. Test with Real Users

One of the main lessons of the IowaReporter mishap was the importance of usability testing – testing with real people who simulate actual user behavior and can find the bugs before users do. 

Even with functional testing, security testing, and load testing, some defects can only be exposed when tested with real people. For example, the IowaReporter was not released in app stores, but instead could only be downloaded through TestFlight and TestFairy, mobile testing apps normally used by developers and testers, but not well known to the public. (In fact, even professional testers getting into mobile testing often need training on how to use these apps.) Thus, app’s actual users — the precinct heads in Iowa — naturally had major issues just downloading and logging into the app.

“Testing through multiple levels and many real testers would have surely brought to light the flaws of the app before reaching the end-users,” says testing lead Brandon Blewett. “Many apps fail before gaining any traction due to a rushed release that does not consider the human element.”

4. Don’t Rush Launches

It’s important to note that the technical issues described above (the code defects and lack of testing) were part of a larger management decision to develop and launch the IowaReporter app on a comparatively short timeline. 

According to The Wall Street Journal, the Iowa Democratic Party (IDP) initially evaluated an entirely different software solution: a vote by phone system. Though online voting software is in use in a handful of countries (notably Testlio’s European HQ of Estonia), the proposal is contentious in the US for security concerns, and the Democratic National Committee rejected the proposal.

This question about security left the IDP looking for other solutions to streamline reporting, and it appears it adopted the IowaReporter as its technology solution a couple of months before the caucuses. Though we don’t know exactly how long / thorough the development process was, it was likely rushed, causing some software experts to call the app an ‘off-the-shelf’ skeleton.

Instead of a couple of months, a more typical timeline for development and testing this type of app would have been a year, according to Testlio’s QA experts. “Testing and bug fixes should have been completed 6 months before first-use to accommodate end-user training,” says tester Natalie Ball.   

Delivering a seamless live event and a successful app experience depend on common sense development decisions, but also a thorough testing process that integrates testing on real users.  

Helpful resources:

Your step-by-step mobile application testing process
9-step mobile app strategy checklist

Jumbotron image

Need to ensure a flawless mobile app experience? Talk to us about how our end-to-end mobile app testing services used by clients like CBS, the NBA, Hotels.com and many more.