Testing solutions that go beyond industry standards: an interview with Summer Weisberg and Kerli Rungi

We sat down to chat with both of them, as leaders at Testlio and in the industry— about the challenge our clients face, how they prioritize services work, onboarding, our different service models, and their favorite client stories.

If you’re a Testlio client, the services team is your first point of contact. Hopefully, this can serve as a meaningful introduction to the people leading our QA solutions charge.

The most common client problem and our solution

What’s the number one challenge faced by our clients? How do we help them?

On top of that, there is the scale of our offering. Global locations, diverse skills, across thousands of real device and OS combinations, using multiple capabilities, all facilitated by a cohesive platform and dedicated services arm. We can scale QA service to any extent, in any way, in moments that matter.

But third and closest to my heart are passion and partnership. We don’t see ourselves merely as a mobile app testing company — we seek true partnership, understanding, service, mutual value. And based on the positive feedback we receive, this definitely radiates through to our customers. Their satisfaction and appreciation is evident in our high NPS scores and G2 ratings.

Summer — why do you think that clients stick with us?

When I spoke to (a client) yesterday they told us, “We are running on Rails 4.x which has been deprecated by the vendor. We have had the new build ready for six months, but we can’t test it because we simply don’t have the resources.” So they are looking to Testlio to perform the testing. This is a significant problem for them that we can solve.

Summer, do the reasons why you joined Testlio align with why clients stick with us?

Absolutely, I believe there is alignment between why I joined Testlio and why we have strong partnerships with clients. As we’ve already discussed Testlio provides tremendous value to customers through networked testing. That fit in the market was an important factor in why I joined Testlio. In my first few weeks, I’ve been able to join QBRs with our customers and each of those interactions reinforced the value we are providing clients.

Prioritizing the work

When we provide QA service solutions, how do we ensure that only the most important bugs are reported to clients to save them time?

We know that false positives and irrelevant bug reports hurt cadences, development cycles. So we review and prioritize bug lists before the client ever sees them. Our goal is to send back bugs that are actionable and high-impact, every time.

To do that, It’s key to understand what is important to our clients. Do they care more about form functionality and flow? Or do they care more about other facets?

For example, yesterday, we tested an app update, which was just an architectural type change. It’s different from testing new business functionality or a new feature for a user, which requires more exploratory work.

This variation from client to client is why we have a continuous feedback loop. Because we truly want to understand, how was it? What was valuable to you? How can we proactively fine-tune and adjust the prioritization, the coverage, and all the other facets of the engagement? As our clients’ needs and expectations change over time, we seek to continuously learn, adjust and improve.

Testlio platform reports overview

Have we ever struggled to prioritize these bugs or learn with clients? Does anything come to mind?

Certainly. As mentioned, aligning with the client’s priorities & needs is important as you start to test a new application. Recently we were working with a client and had our testers doing exhaustive testing including providing design suggestions. We reported 48 issues during the test run. But the client said, “Only 12 of those were meaningful to us.” They were frustrated, triaging bugs that they didn’t feel provided them value.

When we had a follow-up conversation to align with the customer, we realized we were testing a beta application. So in a beta application, sometimes you click a button, and it will pop up a message that says, “This feature isn’t yet available.”

For our testers, they expected full functionality. So it was natural for them to report these items as a bug.

For the client, that’s the current state application. And the tester shouldn’t be bringing them a bug because that’s what they expect their app to do.

So we missed a deep understanding of the state of the application. It’s a mismatch of expectations. We didn’t make that mistake again.


How do we make onboarding easy for clients?

Understanding the customer’s product or app quality challenges will shape the entire onboarding process as a cornerstone of our collaboration and aligned partnership.

We seek to make onboarding as easy and seamless as possible. And therefore, we have a clearly defined onboarding approach to ensure alignment and visibility of the key milestones associated with it. We will lead them through the whole onboarding process in as little as two weeks.

We want to focus onboarding on our strategic understanding of their app and the value we can bring. Clients look to us for our testing expertise so onboarding is a time where we can be prescriptive in our approach and really guide clients on testing best practices.

Co-managed vs. fully-managed

Let’s talk about our different models. What does fully managed and co-managed mean? Why do clients like the choice?

Our Testlio team analyzes the scope, authors the test cases, builds the team, assigns the tasks, reviews results, triages issues, creates reports. This fits extremely well where there is no dedicated QA team in the client organization or where they are looking to partner with Testlio for a whole segment of their work with minimal effort on their end.

And what’s beneficial is that there’s an option to seamlessly shift between those models. We go fully managed for big regression releases, while some midweek, ad hoc releases can be co-managed. So there’s this fantastic flexibility in the system. And soon, in-house testers will be added to the system as the next phase of transparent, collaborative testing processes. For that, clients will have to wait for the Testlio 3.1 platform.

What are we providing no matter what services model we use?

In both, I think we provide clients a sense of relief. By providing them testing before production they are able to make a business decision about deploying the new application based on risk.

And that relieves revenue pressure. At the end of the day, if your application is not succeeding to fulfill end user’s expectations, then that financially impacts your company.

Most challenging projects

What’s the biggest client challenge we’ve overcome? And what was the positive outcome for the client?

That’s a tricky one. All clients are unique with their unique set of challenges. And how do we define bigger?

But one example that comes to mind is this global-scale sports event broadcasted live on mobile devices. We went through a comprehensive multi-month planning, preparation and rehearsal process with the client team. And on game-day, we were literally side-by-side, in the war room saying go or no go from a quality standpoint. To ensure that everybody across the world could experience the game as intended.

We ended up having a record-breaking 2.5 million streaming users. That was a big deal for us and the customer. And truly, building the approach was very unique —engaging testers from so many different media markets in the US, live for nine hours. It was complex but turned out to be super awesome and rewarding.

Another example here is from one of the leading social network apps. They were working on expanding their global outreach with a localized app. To support their quest, we went through a comprehensive and unique recruitment process to bring in nearly 200 (we screened more than 6,000) linguistic experts in specific locations, countries— all in a matter of months!

We released that feature, and we couldn’t have done it without you.

To this day, we still get thank you notes from that customer saying, “we released that feature, and we couldn’t have done it without you.” So these moments of feedback from our customers are truly heart-warming and one of the key reasons we’re doing this job.

Jumbotron image