Testing solutions that go beyond industry standards: an interview with Summer Weisberg and Kerli Rungi

Tim Ryan , Tim Ryan serves as the Director of Marketing for Testlio and spends his time between Austin, TX and New Orleans, LA.

October 26th, 2020

This month, Testlio announced our hiring of Summer Weisberg. We couldn’t be more excited to have her at Testlio as our services leader, ensuring great client experiences for years to come. More, her hire allows Kerli Rungi (Testlio former Head of Services) to move into a new, exciting role as Head of Learning and Growth.

We sat down to chat with both of them, as leaders at Testlio and in the industry— about the challenge our clients face, how they prioritize services work, onboarding, our different service models, and their favorite client stories.

If you’re a Testlio client, the services team is your first point of contact. Hopefully, this can serve as a meaningful introduction to the people leading our QA solutions charge.

The most common client problem and our solution

What’s the number one challenge faced by our clients? How do we help them?

Kerli:
We all know that the number of apps is growing tremendously – there are nearly 9 million mobile apps in the world today. Considering the fragmented world of devices and apps, developers have just a few seconds to impress the end-user and retain them. If the end-user doesn’t perceive the value of the app— maybe it’s buggy or its usability is poor— he or she will opt for an alternative. I think the one key challenge for our clients is quality and retaining end-users. And that’s why they find a partnership with Testlio so rewarding.

Our unique methods, the way our testers are incentivized to work and collaborate together with the industry expertise of our services team, builds an environment that genuinely nourishes and nurtures quality results.

On top of that, there is the scale of our offering. Global locations, diverse skills, across thousands of real device and OS combinations, using multiple capabilities, all facilitated by a cohesive platform and dedicated services arm. We can scale QA service to any extent, in any way, in moments that matter.

But third and closest to my heart are passion and partnership. We don’t see ourselves merely as a mobile app testing company — we seek true partnership, understanding, service, mutual value. And based on the positive feedback we receive, this definitely radiates through to our customers. Their satisfaction and appreciation is evident in our high NPS scores and G2 ratings.

Summer — why do you think that clients stick with us?

Summer:
I would say, based on the networked testing model and my experience elsewhere, it’s mainly Kerli’s point about moments. Having what you need on critical timelines is highly impactful. Clients don’t have to staff up for one big release or application rewrite that needs major testing.

When I spoke to (a client) yesterday they told us, “We are running on Rails 4.x which has been deprecated by the vendor. We have had the new build ready for six months, but we can’t test it because we simply don’t have the resources.” So they are looking to Testlio to perform the testing. This is a significant problem for them that we can solve.

Summer, do the reasons why you joined Testlio align with why clients stick with us?

Summer:
Absolutely, I believe there is alignment between why I joined Testlio and why we have strong partnerships with clients. As we’ve already discussed Testlio provides tremendous value to customers through networked testing. That fit in the market was an important factor in why I joined Testlio. In my first few weeks, I’ve been able to join QBRs with our customers and each of those interactions reinforced the value we are providing clients.

But outside of testing the other important factor for me was Testlio’s values. If you look at this moment in history, all of the social issues, when people go to Testlio’s ‘About Us’ customer page, you’ll see that we value diversity. And I think people are starting to recognize that diversity makes a difference. One, they want to see companies that are taking it seriously. But two, I truly believe that diverse companies work better and have longer term success.

Prioritizing the work

When we provide QA service solutions, how do we ensure that only the most important bugs are reported to clients to save them time?

Kerli:
We know that false positives and irrelevant bug reports hurt cadences, development cycles. So we review and prioritize bug lists before the client ever sees them. Our goal is to send back bugs that are actionable and high-impact, every time.

Summer:
To do that, It’s key to understand what is important to our clients. Do they care more about form functionality and flow? Or do they care more about other facets?

For example, yesterday, we tested an app update, which was just an architectural type change. It’s different from testing new business functionality or a new feature for a user, which requires more exploratory work.

Kerli:
This variation from client to client is why we have a continuous feedback loop. Because we truly want to understand, how was it? What was valuable to you? How can we proactively fine-tune and adjust the prioritization, the coverage, and all the other facets of the engagement? As our clients’ needs and expectations change over time, we seek to continuously learn, adjust and improve.

Have we ever struggled to prioritize these bugs or learn with clients? Does anything come to mind?

Summer:
Certainly. As mentioned, aligning with the client’s priorities & needs is important as you start to test a new application. Recently we were working with a client and had our testers doing exhaustive testing including providing design suggestions. We reported 48 issues during the test run. But the client said, “Only 12 of those were meaningful to us.” They were frustrated, triaging bugs that they didn’t feel provided them value.

When we had a follow-up conversation to align with the customer, we realized we were testing a beta application. So in a beta application, sometimes you click a button, and it will pop up a message that says, “This feature isn’t yet available.”

For our testers, they expected full functionality. So it was natural for them to report these items as a bug.

For the client, that’s the current state application. And the tester shouldn’t be bringing them a bug because that’s what they expect their app to do.

So we missed a deep understanding of the state of the application. It’s a mismatch of expectations. We didn’t make that mistake again.

Onboarding

How do we make onboarding easy for clients?

Kerli:
Understanding the customer’s product or app quality challenges will shape the entire onboarding process as a cornerstone of our collaboration and aligned partnership.

We seek to make onboarding as easy and seamless as possible. And therefore, we have a clearly defined onboarding approach to ensure alignment and visibility of the key milestones associated with it. We will lead them through the whole onboarding process in as little as two weeks.

Summer:
We want to focus onboarding on our strategic understanding of their app and the value we can bring. Clients look to us for our testing expertise so onboarding is a time where we can be prescriptive in our approach and really guide clients on testing best practices.

Co-managed vs. fully-managed

Let’s talk about our different models. What does fully managed and co-managed mean? Why do clients like the choice?

Kerli:
So the fully managed service truly is, “Let us know what you need and we’ll make it happen.”

Our Testlio team analyzes the scope, authors the test cases, builds the team, assigns the tasks, reviews results, triages issues, creates reports. This fits extremely well where there is no dedicated QA team in the client organization or where they are looking to partner with Testlio for a whole segment of their work with minimal effort on their end.

While the co-managed experience brings more optionality into the services model. Our clients join the testing process and take control of the parts they see fit. They might author their own test cases, triage issues, conduct runs on our comprehensive testing management platform when they’re seeking quick ad hoc results. We are an extension of the QA team, partnered on different parts of the workflow.

And what’s beneficial is that there’s an option to seamlessly shift between those models. We go fully managed for big regression releases, while some midweek, ad hoc releases can be co-managed. So there’s this fantastic flexibility in the system. And soon, in-house testers will be added to the system as the next phase of transparent, collaborative testing processes. For that, clients will have to wait for the Testlio 3.1 platform.

Related: Partnering for remote QA: fully-managed vs. co-managed testing approaches

What are we providing no matter what services model we use?

Summer:
In both, I think we provide clients a sense of relief. By providing them testing before production they are able to make a business decision about deploying the new application based on risk.

And that relieves revenue pressure. At the end of the day, if your application is not succeeding to fulfill end user’s expectations, then that financially impacts your company.

Most challenging projects

What’s the biggest client challenge we’ve overcome? And what was the positive outcome for the client?

Kerli:
That’s a tricky one. All clients are unique with their unique set of challenges. And how do we define bigger?

But one example that comes to mind is this global-scale sports event broadcasted live on mobile devices. We went through a comprehensive multi-month planning, preparation and rehearsal process with the client team. And on game-day, we were literally side-by-side, in the war room saying go or no go from a quality standpoint. To ensure that everybody across the world could experience the game as intended.

We ended up having a record-breaking 2.5 million streaming users. That was a big deal for us and the customer. And truly, building the approach was very unique —engaging testers from so many different media markets in the US, live for nine hours. It was complex but turned out to be super awesome and rewarding.

Another example here is from one of the leading social network apps. They were working on expanding their global outreach with a localized app. To support their quest, we went through a comprehensive and unique recruitment process to bring in nearly 200 (we screened more than 6,000) linguistic experts in specific locations, countries— all in a matter of months!

We released that feature, and we couldn’t have done it without you.

To this day, we still get thank you notes from that customer saying, “we released that feature, and we couldn’t have done it without you.” So these moments of feedback from our customers are truly heart-warming and one of the key reasons we’re doing this job.

Testlio offers the industry’s most flexible testing platform and on-demand test capabilities. Quick smoke test turnarounds, overnight functional testing, in-field location testing, real-time events monitoring, multi-country linguistic testing, weekend-oriented regression testing—and more. Get in touch for a free demo.