How to ensure a quality live stream performance Dayana Mayfield , Dayana is a QA engineer turned technology writer living in Milan, Italy. She's always down for a smoothie. June 18th, 2021 Globally, the overall consumption of online video is trending up while television watch time is trending down. In a study of 4,500 consumers from 9 major countries, online video viewing grew 85% between 2016 and 2020. Meanwhile, television view time fell 10%. Previously, the onus of live stream testing and success was on networks and cable providers. But with OTT (over-the-top) streaming on the rise, media organizations face increased demand to deliver high video quality experience. When it comes to live streaming especially, the pressure is on. Viewers want to watch sporting events, concerts, and award shows live—not after the fact. They want no interruptions or issues. They’re usually paying for the service, and may have signed up specifically for that event. Simply put, this live streaming experience can make or break their loyalty. If you’re prepping for a massive upcoming live streaming event, here’s what you need to do to ensure quality and performance on the day of. Scope out all of the requirements Planning a huge live test cycle? It helps to scope out the parameters of testing well in advance. When Testlio partners with large media companies (contact us for media case studies), we plan months ahead to ensure flawless live stream experiences. Video types First up, you’ll need to understand what types of video that testers will need to cover. For example, if there will be 4K capabilities or UHD capabilities. Device types Oculus VR headset lovers want to watch anything with their device, so you can’t rule out VR headsets. Of course, you’ll also want to consider the popular mobile, computer, and tablet devices in your markets. Covering a wide breadth of device options allows testers to cover a range of use cases and test for as many potential customers as possible. Number of testers From our experience working on live events, it’s better to overstaff than understaff. You’ll need to consider the number of testers based on the methods of distribution, the devices you need to test, and the geographical markets you need to cover. Location of testers The location of your testers is critical. For popular live events, have at least one tester in each large metropolitan area. That way, if a CDN or broadcast goes down, you’ll know about it long before a complaint ends up on social media. Run a pre-event practice cycle It’s vital to practice before the live event. Not only will you discover potential issues with CDNs and mobile responsiveness, but you’ll also find plenty of ways to fix your team’s approach to live stream performance testing. Here are some of the things you’ll need to do during the test run: Use common language Most testers have far more experience with software and app testing than live stream testing. Testing managers need to ensure that the network of testers can effectively communicate, especially using specific terms and more familiar language. Testing managers will also often discuss with testers: What CDNs are being served Which APIs are feeding the live event Where different types of issues should be logged (test management system only, or also Slack) The names of key metro areas and their networks Focus on key device types During this simulated test, you’ll also want to ensure that you’re testing on all of the key device types. Make a list of 20 to 50 devices you wish to cover and assign suitable testers who already own these devices. Plan for live event collaboration Have a live war room with your head of QA and your head of engineering, so there’s instant communication. During the live event, they should be in a room (virtual or in-person) together. This way, necessary escalation is instantaneous. The head of engineering can immediately assign fixes to the right people. Consider hooking up testers and engineers in Slack Test management systems are great for most test cycles. When it comes to testing live streams however, Slack is an added benefit to communication. Testers can add issues to the test management system, then drop the link (for major issues) in Slack. This way, engineers can check it live and get notified without delay. To ensure the live stream quality and app experience for your users, download our latest guide. Test for everything, not just the broadcast Get your QA testing team onboard to help with related quality issues that could arise during the stream. In-app marketing messages For media companies that own and offer their own apps, in-app marketing messages can increase customer retention and revenue. You’ll want to test that the banners and notifications are working correctly. You might also want to check your ad analytics to ensure that your advertisers will receive the data they expect. User account credentials Test various types of user accounts, from free trials to paying customers, to people trying to sign up for a trial again. If you only allow streaming on a couple of devices at a time for a single account, you can have testers work together to verify this as well. In light of how common account sharing can be, testers need to monitor and test for multiple devices logging in with a single account. Localization If your live stream is presented in different countries and regions, you’ll want to have testers in those areas check all of the UX copy. The language should go beyond translation correctness, and accurately reflect your brand. Device responsiveness It’s best to also have testers run through the various features of your app using your top priority devices. For example, they might want to check their account settings, viewing options, watchlist, etc. Payment funnels Don’t make it difficult to purchase things. Have testers run through various upgrades and upsell funnels, along with different payment methods. Set your plan into motion during the live event Now, it’s game day. (Or concert day.) It’s time to set into motion everything you’ve been prepping so hard for. While there are tons of technicalities to be mindful of, the most important thing is that you support your testing team in hunting down these issues and resolving them quickly. During the live stream, your testers should be: Watching live Collaborating on issues Efficiently communicating with engineering leads If you get the pre-event testing dress rehearsal and live collaboration right, you won’t leave quality or performance to chance.