With more people cutting the cord on their TV sets and turning to their devices for video content, the rise of streaming media and OTT (Over The Top software) continues to grow in popularity.  According to Statista, OTT video trends shows:

• Revenue in the OTT Video segment is projected to reach US$275.30k in 2022.
• Revenue is expected to show an annual growth rate (CAGR 2022-2027) of 11.60%, resulting in a projected market volume of US$476.60k by 2027.
• The largest segment is OTT Video Advertising with a market volume of US$180.40k in 2022.
• In global comparison, most revenue will be generated in the United States (US$119,100.00 in 2022).
• In the OTT Video segment, the number of users is expected to amount to 0.0m users by 2027.
• User penetration will be 42.9% in 2022 and is expected to hit 53.0% by 2027.
• The average revenue per user (ARPU) in the OTT Video segment is projected to amount to US$84.39 in 2022.

Source: https://www.statista.com/outlook/amo/media/tv-video/ott-video/worldwide

What’s more important is that consumers today are multitasking with more than 1 screen, using both mobile phones, tablets, and laptops to coordinate streaming video streams at the same time, which requires 3.5x more data and bandwidth in the average home in the last 5 years.   Businesses are quick to increase their products and services to capitalize on workstreams which increases the need for more quality and testing of content, quality of service, and performance in this space.    In this post, we’ll discuss a high level of the best approaches to testing streaming media and OTT products.

Streaming Media vs OTT

While both types of streaming services require a strong network bandwidth and a stable internet connection, when it comes to testing, the approach can be slightly different.    To start, streaming media services, also called VOD (Video on Demand), usually involve pre-recorded content, that can be downloadable as files, and favorite shows of the user’s choice. They can be made available at any point in time for the users’ consumption, typically saved on a cloud device or DVR.

OTT software—abbreviated for over-the-top software—is a streaming service provided over the internet rather than by traditional cable and satellite services. Examples like real-time live streams, podcasts, VoIP calls, and audio messaging fall under OTT software. OTT videos are often analogous to cable or TV streaming, nowadays replacing cable and satellite with on-demand services like YouTube TV and Hulu TV.

The biggest difference between OTT and VOD comes into play when you know they aren’t just providing audio and video content, but also offer audio streams like podcasts. And specifically, when you watch a movie from an online video platform, you’re not bound to have to download all the files but enjoy the stream from start to finish.

Test focus areas on streaming platforms

Akamai Research shows that 76% of streaming customers will discontinue their subscription services if the quality of video streaming is poor. This would include significant lag, AV sync issues, and unbearable visual and audio quality. That means the number one objective of quality measurement is the playback of the content. With a combination of OTT performance and automation testing, these areas are the main focus:

Ensure quality video playback.

High-content playback must be smooth and interrupted. Having too many points of lag and pauses will quickly turn off audience engagement. Audio and video need to be in sync and rendered at the decoded level as advertised for an audience to stay engaged. a 4K encoded stream must deliver at the rate of quality else the audience will disengage.

Adapting to bandwidth.

Imagine watching a live sports match and experiencing a lag or delay in the match due to a weak wi-fi signal. With the continued increase of high-quality bandwidth, OTT streaming must keep a continuous media streaming experience, even with high bandwidth isn’t available. Users will compromise a lower video quality but keeping the stream ongoing, versus constant bandwidth interruptions.

Cross-platform interoperability.

Today’s audiences experience streaming services on a variety of platforms and devices. It is expected that I can leave a stream paused on a TV and pick up the rest of the content when moving to a mobile device. It is also a common use case to stream content on more than one device, giving consumers a variety of platforms to use at their disposal when they want. A strong OTT test focus is to maintain streaming quality across all platforms and at every release.

Recovering from network failures.

Because streaming content over networks is expected to be consistent, so does smooth and quick recovery when down networks happen. Customers expect a quality user experience to pick up where things left off, with little patience and desire to pause for too long. A user-friendly test plan is incredibly important to ensure the strength of quality and recovery of a downed stream due to a network hiccup.

Types of Testing

Since streaming media testing is mostly about seamless streaming and visual experiences, the most common bugs revolve around delay, bad UI, and unmanageable solutions for the user. Consider both functional and non-functional approaches.

Performance and Load Testing

Quality of Service with video and audio quality is the number one focus when it comes to streaming media testing. Not only are you checking for A/V sync, lag, latency, heavy network load, video quality playback, encoding and decoding, and network recovery, but you’re checking all the above against network streams versus local streams. Testing non-functional areas like performance and load often require an OTT automation strategy, often simulating and measuring network data while detecting where frames are dropped and recovered. And generating network traffic will require network setup across cloud clusters, so it’s important to be able to account for controlling traffic and settings in your non-functional test plans.

Functional Testing

Aside from the obvious testing of video streaming functions and controls (ie: play, pause, start, stop, forward, replay, etc.), focus strictly on the user experience that supports the streaming content. Formulate test plans around the navigation of the screen, controls, layout, and dashboard functionality in however the content is presented on the platform. A mobile device may have limited controls than a television or a web browser.

Also test multi-operational support like a pause on one screen, and playback on another, and validate if the instance resumes. Test multiple accounts, multiple streams, and min/max balance of thresholds of active and inactive streams when presented. The streaming media experience is purely visible, so representing the customer experience is one of the main areas of functional and regression testing.

Testing tools

To effectively test the areas above, you’ll need to rely on a deep understanding of video streaming services, benchmarked metrics, and an incredible eye for quality drops. Since most of the problems aren’t always obvious to the human eye, you’ll need to rely on test automation tools to assist with your test cycles. Paying attention to a few areas when choosing the right tool:

Scalability

Does the tool help scale up end-to-end testing without having to add more human testers to the project? Find a tool that can help balance tests across platforms, execute tests in multiple development cycles, and allow flexibility in adding/removing tests as functionality changes

Interoperability

Video streaming is now being viewed on devices, tablets, browsers, and TVs, and multi-streamed simultaneously and dynamically. Having a tool that can support all the popular platforms, allow automation of test cases to execute, but also heavy flexibility in controlling settings and gauges. For example, measuring a H.264 stream on a 4k computer monitor will show different results than a stream on a mobile android phone over a 5G connection. The right tool will allow benchmarking directly against those results and allow replaying of the test repeatedly when software iterates over each build.

Conclusion

The goals of OTT testing is largely seamless delivery and quality playback. Developing the right strategy in fulfilling these goals will always begin with an amazing user experience, consistent streaming delivery of content, and critical playback throughout the session. If any of this presents unbearable delays, the audience will quickly disengage and correlate the product with poor performance. Locating the right tools and building the right test cases is the step to successfully testing streaming media services.

If you are interested to work with our team on a streaming media testing project like this, please contact us here.