Video: How Big Buck Bunny helped track down bugs

Early in the development of a streaming service, using a ‘mock’ player is a good way to be able to quickly test your code, in fact, it’s common to create mock API endpoints. The trouble with mocks is that they are too perfect as they lack the complex behaviour of a real player playing real media.

Evan Farina from LinkedIn explains what they’ve found while developing the LinkedIn video player. Communication from media players comes in the form of tracking events that indicate status such as ‘playing’ or ‘duration’. So when a player initiates playing of a video stream, a tracking event would come back to indicate success or otherwise. When using a mock, if your program works, so will the tracking event. In the real world, these values might be empty, have invalid data in or be functioning just as expected.
 

 

Once you’re in production, swapping your mock for real media has benefits. Evan explains that no longer needing to create/maintain a mock is a time saver. After all, it is code and you need to make sure it’s working and keeping up with production changes. But the main benefit is catching player integration issues and race conditions before releasing to production. Evan’s team found errors in tracking events, inefficiencies in the way media was being loaded and a dependency on free CPU which was previously unknown. All of this was possible before deploying into production.

How can you switch from mocks to real media? One goal is to ensure your tests aren’t reliant on any timing meaning your testing and the player need to be fully asynchronous. To do this, Evan’s created ‘helpers’ which control media playback (play, pause, seek etc) and also wait for a result with a timeout in case they never receive the resultant tracking event.

Evan finishes with some tips for scaling from his team. Firstly, there’s no point making your test take longer than necessary so keep media short. Second: store media locally to avoid network traffic not directly related to the test. Log all events since, at some point, these will be critical to understanding a problem. Lastly, he says to make sure you monitor your test servers since it’s important to make sure they’re not struggling to cope with the load which could affect the results of some of your tests.

Watch now!
Speakers

Evan Farina Evan Farina
Software Engineer,
LinkedIn

Video: AV1 and ARM

AV1’s no longer the slow codec it was when it was released. Real-time encodes and decodes are now practical with open-source software implementations called rav1e for encoding and dav1d for decoding. We’ve also seen in previous talks the SVT-AV1 provides real-time encoding and WebRTC now has a real-time version with the AV1 codec.

In this talk, rav1e contributor Vibhoothi explains more about these projects and how the ARM chipset helps speed up encoding. The Dav1d started project started in 2018 with the intention of being a fast, cross-platform AV1 encoder with a small binary which Vibhoothi says is exactly what we have in 2021. Dav1d is the complementary decoder project. AV1 decoding is found in many places now including in Android Q, in Microsoft’s media extension for it, VLC supports AV1 on linux and macOS thanks to dav1d, AV1 is supported in all major browsers, on NVIDIA and AMD GPUs plus Intel Tiger Lake CPUs. Netflix even use dav1d to stream AV1 onto some mobile devices. Overall, then, we see that AV1 has ‘arrived’ in the sense that it’s in common and increasing use.

The ARM CPU architecture underpins nearly all smartphones and most tablets so ARM is found in a vast number of devices. It’s only relatively recently that ARM has made it into mainstream servers. One big milestone has been the release of Neoverse which is an ARM chip for infrastructure. AWS now offer ARM instances that have a 40% higher performance but a 20% reduced cost. These have been snapped up by Netflix but also by a plethora of non-media companies. Recently Apple has made waves with their introduction of the M1 ARM-based chip for desktops which has benchmarks far in excess of the previous x86 offering which shows that the future for ARM-based implementations of the rav1e encoder and dav1d decoder are bright.

Vibhoothi outlines how dav1d works better on ARM then x86 with improved threading support including hand-written asm optimisations and support for 10-bit assembly. rav1e has wide support in VLC, GStreamer, FFmpeg, libavif and others.

The talk finishes with a range of benchmarks showing how better-than-real-time encoding and decoding is possible and how the number of threads relates to the throughput. Vibhoothi’s final thoughts focus on what’s still missing in the ARM implementations.

Watch now!
Speaker

Vibhoothi Vibhoothi
Developer, VideoLAN
Research Assistant, Trinity College Dublin,
Codec Development, rav1e, Mozilla

Video: Netflix – Delivering better video encodes for legacy devices

With over 139 million paying customers, Netflix is very much in the bandwidth optimisation game. It keeps their costs down, it keeps customers’ costs down for those on metered tariffs and a lower bitrate keeps the service more responsive.

As we’ve seen on The Broadcast Knowledge over the years, Netflix has tried hard to find new ways to encode video with Per-Title encoding, VMAF and, more recently, per-shot encoding as well as moving to more efficient codecs such as AV1.

 

Mariana Afonso from Netflix discusses what do you do with devices that decode the latest encoders either because they are too old or can’t get certification? Techniques such as per-title encoding work well because they are wholly managed in the encoder. Whereas with codecs such as AV1, the decoder has to support it too, meaning it’s not as widely applicable an optimisation.

As per-title encoding was developed within Netflix before they got their VMAF metric finished, it still uses PSNR, explains Mariana. This means there is still an opportunity to bring down bitrates by using VMAF. Because VMAF more accurately captures how the video looks, it’s able to lead optimisation algorithms better and shows gains in tests.

Better than per-title is per-chunk. The per-chunk work done modulates the average target bitrate from chunk to chunk. This avoids over-allocating bits for low-complexity scenes and results in a more consistent quality by 6 to 16%.

Watch now!
Speaker

Mariana Alfonso Mariana Afonso
Research Scientist, Video Algorithms,
Netflix

Video: Growing the Next Generation of Us

Hiring is one of the most important things you will do in your company. Bad hires, at best, are a drain of money, time and opportunity costs. Good hires, on the other hand, can be incredible, long-term assets within your company. So when we have the new hire, we want to onboard them in the best way and continue giving them opportunities to learn and develop. This talk from Disney Streaming Services shares their progressive approach to developing engineers so they can handle the toughest moments when the production system is out of commission – AKA ‘the crucible’.

Alexanadria Shealy explains that teams are often made of people with a whole range of backgrounds, often people who are full of transferable skills, but with no specific from your exact domain. As teams grow, the team needs to constantly strive to onboard new people and bring them into the team both to work within the culture and to round off the skill set of the team at large.

Alexandria says their team has the best of intentions at all times and works hard to prevent any problems. As we all know, though, it’s impossible to prevent problems. “Scaling the software is easier than scaling the team,” she continues, and it’s best not to keep going back to the same people time and time again simply because they have become the experts as this isn’t scalable. The trick is to make the difficult things we do into something which is accessible for the inexperienced.
 

 
Kevin Fuhrman introduces ‘the crucible’ as a stressful place to be. It’s the time that you have a production outage which everyone is waiting to be fixed, and they’re repeatedly asking you when, and they’re watching you. But these fixes are never straight forward. They need a lot of focus and a lot of fault-finding. The stress of delivering under pressure adds to the stress of delivering under pressure. The crucible is not an easy place to be but is well known in broadcasters and streaming providers everywhere.

After your next outage, ask yourself how many of your staff would need to be on a bus travelling to their vacation before your team wouldn’t be able to handle it. In an ideal time, you’d have to have pretty much the whole team on holidays before you couldn’t deal with an outage. But many places know that if a few key people weren’t around, their ability to recover would be significantly compromised.

The advice from the Disney Streaming Services team comes in two packages. The first is taking care of onboarding your new colleagues. Looking for highly applicable tasks which have immediate relevance to them and will allow them to contribute quickly. They suggest giving new joiners a history lesson explaining why things are how they are. How did you choose the software your using, either the systems or the langauges. Explain what you would have preferred to do differently and better. This helps people understand what parts of the system they feel able to improve upon, in code as well as in workflow. It’s important, they explain, to help people spot the parts of the system which were put in because something was simply needed and the parts which are there due to a lot of thought and due diligence. Again, true of code as much as workflows.

We all know that mistakes are important in the learning process. One option laid out is to find parts of projects which are difficult enough to allow someone to dip their tow below the surface and to learn. The underlying point is not to shield junior members of the team from projects. In fact, heading a project with all your experienced engineers may be a way to deliver the project with low risk, but the cost of not investing in getting your less experienced team members involved will be paid when the project is delivered and needing support, maintenance and development. It also works against the interest of the less experienced individuals by reducing the speed at which they advance.

Alexandria and Kevin summarise by saying you should create and grow owners, rotate who is on the A-team, give everyone the chance to be in the crucible and share notes and experiences freely. The video finishes by remarking that the technology of today was built by us standing on the shoulders of giants. It’s important, therefore, that the giants of today remember to let people climb aloft.

Watch now!
Speakers

Alexandria Shealy
Director, Technical Project Management
Disney Streaming Services
Kevin Fuhrman Kevin Fuhrman
Staff Software Engineer,
Disney Streaming Services