Facebook

Video: The ROI of Deploying Multiple Codecs

Posted on by Russell Trafford-Jones

Adding a new codec to your streaming service is a big decision. It seems inevitable that H.264 will be around for a long time and that new codecs won’t replace it, but just take their share of the market. In the short term, this means your streaming service may also need to deliver H.264 and your new codec which will add complexity and increase CDN storage requirements. What are the steps to justifying a move to a new codec and what’s the state of play today?

In this Streaming Media panel, Jan Ozer is joined by Facebook’s Colleen Henry, Amnon Cohen-Tidhar from Cloudinary and Anush Moorthy from Netflix talk about their experiences with new codecs and their approach to new Codecs. Anush starts by outlining the need to consider decoder support as a major step to rolling out a new codec. The topic of decoder support came up several times during this panel in discussing the merits of hardware versus software decoding. Colleen points out that running VP9 and VVC is possible, but some members of the panel see a benefit in deploying hardware – sometimes deploying on devices like smart TVs, hardware decoding is a must. When it comes to supporting third party devices, we hear that logging is vitally important since when you can’t get your hands on a device to test with, this is all you have to help improve the experience. It’s best, in Facebook’s view, to work closely with vendors to get the most out of their implementations. Amnon adds that his company is working hard to push forward improved reporting from browsers so they can better indicate their capabilities for decoding.

 

 

Colleen talks about the importance of codec switching to enhance performance at the bottom end of the ABR ladder with codecs like AV1 with H264 at the higher end. This is a good compromise between the computation needed for AV1 and giving the best quality at very low bitrates. But Anush points out that storage will increase when you start using two codecs, particularly in the CDN so this needs to be considered as part of the consideration of onboarding new codecs. Dropping AV1 support at higher bitrates is an acknowledgement that we also need to consider the cost of encoding in terms of computation.

The panel briefly discusses the newer codecs such as MPEG VVC and MPEG LCEVC. Colleen sees promise in VVC in as much as it can be decoded in software today. She also says good things about LCEVC suggesting we call it an enhancement codec due to the way it works. To find out more about these, check out this SMPTE talk. Both of these can be deployed as software decoders which allow for a way to get started while hardware establishes itself in the ecosystem.

Colleen discusses the importance of understanding your assets. If you have live video, your approach is very different to on-demand. If you are lucky enough to have an asset that is getting millions upon millions of views, you’ll want to compress every bit out of that, but for live, there’s a limit to what you can achieve. Also, you need to understand how your long-tail archive is going to be accessed to decide how much effort your business wants to put into compressing the assets further.

The video comes to a close by discussing the Alliance of Open Media’s approach to AV1 encoders and decoders, discussing the hard work optimising the libAV1 research encoder and the other implementations which are ready for production. Colleen points out the benefit of webassembly which allows a full decoder to be pushed into the browser and the discussion ends talking about codec support for HDR delivery technologies such as HDR10+.

Watch now!
Speakers

Colleen Henry Colleen Henry
Cobra Commander of Facebook Video Special Forces.
Anush Moorthy Anush Moorthy
Manager, Video & Imagine Encoding
Netflix
Amnon Cohen-Tidhar Amnon Cohen-Tidhar
Senior Director or Video Architecture,
Cloudinary
Jan Ozer Moderator: Jan Ozer
Principal, Stremaing Learning Center
Contributing Editor, Streaming Media

Video: Providing better video experiences for the next billion users

Posted on by Russell Trafford-Jones

What’s the best way for a billion people all on mobile networks to have a universally great streaming experience? It’s not trivial, and no service is perfect, but Facebook set out to find out what problems existed and find ways to fix them. This video explains their approach and solutions.

Denise Noyes from Facebook spoke at Demuxed 2020 about their work in India over the year. For Facebook, India is unique for this research as it represents such a large number of people almost universally using Android phones and mobile data. Not only does this allow them to understand the low-bitrate performance of video, but the Android penetration level simplifies comparisons.

The problems that Denise and her colleagues identified were gaps in the bitrate ladders where the ABR ladder either wasn’t well optimised or didn’t go low enough. There were also some ABR logic/decisions that were seen to be causing problems along with server delays from the CDN and internal congestion within the app. The research looked at ‘average bad sessions per user’ rather than the overall number of bad sessions which would be skewed by how many videos people generally watched.

Covid had a bearing on the research as this was being conducted by in-person interviews within India. These teams had to come home but the relevance of the research was acutely highlighted by the networks in other countries which worsened in response to the rising amount of traffic making them closer to the Indian example.

Denise’s team worked with colleagues throughout the company to create improvements across the whole network and delivery stack. On the encoding front, they decreased the lowest encoding level to 100kbps. This doesn’t look amazing, as seen by the metric score, but it’s better than buffering and can be watchable dependent on content. The GOP size was also increased from 2 seconds to 5. Longer GOP sizes are known to deliver improved bitrate, in this case up to 8%, but there is a tradeoff to pay in latency and how frequently you can move up/down the ABR ladder. Facebook found that the tradeoffs were worth the improvement for the viewers.

Denise introduces FB-MOS, Facebook’s objective model of the MOS objective metric. The lower the number, the worse the video looks. Facebook have used the fact that encoding resolution ‘A’ at, say, 400kbps and 200kbps can look better than encoding resolution ‘A’ at 400kbps and using a lower resolution ‘B’ for the 200kbps encode. This has lead to the ABR having 360p at two bitrates and 480p at two bitrates.

That FB-MOS score comes in handy for avoiding the lowest rungs of the ABR ladder. As their MOS score is quite low, the player will only choose it if it really has no choice otherwise, it will prefer to settle on a higher quality version if it isn’t able to go up the ladder. Ironically, they have also implemented logic to limit who gets the highest bandwidth streams since most users would prefer to spend less on data than get that disproportionately low improvement in quality.

In playback, Denise explains that they have reduced the impact of occasional anomalies on the bandwidth estimation and adjusted prefetching to prefetch the first chunk of all videos it would like to prefetch before getting the next chunk. This has reduced the chance that someone is able to choose a video which hasn’t yet been buffered and hence have to wait for it to start.

Lastly Denise covers the work done at the network layer seeing a move from HTTP/2 to QUIC. We see how the removal of head-of-line blocking has helped and that, not only has this the move to QUIC seen an overall improvement in performance but as congestion increased, QUIC traffic has shown a disproportionate improvement.

Denise concludes highlighting that this work across the network stack with wide collaboration has not only delivered the desired results but is a vital approach for any company looking to make marked improvements in customer experience.

Watch now!
Speaker

Denise Noyes Denise Noyes
Software Developer,
Facebook

Video: Building an 8k encoder + live streaming platform

Posted on by Russell Trafford-Jones

Streamline is a reference system design for premium quality, end to end live streaming all the way from SDI to a player fed from a CDN that works on the web, iOS, and Android devices. It uses commodity computer hardware, free software, and AWS to create an affordable way to learn how to build a high-quality live streaming system.

Already capable of 4K, this project is ideal for people to use as a learning tool to get first-hand experience of how live video works end to end. Now, the project is being extended to be able to four 4K 60fps feeds, or a single 8K stream. Ths update is called Streamline 2.
 

 
Colleen Henry from Facebook introduces the hardware behind the feat as comprising two NVIDIA QUADRO GPUs and one large CPU – a Ryzen 3990x. The equipment is perfectly capable of 8K, but the goal actually is to have enough power to deal with 10bit, 4K, HDR, high frame-rate feeds. The kit’s also intended to be able to encode AV1, LCEVC and VP9. Colleen suggests considering using the Lenovo ThinkStation P620 as a pre-built Threadripper desktop rather than building yourself.

Code for the project can be found at https://streamline.wtf. After encoding, the rest of the work is done in AWS. Caitlin O’Callaghan talks us through how to set up AWS by setting up an m4.xlarge server with the correct firewall and building the code from the Streamline 2 repository and then shows us how to install the encoder.

Watch now!
Speakers

Colleen Henry Colleen Henry
Cobra Commander of Facebook Video Special Forces.
Caitlin O'Callaghan Caitlin O’Callaghan
Former Software Engineering Co-op,
Facebook

Video: Scaling Video with AV1!

Posted on by Russell Trafford-Jones

A nuanced look at AV1. If we’ve learnt one thing about codecs over the last year or more, it’s that in the modern world pure bitrate efficiency isn’t the only game in town. JPEG 2000 and, now, JPEG XS, have always been excused their high bitrate compared to MPEG codecs because they deliver low latency and high fidelity. Now, it’s clear that we also need to consider the computational demand of codec when evaluating which to use in any one situation.

John Porterfield welcomes Facebook’s David Ronca to understand how AV1’s arriving on the market. David’s the director of Facebook’s video processing team, so is in pole position to understand how useful AV1 is in delivering video to viewers and how well it achieves its goals. The conversation looks at how to encode, the unexpected ways in which AV1 performs better than other codecs and the state of the hardware and software decoder ecosystem.

David starts by looking at the convex hull, explaining that it’s a way of encoding content multiple times at different resolutions and bitrates and graphing the results. This graph allows you to find the best combination of bitrate and resolution for a target quality. This works well, but the multiple encodes burdens the decision with a lot of extra computation to get the best set of encoding parameters. As proof of its effectiveness, David cites a time when a 200kbps max target was given for and encoder of video plus audio. The convex hull method gave a good experience for small screens despite the compromises made in encoding fidelity. The important part is being flexible on which resolution you choose to encode because by allowing the resolution to drift up or down as well as the bitrate, higher fidelity combinations can be found over keeping the resolution fixed. This is called per-title encoding and was pioneered by Netflix as discussed in the linked talk, where David previously worked and authored this blog post on the topic.

It’s an accepted fact that encoder complexity increases for every generation. Whilst this makes sense, particularly in the standard MPEG line where MPEG 2 gave way to AVC which gave way to HEVC which is now being superseded by VVC all of which achieved an approximately 50% compression improvement at the cost of a ten-fold computation increase. But David contends that this buries the lede. Whilst it’s true that the best (read: slowest) compression improves by 50% and has a 10% complexity increase, it’s often missed that at the other end of the curve, one of the fastest settings of the newer codec can now match the best of the old codec with a 90% reduction in computation. For companies working in the software world encoding, this is big news. David demonstrates this by graphing the SVT-AV1 encoder against the x265 HEVC encoder and that against x264.

David touches on an important point, that there is so much video encoding going on in the tech giants and distributed around the world, that it’s important for us to keep reducing the complexity year on year. As it is now, with the complexity increasing with each generation of encoder, something has to give in the future otherwise complexity will go off the scale. The Alliance for Open Media’s AV1 has something to say on the topic as it’s improved on HEVC with only a 5% increase in complexity. Other codecs such as MPEG’s LCEVC also deliver improved bitrate but at lower complexity. There is a clear environmental impact from video encoding and David is focused on reducing this.

AOM is also fighting the commercial problem that codecs have. Companies don’t mind paying for codecs, but they do mind uncertainty. After all, what’s the point in paying for a codec if you still might be approached for more money. Whilst MPEG’s implementation of VVC and EVC aims to give more control to companies to help them control their risk, AOM’s royalty-free codec with a defence fund against legal attacks, arguably, gives the most predictable risk of all. AOM’s aim, David explains, is to allow the web to expand without having to worry about royalty fees.

Next is some disappointing news for AV1 fans. Hardware decoder deployments have been delayed until 2023/24 which probably means no meaningful mobile penetration until 2026/27. In the meantime the very good dav1d decoder and also gav1 are expected to fill the gap. Already quite fast, the aim is for them to be able to do 720p60 decoding for average android devices by 2024.

Watch now!
Speakers

David Ronca David Ronca
Director, Video Encoding,
Facebook
John Porterfield
Freelance Video Webcast Producer and Tech Evangelist
JP’sChalkTalks YouTube Channel