Video: Decoder Complexity Aware AV1 Encoding Optimization

AV1’s been famous for very low encoding speed, but as we’ve seen from panel like this, AV1 encoding times have dropped into a practical range and it’s starting to gain traction. Zoe Liu/strong>, CEO of Visionular, is here to talk at Mile High Video 2020 about how careful use of encoding parameters can deliver faster encodes, smooth decodes, and yet balance that balance with codec efficiency.

Zoe starts by outlining the good work that’s been done with the SVT-AV1 encoder which leaves it ready for deployment, as we heard previously from David Ronca of Facebook. Similarly the Dav1d decoder has recently made many speed improvements, now being able to easily decode 24fps on mobiles using between 1.5 and 3 Snapdragon cores depending on resolution. Power consumption has been measured as higher than AVC decoding but less than HEVC. Further to that, hardware support is arriving in many devices like TVs.

Zoe then continues to show ways in which encoding can be sped up by reducing the calculations done which, in turn, increased decoder speed. Zoe’s work has exposed settings that significantly speed up decoding but have very little effect on the compression efficiency of the codec which opens up use cases where decoding was the blocker and a 5% reduction in the ability to compress is a price worth paying. One example cited is ignoring partition sizes of less than 8×8. These small partitions can be numerous and bog down calculations but their overall contribution to bitrate reduction is very low.

All of these techniques are brought together under the heading of Decoder Complexity Aware AV1 Encoding Optimization which, Zoe explains, can result in an encoding speed-up of over two times the original framerate i.e. twice real-time on an Intel i5. Zoe concludes that this creates a great opportunity to apply AV1 to VOD use cases.

Watch now!
Speaker

Zoe Liu Zoe Liu
CEO,
Visionular

Video: LCEVC, The Compression Enhancement Standard

MPEG released 3 codecs last year, VVC, LCEVC and EVC. Which one was unlike the others? LCEVC is the only one that is an enhancement codec, working in tandem with a second codec running underneath. Each MPEG codec from last year addressed specific needs with VVC aiming at comprehensive bitrate savings while EVC aims to push encoding further whilst having a patent-free base layer.

In this talk, we hear from Guido Meardi from V-Nova who explains why LVECV is needed and how it works. LCEVC was made, Guido explains, to cater to an increasingly crowded network environment with more and more devices sending and receiving video both in residential and enterprise. LCEVC helps by reducing the bitrate needed for a certain quality level but, crucially, reduces the computation needed to achieve good quality video which not only benefits IoT and embedded devices but also general computing.

LCEVC uses a ‘base codec’ which is any other codec, often AVC or HEVC, which runs at a lower resolution than the source video. By using this hybrid technique, LCEVC aims to get the best video compression out of the codec yet by running the encode at a quarter resolution, allowing this to be done on low-power hardware. LCEVC then deals with reconstructing two enhancement layers and a, relatively simple, super-resolution upsample. This is all achieved with a simple toolset and all of the LCEVC computation can be done in CPU, GPU or other types of computation; it’s not bound to hardware acceleration.

Guido presents a number of results from tests against a whole range of codecs from VVC to AV1 to plain old AVC. These tests have been done by a number of people including Jan Ozer who undertook a whole range of tests. All of these tests point to the ability of LCEVC to extend bandwidth savings of existing codecs, new and old.

Guido shows an example of a video only comprising edges (apart from mid-grey) and says that LCEVC encodes this not only better than HEVC but also with an algorithm two orders of magnitude less. We then see an example of a pure upsample and an LCEVC encode. Upsampling alone can look good, but it can’t restore information and when there are small textual elements, the benefit of having an enhancement layer bringing those back into the upsampled video is clear.

On the decode side, Guido presents tests showing that decode is also quicker by at least two times if nor more, and because most of the decoding work is involved in decoding the base layer, this is still done using hardware acceleration (for AVC, HEVC and other codecs depending on platform). Because we can still rely on hardware decoding, battery life isn’t impacted.

Watch now!
Speakers

Guido Meardi Guide Meardi
CEO & Co-Founder,
V-Nova

Video: Video Vectorisation

Yesterday we learnt about machine learning improving VVC. But VVC has a fundamental property which limits its ability to compress: it’s raster-based. Vector graphics are infinitely scalable with no loss of quality and are very efficient. Instead of describing 100 individual pixels, you can just define a line 100 pixels long. This video introduces a vector-based video codec which dramatically reduces bitrate.

Sam Bhattacharyya from Vectorly introduces this technique which uses SVG graphics, a well-established graphics standard available in all major web browsers. It describes shapes with XML and is similar to WebGL. The once universal Adobe Flash was able to animate SVG shapes as part of its distinctive ‘flash animations’. The new aspect here is not to start with SVG shapes and animate them, but to create those shapes from video footage and recreate that same video but with vectors.

Sam isn’t shy to acknowledge that video vectorisation is a technique which works well on animation with solid colours; Peppa Pig being the example shown. But on more complex imagery without solid colours and sharp lines, this technique doesn’t result in useful compression. To deal with shaded animation, he explains a technique of using mesh gradients and diffusion curves to represent gradually changing colours and shades. Sam is interested in exploring a hybrid mode whereby traditional video had graphics overlayed using this low-bandwidth vector-based codec.

The technique uses machine learning/AI techniques to identify the shapes, track them and to put them in to keyframes. The codec plays this back by interpolating the motion. This can produce files playable at HD of only 100kbps. For the right content, this can be a great option given it’s based on established standards, is low bitrate and can be hardware accelerated.

Sam’s looking for interest from the community at large to help move this work forward.

Watch now!
Speaker

Sam Bhattacharyya Sam Bhattacharyya
CEO, Co-founder
Vectorly

Video: Deep Neural Networks for Video Coding

We know AI is going to stick around. Whether it’s AI, Machine Learning, Deep Learning or by another name, it all stacks up to the same thing: we’re breaking away from fixed algorithms where one equation ‘does it all’ to a much more nuanced approached with a better result. This is true across all industries. Within the Broadcast industry, one way it can be used is in video and audio compression. Want to make an image smaller? Downsample it with a Convolutional Neural Network and it will look better than Lanczos. No surprise, then, that this is coming in full force to a compression technology near you.

In this talk from Comcast’s Dan Grois, we hear the ongoing work to super-charge the recently released VVC by replacing functional blocks with neural-networks-based technologies. VVC has already achieved 40-50% improvements over HEVC. From the work Dan’s involved with, we hear that more gains are looking promising by using neural networks.

Dan explains that deep neural networks recognise images in layers. The brain does the same thing having one area sensitive to lines and edges, another to objects, another part of the brain to faces etc. A Deep Neural Network works in a similar way.
 

 

During the development of VVC, Dan explains, neural network techniques were considered but deemed too memory- or computationally-intensive. Now, 6 years on from the inception of VVC, these techniques are now practical and are likely to result in a VVC version 2 with further compression improvements.

Dan enumerates the tests so far swapping out each of the functional blocks in turn: intra- and inter-frame prediction, up- and down-scaling, in-loop filtering etc. He even shows what it would look like in the encoder. Some blocks show improvements of less than 5%, but added together, there are significant gains to be had and whilst this update to VVC is still in the early stages, it seems clear that it will provide real benefits for those that can implement these improvements which, Dan highlights at the end, are likely to require more memory and computation than the current version VVC. For some, this will be well worth the savings.

Watch now!
Speaker

Dan Grois Dan Grois
Principal Researcher,
Comcast