Video: Advanced Video Coding Standards AVC

Whilst the encoding landscape is shifting, AVC (AKA H.264) still dominates many areas of video distribution so, for many, understanding what’s under the hood opens up a whole realm of diagnostics and fault finding that wouldn’t be possible without. Whilst many understand that MPEG video is built around I, B and P frames, this short talk offers deeper details which helps how it behaves both when it’s working well and otherwise.

Christian Timmerer, co-founder of Bitmovin, starts his lesson on AVC with the summary of improvements in AVC over the basic MPEG 2 model people tend to learn as a foundation. Improvements such as variable block size motion compensation, multiple reference frames and improved adaptive entropy coding. We see that, as we would expect the input can use 4:2:0 or 4:2:2 chroma sub-sampling as well as full 4:4:4 representation with 16×16 macroblocks for luminance (8×8 for chroma in 4:2:0). AVC can handle Pictures split into several slices which are self-contained sequences of macroblocks. Slices themselves can then be grouped.

Intra-prediction is the next topic where by an algorithm uses the information within the slice to predict a macroblock. This prediction is then subtracted from the actual block and coded thereby reducing the amount of data that needs to be transferred. The decoder can make the same prediction and reconstruct the full block from the data provided.

The next sections talk about motion prediction and the different sizes of macroblocks. A macroblock is a fixed area on the picture which can be described by a mixture of some basic patterns but the more complex the texture in the block, the more patterns need to be combined to recreate it. By splitting up the 16×16 block, we can often find a simpler way to describe the 8×8 or 8×16 shapes than if they had to encompass a whole 16×16 block.

 

B-frames are fairly well understood by many, but even if they are unfamiliar to you, Christian explains the concept whereby B-frames provide solely motion information of macroblocks both from frames before and after. This allows macroblocks which barely change to be ‘moved around the screen’ so to speak with minimal changes other than location. Whilst P and I frames provide new macroblocks, B-frames are intended just to provide this directional information. Christian explains some of the nuances of B-frame encoding including weighted prediction.

Quantisation is one of the most important parts of the MPEG process since quantisation is the process by which information is removed and the codec becomes lossy. Thus the way this happens, and the optimisations possible are key so Christian covers the way this happens before explaining the deblocking filter available. After splitting the picture up into so many macroblocks which are independently processed, edges between the blocks can become apparent so this filter helps smooth any artefacts to make them more pleasing to the eye. Christian finishes talking about AVC by exploring entropy encoding and thinking about how AVC encoding can and can’t be improved by adding more memory and computation to the encoder.

Watch now!
Speaker

Christian Timmerer Christian Timmerer
CIO & Cofounder, Bitmovin
Associate Professor, Universität Klagenfurt

Video: Video Compression Basics

Video compression is used everywhere we look. So often is it not practical to use uncompressed video, that everything in the consumer space video is delivered compressed so it pays to understand how this works, particularly if part of your job involves using video formats such as AVC, also known as H.264 or HEVC, AKA H.265.

Gisle Sælensminde from Vizrt takes us on this journey of creating compressed video. He starts by explaining why we need uncompressed video and then talks about containers such as MPEG-2 Transport Streams, mp4, MOV and others. He explains that the container’s job is partly to hold metadata such as the framerate, resolution and timestamps among a long list of other things.

Gisle takes some time to look at the past timeline of codecs in order to understand where we’re going from what went before. As many use the same principles, Gisle looks at the different type of frames inside most compressed formats – I, P and B frames which are used in set patterns known as GOPs – Group(s) of Pictures. A GOP defines how long is between I frames. In the talk we learn that I frames are required for a decoder to be able to tune in part way through a feed and still start seeing some pictures. This is because it’s the I frame which holds a whole picture rather than the other types o frame which don’t.

Colours are important, so Gisle looks at the way that colours are represented. Many people know about defining colours by looking at the values of Red, Green and Blue, but fewer about YUV. This is all covered in the talk so we know about conversion between the two types.

Almost synonymous with codecs such as HEVC and AVC are Macroblocks. This is the name given to the parts of the raster which have been spit up into squares, each of which will be analysed independently. We’ll look at who these macro blocks are used, but Gisle also spends some time looking to the future as both HEVC, VP9 and now AV1 use variable-size macro block analysis.

A process which happens throughout broadcast is chroma subsampling. This topic, whereby we keep more of the luminance channel than colours, is explored ahead of looking at DCTs – Discrete Cosine Transforms – which are foundational to most video codecs. We see that by analysing these macro blocks with DCTs. we can express the image in a different way and even cut down on some of the detail we get from DCTs in order to reduce the bitrate.

Before some very useful demos looking at the result of varying quantisation across a picture, the difference signal between the source and encoded picture plus deblocking technology to hide some of the artefacts which can arise from DCT-based codecs when they are pushed for bandwidth.

Gisle finishes this talk at Media City Bergen by taking a number of questions from the floor.

Watch now!
Speaker

Gisle Sælensminde Gisle Sælensminde
Senior Software Engineer,
Vizrt

Video: Things Developers Believe About Video Files (Proven Wrong by User Uploads)


For many transcoding workflows, efficiency or quality are the primary factors defining how they are created. But when ingesting user-generated videos like those uploaded to the online video platform, Vimeo, life gets difficult. Dealing with the wide variety of formats uploaded and the many edge cases in the way that otherwise normal AVC videos are delivered means throwing out any assumptions you ever had and analysing every aspect of the file.

Senior video encoding engineer, Derek Buitenhuis takes us through the many lessons he and his colleagues have learnt over the years. Don’t, he says, assume that properties don’t change between frames – sometimes they change in every single frame. Assuming that you have a single frame rate throughout the video is another ‘no no’ as there are many variable-frame rate videos.

Derek also looks at dealing with samples stamped with negative timestamps, the need for sample durations, the myriad of issues seeking through a file, the fun of having some frames that aren’t displayed and multiple-track videos.

Colour spaces, no surprise to anyone, cause handling difficulties for example if the bitstream colour properties are different to those in the container. As the talk finishes, we’re left considering old MPEG2 files that can have unavoidable banding, replicating looping MOV files, and dealing with QuickTime special effects channels that animate a fire on the screen.

Watch now!
Speakers

Derek Buitenhuis Derek Buitenhuis
Senior Video Encoding Engineer,
Vimeo

Video: Non-standard Codecs with Standard WebRTC

WebRTC is used on a massive scale thanks to Facebook messenger and Google, but when it comes to video streaming services, some find its open source codec VP8 too restrictive. WebRTC is actively evolving to adapt and become codec agnostic though this work is ongoing. In the meantime, Comcast is here to show us there is a way to inject the codec of your choice into WebRTC.

Finding that many of their video capture devices, CCTV cameras and the like, had hardware AVC encoders, Bryan Meissner explains Comcast didn’t feel it had much of a choice in codec, therefore they looked for a way to make WebRTC to carry AVC.

While forcing an unsupported codec into a protocol wasn’t ideal, they were able to leave much of WebRTC unchanged. The RTP and Data channels were established as normal and peering continued to work as ever. With control of both the send and receive side, the team found they could pick out the data from the WebRTC stack ahead of the normal decoder and feed that into Exoplayer using its API. This allowed playback on Android devices. Bryan goes on to explain the approach for iOS and web browsers. As WebRTC is ‘baked in’ to browsers, there really are very few ways to change the signal flow.

At the end of the day, Comcast made this work and used it in production or many years, Jeff Cardillo explains as he wraps up this video. But he also takes time to talk through some of the problems. Having to bypass parts of a program with parts of another library does increase complexity. Not only does the code become more complex but the code becomes platform specific, you need control over the source and keeping the individual parts synchronously up to date can be a balancing act.

Jeff finishes this talk from Demuxed SF 2019 by elaborating on the mobile and browser tradeoffs at play.

Watch now!
Speakers

Bryan Meissner Bryan Meissner
Sr. Director Software Development and Engineering,
Comcast
Jeff Cardillo Jeff Cardillo
Principal Software Engineer,
Comcast Interactive Media