Video: Scaling up Anime with Machine Learning and Smart Real Time Algorithms

Too long has video been dominated by natural scenes and compression has been about optimising for skin tones. Recently we have seen technologies taking care of displaying other types of video correctly like computer displays such as computer games, as seen in VVC and also animation optimisation for upscalers as we explore in this talk.

Anime, a Japanese genre of animation, is not very different from an objective point of video from most video cartoons; the drawing style is black lines on relatively simple, solid areas of colour. Anime itself is a clearly distinct genre whose fans are much more sensitive to quality, but for codecs and scalers, 2D animation, in general, is a style that easily shows artefacts.

Up- and down-scaling is the process of making an image of say 1080 pixels high and 1920 wide larger, for instance 2160×3840 or smaller, say to SD resolution. Achieving this without jagged edges or blurriness is difficult and conventional maths can do a decent job, but often leaves something to be desired. Christopher Kennedy from Crunchyroll explains the testing he’s done looking at a super resolution upscaling technique which uses machine learning to improve the quality of upscaled anime video.

Waifu2x is an opensource algorithm which uses Convolutional Neural Networks (CNNs) to scale images and remove artefacts. To start with, Christopher explains the background of traditional algorithmic upscaling discussing the fact that better-looking algorithms take longer so TVs often choose the fastest leading them to look pretty bad if fed SD video. Better for the streaming provider to spend the time doing an upconversion to 4K so allow the viewer a better final quality on their set.

Machine Learning needs a training set and one thing which has contributed to waifu2x’s success in Anime is that it has been trained only on examples of anime leaving it well practised in improving this type of image. Christopher presents the results of his tests comparing standard bilinear and bicubic scaling with waifu2x showing the VMAF, PSNR and SSIM scores.

Finishing off the video, Christopher talks about the time this waifu2x takes to run, the cost of running it in the cloud and he shares some of the command lines he used.

Reference links:

Watch now!

Christopher Kennedy Christopher Kennedy
Staff Video Engineer,

Video: Harness SSAI’s Superpowers

Server-side Ad Insertion (SSAI) is a great option for streaming services delivering video to a wide variety of devices and for those who need to avoid ad blockers. Whilst ad insertion can happen in the player, this mechanism can be interfered with allowing users to avoid ads. Whilst client-side ad insertion can much more easily create a unique stream for each client, dynamic SSAI can now do the same with a better user experience.

This panel from the OTT Leadership Summit at Streaming Media West 2019 brings together Disney, WarnerMerdia and Crunchyroll to share their experiences with SSAI. They discuss beaconing, ad standards, scaling, SCTE and more.

Beaconing goes hand in hand with ad playback providing metrics on what happened. When you perform certain actions, the player will reach out to a URL. This can be used to indicate such things as users skipping or pausing a video. The beacon information can then be used to verify how much of which ads were seen by whom and charge advertisers accordingly.

The panel moves on to discussing scaling using live sports as an example and cover questions to ask vendors to ensure you and they are ready for maximum scale. Bandwidth, is declared the biggest challenge, but a less obvious problem is that your upstream ad providers can’t always scale well. If you rely on calls from your server to others, then it’s vital to understand their scaling capacity and strategy. They discuss issues with losing beacons when operating at scale and the need for detailed logging and debugging in order to spot errors and reconcile the results.

Some time is next spent on VPAID and VAST 4 which are both messaging specifications to allow ad servers to tell applications which ads to play. The panel discusses the pros and cons in their use for SSAI where the stitcher needs to reach out to and ad server in real time to find out which ads to play.

At the end of the discussion, the panel takes questions from the floor but not before discussing SCTE Markers and ‘content conditioning’ which surrounds taking care of your source videos and encoder such that the two assets fit together properly at I-frame boundaries.

Watch now!

Robert Jameson Robert Jameson
Technical Director, Media Enablement
Turner | WarnerMedia
Stephen Gray Stephen Gray
Director, Ad Tech Systems
Walt Disney Direct-to-Consumer & International
Michael Dale Michael Dale
VP Engineering,
Nadine Krefetz Nadine Krefetz
Consultant, Reality Software
Contributing Editor, Streaming Media