Video: The Fenix Project: Cloud-Based Disaster Recovery

“Moving to the cloud” is different for each broadcaster, some are using it for live production, some for their archives, some just for streaming. While confidence in the cloud is increasing and the products are maturing, many companies are choosing to put their ‘second MCR’ in the cloud or, say, tier-2 playout to test the waters, gain experience and wait for a fuller feature set. Sky Italia, has chosen to put all its disaster recovery transmission capability in the cloud.

Davide Gandino joins us from Mile High 2020 to show – and demo – their disaster recovery deployment which covers playout, processing, distribution and delivery to the end-user. Davide explains this was all driven by a major fire at their facility in Rome. At the time, they managed to move their services to Milan with minimal on-air impact, but with destroyed equipment, they were left to rebuild. It wasn’t long before that rebuild was planned for the cloud.

This is no insignificant project, with 117 channels of which only 39 are third-party pass-through going on to four platforms, the full deployment uses 800 cloud encoders. This amounts to 4Gbps being sent up to the cloud and 8Gbps returning. David highlights the design uses both Google and Amazon cloud infrastructure with 3 availability zones in use for both.

A vital part of this project design is that not all 800 encoders would be working 24×7. This misses the point of the cloud, but the only scalable alternative is fully automated deployment which is exactly what Sky chose to do. The key tenants of the project are:

  • Everything automated – Deployment and configuration are automatic
  • Software Defined – All Applications to be software defined
  • Distributed – Distributed solution to absorb the loss of one site
  • Synchronised – All BAU (business as usual) changes to automatically update the DR configuration. This is done with what Sky call the ‘Service Control Layer’.
  • Observed – Monitoring of the DR system will be as good or better than usual operation

To active the DR, Davide tells us that there is a first stage script which launches a Kubernetes cluster on which the management software sits and 13 Kubernetes clusters across Google and AWS which will run the infrastructure itself. The second script, uses Jenkins jobs to deploy and configure the infrastructure such as encoders and DRM modules etc. Davide finishes the talk showing us a video of the deployment of the infrastructure, explaining what is happening as we see the platform being built.Watch now!

Davide Gandino Davide Gandino
Head of Streaming, Cloud & Computing Systems,
Sky Italia

Video: Moving Live Video Quality Control from the Broadcast Facility to the Living Room

Moving an 24×7 on-site MCR into people’s rooms is not trivial, but Disney Streaming Services, in common with most broadcasters knew they had to move people home, often into their living rooms. Working in an MCR requires watching incoming video to check content which is not easy to do at home, particularly when some of their contribution arrives at 100 Mb/s. These two MCRs in San Francisco and NYC covering Hulu Live & ESPN+ along with other services had two weeks to move remote.

Being a major streaming operator, DSS had their own encoding product called xCoder. DSS soon realised this would be their ticket to making home working viable. As standard, these encoders reject any video which doesn’t match a small range of templates. Michael Rappaport takes us how they wrote scripts to use ffprobe to analyse the desired video and then configure the xCoder just the right way. The incoming video goes straight to xCoder without being ‘groomed’ as it normally wood to add closed captions, ABR etc.

Aside from bandwidth, it was also important to provide these streams as close to real-time as possible, as the operators needed to see ‘right now’ to do their job effectively. This is why the ‘grooming’ section is skipped as that would add latency but also the added functions such as PID normalisation and closed caption insertion aren’t needed. Michael explains that when a feed is needed, it will call out to the whole encoder pool, find an underutilised one and then can program it automatically using an API.

Watching this at home was made possible by some work done by Disney Streaming Services to allow their player to receive feeds directly from an xCoder without having any problems decoder parameters. Michael doesn’t mention what protocol they use, but as the xCoder creates a proprietary video stream, so they could be used that carried over TCP.

Made their own players to the receiver from the xCoders. xCoder, as a standalone, produces a proprietary TCP stream. xCoder exposes an API hook that allows us to quickly determine things like frame rate, resolution, and even whether or not the xCoder is able to subscribe to the stream

Watch now!

Michael Rappaport Michael Rappaport
Senior Manager, Encoding Administration,
Disney Streaming Services

Video: FOX – Uncompressed live sports in the cloud

Is using uncompressed video in the cloud with just 6 frames of latency to get there and back ready for production? WebRTC manages sub-second streaming in one direction and can even deliver AV1 in real-time. The key to getting down to a 100ms round trip is to move down to millisecond encoding and to use uncompressed video in the cloud. This video shows how it can be done.

Fox has a clear direction to move into the cloud and last year joined AWS to explains how they’ve put their delivery distribution into the cloud remuxing feeds for ATSC transmitters, satellite uplinks, cable headends and encoding for internet delivery, In this video, Fox’s Joel Williams, Evan Statton from AWS explain their work together making this a reality. Joel explains that latency is not a very hot topic for distribution as there are many distribution delays. The focus has been on getting the contribution feeds into playout and MCR monitoring quickly. After all, when people are counting down to an ad break, it needs to roll exactly on zero.

Evan explains the approach AWS has taken to solving this latency problem and starts with considering using SMPTE’s ST 2110 in the cloud. ST 2110 has video flows of at least 1 Gbps, typically and when implemented on-premise is typically built on a dedicated network with very strict timing. Cloud datacentres aren’t like that and Evan demonstrates this showing how across 8 video streams, there are video drops of several seconds which is clearly not acceptable. Amazon, however, has a product called ‘Scalable Reliable Datagram’ which is aimed at moving high bitrate data through their cloud. Using a very small retransmission buffer, it’s able to use multiple paths across the network to deliver uncompressed video in real-time. The retransmission buffer here being very small enables just enough healing to redeliver missing packets within the 16.7ms it takes to deliver a frame of 60fps video.

On top of SRD, AWS have introduced CDI, the Cloud Digital Interface, which is able to describe uncompressed video flows in a way already familiar to software developers. This ‘Audio Video Metadata’ layer handles flows in the same way as 2110, for instance keeping essences separate. Evan says this has helped vendors react favourably to this new technology. For them instead of using UDP, SRD can be used with CDI giving them not only normal video data structures but since SRD is implemented in the Nitro network card, packet processing is hidden from the application itself.

The final piece to the puzzle is keeping the journey into and out of the cloud low-latency. This is done using JPEG XS which has an encoding time of a few milliseconds. Rather than using RIST, for instance, to protect this on the way into the cloud, Fox is testing using ST 2022-7. 2022-7 takes in two identical streams on two network interfaces, typically. This way it should end up with two copies of each packet. Where one gets lost, there is still another available. This gives path redundancy which a single stream will never be able to offer. Overall, the test with Fox’s Arizona-based Technology Center is shown in the video to have only 6 frames of latency for the return trip. Assuming they used a California-based AWS data centre, the ping time may have been as low as two frames. This leaves four frames for 2022-7 buffers, XS encoding and uncompressed processing in the cloud.

Watch now!

Joel Williams Joel Wiliams
VP of Architecutre & Engineering,
Fox Corporation
Evan Statton Evan Statton
Principal Architect, Media & Entertainment,

Video: State of the Streaming Market 2021

Streaming Media is back to take the pulse of the Streaming market following on from their recent, mid-year survey measuring the impact of the pandemic. This is the third annual snapshot of the state of the streaming market which will be published by Streaming Media in March. To give us this sneak peak, Eric Schumacher-Rasmussen is joined by colleague Tim Siglin and Harmonic Inc.’s Robert Gambino,

They start off with a look at the demographics of the respondents. It’s no surprise that North America is well represented as Streaming Media is US-based and both the USA and Canada have very strong broadcast markets in terms of publishers and vendors. Europe is represented to the tune of 14% and South America’s representation has doubled which is in line with other trends showing notable growth in the South American market. In terms of individuals, exec-level and ‘engineering’ respondents were equally balanced with a few changes in the types of institutions represented. Education and houses of worship have both grown in representation since the last survey.

Of responding companies, 66% said that they both create and distribute content, a percentage that continues to grow. This is indicative, the panel says, of the barrier to entry of distribution continuing to fall. CDNs are relatively low cost and the time to market can be measured in weeks. Answering which type of streaming they are involved in, live and on-demand were almost equal for the first time in this survey’s history. Robert says that he’s seen a lot of companies taking to using the cloud to deliver popups but also that streaming ecosystems are better attuned to live video than they used to be.

Reading the news, it seems that there’s a large migration into the cloud, but is that shown in the data? When asked about their plans to move to the cloud, around a third had already moved but only a quarter said they had no plans. This means there is plenty of room for growth for both cloud platforms and vendors. In terms of the service itself, video quality was the top ‘challenge’ identified followed by latency, scalability and buffering respectively. Robert points out better codecs delivering lower bitrates helps alleviate all of these problems as well as time to play, bandwidth and storage costs.

There have been a lot of talks on dynamic server-side ad insertion in 2020 including for use with targetted advertising, but who’s actually adopting it. Over half of respondents indicated they weren’t going to move into that sphere and that’s likely because many governmental and educational services don’t need advertising to start with. But 10% are planning to implement it within the next 12 months which represents a doubling of adoption, so growth is not slow. Robert’s experience is that many people in ad sales are still used to selling on aggregate and don’t understand the power of targetted advertising and, indeed, how it works. Education, he feels, is key to continuing growth.

The panel finishes by discussing what companies hope to get out of the move to virtualised or cloud infrastructure. Flexibility comes in just above reliability with cost savings only being third. Robert comes back to pop-up channels which, based on the release of a new film or a sports event, have proved popular and are a good example of the flexibility that companies can easily access and monetise. There are a number of companies that are heavily investing in private cloud as well those who are migrating to public cloud. Either way, these benefits are available to companies who invest and, as we’re seeing in South America, cloud can offer an easy on-ramp to expanding both scale and feature-set of your infrastructure without large Capex projects. Thus it’s the flexibility of the solution which is driving expansion and improvements in quality and production values.

Watch now!

Tim Siglin Tim Siglin
Contributing Editor, Streaming Media Magazine
Founding Executive Director, HelpMeStream
Robert Gambino Robert Gambino
Director of Solutions,
Harmonic Inc.
Eric Schumacher-Rasmussen Moderator: Eric Schumacher-Rasmussen
Editor, Streaming Media