Video: The Fenix Project: Cloud-Based Disaster Recovery

“Moving to the cloud” is different for each broadcaster, some are using it for live production, some for their archives, some just for streaming. While confidence in the cloud is increasing and the products are maturing, many companies are choosing to put their ‘second MCR’ in the cloud or, say, tier-2 playout to test the waters, gain experience and wait for a fuller feature set. Sky Italia, has chosen to put all its disaster recovery transmission capability in the cloud.

Davide Gandino joins us from Mile High 2020 to show – and demo – their disaster recovery deployment which covers playout, processing, distribution and delivery to the end-user. Davide explains this was all driven by a major fire at their facility in Rome. At the time, they managed to move their services to Milan with minimal on-air impact, but with destroyed equipment, they were left to rebuild. It wasn’t long before that rebuild was planned for the cloud.

This is no insignificant project, with 117 channels of which only 39 are third-party pass-through going on to four platforms, the full deployment uses 800 cloud encoders. This amounts to 4Gbps being sent up to the cloud and 8Gbps returning. David highlights the design uses both Google and Amazon cloud infrastructure with 3 availability zones in use for both.

A vital part of this project design is that not all 800 encoders would be working 24×7. This misses the point of the cloud, but the only scalable alternative is fully automated deployment which is exactly what Sky chose to do. The key tenants of the project are:

  • Everything automated – Deployment and configuration are automatic
  • Software Defined – All Applications to be software defined
  • Distributed – Distributed solution to absorb the loss of one site
  • Synchronised – All BAU (business as usual) changes to automatically update the DR configuration. This is done with what Sky call the ‘Service Control Layer’.
  • Observed – Monitoring of the DR system will be as good or better than usual operation

To active the DR, Davide tells us that there is a first stage script which launches a Kubernetes cluster on which the management software sits and 13 Kubernetes clusters across Google and AWS which will run the infrastructure itself. The second script, uses Jenkins jobs to deploy and configure the infrastructure such as encoders and DRM modules etc. Davide finishes the talk showing us a video of the deployment of the infrastructure, explaining what is happening as we see the platform being built.Watch now!

Davide Gandino Davide Gandino
Head of Streaming, Cloud & Computing Systems,
Sky Italia