Video: Growing the Next Generation of Us

Hiring is one of the most important things you will do in your company. Bad hires, at best, are a drain of money, time and opportunity costs. Good hires, on the other hand, can be incredible, long-term assets within your company. So when we have the new hire, we want to onboard them in the best way and continue giving them opportunities to learn and develop. This talk from Disney Streaming Services shares their progressive approach to developing engineers so they can handle the toughest moments when the production system is out of commission – AKA ‘the crucible’.

Alexanadria Shealy explains that teams are often made of people with a whole range of backgrounds, often people who are full of transferable skills, but with no specific from your exact domain. As teams grow, the team needs to constantly strive to onboard new people and bring them into the team both to work within the culture and to round off the skill set of the team at large.

Alexandria says their team has the best of intentions at all times and works hard to prevent any problems. As we all know, though, it’s impossible to prevent problems. “Scaling the software is easier than scaling the team,” she continues, and it’s best not to keep going back to the same people time and time again simply because they have become the experts as this isn’t scalable. The trick is to make the difficult things we do into something which is accessible for the inexperienced.
 

 
Kevin Fuhrman introduces ‘the crucible’ as a stressful place to be. It’s the time that you have a production outage which everyone is waiting to be fixed, and they’re repeatedly asking you when, and they’re watching you. But these fixes are never straight forward. They need a lot of focus and a lot of fault-finding. The stress of delivering under pressure adds to the stress of delivering under pressure. The crucible is not an easy place to be but is well known in broadcasters and streaming providers everywhere.

After your next outage, ask yourself how many of your staff would need to be on a bus travelling to their vacation before your team wouldn’t be able to handle it. In an ideal time, you’d have to have pretty much the whole team on holidays before you couldn’t deal with an outage. But many places know that if a few key people weren’t around, their ability to recover would be significantly compromised.

The advice from the Disney Streaming Services team comes in two packages. The first is taking care of onboarding your new colleagues. Looking for highly applicable tasks which have immediate relevance to them and will allow them to contribute quickly. They suggest giving new joiners a history lesson explaining why things are how they are. How did you choose the software your using, either the systems or the langauges. Explain what you would have preferred to do differently and better. This helps people understand what parts of the system they feel able to improve upon, in code as well as in workflow. It’s important, they explain, to help people spot the parts of the system which were put in because something was simply needed and the parts which are there due to a lot of thought and due diligence. Again, true of code as much as workflows.

We all know that mistakes are important in the learning process. One option laid out is to find parts of projects which are difficult enough to allow someone to dip their tow below the surface and to learn. The underlying point is not to shield junior members of the team from projects. In fact, heading a project with all your experienced engineers may be a way to deliver the project with low risk, but the cost of not investing in getting your less experienced team members involved will be paid when the project is delivered and needing support, maintenance and development. It also works against the interest of the less experienced individuals by reducing the speed at which they advance.

Alexandria and Kevin summarise by saying you should create and grow owners, rotate who is on the A-team, give everyone the chance to be in the crucible and share notes and experiences freely. The video finishes by remarking that the technology of today was built by us standing on the shoulders of giants. It’s important, therefore, that the giants of today remember to let people climb aloft.

Watch now!
Speakers

Alexandria Shealy
Director, Technical Project Management
Disney Streaming Services
Kevin Fuhrman Kevin Fuhrman
Staff Software Engineer,
Disney Streaming Services

Video: Moving Live Video Quality Control from the Broadcast Facility to the Living Room

Moving an 24×7 on-site MCR into people’s rooms is not trivial, but Disney Streaming Services, in common with most broadcasters knew they had to move people home, often into their living rooms. Working in an MCR requires watching incoming video to check content which is not easy to do at home, particularly when some of their contribution arrives at 100 Mb/s. These two MCRs in San Francisco and NYC covering Hulu Live & ESPN+ along with other services had two weeks to move remote.

Being a major streaming operator, DSS had their own encoding product called xCoder. DSS soon realised this would be their ticket to making home working viable. As standard, these encoders reject any video which doesn’t match a small range of templates. Michael Rappaport takes us how they wrote scripts to use ffprobe to analyse the desired video and then configure the xCoder just the right way. The incoming video goes straight to xCoder without being ‘groomed’ as it normally wood to add closed captions, ABR etc.

Aside from bandwidth, it was also important to provide these streams as close to real-time as possible, as the operators needed to see ‘right now’ to do their job effectively. This is why the ‘grooming’ section is skipped as that would add latency but also the added functions such as PID normalisation and closed caption insertion aren’t needed. Michael explains that when a feed is needed, it will call out to the whole encoder pool, find an underutilised one and then can program it automatically using an API.

Watching this at home was made possible by some work done by Disney Streaming Services to allow their player to receive feeds directly from an xCoder without having any problems decoder parameters. Michael doesn’t mention what protocol they use, but as the xCoder creates a proprietary video stream, so they could be used that carried over TCP.

Made their own players to the receiver from the xCoders. xCoder, as a standalone, produces a proprietary TCP stream. xCoder exposes an API hook that allows us to quickly determine things like frame rate, resolution, and even whether or not the xCoder is able to subscribe to the stream

Watch now!
Speakers

Michael Rappaport Michael Rappaport
Senior Manager, Encoding Administration,
Disney Streaming Services

Video: Layer 4 in the CDN

Caching is a critical element of the streaming video delivery infrastructure, but with the proliferation of streaming services, managing caching is complex and problematic. Open Caching is an initiative by the Streaming Video Alliance to bring this under control allowing ISPs and service providers a standard way to operate.

By caching objects as close to the viewer as possible, you can reduce round-trip times which helps reduce latency and can improve playback but, more importantly, moving the point at which content is distributed closer to the customer allows you to reduce your bandwidth costs, and create a more efficient delivery chain.

This video sees Disney Streaming Services, ViaSat and Stackpath discussing Open Caching with Jason Thibeault, Executive Director of the Streaming Video Alliance. Eric Klein from Disney explains that one driver for Open Caching is from content producers which find it hard to scale, to deliver content in a consistent manner across many different networks. Standardising the interfaces will help remove this barrier of scale. Alongside a drive from content producers, are the needs of the network operators who are interested in moving caching on to their network which reduces the back and forth traffic and can help cope with peaks.

Dan Newman from Viasat builds on these points looking at the edge storage project. This is a project to move caching to the edge of the networks which is an extension of the original open caching concept. The idea stretches to putting caching directly into the home. One use of this, he explains, can be used to cache UHD content which otherwise would be too big to be downloaded down lower bandwidth links.

Josh Chesarek from StackPath says that their interest in being involved in the Open Caching initiative is to get consistency and interoperability between CDNs. The Open Caching group is looking at creating these standard APIs for capacity, configuration etc. Also, Eric underlines the interest in interoperability by the close work they are doing with the IETF to find better standards on which to base their work.

Looking at the test results, the average bitrate increases by 10% when using open caching, but also a 20-40% improvement in connection use rebuffer ratio which shows viewers are seeing an improved experience. Viasat have used multicast ABR plus open caching. This shows there’s certainly promise behind the work that’s ongoing. The panel finishes by looking towards what’s next in terms of the project and CDN optimisation.

Watch now!
Speakers

Eric Klein Eric Klein
Director, CDN Technology,
Disney+
Dan Newman Dan Newman
Product Manager,
Viasat
Josh Chesarek Josh Chesarek
VP, Sales Engineering & Support
Stackpath.com
Jason Thibeault Jason Thibeault
Executive Director, Streaming Video Alliance

Video: CMAF And The Future Of OTT

Why is CMAF still ‘the future’ of OTT? Published in 2018, CMAF’s been around for a while now so what are the challenges and hurdles holding up implementation? Are there reasons not to use it at all? CMAF is a way of encoding and packaging media which then can be sent using MPEG DASH and HLS, the latter being the path Disney+ has chosen, for instance.

This panel from Streaming Media West Connect, moderated by Jan Ozer, discusses CMAF use within Akami, Netflix, Disney+, and Hulu. Peter Chave from Akamai starts off making the point that CMAF is important to CDNs because if companies are able to use just one CMAF file as the source for different delivery formats, this reduces storage costs for consumers and makes each individual file more popular thus increasing the chance of having a file available in the CDN (particularly at the edge) and reducing cache misses. They’ve had to do some work to ensure that CMAF is carried throughout the CDN efficiently and ensuring the manifests are correctly checked.

Disney+, explains Bill Zurat, is 100% HLS CMAF. Benefiting from the long experience of the Disney Streaming Services teams (formerly BAMTECH), but also from setting up a new service, Disney were able to bring in CMAF from the start. There are issues ensuring end-device support, but as part of the launch, a number were sunsetted which didn’t have the requirements necessary to support either the protocol or the DRM needed.

Hulu is an aggregator so they have strong motivation to normalise inputs, we hear from Hulu’s Nick Brookins. But they also originate programming along with live streaming so CMAF has an important to play on the way in and the way out. Hulu dynamically regenerates their manifests so can iterate as they roll out easily. They are currently part the way through the rollout and will achieve full CMAF compatibility within the next 18 months.

The conversation turns to DRM. CMAF supports two methods of DRM known as CTR (adopted by Apple) and CBC (also known as CBCS) which has been adopted by others. AV1 supports both, but the recommendation has been to use CBC which appears have been universally followed to date explains Netflix’s Cyril Concolato. Netflix have been using AV1 since it was finalised and are aiming to have most titles transitioned by 2021 to CMAF.

Peter comments from Akamai’s position that they see a number of customers who, like Disney+ and Peacock, have been able to enter the market recently and move straight into CMAF, but there is a whole continuum of companies who are restricted by their workflows and viewer’s devices in moving to CMAF.

Low latency streaming is one topic which invigorates minds and debates for many in the industry. Netflix, being purely video on demand, they are not interested in low-latency streaming. However, Hulu is as is Disney Streaming Services, but Bill cautions us on rushing to the bottom in terms of latency. Quality of experience is improved with extra latency both in terms of reduced rebuffering and, in some cases, picture quality. Much of Disney Streaming Services’ output needs to match cable, rather than meeting over-the-air latencies or less.

The panel session finishes with a quick-fire round of questions from Jan and the audience covering codec strategy, whether their workflows have changed to incorporate CMAF, just-in-time vs static packaging, and what customers get out of CMAF.

Watch now!
Speakers

Cyril Concolato Cyril Concolato
Senior Software Engineer,
Netflix
Peter Chave Peter Chave
Principal Architect,
Akamai
Nick Brookins Nick Brookins
VP, Platform Services Group,
Hulu
Bill Zurat Bill Zurat
VP, Core Technology
Disney Streaming Services
Jan Ozer Moderator: Jan Ozer
Contributing Editor, Streaming Media
Owner, StreamingLearningCenter.com