SCTE 35

Video: Scalable Per-User Ad Insertion in Live OTT

Posted on by Russell Trafford-Jones

Targetted ads are the most valuable ads, but making sure the right person gets the right ad is tricky, not only in deciding who to show which ad to, but in scaling – and keeping track of – the ad infrastructure to thousands or millions of viewers. This video explains how this complexity arises and the techniques that Hulu have implemented to improve the situation.

Zachary Cava from Hulu lays out the way that standard advertising works for live streams. Whilst he uses MPEG DASH as an example, much the same is true of HLS. This starts with cutting up the video into sections which all start with an IDR frame for seeking. SCTE 35 is used to indicate times when ads can be inserted. These are called SCTE Markers. As DASH has the principle of defining a period (exactly as it sounds, just a way of marking a section of time), we can define periods of ‘programme’ and periods for ‘ads’. This allows the possibility of swapping out a whole period for a section of several ads.

If it were as simple as just swapping out whole periods, that would be Server-Side Ad Insertion. For per-user targetted ads, the streaming service has to keep track of every ad which was given to a user so that when they rewind, they have a consistent experience. This can mean remembering millions of ads for services which have a large rewind buffer. Moreover, traffic can become overwhelming as, since the requests are unique, a CDN can’t help in the caching. Whilst you can scale your system, the cost can spiral up beyond the revenue practical.

Enter MPD Patch Requests. This addition to MPEG Dash requires the client to remember the whole of the manifest. Where the client has a gap in its knowledge, it can simply request that section from the server which generates a ‘diff’, returning only the changes, which the client then assimilates into memory. The benefit here is that all the clients end up converging on only requesting what’s happening ‘now’ and so CDNs come back in to play. Zachary explains how this works in more detail and shows examples before explaining how URLQueryInfo helps reduce the complexity of URL parameters, again in order to interoperate better with CDNs and allows the ad system to be scaled separately to the main video assets.

Finally, Zachary takes a look at coming back from an ad break where you may find that your ads were longer then the ad period allotted or that the programme hasn’t returned before the ads finished. During the ad break, the client is still polling for updates so it’s possible to quickly update the manifest and swap back to programme video early. Similarly at the end of a break, if there is still no content, the server can start issuing its own ad or content, effectively moving back to server-side ad insertion. However, this is not necessarily just plain ad insertion, explains Zachary, rather Hulu cal it ‘Server-Guided’ ad insertion. There is no stitching on the server, but the server is informing you where to get the next video from. It also allows for some levels of user separation where some larger geographies can see different ads to those from other areas.

Zachary finishes by outlining the work Hulu is doing to feedback this learning into the DASH spec, via the DASH Industry Forum and their work with the industry at large to bring more consistency to SCTE 35 markers.

Watch now!
Speaker

Zachary Cava Zachary Cava
Software Architect,
Hulu

Video: How CBS Sports Digital streams live events at scale

Posted on by Russell Trafford-Jones

Delivering high scale in streaming really exposes the weaknesses of every point of your workflow, so even those of us who are not streaming at maximum scale, there are many lessons to be learnt. CBS Sports Digital delivered the Super Bowl using the principles of ‘practice, practice, practice’, keeping the solution as simple as possible and making mitigation of problems primary to solving them.

Taylor Busch tells walks us through their solution explaining how it supported their key principles and highlighting the technology used. Starting with Acquisition, he covers the SDI fibre delivery to a backup facility as well as the AWS Direct Connect links for their Elemental Live encoders. The origin servers were in two different regions and both received data from both sets of encoders.

CBS used ‘Output locking’ which ensures that the TS segments are all aligned even across different encoders which is done by respecting the timecode in the SDI and helps in encoder failover situations. QVBR encoding is a method of encoding up to a quality level rather than simply saying ‘7000 kbps’. QVBR provides a more efficient use a bandwidth since in the situations where a scene doesn’t require a lot of bandwidth, it won’t be sent. This variability, even if you run in capped mode to limit the bandwidth of particularly complex scenes, can look like a failing encoder to some systems, so the fact this is now in ‘VBR’ mode, needs to be understood by all the departments and companies who monitor your feed.

Advertising is famously important for the Super Bowl, so Taylor gives an overview of how they used the CableLabs ESAM protocol and SCTE to receive information about and trigger the adverts. This combined SCTE-104, ESAM and SCTE-35 as we’ll as allowing clients to use VAST for tracking. Extra caching was provided by Fastly’s Media Shield which tests for problems with manifests, origin servers and encoders. This fed a Multi-CDN setup using 4 CDNs which could be switched between. There is a decision point for requests to determine which CDN should answer.

Taylor then looks at the tools, such as Mux’s dashboard, which they used to spot problems in the system; both NOC-style tools and multiviewers. They set up three war rooms which looked at different aspects of the system, connectivity, APIs etc. This allowed them to focus on what should be communicated keeping ‘noise’ down to give people the space they needed to do their work at the same time as providing the information required. Taylor then opens up to questions from the floor.

Watch now!
Speaker

Taylor Busch, Sr. Taylor Busch
Senior Director Engineering,
CBS Sports Digital

Video: Harness SSAI’s Superpowers

Posted on by Russell Trafford-Jones

Server-side Ad Insertion (SSAI) is a great option for streaming services delivering video to a wide variety of devices and for those who need to avoid ad blockers. Whilst ad insertion can happen in the player, this mechanism can be interfered with allowing users to avoid ads. Whilst client-side ad insertion can much more easily create a unique stream for each client, dynamic SSAI can now do the same with a better user experience.

This panel from the OTT Leadership Summit at Streaming Media West 2019 brings together Disney, WarnerMerdia and Crunchyroll to share their experiences with SSAI. They discuss beaconing, ad standards, scaling, SCTE and more.

Beaconing goes hand in hand with ad playback providing metrics on what happened. When you perform certain actions, the player will reach out to a URL. This can be used to indicate such things as users skipping or pausing a video. The beacon information can then be used to verify how much of which ads were seen by whom and charge advertisers accordingly.

The panel moves on to discussing scaling using live sports as an example and cover questions to ask vendors to ensure you and they are ready for maximum scale. Bandwidth, is declared the biggest challenge, but a less obvious problem is that your upstream ad providers can’t always scale well. If you rely on calls from your server to others, then it’s vital to understand their scaling capacity and strategy. They discuss issues with losing beacons when operating at scale and the need for detailed logging and debugging in order to spot errors and reconcile the results.

Some time is next spent on VPAID and VAST 4 which are both messaging specifications to allow ad servers to tell applications which ads to play. The panel discusses the pros and cons in their use for SSAI where the stitcher needs to reach out to and ad server in real time to find out which ads to play.

At the end of the discussion, the panel takes questions from the floor but not before discussing SCTE Markers and ‘content conditioning’ which surrounds taking care of your source videos and encoder such that the two assets fit together properly at I-frame boundaries.

Watch now!
Speakers

Robert Jameson Robert Jameson
Technical Director, Media Enablement
Turner | WarnerMedia
Stephen Gray Stephen Gray
Director, Ad Tech Systems
Walt Disney Direct-to-Consumer & International
Michael Dale Michael Dale
VP Engineering,
Crunchyroll
Nadine Krefetz Nadine Krefetz
Consultant, Reality Software
Contributing Editor, Streaming Media

Video: Doing Server-Side Ad Insertion on Live Sports for 25.3M Concurrent Users

Posted on by Russell Trafford-Jones

Delivering ads successfully is done by some services by having the client insert the different ads, and some by inserting the ads at the server end. The choice of which to use requires knowing your customers and how they are most likely to receive your streams. With the prevalence of ad blockers, businesses find the many customers never see the client-side inserted ads. Inserting ads at the server, therefore allows you to get around this as even the ads look like they are a continuation of the same video feed.

The downside of server-side ad insertion (SSAI), whilst rendering the ads unblockable, restricts the ads you can place. Theoretically, in client-side ad insertion, each user can have their own advert. With SSAI, to do that you would need to create a new stream per user which becomes much more computationally hungry. So the sweet spot comes in between the two where viewers are grouped into categories so that only a few tens of streams, for example, are needed to match ten demographics identified to advertisers. This is known as ‘dynamic SSAI’.

Ashutosh Agrawal took to the stage at the Demuxed SF 2019 conference to explain how Hotstar used dynamic SSAI to deliver targeted ads to their 25 million viewers. As an example of your understanding of your viewers driving your choice of ad-delivery technology, Ashutosh explains that close to 85% of their viewing is on mobile and much of that has marginal reception. In hostile network conditions, the requirement for the player to be downloading ads in the background doesn’t work well since the network can only just about support the live video, so a background download pushes the ABR quality down and could even create pausing and rebuffering. It’s for this reason that Hotstar decided that server-side was the way to go.

Ashutosh takes us through how Hotstar approached this large event. In India, cricket is a very popular game which lasts for up to 8 hours a day. This gives rise to a large number of breaks, over 100, which add up to over an hour’s advertising in total so it’s clear to see why this is a massive opportunity for optimisation. Static ad insertion reacts to SCTE 35 markers inserted. This can work well in the sense that for a 40 seconds SCTE marker, the platform can ad an approx 40-second ad or two 20 second ads. However, it isn’t flexible enough to deal with the times when there are far more people watching than that ad agency has paid for which means that Hotstar would end up delivering more viewers than necessary. It would be better for those viewers to see a different ad, triggered by SCTE 35.

As discussed above, doing SSAI for each person is a scalability and cost nightmare, so we quickly see that Targeted SSAI is the way forward. This allows different cohorts of users to be identified. Each cohort will receive its own virtual feed with their own adverts. We then see the architecture of the system showing how the CDN is used. For scaling, we see that they use a cache rather than a database.

Nginx then gets a namecheck as Ashutosh explains how they provide caching, including an nginx memory cache, to deal with up to 50% of the overall load, shared with the CDN if necessary. He then finishes with a look at the best practices they have learnt and what Ashutosh sees is the future for this technique.

Watch now!
Speaker

Ashutosh Agrawal Ashutosh Agrawal
Evangelist/Architect – CTO’s Office,
Hotstar