Video: AES67 Over Wide Area Networks


AES67 is a widely adopted standard for moving PCM audio from place to place. Being a standard, it’s ideal for connecting equipment together from different vendors and delivers almost zero latency, lossless audio from place to place. This video looks at use cases for moving AES from its traditional home on a company’s LAN to the WAN.

Discovery’s Eurosport Technology Transformation (ETT) project is a great example of the compelling use case for moving to operations over the WAN. Eurosport’s Olivier Chambin explains that the idea behind the project is to centralise all the processing technology needed for their productions spread across Europe feeding their 60 playout channels.

Control surfaces and some interface equipment is still necessary in the European production offices and commentary points throughout Europe, but the processing is done in two data centres, one in the Netherlands, the other in the UK. This means audio does need to travel between countries over Discovery’s dual MPLS WAN using IGMPv3 multicast with SSM

From a video perspective, the ETT project has adopted 2110 for all essences with NMOS control. Over the WAN, video is sent as JPEG XS but all audio links are 2022-7 2110-30 with well over 10,000 audio streams in total. Timing is done using PTP aware switches with local GNSS-derived PTP with a unicast-over-WAN as a fallback. For more on PTP over WAN have a look at this RTS webinar and this update from Meinberg’s Daniel Boldt.

 

 

Bolstering the push for standards such as AES67 is self-confessed ‘audioholic’ Anthony P. Kuzub from Canada’s CBC. Chair of the local AES section he makes the point that broadcast workflows have long used AES standards to ensure vendor interoperability from microphones to analogue connectors, from grounding to MADI (AES10). This is why AES67 is important as it will ensure that the next generation of equipment can also interoperate.

Surrounding these two case studies is a presentation from Nicolas Sturmel all about the AES SC-02-12-M working group which aims to define the best ways of working to enable easy use of AES67 on the WAN. The key issue here is that AES67 was written expecting short links on a private network that you can completely control. Moving to a WAN or the internet with long-distance links on which your bandwidth or choice of protocols is limited can make AES67 perform badly if you don’t follow the best practices.

To start with, Nicolas urges anyone to check they actually need AES67 over the WAN to start with. Only if you need precise timing (for lip-sync for example) with PCM quality and low latencies from 250ms down to as a little as 5 milliseconds do you really need AES67 instead of using other protocols such as ACIP, he explains. The problem being that any ping on the internet, even to something fairly close, can easily take 16 to 40ms for the round trip. This means you’re guaranteed 8ms of delay, but any one packet could be as late as 20ms known as the Packet Delay Variation (PDV).

Not only do we need to find a way to transmit AES67, but also PTP. The Precise Time Protocol has ways of coping for jitter and delay, but these don’t work well on WAN links whether the delay in one direction may be different to the delay for a packet in the other direction. PTP also isn’t built to deal with the higher delay and jitter involved. PTP over WAN can be done and is a way to deliver a service but using a GPS receiver at each location, as Eurosport does, is a much better solution only hampered by cost and one’s ability to see enough of the sky.

The internet can lose packets. Given a few hours, the internet will nearly always lose packets. To get around this problem, Nicolas looks at using FEC whereby you are constantly sending redundant data. FEC can send up to around 25% extra data so that if any is lost, the extra information sent can be leveraged to determine the lost values and reconstruct the stream. Whilst this is a solid approach, computing the FEC adds delay and the extra data being constantly sent adds a fixed uplift on your bandwidth need. For circuits that have very few issues, this can seem wasteful but having a fixed percentage can also be advantageous for circuits where a predictable bitrate is much more important. Nicolas also highlights that RIST, SRT or ST 2022-7 are other methods that can also work well. He talks about these longer in his talk with Andreas Hildrebrand

The video concludes with a Q&A.

Watch now!
Speakers

Nicolas Sturmel Nicolas Sturmel
Product Manager – Senior Technologist,
Merging Technologies
Anthony P. Kuzub Anthony P. Kuzub
Senior Systems Designer,
CBC/Radio Canada
Olivier Chambin Olivier Chambin
Audio Broadcast Engineer, AioP and Voice-over-IP
Eurosport Discovery

Video: PTP/ST 2059 Best Practices developed from PTP deployments and experiences

PTP is foundational for SMPTE ST 2110 systems. It provides the accurate timing needed to make the most out of almost zero-latency professional video systems. In the strictest sense, some ST 2110 workflows can work without PTP where they’re not combining signals, but for live production, this is almost never the case. This is why a lot of time and effort goes into getting PTP right from the outset because making it work perfectly from the outset gives you the bedrock on which to build your most valuable infrastructure upon.

In this video, Gerard Phillips from Arista, Leigh Whitcomb from Imagine Communications and Telestream’s Mike Waidson join forces to run down their top 15 best practices of building a PTP infrastructure you can rely on.

Gerard kicks off underlining the importance of PTP but with the reassuring message that if you ‘bake it in’ to your underlying network, with PTP-aware equipment that can support the scale you need, you’ll have the timing system you need. Thinking of scale is important as PTP is a bi-directional protocol. That is, it’s not like the black and burst and TLS that it replaces which are simply waterfall signals. Each endpoint needs to speak to a clock so understanding how many devices you’ll be having and where is important to consider. For a look a look at PTP itself, rather than best practices, have a look at this talk free registration required or this video with Meinberg.

 

 

Gerard’s best practices advice continues as he recommends using a routed network meaning having multiple layer 2 networks with layer 3 routing between This reduces the broadcast domain size which, in turn, increases stability and resilience. JT-NM TR-1001 can help to assist in deployments using this network architecture. Gerard next cautions about layer 2 IGMP snoopers and queriers which should exist on every VLAN. As the multicast traffic is flooded to the snooping querier in layer 2, it’s important to consider traffic flows.

When Gerard says PTP should be ‘baked in’, it’s partly boundary clocks he’s referring to. Use them ‘everywhere you can’ is the advice as they bring simplicity to your design and allow for easier debugging. Part of the simplicity they bring is in helping the scalability as they shed load from your GM, taking the brunt of the bi-directional traffic and can reduce load on the endpoints.

It’s long been known that audio devices, for instance, older versions of Dante before v4.2, use version one of PTP which isn’t compatible with SPMTE ST 2059’s requirement to use PTP v2. Gerard says that, if necessary, you should buy a version 1 to version 2 converter from your audio vendor to join the v1 island to your v2 infrastructure. This is linked to best practice point 6; All GMs must have the same time. Mike makes the point that all GMs should be locked to GPS and that if you have multiple sites, they should all have an active, GPS-locked GM even if they do send PTP to each other over a WAN as that is likely to deliver less accurate timing even if it is useful as a backup.

Even if you are using physically separate networks for your PTP and ST 2110 main and backup networks, it’s important to have a link between the two GMs for ST 2022-7 traffic so a link between the two networks just for PTP traffic should be established.

The next 3 points of advice are about the ongoing stability of the network. Firstly, ST 2059-2 specifies the use of TLV messages as part of a mechanism for media notes to generate drop-frame timecode. Whilst this may not be needed day 1, if you have it running and show your PTP system works well with it on, there shouldn’t be any surprises in a couple of years when you need to introduce an end-point that will use it. Similarly, the advice is to give your PTP domain a number which isn’t a SMPTE or AES default for the sole reason that if you ever have a device join your network which hasn’t been fully configured, if it’s still on defaults it will join your PTP domain and could disrupt it. If, part of the configuration of a new endpoint is changing the domain number, the chances of this are notably reduced. One example of a configuration item which could affect the network is ‘ptp role master’ which will stop a boundary clock from taking part in BCMA and prevents unauthorised end-points taking over.

Gerard lays out the ways in which to do ‘proper commissioning’ which is the way you can verify, at the beginning, that your PTP network is working well-meaning you have designed and built your system correctly. Unfortunately, PTP can appear to be working properly when in reality it is not for reasons of design, the way your devices are acting, configuration or simply due to bugs. To account for this, Gerard advocates separate checklists for GM switches and media nodes with a list of items to check…and this will be a long list. Commissioning should include monitoring the PTP traffic, and taking a packet capture, for a couple of days for analysis with test and measurement gear or simply Wireshark.

Leigh finishes up the video talking about verifying functionality during redundancy switches and on power-up. Commissioning is your chance to characterise the behaviour of the system in these transitory states and to observe how equipment attached is affected. His last point before summarising is to implement a PTP monitoring solution to capture the critical parameters and to detect changes in the system. SMPTE RP 2059-15 will define parameters to monitor, with the aim that monitoring across vendors will provide some sort of consistent metrics. Also, a new version of IEEE-1588, version 2.1, will add monitoring features that should aid in actively monitoring the timing in your ST 2110 system.

This Arista white paper contains further detail on many of these best practices.

Watch now!
Speakers

Gerard Phillips Gerard Phillips
Solutions Engineer,
Arista
Leigh Whitcomb Leigh Whitcomb
Principal Engineer.
Imagine
Michael Waidson Mike Waidson
Application Engineer,
Telestream

Video: ST 2110 The Future of Live Remote Production

Trying to apply the SMPTE ST 2110 hype to the reality of your equipment? This video is here to help. There are many ‘benefits’ of IP which are banded about yet it’s almost impossible to realise them all in one company. For the early adopters, there’s usually one benefit that has been the deal-breaker with other benefits helping boost confidence. Smaller broadcast companies, however, can struggle to get the scale needed for cost savings, don’t require as much flexibility and can’t justify the scalability. But as switches get cheaper and ST 2110 support continues to mature, it’s clear that we’re beyond the early adopter phase.

This panel gives context to ST 2110 and advises on ways to ‘get started’ and skill up. Moderated by Ken Kerschbaumer from the Sports Video Group, Leader’s Steve Holmes, Prinyar Boon from Phabrix join the panel with Arista colleagues Gerard Phillips and Robert Welch and Bridge Technologies’ Chairman Simen Frostad.

The panel quickly starts giving advice. Under the mantra ‘no packet left behind’, Gerard explains that, to him, COTS (Commercial Off The Shelf) means a move to enterprise-grade switches ‘if you want to sleep at night’. Compared to SDI, the move to IT can bring cost savings but don’t skimp on your switch infrastructure if you want a good quality product. Simen was pleased to welcome 2110 as he appreciated the almost instant transmission that analogue gave. The move to digital added a lot of latency, even in the SDI portions of the chain thanks to frame syncs. ST 2110, he says, allows us to get back, most of the way, to no-latency production. He’s also pleased to bid good-bye to embedded data.

It is possible to start small, is the reassuring message next from the panel. The trick here is to start with an island of 2110 and do your learning there. Prinyar lifts up a tote bag saying he has a 2110 system he can fit in there which takes just 10 minutes to get up and running. With two switches, a couple of PTP grandmasters and some 2110 sources, you have what you need to start a small system. There is free software that can help you learn about it, Easy NMOS is a quick-to-deploy NMOS repository that will give you the basics to get your system up and running. You can test NMOS APIs for free with AMWA’s testing tool. The EBU’s LIST project is a suite of software tools that help to inspect, measure and visualize the state of IP-based networks and the high-bitrate media traffic they carry and there’s is also SDPoker which lets you test ST 2110 SDP files. So whilst there are some upfront costs, to get the learning, experience and understanding you need to make decisions on your ST 2110 trajectory, it’s cost-effective and can form part of your staging/test system should you decide to proceed with a project.

The key here is to find your island project. For larger broadcasters or OB companies, a great island is to build an IP OB truck. IP has some big benefits for OB Trucks as we heard in this webinar, such as weight reduction, integration with remote production workflows and scalability to ‘any size’ of event. Few other ‘islands’ are able to benefit in so many ways, but a new self-op studio or small control room may be just the project for learning how to design, install, troubleshoot and maintain a 2110 system. Prinyar cautions that 2110 shouldn’t be just about moving an SDI workflow into IP. The justification should be about improving workflows.

Remote control is big motivator for the move to ST 2110. Far before the pandemic, Discovery chose 2110 for their Eurosport production infrastructure allowing them to centralise into two European locations all equipment controlled in production centres in countries around Europe. During the pandemic, we’ve seen the ability to create new connections without having to physically install new SDI is incredibly useful. Off the back of remote control of resources, some companies are finding they are able to use operators from locations where the hourly rate is low.

Before a Q&A, the panel addresses training. From one quarter we hear that ensuring your home networking knowledge is sound (DHCP, basic IP address details) is a great start and that you can get across the knowledge needed very little time. Prinyar says that he took advantage of a SMPTE Virtual Classroom course teaching the CCNA, whilst Robert from Arista says that there’s a lot in the CCNA that’s not very relevant. The Q&A covers 2110 over WAN, security, hardware life cycles and the reducing carbon footprint of production.

Watch now!
Speakers

Steve Holmes Steve Holmes
Applications Engineer,
Leader
Prinyar Boon Prinyar Boon
Product Manager,
PHABRIX
Gerard Phillips Gerard Phillips
Systems Engineer,
Arista
Simen Frostad Simen Frostad
Chairman,
Bridge Technologies
Robert Welch Robert Welch
Technical Solutions Lead,
Arista
Ken Kerschbaumer Moderator: Ken Kerschbaumer
Chair & Editorial Directo,
Sports Video Group

Video: Time and timing at VidTrans21

Timing is both everything and nothing. Although much fuss is made of timing, often it’s not important. But when it is important, it can be absolutely critical. Helping us navigate through the broadcast chains varying dependence on a central co-ordinated time source is Nevion’s Andy Rayner in this talk at the VSF’s VidTrans21. When it comes down to it, you need time for coordination. In the 1840s, the UK introduced ‘Railway time’ bringing each station’s clock into line with GMT to coordinate people and trains.

For broadcast, working with multiple signals in a low-latency workflow is the time we’re most likely to need synchronisation such as in a vision or audio mixer. Andy shows us some of the original television technology where the camera had to be directly synchronised to the display. This is the era timing came from, built on by analogue video and RF transmission systems which had components whose timing relied on those earlier in the chain. Andy brings us into the digital world reminding us of the ever-useful blanking areas of the video raster which we packed with non-video data. Now, as many people move to SMPTE’s ST 2110 there is still a timing legacy as we see that some devices are still generating data with gaps where the blanking of the video would be even though 2110 has no blanking. This means we have to have timing modes for linear and non-linear delivery of video.
 

 
In ST 2110 every packet is marked with a reduced resolution timestamp from PTP, the Precision Time Protocol (or See all our PTP articles). This allows highly accurate alignment of essences when bringing them together as even a slight offset between audios can create comb filters and destroy the sound. The idea of the PTP timestamp is to stamp the time the source was acquired. But Andy laments that in ST 2110 it’s hard to keep this timestamp since interim functions (e.g. graphics generators) may restamp the PTP breaking the association.

Taking a step back, though, there are delays now up to a minute later delivering content to the home. Which underlines that relative timing is what’s most important. A lesson learnt many years back when VR/AR was first being used in studios where whole sections of the gallery were running several frames delayed to the rest of the facility to account for the processing delay. Today this is more common as is remote production which takes this fixed time offset to the next level. Andy highlights NMOS IS-07 which allows you timestamp button presses and other tally info allowing this type of time-offset working to succeed.

The talk finishes by talking about the work of the GCCG Activity Group at the VSF of which Andy is the co-chair. This group is looking at how to get essences into and out of the cloud. Andy spends some time talking about the tests done to date and the fact that PTP doesn’t exist in the cloud (it may be available for select customers). In fact you may have live with NTP-derived time. Dealing with this is still a lively discussion in progress and Andy is welcoming participants.

Watch now!
Speakers

Andy Rayner Andy Rayner
Co-Chair, Ground-Cloud-Cloud-Ground Activity Group, VSF
Chief Technologist, Nevion