Video: Remote Production in Real Life

Remote production is in heightened demand at the moment, but the trend has been ongoing for several years. For each small advance in technology, it becomes practical for another event to go remote. Remote production solutions have to be extremely flexible as a remote production workflow for one copmany won’t work for the next This is why the move to remote has been gradual over the last decade.

In this video, Dirk Skyora from Lawo gives three examples of remote production projects stretching back as far as 2016 to the present day in this RAVENNA webinar with evangelist Andreas Hildebrand.

The first case study is remote production for Belgian second division football. Working with Belgian telco Proximus along with Videohouse & NEP, LAWO setup remote production for stadia kitted out with 6 cameras and 2 commentary positions. With only 1 gigabit connectivity to the stadiums, they opted to use JPEG 2000 encoding at 100 Mbps for both the cameras out of the stadia but also the two return feeds back in for the commentators.

The project called for two simultaneous matches feeding into an existing gallery/PCR. Deployment was swift with flightcases deployed remotely and a double set of equipment being installed into the fixed PCR. Overall latency was around 2.5 frames one-way, so the camera viewfinders were about 5 frames adrift once transport and en/decoding delay were accounted for.

The main challenges were with the MPLS network into the stadia which would spontanously reroute and be loaded with unrelated traffic at around 21:00. Although there was packet loss, none was noticable on the 100Mbps J2K feeds. Latency for the commentators was a problem so some local mixing was needed and lastly PTP wasn’t possible over the network. Timing was, therefore, derived from the return video feed into the stadium which had come from the PTP-locked gallery. Locally this incoming timing was used to lock a locally generated PTP signal.

The next case study is inter-country links for the European Council connecting the Luxembourg and Brussels buildings for the European Council. The project was to move all production to a single tech control room in Brussells and relied on two 10GbE links between the buildings going through an Arista 7280 carrying 18 videos in one direction and two in return. Although initially reluctant to compress, the Council realised after testing that VC2 which offers around 4x compression would work well and deliver no noticable latency (approx 20ms end to end). Thanks to using VC2, the 10Gig links saw low usage from the project and the Council were able to migrate other business activities onto the link. PTP was generated in Brussels and Luxembourg re-generated their PTP from the Brussels signal, to be distributed locally. Overall latency was 1 frame.

Lastly, Dirk outlines the work done for the Belgium Daily News which had been bought out by DPG Media. This buy-out prompted a move from Brussels to Antwerp where a new building opened. However, all of the techinical equipmennt was already in Brussels. This led to the decision to remote control everything in Brussels from Antwerp. The production staff moved to Antwerp, causing some issues with the disconnect between production and technical, but also due to personnel relocating and getting used to new facilities.

The two locations were connected with a 400GbE, redundant infrastructure using IP<->SDI gateways. Latency was 1 frame and, again, PTP on one site was created from the incoming PTP from the other.

The video finishes with a detailed Q&A.

Watch now!
Speakers

Dirk Sykora Dirk Sykora
Technical Sales Manager,
Lawo
Andreas Hildebrand Andreas Hildebrand
RAVENNA Evangelist,
ALC NetworX

Video: PTP in WAN Applications & Update on PTP v2.1

PTP is evolving as is our ability to use it over WANs. This video explains what’s new in PTP’s second revision and the evolving techniques of using PTP over a wide area network such as the internet. As PTP was built assuming the use of LANs, the longer and more unpredictable latency of WANs throws off the timing calculations, so what can be done to compensate?

In this video from RAVENNA, Andreas Hildebrand from ALC Networx takes us through PTP 2.1, the 2019 revision of PTP following on from PTP 2.0 in 2008 and from the original 1.0 in 2002. Famously, 2.0 and 1.0 were not compatible with each other which has caused problems with some hardware implementations of DANTE which were first written when 1.0 was the only edition. Importantly, Andreas highlights, version 2.1 is backwards compatible with version 2.0. If you’re looking for a PTP primer before digging in, have a look at this intro video from Daniel Boldt, Meinberg

Andreas explains the use of PTP profiles within both AES67 and SMTPTE 2110 which standardise some of the parameters for using PTP such as message frequency. Whilst they do have different defaults, there is an overlap between the two allowing for use of AES67 streams withing both an AES67 ecosystem and with a 2110-30 ecosystem. These overlaps are detailed in the joint AES and SMPTE document, AES-R16-2016.

“What’s new in PTP v2.1?” asks Andreas. Apart from clearer language, accuracy, flexibility and robustness have been enhanced.

Flexibility
Flexibility comes from the ability to mixed multicast and unicast messages. Announce and sync messages are still sent in multicast, but queries like delayresponse & delayrequest can now be sent unicast which provides better scalability as, for many scenarios, the reply never needs to go back to any other computers. Another example of flexibility is modular transparent clocks i.e. ones in SFPs. Another flexibility improvement is that it’s now possible to isolate profiles without using different PTP domain numbers. This is by adding a Profile ID called ‘SdoId’.

Robustness and Security
PTP now allows inter-domain interactions effectively allowing multiple GMs to vote onto a single ‘multi-domain clock’. This becomes a very robust clock as it’s created from the insight of three grandmasters. PTP v2.1 adds source integrity checking to allow identification of false, injected, messages. A long-needed improvement as security, even of a LAN, can’t be assumed nowadays.

Performance and Accuracy
Stats reporting has been added to PTP v2.1 allowing monitoring of the average, minimum, max and standard deviation of a number of metrics from the leader clock, delay metrics and message reception counters. Accuracy has been boosted to sub-nanosecond by the CERN-related White Rabbit Project which is an overall benefit to PTP even if sub-nanosecond timing isn’t needed.

Source: ALC NetworX

The second part of the video features Meinberg’s Daniel Boldt who discusses how to transmit PTP over the WAN. This is more challenging than a WAN because it’s more likely to be affected by queueing delays – particularly if the WAN in question is the internet. Queueing delays happen for a number of reasons but it all boils down to the fact the switches and routers often have to hold packets in a buffer, something that tends to happen more when there is more load. As such, this means that the delay is variable leading to varying jitter on the measurements.

Another problem often encountered is path changes where a switch happens in the network to divert the signal to a different path. Whilst this is a great way to route around problems, it does mean a sudden change in path length and therefore propagation delay. A conventional ping time may change from 100ms to 250ms in a second. This could have a big impact on the accuracy of a PTP timing signal if undetected.

Finally, the PTP timing algorithm assumes that it takes just as long, and no longer, to get the timing information from A to B as it does to get the follow-up reply from B to A. When one direction takes longer than the other, for instance when one direction is forced through a 100Mbps link rather than 1000Mbps, the PTP timing will have a constant timing error.

Source: Meinberg

Daniel explains that these issues can be mitigated by more thorough processing of the incoming packets. For instance, having a high-quality oscillator which can maintain an accurate frequency for a long time without external input is helpful. Having a local GM on GPS is another good way to avoid problems, in the cases when this is practical, where the WAN PTP becomes an ‘aide’ to timing rather than the authority. Finally, the ‘lucky packets’ technique is demonstrated.

Daniel explains that if you look at the delay of each packet incoming over, say, a two-second period, you can collect all the packets that, based on the timestamp, were lucky enough not to be delayed a lot. By discarding those very-delayed packets, the accuracy of the PTP signal becomes much higher and jitter can reduce, as we see from two case studies, by an order of magnitude.

Watch now!
Speakers

Andreas Hildebrand Andreas Hildebrand
RAVENNA Evangelist,
ALC NetworX
Daniel Boldt Daniel Boldt
Head of Software Development,
Meinberg

Video: Case Study: Dropbox HQ ST 2110

Dropbox is embedded in many production workflows – official and otherwise – so it’s a beautiful symmetry that they’re using Broadcast’s latest technology, SMPTE ST 2110, within their own headquarters. Dropbox have AV throughout their building and a desire to create professional video from anywhere. This desire was a driving factor in an IP-based production facility as, to allow mobile production platforms to move from room to room with only a single cable needed to connect to the wall and into the production infrastructure.

David Carroll’s integration company delivered this project and joins Wes Simpson to discuss this case-study with colleague Kevin Gross. David explains that they delivered fibre to seventy locations throughout the building making most places into potential production locations.

Being an IT company at heart, the ST 2110 network was built to perform in the traditional way, but with connections into the corporate network which many broadcasters wouldn’t allow. ST 2110 works best with two separate networks, often called Red and Blue, both delivering the same video. This uses ST 2022-7 to seamlessly failover if one network loses a packet or even if it stops working all together. This is the technique used with dropbox, although there these networks are connected together so are not one hundred per cent isolated. This link, however, has the benefit of allowing PTP traffic between the two networks.

PTP topology typically sees two grandmasters in the facility. It makes sense to connect one to the red network, the other to the blue. In order to have proper redundancy, though, there should really be a path from both grandmasters to both networks. This is usually done with a specially-configured ‘PTP only’ link between the two. In this case, there are other reasons for a wider link between networks which also serves as the PTP link. Another element of PTP topology is acknowledging the need for two PTP domains. A PTP domain allows two PTP systems to operate on the same network but without interfering with one another. Dante requires PTP version 1 whereas 2110, and most other things, require v2. Although this is in the process of improving, the typical way to solve this now is to run the two separately and block v1 from areas of the network in which it’s not needed.

PTP disruptions can also happen with multicast packet loss. If packets are lost at the wrong time, a grandmaster election can happen. Finally, on PTP, they also saw the benefits of using boundary clock switches to isolate the grandmasters. These grandmasters have to send out the time eight times a second. Each end-device then replies to ascertain the propagation delay. Dealing with every single device can overwhelm grandmasters, so boundary clock switches can be very helpful. On a four-core Arista, David and Kevin found that one core would be used dealing with the PTP requests.

A more extensive write-up of the project can be found here from David Carroll

Watch now!

Speakers

Kevin Gross Kevin Gross
Media Network Consultant
AVA Networks
David Carroll David Carroll
President,
David Carroll Associates, Inc.
Wes Simpson Wes Simpson
Owner, LearnIPVideo.com

Video: AES67 over WAN

Deeply embedded in the audio industry and adopted into SMPTE ST 2110, AES67 workflows surround us. Increasingly our workflows are in multiple locations so moving AES67 on the WAN and the internet is essential. If networks were always perfect, this would be easy but as that’s not the case, this RAVENNA talk examines what the problems are and how to solve them.

Andreas Hildebrand introduces the video with an examination of how the WAN, whether that’s a company’s managed wide area network or the internet at large, is different from a LAN. Typical issues are packet loss, varying latency meaning the packets arrive with jitter, lack of PTP and multicast. With this in mind, Nicolas Sturmel from Merging Technologies takes the reins to examine the solutions.

Nicolas explains the typically EBU Tech 3326 (also known as ACIP) is used for WAN contribution which specifies how a sender and receiver communicate and the codecs to be used. Although PCM is available, many codecs such as AptX are also prescribed for use. Nicolas says that ACIP is great for most applications but if you need low-latency, precise timing and PCM-quality staying AES67 may be the best policy, even over the WAN.

Having identified your AES6-over-WAN workflow, the question is how to pull it off. Nicolas looks at three methods, one is FEC whereby you are constantly sending redundant data. FEC can send up to around 25% extra data so that if any is lost, the extra information sent can be leveraged to determine the lost values and reconstruct the stream. This is can work well but requires sending this extra data constantly therefore putting up your bandwidth. It can also only deal with certain losses requiring them to be of a short duration.

Instead of FEC, you can use RIST, SRT or a similar re-transmission technology. These will actively recover any lost packets and have the benefit that you only transmit more data when you have lost data. Lastly, he mentions SMPTE ST 2022-7 which uses two paths of identical data to cover losses in any one of them. Although this is 100% extra data, the benefit is that it can deal with any type of loss including a complete path failure which neither of the others can do. It is, however possible to combine FEC or RIST with a 2022-7 workflow so you can have two levels of protection.

Timing over the WAN is not ideal as PTP loses accuracy over long-latency links and it assumes symmetry. On the internet, it’s possible to get links where the latency is longer in one direction than the other. An easy, though potentially costly, workaround for distributing PTP over the WAN is to use GPS, GLONASS or similar to synchronise grandmaster clocks at each location.

Watch now!
Speakers

Nicolas Sturmel Nicolas Sturmel
Product Manager & Senior Technologist
Merging Technologies
Andreas Hildebrand Andreas Hildebrand
RAVENNA Evangelist,
ALCNetworx