Many of the bottlenecks in processing video today are related to bandwidth but most codecs that solve this problem require a lot of compute power and/or add a lot of latency. For those that wish to work with high-quality video such as within cameras and in TV studios, what’s really needed is a ‘zero’ latency codec that maintains lossless video but drops the data rate from gigabits to megabits. This is what JPEG XS does and Jean-Baptiste Lorent joined the NVIDIA GTC21 conference to explain why this is so powerful.
Created by intoPIX who are not only active in compression intellectual property but also within standards bodies such as JPEG, MPEG, ISO, SMPTE and others, JPEG XS is one of the latest technologies to come to market from the company. Lorent explains that it’s designed both to live inside equipment compressing video as it moves between parts of a device such as a phone where it would enable higher resolutions to be used and minimise energy use, and to drive down bandwidths between equipment in media workflows. We’ve featured case studies of JPEG XS in broadcast workflows previously.
JPEG XS prioritisation of quality & latency over compression. Source: intoPIX
The XS in JPEG XS stands for Xtra Small, Xtra Speed. And this underlines the important part of the technology which looks at compression in a different way to MPEG, AV1 and similar codecs. As discussed in
this interview the codec market is maturing and exploiting other benefits rather than pure bitrate. Nowadays, we need codecs that make life easy for AI/ML algorithms to quickly access video, we need low-complexity codecs for embedded devices like old set-top boxes and new embedded devices like body cams. We also need ulta-low delay codecs, with an encode delay in the microseconds, not milliseconds so that even multiple encodes seem instantaneous. JPEG XS is unique in delivering the latter.
With visually lossless results at compression levels down to 20:1, JPEG XS is expected to be used by most at 10:1 at which point it can render uncompressed HD 1080i at around 200Mbps, down from 1.5Gbps or can bring 76Gbps down to 5Gbps or less. Lorent explains that the maths in the algorithm has low complexity and is highly paralellisable which is a key benefit in modern CPUs which have many cores. Moreover, important for implementation in GPUs and FPGAs, it doesn’t need external memory and is low on logic.
The talk finishes with Lorent highlighting that JPEG XS has been created flexibly to be agnostic to colour space, chroma subsampling, bit depths, resolution and more. It’s also been standardised to be carried in SMPTE ST 2110-22, under ISO IEC 21122, carriage over RTP, in an MPEG TS and in the file domain as MXF, HEIF, JXS and MP4 (in ISO BMFF).
Free Registration required. Easiest way to watch is to click above, register, come back here and click again.
If you have trouble, use the chat on the bottom right of this website and we can send you a link
Director Marketing & Sales
Our ability to work remotely during the pandemic is thanks to the hard work of many people who have developed the technologies which have made it possible. Even before the pandemic struck, this work was still on-going and gaining momentum to overcome more challenges and more hurdles of working in IP both within the broadcast facility and in the cloud.
SMPTE’s Paul Briscoe moderates the discussion surrounding these on-going efforts to make the cloud a better place for broadcasters in this series of presentation from the SMPTE Toronto section. First in the order is Peter Wharton from TAG V.S. talking about ways to innovate workflows to better suit the cloud.
Peter first outlines the challenges of live cloud production, namely keeping latency low, signal quality high and managing the high bandwidths needed at the same time as keeping a handle on the costs. There is an increasing number of cloud-native solutions but how many are truly innovating? Don’t just move workflows into the cloud, advocates Peter, rather take this opportunity to embrace the cloud.
Working with the cloud will be built on new transport interfaces like RIST and SRT using a modular and open architecture. Scalability is the name of the game for ‘the cloud’ but the real trick is in building your workflows and technology so that you can scale during a live event.
Source: TAG V.S.
There are still obstacles to be overcome. Bandwidth for uncompressed video is one, with typical signals up to 3Gbps uncompressed which then drives very high data transfer costs. The lack of PTP in the cloud makes ST 2110 workflows difficult, similarly the lack of multicast.
Tackling bandwidth, Peter looks at the low-latency ways to compress video such as NDI, NDI|HX, JPEG XS and Amazon’s lossless CDI. Peter talks us through some of the considerations in choosing the right codec for the task in hand.
Finishing his talk, Peter asks if this isn’t time for a radical change. Why not rethink the entire process and embrace latency? Peter gives an example of a colour grading workflow which has been able to switch from on-prem colour grading on very high-spec computers to running this same, incredibly intensive process in the cloud. The company’s able to spin up thousands of CPUs in the cloud and use spot pricing to create temporary, low cost, extremely powerful computers. This has brought waiting times down for jobs to be processed significantly and has reduced the cost of processing an order of magnitude.
Lastly Peter looks further to the future examining how saturating the stadium with cameras could change the way we operate cameras. With 360-degree coverage of the stadium, the position of the camera can be changed virtually by AI allowing camera operators to be remote from the stadium. There is already work to develop this from Canon and Intel. Whilst this may not be able to replace all camera operators, sports is the home of bleeding-edge technology. How long can it resist the technology to create any camera angle?
Jean-Baptiste Lorent is next from intoPIX to explain what JPEG XS is. A new, ultra-low-latency, codec it meets the challenges of the industry’s move to IP, its increasing desire to move data rather than people and the continuing trend of COTS servers and cloud infrastructure to be part of the real-time production chain.
As Peter covered, uncompressed data rates are very high. The Tokyo Olympics will be filmed in 8K which racks up close to 80Gbps for 120fps footage. So with JPEG XS standing for Xtra Small and Xtra Speed, it’s no surprise that this new ISO standard is being leant on to help.
Tested as visually lossless to 7 or more encode generations and with latency only a few lines of video, JPEG XS works well in multi-stage live workflows. Jean-Baptiste explains that it’s low complexity and can work well on FPGAs and on CPUs.
JPEG XS can support up to 16-bit values, any chroma and any colour space. It’s been standardised to be carried in MPEG TSes, in SMPTE ST 2110 as 2110-22, over RTP (pending) within HEIF file containers and more. Worst case bitrates are 200Mbps for 1080i, 390Mbps for 1080p60 and 1.4Gbps for 2160p60.
Evolution of Standards-Based IP Workflows Ground-To-Cloud
Last in the presentations is John Mailhot from Imagine Communications and also co-chair of an activity group at the VSF working on standardising interfaces for passing media place to place. Within the data plane, it would be better to avoid vendors repeatedly writing similar drivers. Between ground and cloud, how do we standardise video arriving and the data you need around that. Similarly standardising new technologies like Amazon’s CDI is important.
John outlines the aim of having an interoperability point within the cloud above the low-level data transfer, closer to 7 than to 1 in the OSI model. This work is being done within AIMS, VSF, SMPTE and other organisations based on existing technologies.
The video finishes with a Q&A and includes comments from AWS’s Evan Statton whose talk on CDI that evening is not part of this video. The questions cover comparing NDI with JPEG XS, how CDI uses networking to achieve high bandwidths and high reliability, the balance between minimising network and minimising CPU depending on workflow, the increasingly agile nature of broadcast infrastructure, the need for PTP in the cloud plus the pros and cons of standards versus specifications.
Both broadcast and professional AV are undergoing a transition to IP. For Pro-AV, the benefits are similar to those for Broadcast such as less cabling, cables are bi-directional and reduced space. But for Pro-AV, there is a big incentive to re-use CAT5e cabling unlike the typical SMPTE 2110 deployments which are on 10 gig cabling.
Jean-Baptiste Lorent from intoPIX explains how JPEG XS helps enable cable reuse and, at the same time, brings the advantages of open interoperability and IP to the Pro-AV market. JPEG-XS is a light compression codec which minimises latency. JPEG XS, explains Jean-Baptiste, started in 2016 with light compression format TICO which, working with the JPEG committee has become JPEG-XS. intoPIX have blinded tests showing that at a 10:1 compression ratio, there is no difference observable between the uncompressed source and the JPEG-XS encoder.
Whilst the practical benefits of JPEG-XS are discussed in this talk from Nevion, Laurent outlines that JPEG-XS has a low computational overhead, 0.1 milliseconds encode to decoder.
A very interesting feature is the embedded downscaling meaning if you sent an 8K image as JPEG-XS, a decoder need only decode a UHD or HD resolution version of it – without having to do a computationally expensive downconversion to the desired resolution after having decoded the whole thing 8K image. Jean-Baptiste suggests this would be ideal for devices such as multiviewers which typically show images at much lower than their native resolution.
Looking again at the idea of 1Gbe cable reuse, Jean-Baptiste looks at the data rates achievable with JPEG-XS. HD video under JPEG-XS is 150-390Mbps allowing 2 or more HDs per 1Gbe Cat 5e cable. Using Cat 6 or Cat 5e to run 10Gbe (up to around 40m) allows up to four 8K streams. So it’s practical to use existing infrastructure.
JPEG XS is a brand-new, ultra-low latency standard delivering JPEG 2000 quality with 1000x lower latency; microseconds instead of milliseconds. This mezzanine compression standard promises compression ratios of up to 10:1, resolutions of up to 8K plus HDR and features frame rates from 24 to 120 fps.
Jean-Baptiste Lorent from intoPIX shows how JPEG-XS can be used with SMPTE ST-2110 stack. Part -22 of ST 2110 allows for transport of compressed video essence as an alternative to uncompressed essence – all the other elementary streams stay the same, just the video RTP payload changes. This approach saves a lot of bandwidth and keeps all the existing advantages of moving from SDI to IP at the same time.
Based on TICO which arrived in products four or more years ago allowing HD products to support UHD workflows, JPEG XS was also designed for visually lossless quality and maintaining that quality over multiple re-encoding stages. The combination of very-low microsecond-latency and relatively low bandwidth makes it ideal for remote production of live events.
Director Marketing & Sales
Subscribe to get daily updates
Views and opinions expressed on this website are those of the author(s) and do not necessarily reflect those of SMPTE or SMPTE Members.
This website is presented for informational purposes only. Any reference to specific companies, products or services does not represent promotion, recommendation, or endorsement by SMPTE