Video: All you need to know about video KPIs

KPIs are under the microscope as Milan’s Video Tech meet up fights against the pandemic by having its second event online and focused on measuring, and therefore improving, streaming services.

Looking at ‘Data-Driven Business Decision Making‘, Federico Preli, kicks off the event looking at how to harness user data to improve the user experience. He explains this using Netflix’s House of Cards as an example. Netflix commissioned 2 seasons of House of Cards based not on a pilot, but on data they already have. They knew the British version had been a hit on the platform, they could see that the people who enjoyed that, also watched other films from Kevin Spacey or David Fincher (the director of House of Cards). As such, this large body of data showed that, though success was not guaranteed, there was good cause to expect people to be receptive to this new programme.

Federico goes on to explain how to balance recommendations based upon user data. A balance is necessary, he explains, to avoid a bubble around a viewer where the same things keep on getting recommended and not to exaggerate someone’s interests at the detriment of nuance and not representing the less prominent predilections. He outlines the 5 parts of a balanced recommendations experience: Serendipity, diversity, coverage, fairness & trust. Balancing these equally will provide a rounded experience. Finally, Federico discusses how some platforms may choose to under invest in some of these due to the nature of their platforms. Relevance, for instance, may be less important for an ultra-niche platform where everything has relevance.

Performance Video KPIs at the Edge‘ is the topic of Luca Moglia‘s talk. A media solutions engineer from Akamai, he looks at how to derive more KPI information from logs at the edge. Whilst much data comes from a client-side KPI, data directly reported by the video player itself to the service. Client-side information is vital as only the client knows on which button you clicked, for instance and how long you spent in certain parts of the GUI. But in terms of video playback, there is a lot to be understood by looking at the edge, the part of the CDN which is closest to the client.

One aspect that client-side reporting doesn’t cover is use of the platform by clients which aren’t fully supported meaning they report back less information. Alternatively, for some services, it may be possible to access them with clients which don’t report at all. Depending on how reporting is done, this could be blocked by ad blockers or DNS rules. As such, this is an important gap which can be largely filled by analysis of CDN logs. This allows you to enhance the data analysis done elsewhere and validate it.

Luca gives examples of KPIs that can be measured or inferred from the edge, such as ‘hand-waving latency’ which can be understood from the edge-to-origin latency and time to manifest. He also shows an example graph analysing the number of segments served at the edge within the segment duration time. This helps indicate how many streams weren’t rebuffering. Overall, Luca concludes, analysing data from the edge helps track improvements, gives you better visibility on consumer/global events and allows you to enhance the performance of the platform.

Bitmovin’s Andrea Fassina covers ‘Client KPIs – Five Analytics Metrics That Matter‘ which he summarises at the beginning of his talk ahead of explaining each individually. ‘Impressions & Total Hours Watched’ is first. This metric has really shown its importance as the SARS-CoV-2 pandemic has rolled around the globe. Understanding how much more people are watching is important in understanding how your platform is reacting. After all, if a platform is struggling this could be for many reasons that are correlated with, but not because of, more hours streamed. For instance, in boxing matches, it’s often the payment system which struggles before the streaming does.

Video startup time is next. Andrea explains the statistics of lost viewers as your time-to-play increases. You can look at startup time across each device and see where the low-hanging fruit for improvements and prioritise your work. This metric can be extended to ad playing and DRM load time which need to be brought into the overall equation.

Third is Video Bitrate Heatmap which allows you to see which type of chunks are most used and, similarly, which rungs on your ABR ladder aren’t needed (or could be improved.) The fourth KPI discussed is Error Types and Codes. Analysing codes generated can give you early warning to issues and allow you to understand whether you suffer more problems than the industry average (6.6%) but also proactively talk to connectivity providers to reduce problems. Lastly, Andrea explains how Rebuffering percentage helps understand where there are gaps in your service in terms of devices/apps which are particularly struggling.

Source: Andrea Fassina, Bitmovin

Video Quality Metrics‘ rounds off the session as Fabio Sonnati tackles the tricky problem of how to know what quality of video each viewer is seeing. Given that the publisher has each and every chunk and can view them, many would think this would mean you could see exactly what each stream would look like. But a streaming service can only see what each chunk looks like on their device in their environment. When you view a chunk encoded at 1080i on an underpowered SD device, what does the user actually see and would they have been better receiving a lower resolution, lower bitrate chunk instead?

In order to understand video quality, Fabio briefly explains some objective metrics such as VMAD, SSIM and PSNR. He then discusses the way that Sky Italia have chosen to create their own metric by combining metrics, subjective feedback and model training. The motivation to do this, to tailor your metric to the unique issues that your platform has to contend with. This metric, called SynthEYE, has been expanded to be able to run without a reference – i.e. it doesn’t require the source as well as the encoded version. Fabio shows results of how well SynthEYE Absolute predicts VMAF and MOS scores. He concludes by saying that using an absolute metric is useful because it gives you the ability to analyse chunk-by-chunk and then match that up with resolution and other analytics data to better understand the performance of the platform.

The session concluded with 20 minutes of Q&A

Watch now!
Speakers

Luca Moglia Luca Moglia
Media Solutions Engineer,
Akamai
Andrea Fassina Andrea Fassina
Developer Evangelist,
Bitmovin
Fabio Sonnati Fabio Sonnati
Media Architect and
Encoding & Streaming Specialist
Federico Preli Federico Preli
Senior Solution Architect,
ContentWise
Stefano Morello Moderator: Stefano Morello
Senior Sales Engineer,
ContentWise

Video: Pervasive video deep-links

Google have launched a new initiative allowing publishers to highlight key moments in a video so that search results can jump straight to that moment. Whether you have a video that looks at 3 topics, one which poses questions and provides answers or one which has a big reveal and reaction shots, this could help increase engagement.

The plan is the content creators tell Google about these moments so Paul Smith from theMoment.tv takes to the stage at San Francisco Video Tech to explain how. After looking at a live demo, Paul takes a dive into the webpage code that makes it happen. Hidden in the tag, he shows the script which has its type set to application/ld+json. This holds the metadata for the video as a whole such as the thumbnail URL and the content URL. However it also then defines the highlighted ‘parts’ of the video with URLs for those.

Whiles the programme is currently limited to a small set of content publishers, everyone can benefit from these insights on google video search. It will also look at YouTube descriptions in which some people give links to specific times such as different tracks in a music mix, and bring those into the search results.

Paul looks at what this means for website and player writers. On suggestion is the need to scroll the page to the correct video and make the different videos on a page clearly signposted. Paul also looks towards the future at what could be done to better integrate with this feature. For example updating the player UI to see and create moments or improve the ability to seek to sub-second accuracy. Intriguingly he suggests that it may be advantageous to synchronise segment timings with the beginning of moments for popular video. Certainly food for thought.

Watch now!
Speaker

Paul Smith Paul Smith
Founder,
theMoment.tv

Webinar: Broadcaster VOD: Delivering the next-generation of catch-up viewing

With Amazon, Netflix and so many other VOD services available, broadcasters like the BBC and Discovery are investing a lot in their own VOD services, known as Broadcaster VOD (BVOD) in order to maintain relevance, audiences and revenue.

Commercial broadcasters such as Sky, ITV and Channel 4 are trying hard to attract advertisers and “have all launched new ad formats, struck deals with ad tech vendors to build marketplaces and set up programmatic teams to manage them” according to a report from digiday.com. As such this means that the battle for advertisers wallets is moving more towards VOD from linear.

Date: Thursday 30 January, 14:00 GMT / 9 a.m. ET

With this in mind, IBC365 will discuss the business models, platforms and strategies being used by BVOD platforms. They will look at the BBC’s move to build a deep content library of free-to-view box sets, and to the importance of data, personalisation and addressable advertising models.

Further more, this webinar will talk about the commercial and technical requirements to build a BVOD to a standard that’s going to stand on its own in this increasingly crowded, but well-funded marketplace.

Register now!
Speakers

Richard Davidson-Houston Richard Davidson-Houston
Founder,
Finally Found Ltd.
Roma Kojima Roma Kojima
Senior Director OTT Video (CBC Gem),
Canadian Broadcasting Corporation
Niels Baas Niels Baas
Managing Director, NLZIET

Video: Platform Wars – the De-aggregation and Re-aggregation of Content

How does the move to OTT delivery impact the traditional platforms? Are there too many streaming services? This session looks at the new platforms, the consumer experience, the role of aggregation and the way that operators have been involved in de-aggregation and then re-aggregation of channel packages both in competition and in cooperation.

How many subscription services are too many for a household? There’s some thinking that 3 may be the typical maximum when people tend to switch to a ‘one in, one out’ policy on subscription packages. Colin Dixon says the average is currently 2 in the UK and Germany. The panel asks whether we should have as many and compares the situation with audio where ‘super aggregation’ rules. Services like Apple Music and Spotify rely on aggregating ‘all’ music and consumers don’t subscribe separately to listen to Sony artists one on service and EMI on another, so what is it that drives video to be different and will it stay that way?

The topic then switches to smart TVs discussing the feeling that five to eight years ago they had a go at app stores and ended up disappointing. Not only was it often clunky at the time, but support has now gone on the whole from the manufacturers. Is the current wave of smart TVs any different? From BT’s perspective, explains Colin Phillips, it’s very costly to keep many different versions of app up to date and tested so a uniform platform across multiple TVs would be a lot better.

The talk concludes looking at the future for Disney+, Netflix and other providers ahead of discussing predictions from industry analysts.

Watch now!

This panel was produced by IET Media, a technical network within the IET which runs events, talks and webinars for networking and education within the broadcast industry. More information

Speakers

Colin Phillips Colin Phillips
IPTV and CPE Solutions Architect,
British Telecom
Chris Ambrozic Chris Ambrozic
VP of Product,
TiVo
Colin Dixon Colin Dixon
Founder & Chief Analyst,
nScreenMedia
Brian Paxton Brian Paxton
Managing Director,
Kingsmead Security
Justin Lebbon Moderator: Justin Lebbon
Co-Founder & Director,
Mediatel Events Ltd