Cloudinary Blog

Video Optimization, Part II: Multi-Codec Adaptive Bitrate Streaming

Multi Codec Adaptive Bitrate Streaming

In Part I of this series, we discussed the optimal way to deliver progressive video streams, taking advantage of modern, efficient codecs. That approach works great for short-form videos (under 20 seconds) and for videos that are displayed at a low resolution (such as ads and previews). But what if you're delivering videos that are longer than 20 seconds for a higher-resolution experience? You can certainly still deliver them as a single file (progressive streaming), but you might run into issues, such as buffering or too high a resolution.

As sharp as modern mobile-device screens have become, they can't display all the pixels in high-resolution videos (above 1,920 pixels). Hence, sending down a video too large for the screen is not only a waste of resources and increased bandwidth costs on your part, but also a waste of your users' bandwidth, rendering buffering a likely occurrence.

How can we combine modern codecs with Adaptive Bitrate Streaming to deliver a lightweight and optimal experience to all users, automatically?

Adaptive Bitrate Streaming

Adaptive Bitrate Streaming

A good solution for those problems is Adaptive Bitrate Streaming (ABS). While on YouTube, Netflix, Hulu, or a similar service, if you notice picture quality changing from good to pixelated or vice versa, you've experienced ABS. When encountering slow network conditions, instead of buffering your video, ABS switches dynamically to a lower-bitrate (lower-quality) version of the video, something the connection speed can handle.

ABS also takes into account the available resolution of the viewer. So, if you're viewing the video on an iPhone X, ABS does not deliver the 4K version even if sufficient bandwidth is available to handle such a large file because there are not enough pixels in the display to take advantage of the higher-resolution version. The magic here is that all that happens automatically; you just need to configure the video player with the relevant options.

For a more in-depth discussion of ABS, see the post Plug-and-Play Adaptive Bitrate Streaming With HLS and MPEG-DASH.

Definition of Bitrate

Before diving into how to deliver an optimal adaptive experience, let's talk about the main concept behind it. Bitrate is simply a measure of how many bits it takes to encode a one-second video. Thus, a 10-second video with a 10MB file size has a bitrate of 8 mbps (1 megabyte per second * 8 megabits per megabyte). How can we change the value of the bitrate?

Bitrate = total pixels/second * bytes/pixels

The above equation shows the different levers you can pull to change the bitrate. To achieve a lower bitrate, you can reduce the total number of pixels by reducing the resolution of the video, sampling fewer frames per second, or increasing pixel sharing across frames. You can also reduce the bitrate by compressing the bytes per pixel by lowering the video quality, akin to how you would do that for an image. Part III of this series will suggest the best parameters to adjust for different pieces of content.

Codecs

In Part I, we discussed modern codecs, such as HEVC (h265) and VP9. These codecs render sharing pixels across frames more efficient, encoding the key frames while maintaining the visual fidelity of videos. In fact, they are about 30-50 percent more efficient than the h264 standard. Currently, however, HEVC and VP9 do not enjoy universal browser or device support.

Not to worry. As described in Part I, you can present your video player with multiple codec options in the adaptive bitrate manifest, which would then choose the first codec it knows how to decode. In fact, about 85 percent of Internet traffic uses one of the modern codecs, meaning that your total bandwidth would be reduced by anywhere between 25 to 43 percent. You can easily calculate how cost-effective this approach would be for you.

Creation of Streaming Profiles for Your Video File

To set up codec options, first create the "streaming profiles" to build the manifests. Think of a streaming profile as a list of all the variations you want to create for ABS of your videos. Here, you can define the aspect ratio, the resolutions, the bitrates, and any additional Cloudinary transformations you’d want to perform to the videos, such as watermarks, subtitles, other text, and splicing in other videos.

Create three streaming profiles, one for each codec you want to support: VP9, H265, and H264. Before doing that, be aware that you have two options:

Option 1: Deliver the same visual experience at less bandwidth cost.

Since H265 and VP9 are more efficient, you can get the same picture quality for a lower bitrate. If saving bandwidth costs is your primary concern, this is probably the way to go.

Option 2: Deliver a superior visual experience for the same bandwidth cost.

If providing your users with a superior visual experience is your primary objective, create a streaming profile with higher bitrates than you would in Option 1 for your H265 and VP9 profiles. That means that at a given bandwidth availability, users whose devices support those modern codecs will get a better visual experience.

Part III will address other options. For now, keep in mind that the content of your video could also affect your decision. For instance, for sport or action content, you might want to maintain a higher bitrate at lower resolutions.

Creating Bitrate Profiles

Before creating the profiles for Option 1, which doesn’t strictly preclude some of the benefits in Option 2, see the Cloudinary API code for creating a streaming profile for H264:

Creating Custom Streaming Profiles

cloudinary.v2.api.create_streaming_profile('16x9_h264',
  {
    display_name: "16x9 H264 Codec",
    representations: [
    {transformation:{crop: "pad", width: 1920, aspect_ratio: "16:9", bit_rate: "8500k"}},
    {transformation:{crop: "pad", width: 1280, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", width: 960, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", width: 640, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", width: 480, aspect_ratio: "16:9", bit_rate: "800k"}}
  ] },
  function(error, result){console.log(result);});

Copy that code, paste it into another version, and make the necessary edits for H265, as follows:

cloudinary.v2.api.create_streaming_profile('16x9_h265',
  {
    display_name: "16x9 H265 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "h265", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

Notice the lower bitrates on H265. This profile is meant to maintain the visual fidelity of the H264 version for less bandwidth.

Now create a profile for VP9 by using the same bitrates you did for H265:

  cloudinary.v2.api.create_streaming_profile('16x9_vp9',
  {
    display_name: "16x9 vp9 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "vp9", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

The above bitrates are meant to be an example only. Though they should work fine in most cases, be sure to test them out and tweak the bitrates to your own standards. Keep in mind that a given bitrate at a constant resolution would result in a video of higher visual quality in VP9 and H265 than in H264. Part III will explore this topic further.

Now create all those versions upon upload of a new video, as follows:

CAUTION: This approach will result in a LOT of transformations--something to take into account because video encoding is expensive. Some of this cost might be offset by the decrease in consumed bandwidth, maybe all of it. You might even save money (sometimes a lot!) doing it this way, depending on the ratio of bandwidth consumption to the number of videos you need encoded. Just be aware of that overhead before encoding your entire Cloudinary library this way. Doing so does not make sense for everyone. As stated earlier, by using this method to reduce bandwidth, you can expect a 25 to 43 percent reduction in total outgoing video bandwidth. With Cloudinary, 1 TB of outgoing bandwidth is equal in cost to one million transformations. You can easily calculate the total cost or cost savings you will see.

Uploading Videos

Next, upload a new video and create the various versions. The above example would read like this:

cloudinary.v2.uploader.upload('video-file.mp4', {resource_type: 'video', 
    eager: [{format: 'm3u8', streaming_profile: '16x9_h264'},
      {format: 'm3u8', streaming_profile: '16x9_h265'},
      {format: 'mpd', streaming_profile: '16x9_vp9'}
      ],                                   
    eager_async: true,
    eager_notification_url: 'https://mysite/notify_endpoint', public_id:'yourPublicId'}, function(result) {console.log(result); });

Replace all the variables in bold font with the actual values. When your files are ready, you’ll get an async response to the endpoint you listed under ‘eagernotificationurl`. Remember that you’re creating a lot of files here and, even though VP9 and H265 codecs are extremely efficient consumers of bandwidth, they take about twice as long to encode than H264.

When the upload is complete, you will see the following three manifests, which you’ll need for the next step:

https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h264/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h265/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_vp9/yourPublicId.mpd

Now you need to decide which manifest to pick for each visitor. Thankfully, David Walsh has already figured it out for us!

Check for H265 support first and then VP9 support. If neither of them works, fall back to the H264 manifest.

Check out this Codepen with a real working example.

See the Pen Cloudinary - Multi-Codec Adaptive Bitrate Streaming by Robert Moseley (@mosedart) on CodePen.

Conclusion

At Cloudinary, we’re continuously conducting research on ABS and will make ongoing enhancements to make it easier to execute on multiple codecs. Try it out for yourself and tell us about your experience in the Comments section below. Also, feel free to check out our HTML5 Video Player

Recent Blog Posts

 New Image File Format: FUIF:Lossy, Lossless, and Free

I've been working to create a new image format, which I'm calling FUIF, or Free Universal Image Format. That’s a rather pretentious name, I know. But I couldn’t call it the Free Lossy Image Format (FLIF) because that acronym is not available any more (see below) and FUIF can do lossless, too, so it wouldn’t be accurate either.

Read more
Optimizing Video Streaming and Delivery: Q&A with Doug Sillars

Doug Sillars, a digital nomad and a freelance mobile-performance expert, answers questions about video streaming and delivery, website optimization, and more.

Doug Sillars, a freelance mobile-performance expert and developer advocate, is a Google Developer Expert and the author of O’Reilly’s High Performance Android Apps. Given his extensive travels across the globe—from the UK to Siberia—with his wife, kids, and 11-year-old dog, Max, he has been referred to as a “digital nomad.” So far in 2018, Doug has spoken at more than 75 meetups and conferences!

Read more
Building a Music Discovery Service

In May 2018, Cloudinary sponsored Capitol Music Group’s first hackathon held by its new Capitol360 Innovation Center, which aims at connecting musicians and software technologists to facilitate and stimulate the creation of music. See this interview for details. As a starter project for the hackathon, we built a sample app called Music Discovery Service.

Read more
Once a Hackathon Participant, Now a Judge

Over the past several years, I've had a chance to participate in hackathons, as part of teams that developed a social payment app and helped users decide meals to cook. But it wasn't until last month that I got to experience a hackathon from the other side, as a judge.

Read more
10 Website Videos Mistakes and How to Solve Them

It should come as no surprise that video use on the internet is exploding. You can see the dramatic growth of video on the average site in this SpeedCurve blog post.

With the growth in video comes greater bandwidth use, which is not only costly for your IT budget, but for your visitors as well. Beyond the expense, there is the user experience to consider. The heavier the page, the longer it will take to load, and the greater likelihood visitors will abandon your site. Page load speed is also an important factor in SEO ranking, so clearly video is something we need to take seriously and get right. Video is challenging, presenting terms still unfamiliar to developers - like codecs, bitrate and adaptive bitrate streaming. As a result, mistakes are being made in video implementation.

Read more
Android Data Saver: Optimizing Mobile Data Usage with Cloudinary

Over the life of a mobile device, the cost of a cellular data plan often exceeds that of the device itself. To optimize data usage and purge useless data on their mobile devices, users can enable Data Saver from Android 7.0 (API level 24). To do so, users toggle Data Saver in quick settings under the Notification shade or under Settings > Data usage. With Data Saver enabled, apps that aren't whitelisted cannot use cellular data in the background. They are also directed to consume less data while active.

Read more