Cloudinary Blog

Video Optimization, Part II: Multi-Codec Adaptive Bitrate Streaming

Multi Codec Adaptive Bitrate Streaming

In Part I of this series, we discussed the optimal way to deliver progressive video streams, taking advantage of modern, efficient codecs. That approach works great for short-form videos (under 20 seconds) and for videos that are displayed at a low resolution (such as ads and previews). But what if you're delivering videos that are longer than 20 seconds for a higher-resolution experience? You can certainly still deliver them as a single file (progressive streaming), but you might run into issues, such as buffering or too high a resolution.

As sharp as modern mobile-device screens have become, they can't display all the pixels in high-resolution videos (above 1,920 pixels). Hence, sending down a video too large for the screen is not only a waste of resources and increased bandwidth costs on your part, but also a waste of your users' bandwidth, rendering buffering a likely occurrence.

How can we combine modern codecs with Adaptive Bitrate Streaming to deliver a lightweight and optimal experience to all users, automatically?

Adaptive Bitrate Streaming

Adaptive Bitrate Streaming

A good solution for those problems is Adaptive Bitrate Streaming (ABS). While on YouTube, Netflix, Hulu, or a similar service, if you notice picture quality changing from good to pixelated or vice versa, you've experienced ABS. When encountering slow network conditions, instead of buffering your video, ABS switches dynamically to a lower-bitrate (lower-quality) version of the video, something the connection speed can handle.

ABS also takes into account the available resolution of the viewer. So, if you're viewing the video on an iPhone X, ABS does not deliver the 4K version even if sufficient bandwidth is available to handle such a large file because there are not enough pixels in the display to take advantage of the higher-resolution version. The magic here is that all that happens automatically; you just need to configure the video player with the relevant options.

For a more in-depth discussion of ABS, see the post Plug-and-Play Adaptive Bitrate Streaming With HLS and MPEG-DASH.

Definition of Bitrate

Before diving into how to deliver an optimal adaptive experience, let's talk about the main concept behind it. Bitrate is simply a measure of how many bits it takes to encode a one-second video. Thus, a 10-second video with a 10MB file size has a bitrate of 8 mbps (1 megabyte per second * 8 megabits per megabyte). How can we change the value of the bitrate?

Bitrate = total pixels/second * bytes/pixels

The above equation shows the different levers you can pull to change the bitrate. To achieve a lower bitrate, you can reduce the total number of pixels by reducing the resolution of the video, sampling fewer frames per second, or increasing pixel sharing across frames. You can also reduce the bitrate by compressing the bytes per pixel by lowering the video quality, akin to how you would do that for an image. Part III of this series will suggest the best parameters to adjust for different pieces of content.

Codecs

In Part I, we discussed modern codecs, such as HEVC (h265) and VP9. These codecs render sharing pixels across frames more efficient, encoding the key frames while maintaining the visual fidelity of videos. In fact, they are about 30-50 percent more efficient than the h264 standard. Currently, however, HEVC and VP9 do not enjoy universal browser or device support.

Not to worry. As described in Part I, you can present your video player with multiple codec options in the adaptive bitrate manifest, which would then choose the first codec it knows how to decode. In fact, about 85 percent of Internet traffic uses one of the modern codecs, meaning that your total bandwidth would be reduced by anywhere between 25 to 43 percent. You can easily calculate how cost-effective this approach would be for you.

Creation of Streaming Profiles for Your Video File

To set up codec options, first create the "streaming profiles" to build the manifests. Think of a streaming profile as a list of all the variations you want to create for ABS of your videos. Here, you can define the aspect ratio, the resolutions, the bitrates, and any additional Cloudinary transformations you’d want to perform to the videos, such as watermarks, subtitles, other text, and splicing in other videos.

Create three streaming profiles, one for each codec you want to support: VP9, H265, and H264. Before doing that, be aware that you have two options:

Option 1: Deliver the same visual experience at less bandwidth cost.

Since H265 and VP9 are more efficient, you can get the same picture quality for a lower bitrate. If saving bandwidth costs is your primary concern, this is probably the way to go.

Option 2: Deliver a superior visual experience for the same bandwidth cost.

If providing your users with a superior visual experience is your primary objective, create a streaming profile with higher bitrates than you would in Option 1 for your H265 and VP9 profiles. That means that at a given bandwidth availability, users whose devices support those modern codecs will get a better visual experience.

Part III will address other options. For now, keep in mind that the content of your video could also affect your decision. For instance, for sport or action content, you might want to maintain a higher bitrate at lower resolutions.

Creating Bitrate Profiles

Before creating the profiles for Option 1, which doesn’t strictly preclude some of the benefits in Option 2, see the Cloudinary API code for creating a streaming profile for H264:

Creating Custom Streaming Profiles

cloudinary.v2.api.create_streaming_profile('16x9_h264',
  {
    display_name: "16x9 H264 Codec",
    representations: [
    {transformation:{crop: "pad", width: 1920, aspect_ratio: "16:9", bit_rate: "8500k"}},
    {transformation:{crop: "pad", width: 1280, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", width: 960, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", width: 640, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", width: 480, aspect_ratio: "16:9", bit_rate: "800k"}}
  ] },
  function(error, result){console.log(result);});

Copy that code, paste it into another version, and make the necessary edits for H265, as follows:

cloudinary.v2.api.create_streaming_profile('16x9_h265',
  {
    display_name: "16x9 H265 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "h265", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

Notice the lower bitrates on H265. This profile is meant to maintain the visual fidelity of the H264 version for less bandwidth.

Now create a profile for VP9 by using the same bitrates you did for H265:

  cloudinary.v2.api.create_streaming_profile('16x9_vp9',
  {
    display_name: "16x9 vp9 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "vp9", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

The above bitrates are meant to be an example only. Though they should work fine in most cases, be sure to test them out and tweak the bitrates to your own standards. Keep in mind that a given bitrate at a constant resolution would result in a video of higher visual quality in VP9 and H265 than in H264. Part III will explore this topic further.

Now create all those versions upon upload of a new video, as follows:

CAUTION: This approach will result in a LOT of transformations--something to take into account because video encoding is expensive. Some of this cost might be offset by the decrease in consumed bandwidth, maybe all of it. You might even save money (sometimes a lot!) doing it this way, depending on the ratio of bandwidth consumption to the number of videos you need encoded. Just be aware of that overhead before encoding your entire Cloudinary library this way. Doing so does not make sense for everyone. As stated earlier, by using this method to reduce bandwidth, you can expect a 25 to 43 percent reduction in total outgoing video bandwidth. With Cloudinary, 1 TB of outgoing bandwidth is equal in cost to one million transformations. You can easily calculate the total cost or cost savings you will see.

Uploading Videos

Next, upload a new video and create the various versions. The above example would read like this:

cloudinary.v2.uploader.upload('video-file.mp4', {resource_type: 'video', 
    eager: [{format: 'm3u8', streaming_profile: '16x9_h264'},
      {format: 'm3u8', streaming_profile: '16x9_h265'},
      {format: 'mpd', streaming_profile: '16x9_vp9'}
      ],                                   
    eager_async: true,
    eager_notification_url: 'https://mysite/notify_endpoint', public_id:'yourPublicId'}, function(result) {console.log(result); });

Replace all the variables in bold font with the actual values. When your files are ready, you’ll get an async response to the endpoint you listed under ‘eagernotificationurl`. Remember that you’re creating a lot of files here and, even though VP9 and H265 codecs are extremely efficient consumers of bandwidth, they take about twice as long to encode than H264.

When the upload is complete, you will see the following three manifests, which you’ll need for the next step:

https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h264/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h265/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_vp9/yourPublicId.mpd

Now you need to decide which manifest to pick for each visitor. Thankfully, David Walsh has already figured it out for us!

Check for H265 support first and then VP9 support. If neither of them works, fall back to the H264 manifest.

Check out this Codepen with a real working example.

See the Pen Cloudinary - Multi-Codec Adaptive Bitrate Streaming by Robert Moseley (@mosedart) on CodePen.

Conclusion

At Cloudinary, we’re continuously conducting research on ABS and will make ongoing enhancements to make it easier to execute on multiple codecs. Try it out for yourself and tell us about your experience in the Comments section below.

Recent Blog Posts

Mobile Optimization: Optimize Your Mobile-Web User Experience

TL;DR

We live in a visual world, often while on the go, and consumers expect media-rich web content. Accordingly, the loading speed of images and videos is a big factor in user experience. To optimize customer satisfaction with your mobile content, you must focus on the quality, format, and size of your digital assets. With Cloudinary, optimization is simple, not only enhancing your mobile web and app performance, but also upping your SEO game and boosting customer experience.

Read more
Why Visual Storytelling Matters

In my communications role here at Cloudinary, I get to speak to a lot of experts across a variety of subject matter areas, technology and not, about what it takes to connect to today’s consumer. The power of images and other forms of digital content is often at the center of these conversations given our focus at Cloudinary. And as a hobbyist and professional photographer for more than 20 years, I care deeply about images and their role in creating authentic connections and engagement.

Read more
How to Automatically Generate alt Text for Images

Images and videos are critical for ensuring user engagement on the web. For instance, on a retail website, images of a product from different angles or a 360-degree video of the product can lead to higher conversion rates. For a news website, users are more likely to read articles with visual media accompanying the content. It has been reported that posts that include images produce a 650-percent higher user-engagement rate than text-only posts.

Read more
How to Optimize Media in your Salesforce Commerce Cloud Store

As you may have heard, Cloudinary, the end-to-end media management solution, now has a certified cartridge with Salesforce Commerce Cloud (formerly Demandware), one of the industry's most agile and scalable e-commerce platforms. The Cloudinary cartridge enables you to simplify and automate the process of manipulating, optimizing, and delivering images and videos throughout your entire Commerce Cloud store, including product images and videos, category assets, and site catalogs.

Read more