Cloudinary Blog

Video Optimization, Part II: Multi-Codec Adaptive Bitrate Streaming

Multi Codec Adaptive Bitrate Streaming

In Part I of this series, we discussed the optimal way to deliver progressive video streams, taking advantage of modern, efficient codecs. That approach works great for short-form videos (under 20 seconds) and for videos that are displayed at a low resolution (such as ads and previews). But what if you're delivering videos that are longer than 20 seconds for a higher-resolution experience? You can certainly still deliver them as a single file (progressive streaming), but you might run into issues, such as buffering or too high a resolution.

As sharp as modern mobile-device screens have become, they can't display all the pixels in high-resolution videos (above 1,920 pixels). Hence, sending down a video too large for the screen is not only a waste of resources and increased bandwidth costs on your part, but also a waste of your users' bandwidth, rendering buffering a likely occurrence.

How can we combine modern codecs with Adaptive Bitrate Streaming to deliver a lightweight and optimal experience to all users, automatically?

Adaptive Bitrate Streaming

Adaptive Bitrate Streaming

A good solution for those problems is Adaptive Bitrate Streaming (ABS). While on YouTube, Netflix, Hulu, or a similar service, if you notice picture quality changing from good to pixelated or vice versa, you've experienced ABS. When encountering slow network conditions, instead of buffering your video, ABS switches dynamically to a lower-bitrate (lower-quality) version of the video, something the connection speed can handle.

ABS also takes into account the available resolution of the viewer. So, if you're viewing the video on an iPhone X, ABS does not deliver the 4K version even if sufficient bandwidth is available to handle such a large file because there are not enough pixels in the display to take advantage of the higher-resolution version. The magic here is that all that happens automatically; you just need to configure the video player with the relevant options.

For a more in-depth discussion of ABS, see the post Plug-and-Play Adaptive Bitrate Streaming With HLS and MPEG-DASH.

Definition of Bitrate

Before diving into how to deliver an optimal adaptive experience, let's talk about the main concept behind it. Bitrate is simply a measure of how many bits it takes to encode a one-second video. Thus, a 10-second video with a 10MB file size has a bitrate of 8 mbps (1 megabyte per second * 8 megabits per megabyte). How can we change the value of the bitrate?

Bitrate = total pixels/second * bytes/pixels

The above equation shows the different levers you can pull to change the bitrate. To achieve a lower bitrate, you can reduce the total number of pixels by reducing the resolution of the video, sampling fewer frames per second, or increasing pixel sharing across frames. You can also reduce the bitrate by compressing the bytes per pixel by lowering the video quality, akin to how you would do that for an image. Part III of this series will suggest the best parameters to adjust for different pieces of content.

Codecs

In Part I, we discussed modern codecs, such as HEVC (h265) and VP9. These codecs render sharing pixels across frames more efficient, encoding the key frames while maintaining the visual fidelity of videos. In fact, they are about 30-50 percent more efficient than the h264 standard. Currently, however, HEVC and VP9 do not enjoy universal browser or device support.

Not to worry. As described in Part I, you can present your video player with multiple codec options in the adaptive bitrate manifest, which would then choose the first codec it knows how to decode. In fact, about 85 percent of Internet traffic uses one of the modern codecs, meaning that your total bandwidth would be reduced by anywhere between 25 to 43 percent. You can easily calculate how cost-effective this approach would be for you.

Creation of Streaming Profiles for Your Video File

To set up codec options, first create the "streaming profiles" to build the manifests. Think of a streaming profile as a list of all the variations you want to create for ABS of your videos. Here, you can define the aspect ratio, the resolutions, the bitrates, and any additional Cloudinary transformations you’d want to perform to the videos, such as watermarks, subtitles, other text, and splicing in other videos.

Create three streaming profiles, one for each codec you want to support: VP9, H265, and H264. Before doing that, be aware that you have two options:

Option 1: Deliver the same visual experience at less bandwidth cost.

Since H265 and VP9 are more efficient, you can get the same picture quality for a lower bitrate. If saving bandwidth costs is your primary concern, this is probably the way to go.

Option 2: Deliver a superior visual experience for the same bandwidth cost.

If providing your users with a superior visual experience is your primary objective, create a streaming profile with higher bitrates than you would in Option 1 for your H265 and VP9 profiles. That means that at a given bandwidth availability, users whose devices support those modern codecs will get a better visual experience.

Part III will address other options. For now, keep in mind that the content of your video could also affect your decision. For instance, for sport or action content, you might want to maintain a higher bitrate at lower resolutions.

Creating Bitrate Profiles

Before creating the profiles for Option 1, which doesn’t strictly preclude some of the benefits in Option 2, see the Cloudinary API code for creating a streaming profile for H264:

Creating Custom Streaming Profiles

cloudinary.v2.api.create_streaming_profile('16x9_h264',
  {
    display_name: "16x9 H264 Codec",
    representations: [
    {transformation:{crop: "pad", width: 1920, aspect_ratio: "16:9", bit_rate: "8500k"}},
    {transformation:{crop: "pad", width: 1280, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", width: 960, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", width: 640, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", width: 480, aspect_ratio: "16:9", bit_rate: "800k"}}
  ] },
  function(error, result){console.log(result);});

Copy that code, paste it into another version, and make the necessary edits for H265, as follows:

cloudinary.v2.api.create_streaming_profile('16x9_h265',
  {
    display_name: "16x9 H265 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "h265", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "h265", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

Notice the lower bitrates on H265. This profile is meant to maintain the visual fidelity of the H264 version for less bandwidth.

Now create a profile for VP9 by using the same bitrates you did for H265:

  cloudinary.v2.api.create_streaming_profile('16x9_vp9',
  {
    display_name: "16x9 vp9 Codec",
    representations: [
    {transformation:{crop: "pad", video_codec: "vp9", width: 1920, aspect_ratio: "16:9", bit_rate: "5500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 1280, aspect_ratio: "16:9", bit_rate: "3500k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 960, aspect_ratio: "16:9", bit_rate: "2000k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 640, aspect_ratio: "16:9", bit_rate: "800k"}},
    {transformation:{crop: "pad", video_codec: "vp9", width: 480, aspect_ratio: "16:9", bit_rate: "500k"}}
  ] },
  function(error, result){console.log(result);});

The above bitrates are meant to be an example only. Though they should work fine in most cases, be sure to test them out and tweak the bitrates to your own standards. Keep in mind that a given bitrate at a constant resolution would result in a video of higher visual quality in VP9 and H265 than in H264. Part III will explore this topic further.

Now create all those versions upon upload of a new video, as follows:

CAUTION: This approach will result in a LOT of transformations--something to take into account because video encoding is expensive. Some of this cost might be offset by the decrease in consumed bandwidth, maybe all of it. You might even save money (sometimes a lot!) doing it this way, depending on the ratio of bandwidth consumption to the number of videos you need encoded. Just be aware of that overhead before encoding your entire Cloudinary library this way. Doing so does not make sense for everyone. As stated earlier, by using this method to reduce bandwidth, you can expect a 25 to 43 percent reduction in total outgoing video bandwidth. With Cloudinary, 1 TB of outgoing bandwidth is equal in cost to one million transformations. You can easily calculate the total cost or cost savings you will see.

Uploading Videos

Next, upload a new video and create the various versions. The above example would read like this:

cloudinary.v2.uploader.upload('video-file.mp4', {resource_type: 'video', 
    eager: [{format: 'm3u8', streaming_profile: '16x9_h264'},
      {format: 'm3u8', streaming_profile: '16x9_h265'},
      {format: 'mpd', streaming_profile: '16x9_vp9'}
      ],                                   
    eager_async: true,
    eager_notification_url: 'https://mysite/notify_endpoint', public_id:'yourPublicId'}, function(result) {console.log(result); });

Replace all the variables in bold font with the actual values. When your files are ready, you’ll get an async response to the endpoint you listed under ‘eagernotificationurl`. Remember that you’re creating a lot of files here and, even though VP9 and H265 codecs are extremely efficient consumers of bandwidth, they take about twice as long to encode than H264.

When the upload is complete, you will see the following three manifests, which you’ll need for the next step:

https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h264/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_h265/yourPublicId.m3u8
https://res.cloudinary.com/demo-robert/video/upload/sp_16x9_vp9/yourPublicId.mpd

Now you need to decide which manifest to pick for each visitor. Thankfully, David Walsh has already figured it out for us!

Check for H265 support first and then VP9 support. If neither of them works, fall back to the H264 manifest.

Check out this Codepen with a real working example.

See the Pen Cloudinary - Multi-Codec Adaptive Bitrate Streaming by Robert Moseley (@mosedart) on CodePen.

Conclusion

At Cloudinary, we’re continuously conducting research on ABS and will make ongoing enhancements to make it easier to execute on multiple codecs. Try it out for yourself and tell us about your experience in the Comments section below. Also, feel free to check out our HTML5 Video Player

Recent Blog Posts

The Visual Media Report: Visual Engagement and User Experience

With privacy top of mind, we wondered what we might learn from analyzing the large volume of data. What user behaviors would we discover, what regional differences might exist? What insights or early hints from different industries could we extrapolate? These questions guided us as we analyzed millions of anonymous end-user experiences and asset interactions across our platform.

Read more
How a Cloud-native DAM Platform Optimizes Customer Experiences

In today’s digital age, brands rely heavily on rich media to tell stories, foster engagement, and make emotional connections that drive results. Marketers use videos and images, tuned to customers’ interests, to create dynamic visual experiences. Digital campaigns about trips to Florida, for instance, have separate plot lines, depending on audiences’ passions for golf, deep sea fishing, or kid-friendly versus romantic getaways. Marketers expect to detect preferences, produce personalized experiences highlighting different desires, and turn digital prospects into vacationing customers. The ultimate success of a marketing campaign relies on the consistent delivery of these customer experiences, at scale.

Read more
Cloudinary’s Media Developer Experts Program

Cloudinary was founded by developers and developer-centric thinking is in our DNA. Our work with developers helps them better understand all things rich-media management and delivery is crucial to us. Now, in an effort to recognize, support, and reward the innovative leaders in that technical community, we’re excited to introduce a new Media Developer Experts (MDE) program! MDEs will leverage the Cloudinary platform to foster a community of media-management professionals; receive training and certifications to become experts within their field or audience; advance the state of media management, adoption, and best practices; and make the web more accessible.

Read more
An Eye-Opening Talk: Building Apps for the Next Billion Users in Africa

William (iChuloo) Imoh, who hails from Lagos, Nigeria, recently embarked on a U.S. speaking tour, February 20-March 12, during which he powwowed with technical and product teams and communities at such renowned enterprises as Netlify, Pluralsight, Lucidchart, Twilio, and more in Salt Lake City, Dallas, Las Vegas, and San Francisco. On March 5, he gave an enlightening talk, entitled International Developers and Development: Building for the Next Billion Users at Cloudinary in Santa Clara, California. Below is a synopsis. For details, see the related slides.

Read more
The Debut of the Cloudinary Customer Advisory Board

Focus on customers has always been Cloudinary’s mantra. Because we owe them our success, we are constantly reaching out to our customers, not just for feedback on our offerings, but also for their vision, wish list, and buy-in of what Cloudinary can do to meet their needs and make them succeed. About six months ago, it occurred to us that it would be beneficial if we could meet regularly with those who are behind innovation at our key customers—executives, product gurus, developers, content managers—to swap strategies, product roadmaps, best practices, and such. In particular, we’d like to solicit actionable feedback as a foundation for our plans of product enhancements.

Read more