Cloudinary Blog
High Quality Style Transfer Using Deep Learning Techniques

If you are anything like me, one of the things you love about this digital era is that you can be artistic and creative, even if your drawing skills never made it much past stick figures. The average smartphone can take photographs of astonishing quality. And all sorts of special effects, filters, and overlays enable users to convert their favorite photos into unique expressions of themselves.

But all of this still generally remains within the realm of photographic content.

What if there was an algorithm so smart, that it could actually analyze your favorite work of art and then 'paint' you a replica of any other photo in the same style? Not some pixel-by-pixel analysis, but evaluating it as a talented artist would: taking into account colors, brush styles and other artistic techniques and then applying those styles in a way that blends with the content of the target photograph.

Yep, you got it. There is in-fact such an algorithm. Well, it's more than just an algorithm. It requires a sophisticated, multi-layered neural network, known as VGG 16. The network can be used for a variety of applications in categorization, semantic segmentation, and more. Since Gatys et al first realized the network's potential for separately extracting style and content information from images in 2015, there have been a few attempts to commercialize these style transfer deep learning capabilities (applying the style of one image to the content of another). However, in most cases, the practicality of these services has been quite limited.

Webinar
How to Optimize for Page Load Speed

For example, most services available today limit the source artworks to a few, limit the resolution size of the output image, require unrealistic wait times, appropriate huge amounts of the customers' data allotment and processing resources, and/or have a pricing model for high resolution images that makes it impractical to really commercialize.

But now here's the real "what if": What if you could offer all that genius neural network and style transfer deep learning capability to your web site or mobile app users with only a few lines of code and get reliable results, in seconds, for a nominal cost, with all the processing done in the cloud?

As of today, "Yes you can". You can do it with Cloudinary's new Neural Artworks add-on by simply adding the style_transfer effect to any delivered image while specifying the image ID of any source artwork as it's overlay. And voila! Your users can generate and share their own masterpieces in seconds!

This is all it takes:

Ruby:
Copy to clipboard
cl_image_tag("golden_gate.jpg", :transformation=>[
  {:width=>700, :height=>700, :crop=>"fill"},
  {:effect=>"style_transfer", :overlay=>"sailing_angel"}
  ])
PHP:
Copy to clipboard
cl_image_tag("golden_gate.jpg", array("transformation"=>array(
  array("width"=>700, "height"=>700, "crop"=>"fill"),
  array("effect"=>"style_transfer", "overlay"=>"sailing_angel")
  )))
Python:
Copy to clipboard
CloudinaryImage("golden_gate.jpg").image(transformation=[
  {'width': 700, 'height': 700, 'crop': "fill"},
  {'effect': "style_transfer", 'overlay': "sailing_angel"}
  ])
Node.js:
Copy to clipboard
cloudinary.image("golden_gate.jpg", {transformation: [
  {width: 700, height: 700, crop: "fill"},
  {effect: "style_transfer", overlay: "sailing_angel"}
  ]})
Java:
Copy to clipboard
cloudinary.url().transformation(new Transformation()
  .width(700).height(700).crop("fill").chain()
  .effect("style_transfer").overlay(new Layer().publicId("sailing_angel"))).imageTag("golden_gate.jpg");
JS:
Copy to clipboard
cloudinary.imageTag('golden_gate.jpg', {transformation: [
  {width: 700, height: 700, crop: "fill"},
  {effect: "style_transfer", overlay: new cloudinary.Layer().publicId("sailing_angel")}
  ]}).toHtml();
jQuery:
Copy to clipboard
$.cloudinary.image("golden_gate.jpg", {transformation: [
  {width: 700, height: 700, crop: "fill"},
  {effect: "style_transfer", overlay: new cloudinary.Layer().publicId("sailing_angel")}
  ]})
React:
Copy to clipboard
<Image publicId="golden_gate.jpg" >
  <Transformation width="700" height="700" crop="fill" />
  <Transformation effect="style_transfer" overlay="sailing_angel" />
</Image>
Vue.js:
Copy to clipboard
<cld-image publicId="golden_gate.jpg" >
  <cld-transformation width="700" height="700" crop="fill" />
  <cld-transformation effect="style_transfer" overlay="sailing_angel" />
</cld-image>
Angular:
Copy to clipboard
<cl-image public-id="golden_gate.jpg" >
  <cl-transformation width="700" height="700" crop="fill">
  </cl-transformation>
  <cl-transformation effect="style_transfer" overlay="sailing_angel">
  </cl-transformation>
</cl-image>
.Net:
Copy to clipboard
cloudinary.Api.UrlImgUp.Transform(new Transformation()
  .Width(700).Height(700).Crop("fill").Chain()
  .Effect("style_transfer").Overlay(new Layer().PublicId("sailing_angel"))).BuildImageTag("golden_gate.jpg")
Android:
Copy to clipboard
MediaManager.get().url().transformation(new Transformation()
  .width(700).height(700).crop("fill").chain()
  .effect("style_transfer").overlay(new Layer().publicId("sailing_angel"))).generate("golden_gate.jpg");
iOS:
Copy to clipboard
imageView.cldSetImage(cloudinary.createUrl().setTransformation(CLDTransformation()
  .setWidth(700).setHeight(700).setCrop("fill").chain()
  .setEffect("style_transfer").setOverlay("sailing_angel")).generate("golden_gate.jpg")!, cloudinary: cloudinary)

Sailing angel - source artwork Source artwork Golden gate bridge - target photo Target photo Style transfer result Style transfer result

The applications for this functionality are endless, from marketing contests to cool profile pics to unique personalized canvas prints, and more. I'm sure your mind is already jumping with ideas.

Wait, what's Cloudinary, anyway?

Cloudinary is a SaaS offering that provides solutions for image and video management, including server or client-side upload, a huge range of on-the-fly image and video manipulation options, quick CDN delivery, and powerful asset management options.

Cloudinary enables web and mobile developers to address all their media management needs with simple bits of code in their favorite programming languages or frameworks, leaving them free to focus primarily on their own product's value proposition.

The style_transfer transformation effect provided with the new add-on integrates seamlessly with all other Cloudinary features.

Tell me more about that neural network

Deep neural networks like the one used by our add-on use a type of artificial intelligence (AI) that's designed to mimic the behaviors of the neurons in our brain. Like young children, they 'learn' by 'viewing' large amounts of data until they begin to recognize common patterns.

Neural networks that focus on style transfer attempt to synthesize textures and colors from the source artwork within constraints that enable it to still preserve the main semantic content of the target photograph, and thus allow, to some extent, a separation of image content from image style.

This very simplified diagram, taken from the paper 'A Neural Algorithm of Artistic Style' by Leon A. Gatys et al, provides a conceptual demonstration of how different abstractions of the source and target images can be filtered at different layers. The target photograph is downsampled so that the essential elements of the input image can be applied without losing the essence of the original content.

neural network diagram by Leon A. Gatys

 

The more the neural network is allowed to process a particular source artwork, the better the replication of the style can be. But each incremental improvement comes at a huge performance cost.

Cloudinary's algorithm takes advantage of Xun Huange and Serge Belongie's enhancement on the Gatys algorithm, which make it possible to use any image for both source and target, and still deliver a good quality style transfer in real time, using a single feed-forward neural network. While the style match is not as precise as some other available services, Cloudinary's implementation is much faster, is not limited to pre-learned images, and even supports high resolution outputs that are out of scope for most similar services currently available.

Your turn!

By now, I'm sure you are ready to get your hands on this artistic magic, so here's a small demo for you to play with.

Note: Although the feature supports using any image for the source artwork, we've provided a limited set of sources here to avoid uploads of unlicensed art. When you start using this feature on your own account, do make sure that all source artworks have valid usage licenses.

Pick a source artwork:

  Pick a target photo:Or pick your own:   

golf course target photo coffee target photo old man target photo custom target photo

  Press the magic button!

The demo above shows the default style_transfer effect. If you want to retain more of the target photograph's colors or photographic essense, you can include the Boolean preserve_color option or adjust the style_strengthof the effect. Learn more about the style_transfer effect and these additional options in the Neural Artwork Style Transfer documentation

I want to jump on this bandwagon too!

Are you ready to offer Neural Artworks to your users? Or just want to take advantage of this powerful deep learning feature to create some masterpieces for your own site or blog page?

Anyone with a Cloudinary account can register for the Neural Artworks add-on. Take advantage of the add-on's free quota to start playing around with it within your own app.

Share your ideas!

We can't wait to see what amazing ideas you come up with! We'd love if you would share how you envision using style transfer in your app and better yet, screencaps of your own creations, right here in the comments for everyone to enjoy!

The future is so bright...

It's clear that this is just the beginning of the line. Deep learning and neural networks are undoubtedly going to fundamentally change the way we create media and art of all kinds in the very foreseeable future. So, jump on the bandwagon now. It's going to be one heck of a ride!

Recent Blog Posts

Maya Shavin: How I Built My Website

Besides working as a senior front-end developer at Cloudinary, I'm also a content creator, a blogger, and an open-source developer. Follow me at @mayashavin and on mayashavin.com.

In the beginning, my website, mayashavin.com, was mainly for showcasing the status of my development projects and keeping me organized with my speaking schedule. Initially, I built it with Vue.js, later on switching to Nuxt.js (aka Nuxt) for a higher SEO score, and deployed it with Netlify. After some time, I added a blog section with Netlify CMS as the content management system (CMS). Everything was fine until I added more content and features, which led to a significant decline in the site’s performance. Also, the site design needed a modern look. So, I gave the site a makeover.

Read more
Automation Frees Up PetRescue’s Staff to Help Pets Find Their Forever Homes

As we spend more time at home, many of us are adopting pets for the joy, companionship and a surprising range of health benefits. In Australia, where our nonprofit customer PetRescue is located, there’s a shortage of pets to adopt. Last August, the Guardian reported that dog shelters in Australia emptied and adoption fees for puppies were running as high as $AUS1800.

Read more
Cloudinary and Contentful Make Modern Content Management Easier

I am pleased to share that Cloudinary and Contentful have joined forces to further streamline the creation, processing, and delivery of online content through Cloudinary’s digital asset management (DAM) solution and advanced transformation and delivery capabilities for images and video. What’s more, the partnership delivers a headless approach to DAM. By leveraging APIs for media management tasks, marketers and developers alike benefit from an integrated stack of optimized assets for optimization and automation. As a result, page loads are fast and beautiful, and at scale—with less overhead and effort.

Read more
Introducing Cloudinary's Nuxt Module

Since its initial release in October 2016 by the Chopin brothers as a server-side framework that runs on top of Vue.js, Nuxt (aka Nuxt.js) has gained prominence in both intuitiveness and performance. The framework offers numerous built-in features based on a modular architecture, bringing ease and simplicity to web development. Not surprisingly, Nuxt.js has seen remarkable growth in adoption by the developer community along with accolades galore. At this writing, Nuxt has earned over 30K stars on GitHub and 96 active modules with over a million downloads per month. And the upward trend is ongoing.

Read more
How Quality and Quantity can go Hand in Hand

When it comes to quality versus quantity, you’ll often hear people say, “It’s the quality that counts, not the quantity”. While that’s true in many situations, there are also cases where you want both quality and quantity. You may have thousands of images on your website and you want them all to look great. This is especially important if your website allows users to upload their own content, for example, to sell their own products or services. You don't want their poor quality images to reflect badly on your brand.

Read more