Cloudinary Blog

Serverless Tutorial: File Storage with Webtask and Cloudinary

Serverless Tutorial: File Storage with Webtask and Cloudinary

Media makes up the majority of today's website content. While it makes websites more interesting for visitors, media presents challenges because these images and videos are more difficult to store, deliver and manipulate on-the-fly to suit any given situation.

One particularly challenging topic is storage. Traditionally - before the cloud era - FTP was the the go-to solution. We relied on FTP for transferring files to our remote computers. These files are mapped to a URL that we could just deliver to the browser.

What if you have hundreds of images of varying quality and size that need to be stored and delivered. If you do not carefully manage them, storing these files could become difficult, which could lead to unforeseen complexities.

The introduction of cloud storage and hosting helped address the storage problem. Notwithstanding, the cloud (and DevOps engineering generally) still remain a mystery for those of us developers who just write code.

This mystery, however, is about to be solved. Today, we are going to attempt to go completely serverless by deploying Functions to Webtask and storing media files on Cloudinary.

The term serverless does not imply that servers will no longer exist. Rather, it implies that we, the developers, no longer have to care about their existence. We won’t have to set anything up, or bother about IP address and all those terms (like load balancing, containers, etc) that we aren’t accustomed to. We will be able to write and ship functions, as well as upload images to an existing server, for free by creating a Cloudinary account.

Why Serverless

Let's now look in more detail about why you may want to consider this serverless option:

"Serverless" in this article refers to both deployable functions (Function as a Service) and cloud platforms (aka Backend as a Service) like Cloudinary

  • Affordability: Not just serverless, but generally, PaaS, SaaS, IaaS, and *aaS are affordable on a large scale when compared to the cost of doing it yourself. In fact, on a small scale, most services are made free, much like Cloudinary
  • Risk Free: Managing backups, security and other sensitive aspects of your project lifecycle shouldn’t be something you have to worry about. As a developer, your sole job is to write code, so you can focus on that while the platforms take care of these other tasks.
  • Taking the serverless route enables you to focus on what matters in your development cycle as a developer. Sticking to your job of writing code and using simple APIs to store and ship code/assets.

Hosting Functions with Webtask

To get started, let's create a Hello World example with Webtask and see how simple it is.

To do so, we need to install the Webtask CLI tool. The tool requires an account with Webtask so we can create one before installation:

npm install wt-cli -g

Login to the CLI using your sign-up email address:

wt init <Your Email>

Create an index.js file with the following Function as a Service:

module.exports = function (cb) {
    cb(null, 'Hello World');
}

Deploy and run your function:

wt create index.js

Amazing! You have a deployed your app, which now runs on the Webtask server. Go to the URL logged to your console after running the last command to see your deployed app running.

Two things you might want to take note of:

  1. We just wrote a function. A function that takes a callback and sends content as a response. You could already imagine the power.
  2. The create command deploys our app (function) and serves us a URL to interact with.

Let's employ Express to make endpoints which you might be more familiar with:

// ./index.js
var Express = require('express');
var Webtask = require('webtask-tools');
var bodyParser = require('body-parser')
var app = Express();

app.use(bodyParser.urlencoded({ extended: false }))
app.use(bodyParser.json())

// yet to be created
app.use(require('./middleware/cl').config);
require('./routes/galleries')(app);

module.exports = Webtask.fromExpress(app);

Endeavour to install the dependencies: express, webtask-tools, and body-parser.

  • express is a fast Node framework for handling HTTP requests/responses

  • body-parser parses the HTTP request body and exposes it to express req.body

  • webtask-tools simplifies integrating Express and Webtask. No need for the previous function we wrote, we can now write Express routes instead.

Let's create the routes:

// ./routes/galleries.js

module.exports = (app) => {
  app.get('/images', (req, res) => {
      res.json({msg: 'Images are coming soon'})
  });
}

Storing and Delivering Images with Cloudinary

Our next major concern is that Webtask might do a great job at hosting functions, but it may be terrible at file storage, especially media files. Therefore, we need a media storage solution to take care of this for us. Cloudinary offers media storage, but that's not where it shines.

Cloudinary is a media server that manages your media storage and delivery. What’s most impressive is Cloudinary’s ability to transform these media during upload or delivery by simply adjusting the image's delivery URL. Using Cloudinary, you can specify width, height, filters, overlays, and enable a lot of other cool features by editing an image/video transformation URL, which you can learn about here.

To use Cloudinary effectively, we recommend using the SDKs rather than interact with the APIs directly (which you are allowed to do). You can install the SDK to your current project:

npm install --save cloudinary_js

You also need to create a FREE Cloudinary account to get credentials that you will supply to the SDK in order to interact with Cloudinary.

Configuring Cloudinary with Webtask Context

Webtask allows you to dynamically adjust its behavior based on the information provided via the Webtask context. We could use the context to maintain dynamic state of our application. Things like query parameters, secrets and environmental variables are a good fit for what could live in the Webtask context. Speaking of environmental variables, let's rely on those to configure Cloudinary:

// ./middlewares/cl.js
var cloudinary = require('cloudinary');

module.exports = {
    config: (req, res, next) => {
        cloudinary.config({
            cloud_name: req.webtaskContext.secrets.CL_CLOUD_NAME,
            api_key: req.webtaskContext.secrets.CL_API_KEY,
            api_secret: req.webtaskContext.secrets.CL_SECRET
        });
        req.cloudinary = cloudinary;
        next()
    },
}

A simple Express middleware that configures an instance of cloudinary using credentials provisioned via the Webtask Context Secret. While running the functions, you are expected to supply the credentials:

wt create index --secret CL_CLOUD_NAME=<CLOURINARY_CLOUD_NAME> --secret CL_API_KEY=<CLOURINARY_API_KEY> --secret CL_SECRET=<CLOURINARY_API_SECRET> --bundle --watch

Remember to fetch the credentials from your Cloudinary dashboard.

Notice how we are passing each credential using the --secret option, as well as telling Webtask to bundle our file and watch for changes using --bundle and --watch flags, respectively. Bundling is only necessary when you have multiple local modules.

Uploading Images to Cloudinary

Now that we have everything configured, let's store our image on Cloudinary by uploading it from our computers. We can do this by adding an endpoint that its logic will process the image and send it to Cloudinary using the SDK:

// ./routes/galleries.js
var multipart = require('connect-multiparty');
var multipartMiddleware = multipart();

module.exports = (app) => {
    ...
    app.post('/images', multipartMiddleware, (req, res) => {
        console.log(req.files)
          req.cloudinary.uploader.upload(req.files.image.path, function(result) {
             res.status(200).send('Image uploaded to Cloudinary')
         });
     })
}

When the image arrives from the client, it is parsed and processed with multipart and uploaded to Cloudinary using the upload method. Remember that req.cloudinary is provided via the middleware we created earlier.

Serverless

Source Code

Notice how stayed away from creating any local host/server and built everything using the actual environment that our code and media will live. This is where the power of serverless lies.

Cloudinary exposes lots of APIs and transformation options that you can learn about in the documentation. Feel free to dig these docs and find what suits your situation.

Recent Blog Posts

Hipcamp Optimizes Images and Improves Page Load Times With Cloudinary

When creating a website that allows campers to discover great destinations, Hipcamp put a strong emphasis on featuring high-quality images that showcased the list of beautiful locations, regardless of whether users accessed the site on a desktop, tablet, or phone. Since 2015, Hipcamp has relied on Cloudinary’s image management solution to automate cropping and image optimization, enabling instant public delivery of photos, automatic tagging based on content recognition, and faster loading of webpages. In addition, Hipcamp was able to maintain the high standards it holds for the look and feel of its website.

Read more
New Image File Format: FUIF: Why Do We Need a New Image Format

In my last post, I introduced FUIF, a new, free, and universal image format I’ve created. In this post and other follow-up pieces, I will explain the why, what, and how of FUIF.

Even though JPEG is still the most widely-used image file format on the web, it has limitations, especially the subset of the format that has been implemented in browsers and that has, therefore, become the de facto standard. Because JPEG has a relatively verbose header, it cannot be used (at least not as is) for low-quality image placeholders (LQIP), for which you need a budget of a few hundred bytes. JPEG cannot encode alpha channels (transparency); it is restricted to 8 bits per channel; and its entropy coding is no longer state of the art. Also, JPEG is not fully “responsive by design.” There is no easy way to find a file’s truncation offsets and it is limited to a 1:8 downscale (the DC coefficients). If you want to use the same file for an 8K UHD display (7,680 pixels wide) and for a smart watch (320 pixels wide), 1:8 is not enough. And finally, JPEG does not work well with nonphotographic images and cannot do fully lossless compression.

Read more
 New Image File Format: FUIF:Lossy, Lossless, and Free

I've been working to create a new image format, which I'm calling FUIF, or Free Universal Image Format. That’s a rather pretentious name, I know. But I couldn’t call it the Free Lossy Image Format (FLIF) because that acronym is not available any more (see below) and FUIF can do lossless, too, so it wouldn’t be accurate either.

Read more
Optimizing Video Streaming and Delivery: Q&A with Doug Sillars

Doug Sillars, a digital nomad and a freelance mobile-performance expert, answers questions about video streaming and delivery, website optimization, and more.

Doug Sillars, a freelance mobile-performance expert and developer advocate, is a Google Developer Expert and the author of O’Reilly’s High Performance Android Apps. Given his extensive travels across the globe—from the UK to Siberia—with his wife, kids, and 11-year-old dog, Max, he has been referred to as a “digital nomad.” So far in 2018, Doug has spoken at more than 75 meetups and conferences!

Read more