Cloudinary Blog

Building a Smart AI Image Search Tool Using React - Part 1: App Structure and Container Component

Building a Smart AI Image Search Tool Using React Part 1

What if we could create a search service for images? Type in a word and get images with titles or descriptions matching our search. Better yet, what if we could create a search service for images but rather than matching just titles and image description, we can search for something in an image, regardless of the given image title or description. For example, find one with a dog in it, or those that may have a street lamp or a bus (more like an image search tool).

Few artificial intelligence examples out there show what their underlying technology looks like. Hence, in this two-part article, we will show how to build a smart search service to serve as an image search tool, not just based on the title but also on image content. This app will mainly use React, Cloudinary and Algolia Search. In this first article, we will be building the parent component of our app, which is a stateful component.

Prerequisites

You don’t need to know how to write artificial intelligence code to follow along (in as much as we are using AI technology). Basic knowledge of HTML, CSS, JavaScript and React is required. Also knowledge of technologies such as Express, Axios and Webtask offers added advantage, but is not necessary.

Installation

Node.js is used to build the backend server in this service and its package manager npm is required to install essential node modules for this project.

Download Node and npm here. You can verify if they are installed by running this on your command line:

node -v
npm -v

Versions of both tools should display on your console.

React, a front-end JavaScript library, is used for front-end user interaction. The popular create-react-app build tool will be used for quick build. Create a React project by installing create-react-app with:

npm install -g create-react-app

Now, we can build a React app from scratch quickly with:

create-react-app smart-search

This creates a simple React app. cd into smart-search and run the following command in the console to start the app:

    npm run start

React

You have successfully created a simply React app. Now we will tailor it to our app, which has two primary functions - upload and search.

This app has several dependencies and modules required to function. The dependencies are:

  • Algoliasearch: Used to initiate Algolia AI search in our app
  • axios: Required to make communications to the server from the client side
  • body-parser: Parses incoming requests in a middleware
  • classnames: Used to dynamically join classnames in JavaScript
  • Cloudinary: Cloudinary provides an awesome image storage, transformation and delivery solution for both web and mobile
  • connect-multiparty: Connect middleware for multiparty
  • Express: A Node.js framework for creating node servers
  • react-instantsearch: React search library by Algolia
  • webtask-tools: Webtask for a serverless architecture

Install these dependencies by running this on your command line:

npm install --save algoliasearch axios body-parser classnames cloudinary cloudinary-react connect-multiparty express react-instantsearch webtask-tools

Note
Using the —save flag ensures that the dependencies are saved locally in the package.json and can be found in the node_modules folder.

On the client side, in the index.html file, Cloudinary is required and included in script tags at the bottom of the HTML script just before the closing </body> tag. Include these scripts:

...
<script 
    src="//widget.cloudinary.com/global/all.js" 
    type="text/javascript"></script>
...

Also, in the head tag of the HTML script, Bulma and the react-instant-search-theme-algolia css files are imported. These are used to style our app. Bulma is a free and open-source CSS framework based on flexbox. Include these link tags in the head:

...
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/bulma/0.6.1/css/bulma.min.css">
<link rel="stylesheet" href="https://unpkg.com/react-instantsearch-theme-algolia@4.0.0/style.min.css">
...

Still, in the index.html file in the public folder, you can change the title of the page to whichever title you prefer.

Now that we have all dependencies installed, it would be nice to build out the front-end first, before building the server on the backend.

Building the Front-end

In the front-end part of our image search tool, we will be making use of the React component architecture, which includes the stateful and stateless components. In the app currently, the create-react-app tool has helped with creating a singular stateful component in App.js and its corresponding CSS file App.css. This is where the majority of the stateful configuration will occur.

For the stateless components, we will require components for the images modal and imagelist. In the src folder, create a folder called components. In the components folder, create the required components - Modal.js, ImageList.js and their corresponding CSS files - Modal.css, ImageList.js.

Now we have all the build files required, we willl develop the parent App.js component.!

Configure App.js

In the App.js file located in the src folder, clear all the code written so we can start fresh. First, import all required modules with:

import React, { Component } from 'react';
import {
  InstantSearch,
  SearchBox,
  InfiniteHits
} from 'react-instantsearch/dom';
import axios from 'axios';
import ImageList from './components/ImageList';
import Modal from './components/Modal';
import './App.css';
...

React, axios InstantSearch, SearchBox, and InfiniteHits are imported, as well as the components created (ImageList and Modal). Also, the style sheet for the parent App.js file is imported.

Next, we configure the App component. The App component is merely an extension of the React component class with defined constructor function and methods. The state of the app is defined in the constructor functions, as well as some required properties of the App class.

Create an empty class extending React component imported earlier and export it:

...
class App extends Component{
  //configuration goes in here
}
export default App;

In the App class, create a constructor function, and in here the state variables will be defined, as well as constructor properties.

...
class App extends Component {
  constructor(props){
    super(props);
    this.state = {
      modalIsActive: false,
      preview: '',
      previewPublicId: '',
      description: '',
      previewPayload: '',
      pending: false
    };
  }

}
export default App

In the state object defined in the constructor, you can see the various variables defined. Their individual states can be altered and events are based on their state at that time. The states defined are:

  • modalisActive: This represents the state of the modal used in uploading the images, it is assigned a Boolean and when it is true, the modal should be open and vice-versa.
  • preview: This is the URL of the uploaded asset available for preview.
  • previewPublicId: PublicId assigned to the uploaded image and serves as a unique identifier for each image.
  • description: A user-defined description for the image.
  • previewPayload: Once an image is uploaded, the data received from the Cloudinary servers are assigned to the preview payload.
  • pending: This is used to create some “blocking” ability in our app to prevent some other process from running when pending is true or run when pending is false.

Now we have our state props defined, let’s define some key constructor variables we will require in our app.

Include this in the constructor function:

...
this.requestURL =
      'https://wt-nwambachristian-gmail_com-0.run.webtask.io/ai-search';
this.cl = window.cloudinary;
this.uploadWidget = this.cl.createUploadWidget(
  { cloud_name: 'christekh', upload_preset: 'idcidr0h' },
  (error, [data]) => {
    console.log(data);
    this.setState({
      preview: data.secure_url,
      previewPayload: data,
      previewPublicId: data.public_id
    });
  }
);
...

First, we assign the webtask address for our server to the requestURL variable, find out about webtasks, serverless architectures and how to create a webtask here. However, you may want to hold off until the server is built at the end before shipping to webtask. The Cloudinary property is called on the window object and assigned to this.cl. Now we can use the Cloudinary upload widget to handle uploads from both filesystem sources, URL and camera!

The createUploadWidget method is called on the Cloudinary window object and assigned to the this.uploadWidget variable. In createUploadWidget, the first parameter is an object of user details obtained from the Cloudinary dashboard. Create an account on Cloudinary and navigate to the dashboard to see your cloud name and upload_preset. The second parameter is a callback function with error as its first parameter and the returned data array as its second parameter.

In the function, we first log the returned data to the console (comes in handy when debugging) and change the state of the app by assigning properties from the returned data to our state variables. We are done with our constructor, let’s get to creating class methods required. In the App class, add:

...
toggleModal() {
    this.setState({ modalIsActive: !this.state.modalIsActive });
}
handleDropZoneClick(event) {
  this.uploadWidget.open();
}
handleDescriptionChange(event) {
  const { value } = event.target;
  if (value.length <= 150) {
    this.setState({ description: value });
  }
}
...

The toggleModal() function is used to change the state of the modalisActive so whenever the function is called the modal box is closed or if it is closed the modal box is open on the function call. Sounds like a good feature for close and open modal button.

The handleDropZoneClick() function takes event as a parameter and executes the open() method on the Cloudinary uploadWidget window.

Every image should require some form of short description to give details or a title for an image. The handleDropZoneClick() function also receives an event as a parameter, the ES6 object destructuring syntax is used to assign the value property to the value variable and a conditional statement is used to very if value is less than 150 characters, then the value of the description variable in state is changed to that put in by the user. At this point, we can upload and describe the image. Now it’s time to save it to our servers. Create a saveImage() function in the App class with:

saveImage() {
  const payload = {...this.state.previewPayload, {
    description: this.state.description
  };
  this.setState({ pending: true });
  // Post to server
  axios.post(this.requestURL + '/save', payload).then(data => {
    // Set preview state with uploaded image
    this.setState({ pending: false });
    this.toggleModal();
  }).catch(e => console.log(error));
}

What are we trying to save? The payload, in doing that the Object.assign() method is used to transfer the source object(second parameter) and the value of this.state.previewPayload into the object, payload. When saving the image, other possible operations are paused so the pending value is set to true and Axios is used to send a post request to the API endpoint on the server handling image receipt. Since this is an asynchronous operation the .then() method is used to change the state of the pending to false (so it is back to normal) and the modal is closed by calling the toggleModal() function. Let’s render elements to the DOM.

In the App class, create a render function next to return JSX elements with:

...
render() {
  return (
    <div className="App">
      <InstantSearch
        appId="TBD94M93EW"
        apiKey="e5cc0ddac97812e91b8459b99b52bc30"
        indexName="ai_search"
      >
        <section className="hero is-info">
          <div className="hero-body">
            <div className="container">
              <div className="level">
                <div className="level-left">
                  <div className="level-item">
                    <div>
                      <h1 className="title">Smart Search</h1>
                      <h2 className="subtitle">
                        Smart auto tagging & search. Search results depend on
                        images' actual contents.
                      </h2>
                    </div>
                  </div>
                </div>
                <div className="level-right">
                  <div className="level-item">
                    <button
                      onClick={this.toggleModal}
                      className="button is-info is-inverted is-medium"
                    >
                      Add a new image
                    </button>
                  </div>
                </div>
              </div>
            </div>
          </div>
        </section>
      </InstantSearch>
    </div>
  );
}

Here, the InstantSearch element created by the react-instantsearch package wraps the whole elements after the parent div with the className of App and credentials are passed as props to the InstantSearch element. The rest are HTML elements used to create a structure for the app with Bulma classes for styling. Notice the button that says ‘Add a new image’? This is responsible for opening and closing the upload modal. Just after the closing section tag, create another section for the SearchBox and the InfinteHits tag for the images and the upload modal with:

...
<section className="hero is-light">
  <div className="hero-body">
    <div className="container search-wrapper">
      <SearchBox />
    </div>
  </div>
</section>
<InfiniteHits hitComponent={ImageList} />
<Modal
  isActive={this.state.modalIsActive}
  toggleModal={this.toggleModal}
  onDrop={this.onDrop}
  preview={this.state.preview}
  description={this.state.description}
  handleDescriptionChange={this.handleDescriptionChange}
  saveImage={this.saveImage}
  pending={this.state.pending}
  handleDropZoneClick={this.handleDropZoneClick}
/>
...

The top section holds the SearchBox component. Next is the InfiniteHits component which has a prop ImageList which is a component created. The infiniteHits components receive context from its parent component(InstantSearch) and return the images saved in a specified component(ImageList). The upload modal is created in the modal component and passed all the required props. Since we have this class methods as props, it will lose scope, so we have to rebind it to the constructor. In the constructor function, rebind the functions with:

this.toggleModal = this.toggleModal.bind(this);
this.saveImage = this.saveImage.bind(this);
this.handleDescriptionChange = this.handleDescriptionChange.bind(this);
this.handleDropZoneClick = this.handleDropZoneClick.bind(this);

In App.css, clear it and replace it with this styling:

.title:not(.is-spaced)+.subtitle {
  margin-top: 0;
}
.ais-InfiniteHits__root {
  column-count: 3;
  column-gap: 1em;
  width: 60%;
  margin: 50px auto;
}
.search-wrapper {
  display: flex;
  /* align-items: center; */
  justify-content: center;
}
.hero-body {
  padding: 1.5rem 1.5rem;
}

Conclusion

So far we have been able to build out the app structure for our image search tool with the main focus on the stateful component, App.js. In part two, we will show how to develop the stateful components required to render the images and the upload modal, as well as the backend server to handle requests and responses from Algolia-search, even though the server will ultimately be bundled into a Webtask.

Recent Blog Posts

Hipcamp Optimizes Images and Improves Page Load Times With Cloudinary

When creating a website that allows campers to discover great destinations, Hipcamp put a strong emphasis on featuring high-quality images that showcased the list of beautiful locations, regardless of whether users accessed the site on a desktop, tablet, or phone. Since 2015, Hipcamp has relied on Cloudinary’s image management solution to automate cropping and image optimization, enabling instant public delivery of photos, automatic tagging based on content recognition, and faster loading of webpages. In addition, Hipcamp was able to maintain the high standards it holds for the look and feel of its website.

Read more
New Image File Format: FUIF: Why Do We Need a New Image Format

In my last post, I introduced FUIF, a new, free, and universal image format I’ve created. In this post and other follow-up pieces, I will explain the why, what, and how of FUIF.

Even though JPEG is still the most widely-used image file format on the web, it has limitations, especially the subset of the format that has been implemented in browsers and that has, therefore, become the de facto standard. Because JPEG has a relatively verbose header, it cannot be used (at least not as is) for low-quality image placeholders (LQIP), for which you need a budget of a few hundred bytes. JPEG cannot encode alpha channels (transparency); it is restricted to 8 bits per channel; and its entropy coding is no longer state of the art. Also, JPEG is not fully “responsive by design.” There is no easy way to find a file’s truncation offsets and it is limited to a 1:8 downscale (the DC coefficients). If you want to use the same file for an 8K UHD display (7,680 pixels wide) and for a smart watch (320 pixels wide), 1:8 is not enough. And finally, JPEG does not work well with nonphotographic images and cannot do fully lossless compression.

Read more
 New Image File Format: FUIF:Lossy, Lossless, and Free

I've been working to create a new image format, which I'm calling FUIF, or Free Universal Image Format. That’s a rather pretentious name, I know. But I couldn’t call it the Free Lossy Image Format (FLIF) because that acronym is not available any more (see below) and FUIF can do lossless, too, so it wouldn’t be accurate either.

Read more
Optimizing Video Streaming and Delivery: Q&A with Doug Sillars

Doug Sillars, a digital nomad and a freelance mobile-performance expert, answers questions about video streaming and delivery, website optimization, and more.

Doug Sillars, a freelance mobile-performance expert and developer advocate, is a Google Developer Expert and the author of O’Reilly’s High Performance Android Apps. Given his extensive travels across the globe—from the UK to Siberia—with his wife, kids, and 11-year-old dog, Max, he has been referred to as a “digital nomad.” So far in 2018, Doug has spoken at more than 75 meetups and conferences!

Read more