Cloudinary Blog

Jason Grigsby on Responsive Images: Where It All Started

An interview with Jason Grigsby about Responsive Images
 
Portland-based web developer Jason Grigsby co-founded Cloud Four. He’s a prolific writer and speaks about front-end development as thoughtfully as anyone in the business. He’s been writing about responsive images since there were responsive images to write about, and recently, he and I had a chance to sit down and talk about the past, present, and future of images on the web.
Here, in Part 1 (of a three-part series), we discuss how the industry slowly came to understand the surprisingly hard problems around “flexible media,” what’s been done to solve them, and the pain points and open questions that remain.
 
Eric Portis:  One of the interesting points I’ve heard you make — which is a nice entry point into all of this — is that images on the web have always been complicated. Can you give me a sense of what problems developers were facing in the early days of the web?  What did we have to worry about then, that we don’t have to worry about now?
 
Jason Grigsby: So basically, you’re starting off by asking me, okay grandpa—
 
EP: [Laughs]

JG: —Grandpa, tell me what it was like, I’m too young to understand. I see how you are! Well, in the earliest days, the web started without any images at all, and when we first started incorporating them, the problems were tough.  There were differences in color space between Macs and PCs, and we had to make sure that images were compressed well enough to download on connections that are much slower than we have now. Lynda Weinman, of Lynda.com, wrote books — I think it was five books — that were each several hundred pages long, all on images, and they were best sellers. In addition there were a number of specialized tools — I remember a piece of software called Debabelizer — that allowed us to handle images and GIF compression much better than Photoshop did.  It took awhile for Photoshop to add the ability to export for the web. 
 
Linda Wineman’s books
Linda Wineman’s Designing Web Graphics, Coloring Web Graphics, Preparing Web Graphics, and Deconstructing Web Graphics - 1,333 pages all about images!
 
So yes, I think there is a knee jerk reaction that a lot of people have when they start looking at responsive images. They think that the responsive images syntax is too complex and that it should be simple like images have always been. I reject that. Images are complex by their very nature. We’ve gotten good at automating some things and we’ll get better at automating others in the future. But to think that they’re not complex is to deceive yourself. They’ve always been complex.
 
Even after all of the time that I’ve spent on them, I still don’t feel like I understand the full complexity of images. And I think about them far too much.
 
EP: Let’s fast forward to 2010 or so, when developers started asking hard questions about images in responsive layouts.  You were right there at the beginning of those conversations, saying that, hey, something here is broken. What were the first problems that you and other developers ran up against? What were some of the early ideas floating around about how to solve those problems?
 
JG: Back in 2010, I wrote an article with the unfortunate title, “CSS Media Query for Mobile is Fool’s Gold.” I wrote it because about a year before – in 2009 – my company Cloud Four was brought in to consult on a site that was having a ton of performance problems. The site used media queries to do what we now call “responsive design,” before Ethan Marcotte coined that phrase.
 
We dug into what was going on, and found that the media queries were hiding assets. On this site, it was a Google map, which wasn’t necessary on small screens. But we discovered that the map was still loading – along with 47 other files, tiles and a bunch of other stuff – which made the page slow to a crawl.  
 
After Ethan wrote “Responsive Web Design”, I started seeing similar problems everywhere. There were pages where images were hidden but still downloading in the background. Similarly, images that were far too large were being shrunk down for small screens. As a result, you’d have a page that looked great, but it was 5MB in size.  
 
The analogy I make for this problem is that it’s akin to having a dinner party, but not having time to clean, so you shove everything into the closet. It may look like your house is clean, but you’ve really just hidden the problem and you’re ok if no one opens that door.  But on a website, even if it looks good, you can’t hide the performance impact of large assets.
 
About a year later my co-founder Lyza Gardner and I started writing “Head First Mobile Web,” and we wanted to have a chapter on “mobile first” responsive design. At the time people thought, ok, if we start with a minimal set of small assets and only load larger assets if they’re needed, we can solve the responsive image problem.  So I started looking at the techniques people that were using to do this, because I assumed that enough time had gone by — that by then somebody had to have a good solution. But I found that every single technique had problems.  They were all insufficient, even on high-profile sites. The Filament Group built the Boston Globe site with assumptions based on browser behavior that they saw in 2010, when they started working on the project.  But by 2011, when the site launched, that behavior had changed and their attempts at responsive images were no longer working. Experiences like that showed us that this needed to be fixed at a lower level. Things needed to change from a standards perspective. 
 
Boston Globe responsive website

The Filament Group’s landmark 2011 Boston Globe redesign proved that responsive design could be practical (and performant!) at scale. It also highlighted the “Responsive Images” problem – Filament Group developer Mat Marquis wrote prominently about the image problems that the Globe team encountered (and, later, founded/chaired the W3C Responsive Images Community Group, which solved them).

EP:  One thing I’ve found curious is that the push for responsive images really came from the bottom up — from developers like Cloud Four and The Filament Group — rather than from the top down. And initially, there was a lot of push back from browser vendors. Do you have any insight into why this problem was more visible to, more impactful for, developers? 
 
JG: Well, first off, not everybody was pushing back. Bruce Lawson [who works at Oprea] gets credit for coming up with the <picture> syntax. Other people were resistant, because they may not have seen the responsive design tidal wave that was coming.  So you have people who aren’t really thinking about mobile yet, or who were used to sites that were using device detection to route visitors to different, device-specific pages – and they couldn’t see how important this problem was, they didn’t see it as an issue.
 
Plus there are some tradeoffs that happen in responsive design that we still haven’t quite resolved.  By not supplying the height and width of images, we’re affecting browser’s ability to lay out pages quickly, and there’s a jumpy effect that happens when images come in. Browser makers were more attuned to those issues. And they had just finished doing a ton of performance optimization work to try to load images as quickly as possible, and, at first, it seemed like there was no way to do responsive images without breaking those optimizations.
 
So it wasn’t like browser vendors were saying, “we don’t want you in the clubhouse.” They really just had a different worldview. But the people who were building sites had come to realize that responsive design was inevitable and really important; that device diversity had gotten to the point where there was no point in making distinctions between a mobile site or a tablet site because some mobile devices were large and some tablets were small. But browser vendors weren’t there yet.
 
EP: Speaking for myself, and coming at it from the developer side, I know that I didn’t understand anything about low-level browser performance, and had no concept of how layout really worked at a deep level.  I was demanding tools and not understanding their potential impact – I couldn’t see the bigger picture. And it took a while to have that conversation – for the people who make web sites and the people who make web browsers to get into each other’s heads.
 
JG: In May 2012 I wrote about the preparser, two years after I started looking at responsive image issues. It took me that long to get to the point where I fundamentally understood that what we were talking about was a conflict between the browser, which wants to start downloading assets as soon as possible – before it starts laying out the page – and developers working on responsive designs, who want to delay and only start downloading images after layout. Because if we wait until after layout, we can download an image that fits. That fundamental conflict is what took so long to solve, from 2010 when we started identifying these problems, until 2014 when solutions started rolling out in browsers.
 
EP: And full support for the responsive images spec only rolled out in the last couple of browsers – mobile and desktop Safari – this past February. That felt amazing, like — ok, finally, it took four or five years but — we did it! This is complete.
 
JG: You know I had to write a bio for an article that just came out in A List Apart, and I ended it with: “He never wants to talk about responsive images again—”
 
EP: [Laughs] Sorry.
 
JG:“—he never wants to talk about responsive images again, but he has ideas for two more blog posts.”
 
[More laughs]
 
JG: Which is true. I really do have ideas for two more.
 
Thus concludes part 1 of my three-part interview with Jason Grigsby. Stay tuned for part 2, where Jason and I dive into today’s solutions and the prickly problem of Responsive Image Breakpoints.
 

Recent Blog Posts

Build the Back-End For Your Own Instagram-style App with Cloudinary

Github Repo

Managing media files (processing, storage and manipulation) is one of the biggest challenges we encounter as practical developers. These challenges include:

A great service called Cloudinary can help us overcome many of these challenges. Together with Cloudinary, let's work on solutions to these challenges and hopefully have a simpler mental model towards media management.

Read more

Build A Miniflix in 10 Minutes

By Prosper Otemuyiwa
Build A Miniflix in 10 Minutes

Developers are constantly faced with challenges of building complex products every single day. And there are constraints on the time needed to build out the features of these products.

Engineering and Product managers want to beat deadlines for projects daily. CEOs want to roll out new products as fast as possible. Entrepreneurs need their MVPs like yesterday. With this in mind, what should developers do?

Read more

Your Web Image is Unnecessarily Bloated

By Christian Nwamba
Your Web Image is Unnecessarily Bloated

As a developer, it seems inefficient to serve a 2000kb JPEG image when we could compress images to optimize the performance without degrading the visual quality.

We are not new to this kind of responsibility. But our productivity will end up being questioned if we do not deliver fast. In order to do so, the community has devised several patterns to help improve productivity. Let's review few of these patterns based on their categories:

Read more

Google For Nigeria: We saw it all…

By Christian Nwamba
Google For Nigeria: We saw it all…

Note from Cloudinary: Christian Nwamba, a frequent Cloudinary contributor, recently attended, and was a main speaker, at the Google Developer Group (GDG) Conference in Lagos, Nigeria. Christian led a session teaching more than 500 developers how to “Build Offline Apps for the Next Billion Users.” The stack he used included JS (Vue), Firebase, Service Workers and Cloudinary. Below is his account of the conference and his talk.

Read more
Viral Images: Securing Images and Video uploads to your systems

When was the last time you got paid $40,000 for a few days of work? That is what happened last year to Russian independent security researcher Andrey Leonov, who discovered that if you upload a specially constructed image file to Facebook, you can make Facebook's internal servers, nested deep within their firewalls, run arbitrary commands to expose sensitive internal files in a way that could easily lead to a data breach.

Read more