In Defense of Data-Driven Design

Web designers: I'm writing a book! Sign up to be notified here.
It's about web design in the age of A/B testing, web analytics and more!

I’m speaking on this topic at Web Directions South in October this year, and given the topic has now made The New York Times, I thought I should briefly dust off this blog and weigh in on behalf of data in design.

Zeldman has been keeping the debate bubbling along as well, and it’s obviously of interest to many web designers who aren’t quite sure what to make of this whole data thing.

I’m a garden variety web designer who happens to think data-driven design is probably the future of web design, I’ve been thinking about it for the last couple of years, and hopefully I can address some the issues involved with a foot in each camp.

While some aspects of the debate have been disappointing, it’s also been very exciting in some respects to see designers thinking about the role data can play in design, because it seems to be largely a giant unknown at this point.

Who kicked this off?

In short, Doug Bowman, who’s many accomplishments in the web design world include crafting one of the first high-profile, standards-based layouts for Wired in 2002. Google hired him in 2006. Jason Fried of 37signals called this “their best acquisition to date”. In 2009, Bowman left Google citing irreconcilable differences between a ‘classically trained’ designer such as himself, and Google’s obsessive, data-driven culture. Bowman subsequently moved to Twitter.

Debate ensued, and just this Sunday The New York Times picked it up.

A polarized debate on the Internet? Surely you jest!

In the red corner we have the little guy - classicaly trained, famed web standards advocate and all-round nice guy Doug Bowman.

In the blue corner we have the corporate giant - the engineer-driven Internet behemoth that is Google, described rather uncharitably (to put it mildly!) by Joe Clark as being filled with “undersocialized, uncultured, pampered, arrogant faux-savants who have cultivated an arrested adolescence” in Google’s working culture.

The stage is set.

Now you might think if I’m stepping up to defend data-driven design, I’m going to bash Bowman for not getting it, and praise Google for their forward-thinking, data-driven culture.

I’m not.

Personally, I think the whole face-off between design and data is, frankly, stupid. It makes for a good conflict story, but it doesn’t reflect reality. It’s a poor way to frame the debate. Data and design should be friends, not enemies. In web design, people click or they don’t. They stay or they go. That’s good to know. But I’ll get to that later.

First, let’s look at Bowman and Google.

I totally understand Bowman’s position. If I were in his shoes, I would have done the same thing. And so have a number of others it appears - while there are a bunch of designers working at Google (contrary to popular belief), and were before Bowman’s time, there is something of a design brain-drain going on at Google too.

Not waving, drowning

Why are very talented designers having a hard time at Google?

First, lets be careful when we generalize about Google. Google is made up of many different teams of different backgrounds working under one big roof. What is true of one Google team or department might not be true for others, or the organisation on the whole. This is an organisation with, as of early 2009, 20,000 employees around the world. That’s a lot of people. That is certainly, to use Bowman’s analogy, a corporate ship of aircraft carrier proportions.

So it’s hard to say “Google this” or “Google that,” but we can look at the specific circumstances that Bowman apparently faced, and have surfaced elsewhere.

The Bowman Ultimatum

In Bowman’s case, he says he grew tired of debating miniscule details, and in one instance being asked to prove his case - with data - for the exact pixel width of a border. Who wouldn’t grow tired of that? It would be absolutely maddening. But is it the fault of a data-driven culture?

In my opinion, no. It’s the fault of a broken decision making culture.

The problem with testing 41 shades of blue isn’t the testing. If you can, why not? A couple of hours of an engineer’s time is a small price for the micro-improvement extrapolated across millions and millions of users. The problem is that two Google camps couldn’t make a decision and Marissa Mayer, the VP of user experience had to get involved. A VP had to be called in to adjudicate on a shade of colour. The problem isn’t the testing - testing actually is, in my opinion, quite a clever solution - the problem is that a Google VP is wasting her time on such a trivial thing because the other players couldn’t sort it out. What kind of culture is that?

Likewise Bowman’s time was being wasted fighting over utter minutiae. Again, what kind of a culture is that?

Google does this sort of thing because a micro-improvement is significant over Google’s enormous user base. But if it’s true of a micro-improvement, then consider just how much more significant a big improvement would be. Shouldn’t that be the primary concern?

Maybe it is, sometimes. But from Bowman’s experience, it certainly doesn’t sound like it is as often as it should be.

D is for Denial

Here’s the thing with design: any piece of design consists of hundreds of micro-decisions about what goes in, and most of all what is left out. What it’s not. To be asked to make your case for every micro-decision along the way would, I imagine, be a quick path to a padded cell.

But this has nothing to do with data. You can substitute ‘data’ for being asked to write up your reasoning from classic design principles (how is it informed by the golden ratio?), or having an executive committee weigh in with their thoughts. If you slow down the design process like this, where you’re scrutinizing every step you take from A to B, then you’re never going to get to B.

Relying on data in this case is just denial about not having the guts to get on with it.

Horse, meet cart

This is a classic case of putting the cart before the horse. When you’re doing mass-market, high volume sites and products, it is fair enough that you want to optimize the hell out of your products. And Google does. When you’ve got the enormous traffic necessary to work out if miniscule changes have some minor, statistically significant effect, then sure, if you can do it quickly, why wouldn’t you? But that’s optimization that should happen at the very end of the design cycle. The cart goes after the horse.

Put it the other way ‘round and you have a broken setup. It doesn’t mean horses suck. It doesn’t mean carts suck. Carts are not the enemy of horses. Optimization is not the enemy of design. Get them in the right order and you have something really useful. Get them the wrong way around and you have something broken.

Google represents (surprise!) Google

It seems to me that there is some inference that Google’s problems with data-driven design are somehow relevant to you or I. However if Google is going to be used as the representative of ‘data-driven design’ then it’s worth taking a step back and asking whether they actually do represent data-driven design.

In my view, they don’t. Here’s why:

Google’s key interest is in being Google.

Saying Google is anti-design is wrong. Google has a very stringent, tightly defined ‘Googley’ aesthetic. You can read more about it in this interview BusinessWeek did with Irene Au, Google’s user experience director. Or you can just check out their web pages.

A Google product page wont be confused for a Microsoft page or an Apple page. To consider like for like, check out the home pages of Chrome vs IE vs Safari. To see how consistent Google’s product pages are, consider Picasa or Website Optimizer or Gmail . You may not like it, but that’s their brand. That’s Google being Google.

Data-driven design isn’t slave to any aesthetic. But Google is, as is any company trying to further their brand. Google isn’t setting out to represent data-drive design, they’re setting out to pursue their brand and corporate philosophy. To the extent data helps them further that brand, that’s great for them. But we shouldn’t take it much further than that.

It may be interesting to discuss the rights and wrongs of the strategy of a multinational corporation, but let’s not pretend that such a discussion has great relevance for how your average garden variety designer like you or me goes about their work.

Why the broken culture of a 20,000 employee, multi-national company represent anything other than the problems of a 20,000 employee, multi-national company, I’m not sure.

Don’t get me wrong - we, the designers, do and will use data to drive our designs. A/B and multivariate testing will become commonplace, like Content Management Systems and web standards. But Google, as a company, doesn’t have much to do with it.

So let’s not confuse the excesses of one company with a whole notion of data-driven design.

Obey your master

Here’s the heart of the problem: Google doesn’t actually get data-driven design.

(Data-driven design, for what it’s worth, is usually testing different design treatments concurrently with different segments of web traffic, measuring which treatment performs best, and picking the winner. Rinse, wash and repeat. You may know it as A/B, multivariate or bucket testing.)

Google gets end-stage optimization, to a fault it seems. But there’s so much more to data-driven design.

Data serves its master. In Google’s case, as we’ve discussed, the master is being ‘Googley.’

However, being purely data-driven means saying to hell with the overriding philosophy; all you care about is performance. That’s not what Google’s about. That’s why Bowman et al were so stifled.

Consider their product pages, linked above. How do they compare with, say, the 37signals product pages? Or even, say, Dropbox’s ultra-minimal approach?

I’d wager that it wouldn’t be that hard to find pages that convert at a higher rate than Google’s product pages. (Conversion rate is, by the way, the holy grail of data-driven design).

To be truly data-driven is to explore those options - to do radical redesigns and see what happens. Big improvements require big changes.

And that’s the thing: being data-driven allows you to take these risks. Being data-driven is the safety net that allows you to make the big acrobatic leap and if you happen to miss the bar, you don’t land flat on your face. You can climb right back up and try again. That is a wonderful thing!

It should open the door to all sorts of creativity and design risk taking. All the kinds of daring, bold design I bet Bowman wishes he could have done.

That’s what has been, for me at least, another frustrating aspect of this debate. Google, with their enormous amount of traffic, variety of products, and copious talent, could be home to the most extraordinary, exciting design explorations and the hard-nosed, data-backed research of what actually does work. That’s true data-driven design - it’s taking real risks for real rewards. It’s not either/or. You don’t have to choose between design exploration and data. They go hand in hand.

Instead, Google is focused on being ‘Googley’ and is stuck - in some cases at least - with an apparently dysfunctional decision-making culture that gets fixated on the trivial at the expense of the big picture.

Back to Doug Bowman. Was this potential design nirvana I’ve outlined his expectation? I don’t think so - I imagine he did see the potential for Google to go from where they are now to somewhere with a different, better design sensibility. But when you spend three years having every single step you take scrutinized, and after a couple of years you look up from your feet, and find you’ve barely walked half way down the street on a very long journey to where you want to go, of course you’re going to wonder if you’re time couldn’t be better spent elsewhere. If disappointment is the gap between expectation and reality, then I can certainly understand the disappointment!

Twitter, data and you

So Bowman moved to Twitter.

When it comes to Twitter, the future is largely unwritten. That makes it infinitely more exciting to a designer leading the design effort.

Is Twitter the antithesis of data-driven design? Apparently not. In the Times piece Bowman says “Using data is fundamental to what we do [at Twitter] ,” but they take it with a grain of salt. And fair enough.

The truth is data is fundamental to what all web designers do.

From screen resolution to browser popularity, there are a bunch of a data points that web designers already take into account when creating something.

And in the not-too-distant future there will be a lot more. A/B and multivariate testing will become routine.

With that in mind, I do have one nit to pick with Doug Bowman. He finished his piece on leaving Google by saying “But I won’t miss a design philosophy that lives or dies strictly by the sword of data.”

Web design, in my view, should live or die by the sword of data.

I’ll tell you why: every bit of ‘data’ isn’t an abstract thing. It’s a person like you or me trying to do something, and succeeding or failing.

On the web, you can measure what people do. You can measure what they click, how long they stay for, if they scroll, how many pages they view, if they ‘bounce’, if they return, and so on. You can ask if they were successful or not. For perhaps the first time in history we can accurately measure all interactions with a piece of design.

If it can be measured, it can be improved. And each of those improvements represents helping someone do something a little more successfully.

That’s a measure web design should live or die by.

To be fair to Bowman, I assume he means “But I won’t miss a design philosophy that lives or dies strictly by the sword of [Google’s warped, excessive reliance on] data [in the absense of being able to get things done].” Which I can understand.

But still the point is worth making. Web design data is just counting human interactions. That’s all it is. But it matters. A lot.

The future is what we make it

It wasn’t that long ago that databases and content management systems were the domain of engineers only. Now even the simplest of blogs use them, and almost all web designers implement them.

Measuring and testing design performance wont remain the domain of engineer-heavy tech companies either (or marketers for that matter, who actually do much more of this than we do as designers, to our shame), and soon enough it will go mainstream.

I said much the same thing two years ago - I believed it then, and I believe it even more so now. More and more signs are pointing in that direction, from web designers, developers, I/As and more.

Some people have tried to hold it back, but in 2009 data-driven design is in the air.  Small companies like 37signals have finally started dipping their toes into the testing pool, and it turns out the water is fine.

There’s so much more to say on the topic, so I’m knuckling down to write more about it in the coming months to publish later in the year. It is, in my opinion, the future of web design. I’ll be talking about it at Web Directions South in Sydney later this year, and I’ll give this blog a proper burial and start blogging more about data-driven design later this year as well.

In the mean time, if you’re a designer and are intrigued about data and design, here are some resources you might find interesting:
- Follow KISSmetrics on Twitter for plenty of design, analytics and testing link goodness.
- Read the Marketing Experiments blog to see data-driven design in action.
- Dive in yourself right now and do a test with Google Website Optimizer (I didn’t say every Google team didn’t get it!). It’s easy and they provide a boatload of resources to get you started. Ironic? Maybe.

Until next time. Thanks for reading!

Luke Stevens

comment? | permalink
rss feed for this site

Comments

what a great read!
really well thought out and constructed piece.
very much looking forward to yr talk at web directions.

in regards to this bit:
“The problem is that two Google camps couldn’t make a decision and Marissa Mayer, the VP of user experience had to get involved.”

that has really stuck a cord with me too, and in a blatant plug, i’m also talking at web directions - about (at least in part) earning trust from managers/clients to avoid situations like that.

yr article has helped to shape some ideas. thanks :)

Pete

- Pete Ottery on 12 May 2009

Excellent, glad you found it useful Pete, and glad you’re speaking at WDS too! :)

I actually used to work for a little media org a few years ago & looked after their news site, and followed your editorial design work at Fairfax (and now News Ltd) pretty closely. I still follow the debates about the future of online journalism etc too. Cool stuff!

Cheers :)

- Luke Stevens on 12 May 2009

Good points. I’m just interested in your views on Google being ‘Googley’. Does it ultimately means Google can’t be changed/evolved into a more user design than data design?

I would argue since design is subjective, so is data. It’s the quality of data, not the quantity that makes the design testing worthy.

Doug’s statement regarding ‘live and die by the data sword’ may have to do more with that ‘Googley’ data and not all data-drive design. Doug, like all good designers, would’ve have that covered.

- Benson on 12 May 2009

Hi Benson,

Being ‘Googley’ defines Google’s corporate culture (for better or worse), but when it comes to design they’ve produced a list of ‘What makes a design “Googley”?’ on the official Google blog.

Some of them are complete fluff ("Dare to innovate"), but it gives you a bit of an idea of what that’s about.

In terms of quality v quantity of data and subjectivity, my view is that to make sensible decisions you need high quality data (ie your measurements have to be accurate), and you need enough of it to be statistically significant, otherwise you might be looking at random chance. So quantity does improve quality, in that sense.

If you lack those things, then you’re just making decisions based on statistical noise, which obviously isn’t a good idea!

In terms of subjectivity, there’s always the matter of interpretation, for instance if a user stays a short time on a page, is that because they completed their task quickly, or got frustrated and left? To answer that question you need more information, and that’s where usability studies come in to fill in the blanks, so you get your interpretation right.

But that gets back to quality and quantity of data - if you don’t do a usability study with enough users, then you’re not going to have a representative sample, your findings wont be necessarily significant, and you can muddy the issue further!

So it’s always best to stick to the basics: Are the measurements accurate? Are we measuring the right thing? Do we have a representative sample? Are our findings statistically significant?
With those in mind, you can’t go too far wrong.

Getting designers up to speed with things like the importance of statistical significance will be important in the long run too, as data plays a greater role in design decisions.

- Luke Stevens on 13 May 2009

Data, of this sort, *informs* design. It should not *dictate* it. Just like Gallop polls do not run government. Ideally.

Design is dictatorial by nature. You put your problem in the hands of the one person (or small team) best qualified to make a decision (based on data and… OMG! experience! and… OMG! hunches!)

We are human beings, not replicants.

- Boris Anthony on 17 May 2009

Hi Boris, thanks for the comment.

I agree we’re humans, not replicants, but that’s kinda my point - we’re not all knowing either.

When we design something, especially for the web, we can’t really predict how one piece of design will perform vs another (assuming both are of a reasonable standard) when we’ve got all kinds of people visiting from all over the world. We simply can’t read their minds. Our hunches and intuition can and will be wrong.

Therefore, the only way to know what will perform best is to a/b or multivariate test it with real live users until some statistical significant outcome is reached.  Otherwise you’re just flipping a coin. You might guess right sometimes, but you’re still just guessing.

In that sense, data doesn’t just inform a design, the data from experiments (ie, which design the most users have the most success with) determines the final ‘winning’ design.

Intuition will take us pretty far, certainly, but the numbers have the final say, and should dictate the final design.

- Luke Stevens (author) on 21 May 2009

About

Hi! I'm Luke Stevens and this is/was where I write about design on the web. This blog has been dead for about two years while I've been busy doing the freelance web design thing. Later this year it will be reborn as something new. Until then, enjoy the latest post and feel free to comment with your thoughts!

I'm also speaking on data-driven design at Web Directions South in October 09. If you can make it to Sydney, come along!

RSS Feed

Grab the feed if you want to find out when this site is reborn anew.