Testing search

My sense is that information architects generally have less influence over the core experience of search than they do over other aspects of a website’s interface, and I think that’s a bit of a shame since it’s such a critical resource for so many users. Sometimes a website’s search engine will be a user’s primary experience of the site’s architecture, but here we are on the sidelines watching as the implementation teams wield control over the search experience.

In our work on Vanguard’s search engine, we developed a collection of methods for testing the quality of the results it was bringing back. These became critical tools underlying optimization efforts and functional design strategy, and ingrained the IA’s deeply in the quality of search. I think that these testing methods can bring a lot of value to the user experience community, so I’ve proposed a pair of articles for Boxes and Arrows explaining them.

Both tests are premised on the idea that the search engine should return the best matches available for a user’s expressed interest at the top of the results page. That sounds really simple, but there are a lot of complicating factors that weigh into it:

  • How closely does any of the available content actually match the user’s real interest?
  • Has the user done a good enough job describing what he or she wants?
  • How does the search engine parse the user’s input?
  • What standards does the search engine use to determine what constitutes a best match?

As a result of these kinks in the system, search is inevitably much more hit-or-miss than we’d like it to be. So from the user experience perspective, the first questions we should ask are how often does the engine return the very best matches at the top of the results list, and how good are the matches it does return?

Well it turns out we can measure those things, and it’s not especially hard to do. You can put a number to it, then set objectives for how much you’ll improve over time. (The critical resource here is search logs — which is why I’m psyched that Lou Rosenfeld has a book coming out on just that.) In the process, the strategy shakes out too. You’ll find exactly which searches are underperforming, which among those are the most important to people, and the best ways to fix the system. That puts IA’s right back in the center of the search experience.

Those evaluation methods are what I’ll cover in the articles. I’d greatly appreciate any feedback submitted on the B&A site, and incorporate any input into the text when writing them.


No comments yet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: