Image Image member login  |  privacy policy  |  contact us
      Home  |  FAQ  |  Awards  |  Reports  |  Forum  |  Jobs  |  Consultants  |  Join  |  Order
  Forum Questions
Have a question?
Send it to
          Jane Farber.

ASP Forum

Benchmarking Knowledgebases

"We'd like to compare the success rate of our online knowledgebase against other companies, but we've found that there isn't a standard industry definition of 'success rate.' How can we benchmark our results?"

—Gretchen from Great Neck                           


David Kay's suggestions are right on the mark (see feedback below). Self-benchmarking may be the best and the only option available today. It is almost like the game of Golf, where you are trying to improve your game and continuously. I would like to add a couple of things for you to consider.

1) A typical search entry in a Knowledgebase often fetches too many items to read and it can be overwhelming for someone who needs a quick diagnosis and resolution. Our own research focuses on the total time to resolution from start of a user query and comparing it to live support with an expert. Ideally, the end user should want to go to knowledgebase first before calling live support. We are finding that a good old database of use cases and "frequently encountered problems" related to those use-cases often help diagnose the issue rapidly rather than free-form query into text files. Ninety percent of the time delays in resolution seems to involve diagnosis.

2) Often, the cost of support and help-desk and maintenance fees pale in comparison to the cost of downtime for client enterprises. If the user is frustrated they will abandon. This abandonment rate is a good measure to benchmark. This requires that you track the exact path taken by a user using bread-crumbs to determine if they abandoned the Knwledgebase. In addition, if the same person followed up with a call for live support, then it is telling.

The ultimate goal is to reduce the stress levels of the user who is under the gun to fix a problem. They are generally on intense time-deadlines to get things back on track.

—Nat Kannan
    Jeeva Portals
    Managed Product Support Portals

You're right, there is no industry standard model for benchmarking knowledgebase success, and in fact, there's a great deal of pernicious nonsense propagated on the topic.. For example, you sometimes see people assume:

– You can extrapolate the percentage of people who answered "yes" to a "Did this knowledgebase article help you" question, even though a tiny fraction of people provide feedback
– An opened document means the session was successful
– A successful self-service experience is the same thing as an avoided call. n fact, depending on the business, only a small fraction of successful self-service interactions are deflected calls. This doesn't mean they're not important...they are! They satisfy demand that would otherwise have gone unmet, increasing the value of your product, loyalty, and—over time—they drive the top line.

So although companies do provide their data to a few benchmark sources, you're right that they're calculating their percentages differently and it's hard to compare.

I'd offer two suggestions that might get you part of the way to your business goal:

1. Self-benchmark.
Even if you're not sure how others are doing, it can't hurt to continually raise your bar on your own performance. Pick a measurement technique—I like calling a few hundred customers who use your web portal and asking them, "What was your objective in coming to the site," "Were you successful in accomplishing that objective," (if yes) "Would you have opened a case if you had not been," (if no) "Did you open a case?" and finally "What words of advice do you have for us in making the site better?"

It's a short conversation, but you get great data: what was their interaction goal, were they successful, what's the escalation/deflection rate, and other feedback. If you don't have time to do this yourself, well, summer's coming and you can hire an intern, but doing this for a half-hour a day is a great way to hear directly from your customers... there's no substitute for the wisdom you'll get! Of course, make sure you're coordinating with other people doing surveys and the other powers that be, but don't let them put you off by offering to add a question to a larger relationship survey. This is a very tactical, very focused activity.

2. Rate comparable use cases with your competitors.
Assuming you have access to your competitors' customer support sites, you can come up with a list of the top ten or twenty customer interaction goals, try them on your site, try them on your competitors' sites, and see how each one scores. Sure, this is just a sample, and sure, you may know too much about how your own site works, but it's still a source of great insight.

Finally, I can't leave this post without recommending benchmarking yourself against the ASP Online Ten Best Judging Criteria—and, of course, entering the 2010 competition. I recommend this to all of my customers. From what I've seen, the Top Ten winners are doing an increasingly good job at scoring high across the board, and if you're not sure you are, you may have just answered your own question.

—David Kay
    DB Kay & Associates

[If you have any other advice on this question, please send an email to membership director Jane Farber at, and we'll post your feedback.]