Data mining, usability and stories to share.
I've been chatting with fabulous IA Paula Thornton about data mining: how usability people are generally fairly ignorant about these techniques, and how they can lead to powerful insights, yet are often misused.
Paula has a lot of insights to share, and since stories are one of the best ways to share them, here goes one:
Paula Thornton: "I was at a conference. SGI (Silicon Graphics) was there showing one of their 'mapping' tools. They showed how they could evaluate the data and find 'hot spots'
that might need attention, by representing the data in red. In this particular case they had drawn a correlation between 'low spending
customers' and 'high contact volume'. The conclusion they were trying to make was that you might want to get rid of customer (dis-incent) that are 'costing' you more than they're worth (they're calling you a lot but not
spending much).
Happens that I had an experience that very week that would have put me in that classification and it would have been 'wrong'. I was establishing new phone service (we had moved and had been guaranteed our phone would be ready...having made the request months in advance). We got to the new
address; no dial tone. Here's the scenario:
1. I call customer service. I'm told that I must have either had a locked fence or a dog in the yard. I explain the yard is dirt, there is no fence
and we have no dog. No reasonable answer is given.
It's been 7 years, I can't remember the details of all the others except that on call 4 the rep hangs up on me. It isn't until call 5 that I'm told
that there aren't enough connections in the neighborhood for them to give me service and they have no idea when I'll even get service (couldn't they have told me that 2 months ago?). 5 calls in one day...but the data wouldn't have
the reasons and show how it took me 5 calls to get a reasonable answer.
Action on such 'mining' would have been grossly flawed."
Another story she told me was about reusing usability research. I've wondered for a while how you should conduct and present your research so it can be reused effectively.
Paula: "Per your 'experiences' regarding usability research, you've hit on what I see as a 'fundamental flaw' with most usability practitioners. Most have not had sufficient business training. Many 'go through the motions' and simply 'report' without an ability to 'evaluate'.
Literally, I always (as you did) rework from the raw data to draw new conclusions. Often I can't even do valid analysis from the data. [A large airline company] was quite good about spending 'lots' of money on usability tests. I sat through several hours of videos to gather my own data. Here's the challenges:
The tests were 'wholly contrived'. The tester had a goal and drove the participant to that goal. The best data is generally to be found in the 'exceptions', the 'unexpected'. When the environments are too contrived and controlled this data doesn't exhibit itself.
Also, the 'goal' of the experiment will generally not be aligned to what I consider a significant starting point (if you haven't started here, you're not going anywhere meaningful): the value propositions of the stakeholders. The tests shouldn't be about usability (although that will be a component), but whether or not the activity is even remotely focused on something that will return significant value to the stakeholder (based on a list of individual priorities). Testing the 'economics' of the interaction, to me, is far more valuable than testing for usability (and from the stakeholder's perspective, is a far more valid use of their time...in spite of being paid...which can increase the 'quality' of testing participants).
Bottom line: most usability tests I have seen the results of were a huge waste of investment from several angles."
Now control your urge to jump to the defense of usability testing. I think this a valid critique. Think about how the usability testing ethic compares to the data mining ethic, and how that's similar to how the advertiser's ethic compares to the direct marketer's ethic.
Paula recommended reading this and other books by Gordon Linoff and Michael Berry for good explanations on the topic: the books are on their way. Thanks Paula!