标签: usability testing

When the Consumer Electronics Show comes around every January, I bear a complex set of emotional reactions, the same ones that I have when I go into a local Big Box retailer to buy a personal electronics gadget. First there is anticipation of all the new features and capabilities that developers have come up with. Then there is resignation because over the years I have learned there will always be some flaw or oversight for which I will have to find a workaround. Third comes resentment at whatever has caused this to occur: penny-pinching, lack of planning, or ignorance – sometimes willful—of the consumers' needs and wants. The flaws I see in various consumer electronics devices are many, some unique to each type and some common to all. To deal with them I have a kit of tools that I keep at hand to work around the problems. The kit includes: Velcro, Super Glue, Elmer's water soluble glue, a variety of hooks and rings, various kinds of tape, soldering iron and solder, wire cutter and splicer, and safety pins and paper clips of various sizes. And for really tough cases, duct tape. For brevity's sake, here I will focus on just simple MP3 players, of which there must be a dozen different brands and within each brand, a dozen variations. But from what I have learned elsewhere, such as in a recent Jack Ganssle column on Android devices , the same sorts of problems occur no matter how far up the consumer electronics food chain you go in complexity and rampant " featureitis ". Invariably, MP3 players suffer from the same "more is better" malady that infects desktop computers and many consumer electronics devices. MP3 players have been stuffed with as large a screen as possible in a one or two inch square device, a USB connector, audio output, a microscopically sized microphone, and an FM Radio. On MP3 players, I appreciate the large screens that are now standard for viewing the titles of music and audio books. But most such devices also include a capability for viewing your favorite music videos. Images of a postage stamp-sized singer in which there is no detail? No, thanks. I would rather they had used the space to include additional gigabytes of solid state flash for storing even more books and songs. And the weaselly little pin-hole sized microphone capability? Be serious. The FM radio functionality is a nice extra, or would be if it really worked reliably. But reception is always iffy in such a small device. It is also the most prone to failure and, unfortunately, takes along much of the other functionality of the device with it. On one MP3 player, after the FM radio went out, I could still play my stored selections, but, I could not change the volume. On another, the FM radio worked OK and the volume worked – once – after which it would turn itself off and I would have to turn it back on and find my place in my selection again. The designers of the devices, and the companies that take them to market, must know that these problems could occur, because some of them include a reset button someplace on the device – usually a microscopically small pin hole into which you must stick a sharp point to restart the device. They know that they are pushing reliable operation to the edge, but still they try to cram in as much counter-productive functionality as they can. This is where one of the tools in my kit is useful: a safety pin that I have attached to my keychain for easy access. With it, I have an MP3 player that works most of the time. But regularly – and arbitrarily – it shuts down, until I bring it back to normal functioning by poking at the reset with my safety pin. The reliability and usability issues also extend to the accessories. My big bugaboos right now are the headsets and earplugs, both the ones that come with the device and the more expensive ones you have to go out and buy when the supplied ones stop functioning. For example, ear buds. Maybe my ear canals are abnormally small, but none of the buds fit into my ears and stay there. One of my solutions is the Elmer's water soluble glue. I rub a small amount around the edge of my ear entrance and the bud stays there nice and snug, Otherwise I have to spend more money to get an expensive set with a number of ear bud sizes and fits. Despite that, on some ear buds, when I push it in to fit snugly, the result is an inoperative ear speaker, full of static. Then there is the length of the ear phone cord. On average, the ones supplied with the MP3 player are 3 to 4 feet (convert to m) in length. I don't know about you, but my ears are not somewhere between my knees. Some of the more thoughtful companies supply the cord with an attached clip and direct you to fold up the unnecessary cord and attach it to this clip. Thanks for the thought, but it is an awkward solution and gets in my way. For a while my solution was to use my kit and cut out the extra length and splice what is left to get a shorter cord. But I stopped doing that. Too much trouble for something that has a lifetime of a few months. The cords on earbuds are extraordinarily thin and flimsy and have a tendency to break at the points where it attaches to the ear bud and to the earphone jack you plug into the MP3 player. I do what I can to extend the lifetime, using my Superglue to reinforce those weak points at regular intervals. That does not do anything about the cumbersome length of the cord. My first solution was to simply wrap it around my neck until the excess was gone. Now, I have kludged a solution that reduces effective length by half. It still hangs around my neck, earplugs on one side and earphone jack on the other. But I have used a large paper clip to create a sliding attachment much like on a trombone, with the cord from the ear buds inserted on one side and portion of the cord with the electrical plug to the MP3 player on the other, and tape in between to keep them separate. I now have a setup that is much less cumbersome, but flexible enough to fit the player itself into the pocket of my pants, coat ,or shirt, on the left or on the right, or attached to a neck chain—all without pulling the bud end from my ears. But the problems don't end there. Now suppose I have the MP3 player attached to a neck chain to keep it handy and easily viewable? Have you looked at the size of the holes for attachment, or where they are positioned?. For me, they're in the wrong place—at the "top" of the device. So if I have it on a neck chain and I reach down to see what is on the display, the image is upside down. To solve that I use one of the safety pins in my kit and attach it at the bottom of the device with Velcro so I can see the book and song titles right side up, Then I use Superglue to connect it permanently, attachment clamp down, so the safety pin ring hole at one end is available to attach to my neck chain. ( Are you following all of this? Perhaps I should have diagrammed some of it ). These are some of the more mundane and trivial—and least complicated—problems I have with consumer electronics devices. I have dealt with some of the more serious drawbacks from this user's point of view elsewhere in a column written several years ago titled "IPv6, RFID, GPS and finding lost devices. " Nothing was done then nor since about those problems that I can find. Nor do I expect anything to be done. And many years of experience with these devices tell me nothing will be done to deal with what I have written about here as well, sadly. Why? There are three possible reasons. First, perhaps no one but me sees them as problems, but I find that hard to believe. Second, as noted in a previous column on " It's usability testing, stupid! ", I don't think companies do the right kind of research to understand the users of their products. I know they do not understand me . Finally, it may not be in the interest of the device maker or accessory supplier to do so. If they fix it and make it more reliable, it will not malfunction and then there will be no reason to buy a replacement. Welcome to 21st Century consumer electronics!!  

What went wrong on the first release of Apple iPhone 4? Apparently none of the traditional white box, black box or usability testing procedures caught it. Is a more free form kind of ad hoc usability testing necessary? iPhone 4's wireless antenna problems have been a public relations fiasco for Apple Inc. No doubt you've followed all the press on this, but if you haven't witnessed the disaster for yourself, there are video demos on YouTube, including one, from Consumer Reports, demonstrating that when the iPhone 4 is held in a certain way, signal strength drops from four bars to one. The issue underscores the importance of usability testing on any product targeted at the consumer market. Steve Jobs has said that his company subjects its iPhones, iPads and iPods to many variations of the traditional white box and black box testing procedures. Yet these tests apparently did not identify the antenna problem. In my investigation of the iPhone 4 antenna problem I could not find any definitive statement that usability testing was done, except one anonymous rumor that the company does indeed conduct such tests on their consumer products, but using their own trained engineers. If that is the case and it had been done properly, it would have increased the odds that the problem would be identified. But Apple iPhone 4 developers did not spot it. So what went wrong? Black-box testing, the most common form of testing, exercises the functionality of a device, a software program or a system, as opposed to its internal structures or workings. Test cases are built around specifications and requirements, i.e., what the application is supposed to do. It uses external descriptions of the software, the hardware and the application to derive test cases. These tests can be functional or non-functional, though usually functional. The test object's internal structure is irrelevant. White box testing, on the other hand, allows designers to test a device's internal hardware and software structures rather than its functionality. An internal perspective of the system, as well as programming and engineering knowledge, are required in order to exercise paths through the code and to subject the hardware to all of its various operating modes. Neither of these testing modalities subjects the device to operation in a real-world environment in the hands of a consumer. For that you need usability testing with real consumers, not engineers. If Apple had done such a test, it's likely they would have spotted the fact that, as multiple videos on YouTube attest, the way a user grips the mobile phone determines how much its antenna reception is degraded. Many companies seem to confuse market research with true usability testing. Usability testing is not gathering opinions from typical users about a device's features or whether or not it is hard or easy to use. Rather, usability testing as it is usually carried out involves watching people—usually under controlled conditions—trying to use something for its intended purpose. For example, when testing instructions for assembling a toy, or using a cellphone, the test subjects are given the instructions to read, and then asked to proceed – that is, to build the toy or operate the cellphone. They are then observed in action, using the device correctly or incorrectly. The trick in such usability testing is in your definition of typical users. Apple apparently thought trained engineers would be the best ones to conduct usability tests. For black and white box testing, maybe. But to get valid usability test results who do you want? The user experienced in using your product or a similar one? The user who has taken the time to read the user manual? The user who has not read the manual? Or the mobile user from Hell, who has no experience at all, the one who is apt to break the product when, through ignorance or frustration, tries to make it work in a way he or she thinks it should? And the use of a controlled environment further skews results by forcing users to work in conditions that do not necessarily reflect the real world. The iPhone 4 is not the only example where all of these more traditional forms of testing fall short. My life is full of consumer devices—from remote TV channel changers to mobile phones – that do not seem to have been subjected to any serious usability testing. Using a new cellphone is like joining a secret society with a secret handshake. The handshake may be somewhat similar to the one in another secret society, with a common set of movements and procedures that require passwords, but the sequence, rhythm and the timing may be completely different. The secret handshake for an Apple iPhone, for example, is very different from the ones used on any number of Android sort-of look-alikes that are becoming available. In such an ad hoc world of consumer product design, where there is no common set of procedures and protocols that everyone agrees to, maybe a more ad hoc, free form method of usability testing is necessary as well. What is needed is usability testing under completely uncontrolled conditions, in the field, watching users in their own environments, trying out devices they think they know how to use but do not. In the case of mobile phones, the way a consumer thinks it should be used may not be the way that particular device should be used. I have done a lot of reading on all the different types of white box, black box and usability testing and I can't find anything in the formal technical literature that resembles this kind of freeform testing. But that does not mean it does not exist and is not being used. I know for a fact that it does. The best practitioners of the kind of usability testing I'm suggesting were the field engineers I hung out with in the early 1990s while in Japan visiting some of the major consumer electronics companies. A major portion of their job was to go into the consumer electronics stores that lined the streets in the downtown areas of many of the major cities. We watched how consumers of different types used the devices they were considering buying: how much time and how many steps did it take to complete basic tasks? How many mistakes did they make? Were the mistakes fatal; that is, did they break the devices or just temporarily put them out of action? Was it possible for them to easily back out of a situation and try something else? What was their emotional response: frustration? anger? confidence? stress? When the engineers talked to the users they were observing in these totally uncontrolled situations, they did not ask their opinion of what and how to improve the device. Nor did they work from any prepared list of questions. Rather, their queries emerged out of their observations: why did you take this or that action? Were you aware of this or that function? After a day of all of this ad hoc, informal user testing and observation, the engineers then got together and discussed what they had seen users do "in the wild" and what the users told them about what they did and why they had done it that way. But the engineers did not then do what I expected them to do, which would have been to try to quantify their results for a report to be submitted to management, who would then try to derive some general principles. Instead they kept their focus on the specifics of what the users had to tell them and redesigned their products to make them easier to use. Not only that, they tried to make it make it impossible for the consumer devices to be used in any way except the correct way. If I had closed my eyes, I would have mistaken them for field anthropologists and social psychologists observing some remote tribe in the jungle. Nowadays, with only a few exceptions (most of them Japanese), few if any of the consumer products and mobile devices—Apple's iPhone included—seem to have been subjected to the kind of freeform usability testing I saw practiced by those engineers. My closets are full of devices that can attest to that. The reasons probably have to do with cost, time to market, and competitive pressures. No doubt this ad hoc free form and individualistic approach to device usability testing is expensive and time consuming. But as Apple's iPhone 4 experience illustrates, what you "save" early on in time, effort and money could come back to haunt you in lost reputation, lost customers and losses in company valuation on the stock market.