标签: toyota

As a technology journalist, I get the equivalent of a postgraduate education reading good, detailed papers and reports, though some are mind-numbingly difficult. Knowing my interest in such papers, several software developers and tool vendors have independently referred me to "Quantitative Evaluation of Static Analysis Tools," by Shin’ichi Shirashi, Veena Mohan and Hemalatha Marimuthu, at the Toyota InfoTechnology Center in Mountain View, Ca.   In this paper, the authors describe the task of selecting the optimum code analysis tool for doing run time code analysis of software for use in Toyota's vehicles. Starting with tools from about 170 vendors of proprietary tools as well as a range of free and open source versions, they narrowed their choices down to six, those from Coverity, GrammaTech, PRQA, MathWorks, Monoidics and Klocworks to make their selections they used a complex methodology to first make their assessment and from that also derive a set of coding guidelines to help the company's development teams avoid defects proactively.   Readers of the report may disagree with the types of tests and metrics the Toyota team used to make its choices. Some developers I have talked to complain that despite the data-driven quantitative approach, at the beginning of their efforts the team made use of the qualitative and subjective judgements of a few experts they trusted. However given the number of such tools they had to evaluate, that was a choice they were almost forced to make. Even after limiting their choices in that way, there were many alternatives to evaluate and test, even after they narrowed their choices further by excluding noncommercial tools that provided no technical support, those that did not support safety critical applications, and including only those that supported the C language.   The paper describes how the Toyota team first created a set of test suites incorporating a wide variety of software defects that might cause problems in safety critical applications. They then tested the tools they had selected against that suite. Finally, because of the importance of driving buggy software out of the automobile environment, they went one step further: they used the data already collected to create several new metrics to further hone down the performance of the tools. For information on the various tests they used, they depended on several reports from the U.S. National Institute of Standards and Technology (NIST), supplemented with information from various tool vendors and users.   The choices they made and the process they came up with made it clear how many things can go wrong in a software design, especially a safety critical one, and how hard it is to pin things down. Drawing on every piece of literature they could find, they identified eight defect types that were important, including: static and dynamic memory, resource management, pointer-related, and concurrency defects as well as the use of inappropriate and dead code. From that they identified 39 defect sub-types, and from those they created 841 variations that they used in their tests. The methodology is about as comprehensive as any I have ever seen.     In addition to the defect code bases they created for their test suites, the researchers created an identical set without defects. The first code base was for evaluating true positives from the analysis results, while the second set without defects was used for evaluating false positives. The researchers also tried to keep the test suites as simple as possible while keeping in mind the nature of the automotive software environment, especially in relation to the use of global variables and stack memory. (The test suite benchmarks are available on Github .)   On average, the six static analysis tool finalists were correct in their detection of flaws in the code about 34 to 65 percent of the time, with Grammatech's Code Sonar ranking first. But a more important measure is how they ranked on various kinds of other tests. For example, on static memory defects, Gramatech's Code Sonar and the MathWorks Polyspace ranked highest. On pointer-related defects, PRQA ranked highest. On detecting concurrency defects, Code Sonar came out on top, while on numerical defects, Mathwork's tool did best. The report goes into much more detail, of course. Dry and matter-of-fact as it is, though, their report is worth going through slowly line by line because of its value to developers concerned about writing reliable, safe code.   The paper is a gold mine of information that I will refer to over and over again, not only for insight into which tools are best, but for the nature of the defects being hunted and the impact they have on overall software reliability. It is also valuable if you are interested in learning how to develop a methodology for assessing the complex trade-offs that must be made. Perhaps the most valuable lesson to be learned from this paper is the clear inherent message that if you are serious about detecting software defects, more than one tool is necessary, no matter how much the design team manager or the company accountant complains about the expense.

Oh, well, never mind. . . . if you haven't seen the news last February, the in-depth investigation of the alleged "sudden acceleration" of some Toyota vehicles has concluded that neither electronics hardware nor software was at fault. To quote from The Wall Street Journal story (February 8), "The report, released by the Transportation Department to settle persistent questions over the Toyota recalls, concluded that the auto maker had identified the only two causes of the incidents: defective gas pedals and interfering floor mats." It also appears that user confusion was also in the mix, all fueled by lots of misreporting (some intentional, some not).   This news, of course, will likely get hardly any attention, unlike the "sudden acceleration" stories did.   But wait a minute? Didn't most of the pundits and instant experts conclude that it was an electronics problems or software bug? How is this turn-about possible?   What happened is the customary script: the media and lawyers piled in and piled on Toyota, in hopes of boosting their ratings in the first case, and gaining settlements in the second. In such situations, there is no time to wait for a thorough, competent, conscientious investigation. First and loudest to market with the hype is the priority; we can worry about the facts later.   The reality is this: Every engineer with any experience knows that debugging and troubleshooting are the toughest challenges in most designs. The evidence is sketchy or intermittent, the context is often muddled, and lack of consistency or repeatability of the problem often magnifies the challenge. It's even more difficult when you are dealing with just a few hundred reported problems among millions of cars.   Very often, your first debug hunch is either wrong, or what you think is the cause is really just the consequence of the real, deeper cause. Or maybe there is more the one problem happening at the same time? In a complex system, you would expect that single-point problems would be the exception, not the rule.   Look at a simple example: a light bulb goes dark in your house. The problem itself is obvious (the bulb is dark), but the source may not be. It's probably that the bulb has burned out, but it could be a faulty line switch, a problem with the wiring in the walls, a fuse/breaker has opened for unknown reasons, or maybe AC mains have failed completely. Sure, it's easy enough to check these possibilities out, but since it is probably the bulb, that's what you change or check first.   But complex electronics systems are not so easily segmented and testable. Subtle problems and interactions are normal, and often what you think you see happening is not what is really there.   Problem is, it's so easy to dump on, and beat up, a target company and its engineers, and much more fun that actual doing real work. Being a sideline expert and critic beats the hard work of system design and debug, for sure.   But when the hard evaluation work is done, and a credible report issued which contradicts the breathless initial coverage, where do the engineers go to get their reputation and stature restored? Can we at least get a basic "we're sorry?"   That will happen "in my dreams", as they say: these folks have other news and crises to hype, and engineers to blame for not producing perfection along with all those amazing technologies and products.  

Oh, well, never mind. . . . if you haven't seen the news last February, the in-depth investigation of the alleged "sudden acceleration" of some Toyota vehicles has concluded that neither electronics hardware nor software was at fault. To quote from The Wall Street Journal story (February 8), "The report, released by the Transportation Department to settle persistent questions over the Toyota recalls, concluded that the auto maker had identified the only two causes of the incidents: defective gas pedals and interfering floor mats." It also appears that user confusion was also in the mix, all fueled by lots of misreporting (some intentional, some not).   This news, of course, will likely get hardly any attention, unlike the "sudden acceleration" stories did.   But wait a minute? Didn't most of the pundits and instant experts conclude that it was an electronics problems or software bug? How is this turn-about possible?   What happened is the customary script: the media and lawyers piled in and piled on Toyota, in hopes of boosting their ratings in the first case, and gaining settlements in the second. In such situations, there is no time to wait for a thorough, competent, conscientious investigation. First and loudest to market with the hype is the priority; we can worry about the facts later.   The reality is this: Every engineer with any experience knows that debugging and troubleshooting are the toughest challenges in most designs. The evidence is sketchy or intermittent, the context is often muddled, and lack of consistency or repeatability of the problem often magnifies the challenge. It's even more difficult when you are dealing with just a few hundred reported problems among millions of cars.   Very often, your first debug hunch is either wrong, or what you think is the cause is really just the consequence of the real, deeper cause. Or maybe there is more the one problem happening at the same time? In a complex system, you would expect that single-point problems would be the exception, not the rule.   Look at a simple example: a light bulb goes dark in your house. The problem itself is obvious (the bulb is dark), but the source may not be. It's probably that the bulb has burned out, but it could be a faulty line switch, a problem with the wiring in the walls, a fuse/breaker has opened for unknown reasons, or maybe AC mains have failed completely. Sure, it's easy enough to check these possibilities out, but since it is probably the bulb, that's what you change or check first.   But complex electronics systems are not so easily segmented and testable. Subtle problems and interactions are normal, and often what you think you see happening is not what is really there.   Problem is, it's so easy to dump on, and beat up, a target company and its engineers, and much more fun that actual doing real work. Being a sideline expert and critic beats the hard work of system design and debug, for sure.   But when the hard evaluation work is done, and a credible report issued which contradicts the breathless initial coverage, where do the engineers go to get their reputation and stature restored? Can we at least get a basic "we're sorry?"   That will happen "in my dreams", as they say: these folks have other news and crises to hype, and engineers to blame for not producing perfection along with all those amazing technologies and products.