标签: kozio

A few years ago, intrigued by their technology, I visited a little company named Kozio in Colorado. They produce software that diagnoses faults in hardware. It's aimed at a couple of markets: the engineer who is bringing up a new design, and production test. I found the notion interesting as in a previous life I discovered that a huge number of embedded systems that use DRAM and large SRAM arrays are poorly-designed. They pass the limited tests their creators use, but hit that memory with the perfect storm of bits flipping and those highly-capacitive arrays start acting like, well, capacitors. Occasional errors result that are very hard to diagnose. Recently Kozio expanded and reconfigured their product line. The flagship offering is now VTOS (Verification and Test Operating System), a name that is a bit confusing. An OS? Just what we need, another OS! Turns out, VTOS is a sort of minimal OS that boots with far few resource than the more bloated Linux or Windows. And, of course, it comes with a lot less functionality, the point being that one needs just enough code running to support executing a wide range of hardware tests. What kind of hardware tests? Well, all sorts of memory tests, of course, which will isolate not only failures, but design weaknesses. Perhaps there's not enough margin in the timing—the tests hammer at the memory controller. Today, of course, we use these elaborate DDR-style SDRAMs which have access times that are just about impossible to predict or measure, but VTOS's tools will give detailed performance data. VTOS is U-Boot compatible, so a production system can have it ready and lurking, bootable when necessary. The company tells me that the typical high-end development effort that doesn't use VTOS requires 1.5 to two person years to build a verification package. They also claim their customers can get VTOS up and running in only 30 minutes. I can't check those numbers but scale them by even an order of magnitude and the delta is huge. Further they say there are $17 billion in returns of supposedly defective consumer electronics every year, and most of those show no problems. Presumably some large fraction then are due to flaky designs whose hardware problems weren't fleshed out because of inadequate testing code. Then there's the increasing problem of counterfeit parts; VTOS on the production line can identify boards with parts of suspect origin that fail the aggressive tests. One doesn't port VTOS to a new target system; instead it's essentially pre-ported (versions come for different processor families). VTOS Builder creates a custom version matching the target's memory and peripheral configurations. It knows about hundreds of devices, but, from the GoToMeeting demo, it looks rather easy accommodate a new peripheral by filling in form fields. One customer spent 10 weeks troubleshooting crashes. Three days after installing VTOS the error was identified: the CPU's PLL occasionally lost sync. In another case a customer had had problems for two years with memory errors. An hour of exploration with VTOS determined that the ECC had never been turned on! The cost depends on which packages one buys, but figure on a bit under $10k to twice that. That's a ton cheaper than 1.5 to 2 development years for an in-house solution. All in all I think this is a nice solution to a too-often-neglected problem.  

Does your hardware work?   One of the most fascinating aspects of embedded development is that we're working in two very distinct camps: the firmware and the hardware, which, until the invention of the embedded systems, were two non-intersecting Venn circles.   One of the most frustrating aspects of embedded development is that we're working in those two very distinct camps. The hardware guys build a board, "verify" it, and toss it over the wall into the firmware group with a Good Hardware Seal of Approval stamped on the thing. Then the software group tries to test their code only to find a series of problems with the PCB.   To double the Embedded Excedrin headache, the problems manifest themselves as software issues. Troubleshooting becomes a nightmare because people are looking in all of the wrong places. Or maybe the board is intermittent, working at times but failing only when specific and hard-to-reproduce events occur.   In the best of cases, how do you even know if your hardware works? Crank out some diagnostics and, sure, with enough time one can insure that everything functions correctly. Yet some kinds of failures " say, with complex SDRAM logic " require quite complex test code, as failures occur in perverse ways, ways that few of us non-test engineers really understand.   I visited a Colorado company last year which aims to break this bottleneck. Kozio offers a variety of products that perform excruciatingly-detailed tests on custom hardware.   Their kDiagnsotics is a program they customize (often in just a few days) to your hardware design. That's possible and cost-effective since the company has a huge library of canned tests they've written for many types of peripherals. Given a board with not much more than a running CPU and a bit of memory, kDiagnostics can exercise the rest of the board and pinpoint design or production errors.   With the use of scripts the hardware engineer can loop through tests, or even subsets of tests, to create known and repeating stimuli to guide scopes and logic analyzers through the troubleshooting process. Said loops can run for an extended period of time to look for glitches, perhaps while cycling the board in an environmental chamber or around ESD discharges.   The upside: the hardware people deliver a known good board to the firmware engineers. The latter can use the tests to prove that a problem really is in the code, not on the board (all hope is not lost; we can still blame the compiler).   Other offerings include a royalty-free POST that can be embedded into your product, a utility that dumps peripheral register contents so you can get insight into how to set these up, and more.   But with hardware getting more complex I think validating it as thoroughly as we test the code is a good idea. Check them out at www.kozio.com .      

Does your hardware work?   One of the most fascinating aspects of embedded development is that we're working in two very distinct camps: the firmware and the hardware, which, until the invention of the embedded systems, were two non-intersecting Venn circles.   One of the most frustrating aspects of embedded development is that we're working in those two very distinct camps. The hardware guys build a board, "verify" it, and toss it over the wall into the firmware group with a Good Hardware Seal of Approval stamped on the thing. Then the software group tries to test their code only to find a series of problems with the PCB.   To double the Embedded Excedrin headache, the problems manifest themselves as software issues. Troubleshooting becomes a nightmare because people are looking in all of the wrong places. Or maybe the board is intermittent, working at times but failing only when specific and hard-to-reproduce events occur.   In the best of cases, how do you even know if your hardware works? Crank out some diagnostics and, sure, with enough time one can insure that everything functions correctly. Yet some kinds of failures " say, with complex SDRAM logic " require quite complex test code, as failures occur in perverse ways, ways that few of us non-test engineers really understand.   I visited a Colorado company last year which aims to break this bottleneck. Kozio offers a variety of products that perform excruciatingly-detailed tests on custom hardware.   Their kDiagnsotics is a program they customize (often in just a few days) to your hardware design. That's possible and cost-effective since the company has a huge library of canned tests they've written for many types of peripherals. Given a board with not much more than a running CPU and a bit of memory, kDiagnostics can exercise the rest of the board and pinpoint design or production errors.   With the use of scripts the hardware engineer can loop through tests, or even subsets of tests, to create known and repeating stimuli to guide scopes and logic analyzers through the troubleshooting process. Said loops can run for an extended period of time to look for glitches, perhaps while cycling the board in an environmental chamber or around ESD discharges.   The upside: the hardware people deliver a known good board to the firmware engineers. The latter can use the tests to prove that a problem really is in the code, not on the board (all hope is not lost; we can still blame the compiler).   Other offerings include a royalty-free POST that can be embedded into your product, a utility that dumps peripheral register contents so you can get insight into how to set these up, and more.   But with hardware getting more complex I think validating it as thoroughly as we test the code is a good idea. Check them out at www.kozio.com .