原创 Bringing back watchdogs (Part 2)

[Continued from Bringing back watchdogs (Part 1)]

ARM supplies a bewildering array of debugging IP, including a macrocell that sends trace data out just two pins. The so-called Program Flow Trace (PFT) is described at ARM's infocenter.⁶ It can be set up in a zillion different configurations, but will at the very least emit compressed information that lets one track changes in program flow. For instance, the execution of a branch instruction pushes address data out. So does Branch with Link, which is ARM's way of calling a subroutine. Similar instructions in Thumb mode do the same.

There's a lot to like about the Shankar and Lysecky's approach. First, modern processors are barricaded behind caches, pipelines, and speculative execution logic. Even if you're using a CPU that has address pins (as opposed to a self-contained microcontroller or IP on an FPGA), those address lines simply don't match anything that is going on in the processor's gray cells.

Second, pins are valuable. By squeezing addresses through a couple of debug pins few of these valuable resources are consumed. And logic is cheaper than pins; adding circuitry to decompress the trace stream and make sense of it may cost very little. In the paper, the authors state that the overhead for their particular approach was only 3% of the silicon area of an ARM1156T.

Third, most watchdogs use ad hoc approaches that just are not reliable indicators of system operation. This new approach lets the designer decide just how fine-grained the monitoring should be. Check an occasional call... or watch every instance of every branch and call.

Fourth, there's no overhead in the system's firmware. Zilch. Unlike traditional watchdog approaches, which require one to seed the code with instructions to periodically kick the dog to keep it from resetting the system, address profiling is transparent to the software.

There are a few caveats, of course. The logic needed to make sense of the address information is substantial, and is probably impractical unless implemented in an FPGA. Building such a monitor would be a lot of work. But it needs to be done once, and then can ever after be used in a succession of products. I can see this as packaged IP sold by a third party, or as an open source project.

Remember, though, there's no guarantee that your ARM CPU will have the trace logic needed. Every vendor is free to include the IP or not.

Going deeper
I haven't thought out all of the implications or possible ways to actually use this idea in a real embedded system, but here are a few ideas.

One problem with the proposed approach is that every recompile will shift all of the system's addresses, requiring the developer to re-profile the code to determine where the branches are. An alternative is to monitor just function calls, but use a level of indirection. Build a table of jump instructions that lives at a fixed address in memory; each entry corresponds to a particular function. Make calls through that table. Then monitor those jumps. One would have to be careful that the compiler didn't optimize the jumps away.

This does mean the software changes a bit, of course. But more than a few of us use these jump tables anyway. They can aid debugging and sometimes simplify on-the-go firmware updating.

ARM's program flow trace also very intriguingly sends address information out when a DSB, DMB, or ISB instruction is executed. DSB (Data Synchronization Barrier) holds up program flow until all of the instructions before it complete; DMB (Data Memory Barrier) ensures that memory accesses before it compete before one after it starts, and ISB (Instruction Synchronization Barrier) flushes the instruction pipeline, ensuring that the instructions after the ISB are fetched from memory or the cache.

Why are these interesting instructions? ARM mandates that at the very least a DMB be issued before locking or unlocking a mutex, and before incrementing or decrementing a semaphore. One could monitor these, just like watching branches, to ensure that the code is properly going through all of the activities mediated by the RTOS. In fact, monitoring only these actions may be simpler than the addresses, because program flow can change quite a lot even from a simple change (requiring re-profiling), but resource management tends to change much less frequently.

As far as the DSB and ISB instructions, I'm not quite sure how they could offer useful watchdog information, but something makes me think they could offer some interesting options.

Parting thoughts
If you're building an electronic toothbrush, watchdogs are probably not terribly important. But an automated reset helps boost consumer confidence in our products' quality. Everyone hates the "remove batteries and wait 30 seconds" dance.

Many vendors are putting more thought into their WDT designs; some are doing a pretty good job. But we have a long way to go, and the wise developer will apply sound engineering practices to this often-neglected part of the system.

The article I cited shows that some ingenious approaches are being used. Consider adding a bit of hardware support if robustness is an important requirement.

Endnotes:
1. Ganssle, Jack. "Born to Fail" Embedded.com, December 12, 2002. Available at www.eetimes.com/4024495.
2. Texas Instruments. "MSP430x5xx/ MSP430x6xx Family User's Guide" Literature Number: SLAU208H, June 2008–revised December 2010. Available at http://focus.ti.com/lit/ug/slau208h/slau208h.pdf
3. Freescale Semiconductor. "MCF51QM128 Reference Manual: Supports the MCF51QM32, MCF51QM64, and MCF51QM128," Document Number: MCF51QM128RM, Rev. 0, November 11, 2010. http://cache.freescale.com/files/32bit/doc/ref_manual/MCF51QM128RM.pdf
4. Shankar, Karthik and Roman Lysecky. "Control Focused Soft Error Detection for Embedded Applications," IEEE Embedded Systems Letters, December 2010, Volume 2 Number 4.
5. The Nexus 5001 Forum—A Program of the IEEE–ISTO. See http://www.nexus5001.org.
6. "CoreSight Program Flow Trace Architecture Specification, v1.0." ARM, 2008. Available at http://infocenter.arm.com/help/topic/com.arm.doc.ihi0035a/IHI0035A_coresight_pft_architecture_spec.pdf.