标签: troubleshooting

Debugging and troubleshooting are major parts of the engineer's world. I've always regarded them as the most difficult of engineering challenges because it takes a combination of factors: technical expertise, experience and practice, lessons learned from mistakes, ability to think in both big picture and detail modes, willingness to question everything, ability to think outdid the box, a dose of luck, and more.   Troubleshooting a once-working product is a humbling, constant reminder that even "simple" can be tricky. Obviously, some troubleshooting challenges are more difficult than others (I define "troubleshooting" as finding out what's wrong with something that has worked well and been in the field for a while, while "debugging" is trying to get a prototype or pilot run unit working to spec). Intermittent problems are often the worst, not only because they are hard to track down, but you are often not sure that you really found the source—or the problem just go away for a while on its own?   I was reminded of this lesson when my flashlight started acting strange. It's a quality Maglite unit from Mag Instrument Inc., not some cheap throwaway, and it became frustratingly intermittent. The incandescent bulb would flicker, go off and then back on when jostled or even when just resting on the counter. I figured this would be an easy problem to resolve—after all, it's just a flashlight, and has no software, no active components, nothing except a bulb, alternate-action sealed switch, and two batteries, that's it. Sponsor video, mouseover for sound     The nearly invisible grit on the threads of the end cap prevented solid electrical contact between the internal and external threads, and also prevented the end cap from screwing down and seating properly onto the body, resulting in strange intermittent operation of this high-quality flashlight.   I checked for corrosion of the contacts (there was none), and even rigged up a separate circuit so I could check the bulb for an internal intermittent by tapping the glass enclosure (it was solid). I tested the sealed on/off switch with a continuity checker and that was fine. My summary: every component was fine, but the assembled product was not. Adding to the confusion was that at one point, I thought the unit was off, but it wasn't.   Later, it came back on by itself and drained the batteries, so I had an added problem of now-dead batteries, which added to the confusion while I was collected my data and evidence. Still, I was determined not be defeated by a mere flashlight.   Long story short: I looked at the end-cap which screws onto the body, Figure, and which is the current path from the negative side of the battery to the body and then up to the switch. It seemed to screw down solidly but I had to check everything. I cleaned the machined external threads of the end cap and the internal threads of the body, sprinkled some powdered graphite onto them to both enhance conductivity and also "grease" them for smoother torque-down, and felt a nice, solid "thunk" as the bottom cap seated on the body when I threaded it on. (The instructor at the Loctite mini-course I took on threaded fasteners emphasized that dirt on threads not only result in a poor final assembly, it fools you into thinking the nut is on tight because it increases the apparent torque.)   That did it. The flashlight’s performance was now steady, no intermittent behavior, and the bulb was also brighter. Problem solved—and the diagnosis was obvious, in retrospect: the screw-on end cap was not making good contact. That's really no surprise, since so many intermittents in electronic products are due to mechanical issues. In fact, the first rules of troubleshooting are to check the power source and all connections, and I should have known better!   While there are a lot of case studies on debugging and troubleshooting, there isn't much technical literature about these as a discipline, and it's easy to understand why: the subject is tough to get your hands and mind around. The only resource I know is the excellent book Debugging: The 9 Indispensable Rules for Finding Even the Most Elusive Software and Hardware Problems by David Agans. I strongly recommend you take a look at it, even if you are an experienced engineer. (Don't be put off because it is published by the American Management Association—it's a down-in-the-trenches book, not some high-level abstract treatise).   Have you ever had a simple product with a vexing problem that took a significant amount of time and mental energy to troubleshoot? Did you feel relieved, foolish, or some other emotion when you finally found and resolved the problem—that is, assuming you did?

Here's my oscilloscope: a 1966 15Mhz Tektronix Type 422.   Vintage 1966 15Mhz Tektronix Type 422 When I bought my oscilloscope for twenty dollars, it was dead. Someone had tried to stick ten pounds of fuse into a one-pound fuse holder and subsequently broke the fuse holder. I replaced the fuse holder with a non-standard fuse holder and a fuse of the recommended type and rating, thus returning the instrument to a functioning capacity.   Vintage scope with new fuse holder. Now, the Tektronix Type 422 Oscilloscope is a fine instrument of quality manufacture, but I would like to try some of those new features they've come up with since the moon landing—features like 200MHz bandwidth, 1GS/s, 4 analogue channels, 16 digital channels, automated measurements, and FFT analysis that are provided by the Tektronix MSO2024B digital oscilloscope that is the prize of this contest, so I'm going to tell you this story. One day, I received a message from manufacturing that they had an instrument that was dead and that they required the assistance of my superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience. So, I put on my anti-static smock and my anti-static shoe straps and grabbed my trusty DVM with the very pointy probes and headed down to manufacturing to render said assistance. Upon arriving in manufacturing and being shown the instrument, I asked the manufacturing engineer what was wrong with the device. He replied, "If I knew that, I would not require the assistance of your superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience." So, I asked, "Well what's the problem?" He said, "It's dead." I said, "Well, what did you try?" He said, "I replaced all the boards with known good boards and the power supply with a known good power supply and it is still dead." He spilled the beans. I knew he would crack under my relentless questioning. As I sat down by the instrument, I bumped the table, which caused the screen to blink. So I banged on the table, causing the screen to blink again. I continued to bang on the table making the screen blink much to the amusement of the technicians working behind me. I did not care because I had a nibble and I was going to play it out before I got lost in the labyrinth of the innards of the device.   Handy troubleshooting technique. After several minutes of judicious banging, the screen came up and stayed on. I heard from behind me someone guffaw, "He's got it working," followed by laughter from the peanut gallery. I then took my pen and poked at the cables in the device until I found one that caused the screen to malfunction. I shut down the instrument and removed the cable. Then, using my superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience, and using my trusty DVM with the very pointy probes, I determined that the cable was indeed defective; knowledge of which I imparted on the manufacturing engineer, thus proving the old adage that it is not enough to think outside the box. Sometimes you have to bang on the side. So please kindly consider my submission, though it is as badly stitched together as Frankenstein's monster (if Dr. Frankenstein had used duct tape) and send me that fancy new Tektronix MSO2024B digital oscilloscope that is the prize of this contest. I'm sure it will help in repairing those instruments I cannot fix by banging on the table. This article was submitted by P.C. (name withheld by request) as part of Frankenstein's Fix, a design contest hosted by EE Times (US).  

When talking about high-speed circuits, it is imperative to take proper care of signal terminations. Yes, I'm talking about transmission lines. We all know that, even at the printed-circuit level, improper termination of a high-speed signal can lead to all kinds of nasty realities in RF, analogue signalling, or digital circuitry. In RF circuitry, it's often called the voltage standing wave ratio, or (V)SWR, which represents the amount of reflected energy seen by the output amplifier. In an ideal system, regardless of type, the SWR is 1:1. That is, the power sourced by the driver is the same as the power sunk by the receiver. Again, regardless of system type, if the (V)SWR is too bad, the reflected power not only can cause distortions, but also can burn out the driver or even the receiver. What am I talking about? Well, this video explains what I did in a recent project. I have a digital signal with a programmable frequency being output on a pin. At low frequencies, you can see that the output looks pretty good, but as the frequency increases, you can see quite a lot of oddball behaviour. What causes reflections? I won't go into great detail, because many books have been written on this subject. For now, I'll say it's a result of transmission line theory. It's caused by an impedance mismatch between different parts of the circuit. An antenna is actually matching the impedance of free space to the impedance of your transmitter's output driver or receiver's input stage. I think (and correct me if you know better, but please do it in nice, easy terms that I can understand) that this has to do with the propagation velocity of a signal through the medium. Energy travels through the medium at a certain speed but is not absorbed at the same rate as it is being received. This causes a reflection (echo) to go back through the medium. Reflections will occur at any point where there is an impedance mismatch. When a wire, cable, or PCB trace becomes a transmission line is a function of frequency and distance. A rule of thumb is that, when the length of the transmission line is more than about 2x the wavelength of the highest frequency being carried, you have a transmission line. That is, if my frequency is 1MHz (wavelength = 300m), then I have a transmission line when my cable becomes somewhere in the neighbourhood of 600m long. This is because there are now multiple values on my line at any given time, and the reflection of one value can interfere with the value at any given point on the line, distorting the signal. Each time the signal echoes, some energy is lost (heat), but as the signal bounces back and forth between the various discontinuities, the signals can stack up because of superposition. Now, let's talk specifically about digital circuits. Given the explanation on the previous page, you'd think you never need to worry about a 1MHz digital signal, right? Well, there's another rule of thumb specifically for digital circuits: You should treat a wire or interconnect as a transmission line if the propagation delay is more than 1/6 of the rise time of the digital signal. A square wave is made of many frequencies. Recall your college days (way back when) when your instructor talked about the Fourier Series? This is a perfect example. A square wave is the prime frequency plus the odd harmonics. The propagation delay of a signal through a medium is also dependent on the signal's frequency. Different components of our square wave reach the end of the line at different times. If I have a square wave with a slow edge (say, 5µs), then a three-inch trace is no big deal. The distance traveled is so small that all the frequencies get there at almost the same time. But if the edge is, say, 1ns, maybe we'll have ringing on the signal. "Yes, we already know all this, and your explanation isn't all that good," you say. "Why are you bugging us with this ancient news?" When I was working on those F-4s, we occasionally had a broken wire. Look closely at the picture below. This bird has cables that go from the near the tip of the nose all the way to the tip of the tail. That would be more than 40 feet (convert to m) if it were a straight line, but there's nothing straight about it. Believe me.   A real F-4 aircraft. There are two ways to find the location of a fault in one of those wires. The first is long, hard, and laborious. It involves finding an access panel where the cable runs. You open the panel, break the wire (this is usually a splice point), and use an ohm meter ('re' for unit of measurement) to shoot the cable both ways. Now you know if the problem is in the front half or the back half. You keep going like this until you isolate the bad section and replace it. Then you go about repairing all the breaks you made and sealing the jet back up. Now, let's talk about the second way to find the fault—one of the greatest troubleshooting tools of all time. It's a time domain reflectometer (TDR). It makes use of the principle of reflections to help find faults in long cables. It is really an oscilloscope combined with a pulse generator.   A TDR is really an oscilloscope with a signal generator. (Source: ePanorama.net) The pulse is sent down the cable, and you use the oscilloscope to measure the signals on the wire. A break in the wire looks like a pulse up. A short looks like a pulse down, and a connector usually looks like a glitch. Places where the cable is going bad (e.g., due to corrosion from water infiltration) look, well, crummy. TDRs are essentially oscilloscopes, except the time-per-division knob is continuously variable and is calibrated in tenths of feet. We could just start dialling the knob until we saw the fault, look at the knob, and know it's about 37.5 feet down the cable (likely in the engine compartment). That's a lot easier than the other way of doing it. Some people think TDRs can be used only with coax cables, but I'll tell you that just isn't true. Of course, that's how they work best, but they work well on twisted pairs, as well. In fact, they work rather well if you have any kind of multi-conductor cable where you can probe both lines together. In fact, the TDR is so sensitive that you could just lay a long wire on the ground outside the plane and still get a pretty decent reading on the location of the fault—within a few feet. (I've done this once, and the manual for the TDR referenced it.) TDRs come in two varieties—electronic (like I've been discussing) and optical (for optical fibres). If you ever need to deplay or troublwshoot some long cables, look into one of these devices. Unfortunately, they're kind of expensive (imagine that) to have one of your own, especially given how rarely you need them. The cheapest electrical TDR I've seen recently costs more than $400. However, if you'd like to do a fun project at home, take a look at this . This guy is basically building his own and telling us all about it. As for my opinion about the best electrical troubleshooting tool of all time, take a look at this video . What's your favourite troubleshooting tool? Tom Burke Senior Systems Engineer  

Here's a picture of my oscilloscope: a 1966 15Mhz Tektronix Type 422.   Vintage 1966 15Mhz Tektronix Type 422 When I bought my oscilloscope for twenty dollars, it was dead. Someone had tried to stick ten pounds of fuse into a one-pound fuse holder and subsequently broke the fuse holder. I replaced the fuse holder with a non-standard fuse holder and a fuse of the recommended type and rating, thus returning the instrument to a functioning capacity.   Vintage scope with new fuse holder. Now, the Tektronix Type 422 Oscilloscope is a fine instrument of quality manufacture, but I would like to try some of those new features they've come up with since the moon landing—features like 200MHz bandwidth, 1GS/s, 4 analogue channels, 16 digital channels, automated measurements, and FFT analysis that are provided by the Tektronix MSO2024B digital oscilloscope that is the prize of this contest, so I'm going to tell you this story. One day, I received a message from manufacturing that they had an instrument that was dead and that they required the assistance of my superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience. So, I put on my anti-static smock and my anti-static shoe straps and grabbed my trusty DVM with the very pointy probes and headed down to manufacturing to render said assistance. Upon arriving in manufacturing and being shown the instrument, I asked the manufacturing engineer what was wrong with the device. He replied, "If I knew that, I would not require the assistance of your superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience." So, I asked, "Well what's the problem?" He said, "It's dead." I said, "Well, what did you try?" He said, "I replaced all the boards with known good boards and the power supply with a known good power supply and it is still dead." He spilled the beans. I knew he would crack under my relentless questioning. As I sat down by the instrument, I bumped the table, which caused the screen to blink. So I banged on the table, causing the screen to blink again. I continued to bang on the table making the screen blink much to the amusement of the technicians working behind me. I did not care because I had a nibble and I was going to play it out before I got lost in the labyrinth of the innards of the device.   Handy troubleshooting technique. After several minutes of judicious banging, the screen came up and stayed on. I heard from behind me someone guffaw, "He's got it working," followed by laughter from the peanut gallery. I then took my pen and poked at the cables in the device until I found one that caused the screen to malfunction. I shut down the instrument and removed the cable. Then, using my superior technical skills and extensive engineering knowledge acquired through advanced education and years of experience, and using my trusty DVM with the very pointy probes, I determined that the cable was indeed defective; knowledge of which I imparted on the manufacturing engineer, thus proving the old adage that it is not enough to think outside the box. Sometimes you have to bang on the side. So please kindly consider my submission, though it is as badly stitched together as Frankenstein's monster (if Dr. Frankenstein had used duct tape) and send me that fancy new Tektronix MSO2024B digital oscilloscope that is the prize of this contest. I'm sure it will help in repairing those instruments I cannot fix by banging on the table. This article was submitted by P.C. (name withheld by request) as part of Frankenstein's Fix, a design contest hosted by EE Times (US).  

Recently, I had a modest troubleshooting assignment. I was involved in the mechanical repair of an interface box which had been removed from service, with four identical connectors hand-labeled as A, B, C and D, corresponding to internal channels. I was told that only channels A and B were live, and that C and D were disconnected internally. After I did the repair, I went on-site and reconnected the cables—which were conveniently labeled as a, b, c, and d—to their respective connectors. "All done," or so I thought. Well, not quite. The signals on cable "a" appeared at channel "B", while the signal of cable "b" didn't appear anywhere. It took a few minutes to realize the problem: those thoughtful labels A, B, C, and D on the box were wrong; I swapped things around, re-labeled the box's connectors correctly, and my work was done. But the incident got me thinking . This was a relatively simple case of mislabeling and incorrect documentation. But in many cases, the situation is far more complex, and the requisite documentation more vital. Yet often, we are given documentation which is incomplete or, perhaps worse, inaccurate. Then we spend our time trying to figure out which part of the documentation is correct and which is not, at the same time we are trying to debug the system itself. This documentation can be a system-level block diagram, a circuit schematic, or even a code listing. The other scenario is to have minimal or no documentation at all. In that case, you have fewer misconceptions about the correctness of what you've got, so you can't be misled. That's the good news. But you do need to unravel the design with a blank sheet of paper in front of you, so to speak, and that can be very tough. So here's the philosophical engineering question: is it better to have incorrect documentation which may help you get started, but which also may lead you astray; or to have no (or minimal) documentation which forces you to figure it out from scratch? What's been your experience? What's your preference?