A very weird digital sidecar problem...
 

So our robot was working perfectly yesterday on the practice field, but not today: when we plug in our two encoders to the digital sidecar (DIO slots 1-4), the 5V power on the sidecar fails (and thus we lose all pwm control and become a cute purple brick). Without the encoders plugged in, there are no shorts between power/ground and power/signal...

We tried replacing the encoders, the wires from the encoders to the sidecar, the sidecar, the cable from the sidecar to the bumper, and the bumper... Even replacing *ALL* of them yielded the same result.

Any ideas? It's rather too late for FLR -- we've established ourselves as a rather good fender defense and are for this regional comfortable in that role -- but for Buckeye in two weeks it would be nice to have actual speed control on our shooter...

We're stumped. Any assistance anyone could provide would be most appreciated.

Thanks!

Re: A very weird digital sidecar problem...
 

Check the battery wiring to the Digital Sidecar. Make sure that the connections at both ends of both wires are grabbing all copper and not insulation.

Re: A very weird digital sidecar problem...
 

Already done.

Re: A very weird digital sidecar problem...
 

Send a private message to EricVanWyk here on CD to bring his attention to this post. He is VERY familiar with the digital sidecars and its various failure modes.

Re: A very weird digital sidecar problem...
 

I'll do that!

Re: A very weird digital sidecar problem...
 

Without having hands on, this is difficult to help troubleshoot, but what the heck, troubleshooting systems is what I've done for a living for 30+ years.

The very first thing you need to ask is: "What changed between yesterday and today? What was the last thing you did?" That might point you in the right direction. The fact that it was working yesterday and is not today is baffling. Something must have changed. If you can't think of anything at all, try the steps below, maybe you'll find something.

Start by plugging in one encoder at a time. If either one makes you loose the +5vdc, you have narrowed down the cause. If neither does by it's self, what about trying to move them to a different input,. Same result?

If neither kills it by it's self, then maybe the power supplying the board it the issue.

Re: A very weird digital sidecar problem...
 

What kind of encoders are you using, and how are you wiring them? Are you positive that the encoder wiring is correct?

I don't have any magic answers, but can suggest a few things to try...

We use the USDigital encoders exclusively, although different models (generally E4P and E7P) and wire them with the 4-pin USDigital cable going to two separate 3-pin PWM-style connectors, but with each connector only using two conductors. One connector will have "A" and 5V; the other connector will have "B" and GND. With the way that we wire them up, it is essential that the connectors be wired the correct way round, and it is easy to get them wrong. When they are wired incorrectly, wierd sidecar behavior can result.

Another thing to check -- is the ribbon cable from the cRIO module to the Digital Sidecar ok? We had a cable which had been damaged due to a robot accident and had intermittent shorts in a couple of the conductors.

I would also suggest checking the power connections to the digital sidecar very carefully. Replace the circuit breaker if you have any reason to suspect it. Consider temporarily replacing the circuit breaker with a small (10A?) automotive fuse -- if it blows, it will do so completely and finally, making it easy to diagnose the problem.

Re: A very weird digital sidecar problem...
 

This was literally a question of "it worked last night, we turned it on and went out to the field and it stopped working halfway through a match".

Indeed. Whatever it was, we can't find it.

Did that. Both do it. ...and they do it if they're new or old encoders, new or old cables, old or new sidecars, and old or new DIO ports... All with new or old sidecar-to-bumper cables and new or old CRIo bumpers.

We unplug the encoders, everything else works fine. We can't seem to even come up with something else to check.

Re: A very weird digital sidecar problem...
 

Another thought -- have you tried moving the encoder pins to different pins -- say 7-10 instead of 1-4? There may be a physical problem with the pins you are using on the sidecar.

Replacing the sidecar is a painful change (with the many cables for PWM robots) but might be worthwhile. You should be able to get one from the "spare parts" desk at the tournament.

Re: A very weird digital sidecar problem...
 

Tried that.

And that.

I like to think that at this point we're good at troubleshooting, but this one has us extremely stumped!

Re: A very weird digital sidecar problem...
 

Tried replacing the power distribution board?

Re: A very weird digital sidecar problem...
 

Considering he helped design it... :P

That being said, a few questions (some elementary, but bear with me):
- Which encoders are you using? (I assume the E4P from USDigital, which is what's standard in the KoP)
- Is this behavior consistent no matter which DIO (technically, the term is GPIO ;)) pins are used? That is, it doesn't matter which pair of pins is connected to the encoder?
- Have you double checked the encoder wiring setup? (i.e. made sure each line is isolated and not shorting to something else, and that you have +5 to +5, A and B to GPIO, and GND to GND)
- Do other digital sensors work on the GPIO pins? (If you don't have any on hand, see if any other teams have spare photoelectric sensors that you could play with).
- You said the 5V power fails; I assume this means that the 5V light turns off while the 6V and BAT lights stay on? (note to Eric: p3 of the spec sheet seems to have the 6V and "Power Input good" labels reversed)
- What do you mean by bumper? The 9403 module (the 37-pin module that plugs into the cRIO)? Something tells me that you're not plugging the DB37 into your bumper... :ahh:
- Since the only remaining common at this point is the cRIO port, have you tried using a different slot for the module? Although the cRIO probably isn't your point of failure, replacing everything else does nothing else, soo...
- When you replaced parts, did you restore the original after finding that the replacement didn't fix the problem? (If a replacement has a problem, it introduces another problem to the system that you're likely to not notice, because fixing the original problem won't fix the system).

Re: A very weird digital sidecar problem...
 

Pat,
The power supply in the DSC is capable of several amps so it should handle two encoders. Is there any chance that someone unplugged the encoder connectors and put them back reversed? Some devices are reverse polarity protected and would produce a short if wired backwards and thereby cause the 5 volt supply to shut down. It is also possible that the power feeding the DSC has a loose wire. Does the 6 volt supply also shut down? If multiple power supplies shutdown at the same time, the input power is the likely cause.

Re: A very weird digital sidecar problem...
 

Like mentioned before, the DB-37 cable is plugged into a module (not a bumper), the Analog board that attaches to the Analog module is called a bumper.
Also when you tested it in the morning, was there new code deployed, or was it "permanent" code with no new deploying?
Also, did you test different power wires/connectors from the Power Distribution Board to the DSC?

And finally recently our robot's wifi would decide to restart whenever I enabled the robot (happened to be for only 1 battery), but that battery's main terminal was loose which let the robot boot fine but when it came to draw power, it crumbled. So I would check any loose connections starting with the battery. Also, test the 12V dedicated supply for the Wifi but use it on the DSC and see if it does that any more.

Re: A very weird digital sidecar problem...
 

Hopefully I understand all of this correctly, because I haven't seen anyone say this.

It still sounds like a short in the encoder circuit. Whether it's wired wrong now, it's shorting somewhere, not sure. The power shouldn't just go out because of plugging in something unless it is shorting the power supply. Encoders are pretty dumb devices in the terms of sensors. They take in power, and give out a pulse that the cRIO counts. Plugging them into the sidecar should yield no changes at all, ever.

You should take a continuity checker, and check the black and red wire. I have a feeling that you will see 0Ω, or in other words, short circuit.

I feel that you will find that it is shorting.
I'd recommend doing the wiring again. I'd recommend YOU doing the wiring again. Only then do you know if it was a wiring fault.

Re: A very weird digital sidecar problem...
 

Could you check the LEDs on the DSC (BAT, 5V, 6V) with the encoders attached and removed?

My current bet is either that the 5V supply is being shorted by the encoder, or that the 12V power isn't attached correctly and the DSC is being phantom powered by the cRIO.

Re: A very weird digital sidecar problem...
 

Bingo! Something happened during that match. A hard hit or bounce, etc.

I have seen this behavior, now that Eric mentioned the "phantom power", I remember this during our build season.
The three LEDS might be lit, but weakly. Check the power (+12v) coming in. Maybe a bad breaker or bad connector on the PDC. Move to a new slot on the PDC and new breaker on the PDC and you may have better luck.

Re: A very weird digital sidecar problem...
 

So after we were eliminated and went to the practice field to diagnose, everything worked fine -- we plugged 'em back in to start troubleshooting and it simply worked; indeed, we couldn't recreate the problem. This is troubling because we didn't actually diagnose and solve the problem, so it's far too likely that it will reappear at a particularly inconvenient time (say, Buckeye). I'm happy if it doesn't, but don't trust that it won't.

A hypothesis (based on, as far as I can tell, a hunch and nothing more from one of my students talking with another mentor) is that the encoders spin so fast that they are generating too constant a high voltage signal, and that this is tripping some kind of internal breaker in the sidecar or module. I don't even know how I'd begin to test that hypothesis, or what we could do about it if it were true -- seems fishy to me anyway, as many other teams are using kit encoders on their shooters without the same problem.

Thanks so much, everyone, for taking the time to ask and answer. We're hoping we can narrow it down fast so practice day results in a fast fix!

I'm answering all questions below; please forgive the terse tone -- I'm exhausted and going to bed just as soon as I finish replying to this!

Correct.

Also correct.

Yes. We successfully used encoders last year, and they were working through weeks of testing. The wiring was double-triple-quadruple checked for continuity, shorts, and correct pin positions. DSC pins were checked for shorts.

Yes. We use some digital photoelectric sensors on our ball management system, and they're working fine (as long as the encoders are not plugged in).

Correct -- and all Victors start flashing the "no signal" flash.

Module, sorry -- the DIO module has no bumper, and I know that.

Yes.

Yes.

We repeatedly verified that they are not reversed.

Only the 5V shuts down. The 6V and 12V LEDs stay bright and solid -- and we can still fire relays (the compressor doesn't run because of the pressure switch not working, but our LED ring on our camera lights up as normal. Swapping the two caused the compressor to run.).

"Run at startup" from the problem-free night before.

Yes.

Done and done and done.

I'm not sure what you mean here.

Yup, we know all that. There are no shorts anywhere that we can find, either on the encoders, the sidecar pins, or the cables.

Done and done; same problem.

Wouldn't the latter cause a loss of functionality altogether, and not just when the encoders are plugged in? We had full pwm functionality when the encoders weren't plugged in, and none when they were.

As to the former: that was our bet, too, but we cannot find any evidence of a short on any component.

Perhaps, though not that we recall -- either way, it's odd that the short (if it is indeed a short) seems to be undetectable by mortal (that is, 1551) means.

I saw that last year and looked for it. Our LEDs are not weak, they're bright and strong as normal. And as I said before, we can fire Spike relays.

--------

Thanks and goodnight!

There are a max number of counter channels of different types, is it possible you you exceeded them?

Monitor the crio console as it boots. Any error or diagnostic messages?

Try a new digital module?

Good luck!

Re: A very weird digital sidecar problem...
 

From your description of the diagnostic LEDs, this is an electrical problem in either the DSC or your encoder: Your 5V rail is getting shorted out. You can eliminate any theories related to software, the PD, or the cRIO.

I'm betting you have an intermittent short in your encoder wiring. This would explain why it magically fixes itself.

Re: A very weird digital sidecar problem...
 

Perhaps accumulated static electricity? It's a ridiculous hypothesis, but it's all I can think of.

The 9403 can withstand +-30V, and since the encoders are just a digital signal, I can't see how this would cause a problem.

The only suggestion that I have is to prepare a secondary system with Jaguars on CAN (the black ones, not the tan ones) for the explicit purpose of encoder input (and if so desired, to use them to control your shooter as well) in the event that this problem recurs.

Re: A very weird digital sidecar problem...
 

Maybe try replacing the encoders? (Didn't see it suggested here, sorry if it was). It could be something inside the encoder shorting that may have broken from a hard hit. Try borrowing one or just plugging an extra one in using the same wiring to your current encoders and see what happens?

Re: A very weird digital sidecar problem...
 

See the OP.

Re: A very weird digital sidecar problem...
 

Does the issue persist when the encoders are plugged in but not mounted to the robot?

Re: A very weird digital sidecar problem...
 

Good question -- we don't know. We did test to see if the encoders (and everything else) were electrically isolated from the frame, and they are. If the problem pops up again as I suspect it will, that's something new we can check.

It looks very much like, as you said, a short in either the encoder wiring or the DSC -- which is why it's frustrating that we replaced both encoders, their wiring, and the digital sidecar and yet still had the same problem.

We'll try another set of brand new encoders with brand new wiring when we get to Buckeye and can take the bot out of the bag.

Meantime, does anyone have any suggestions on alternate sensors we could easily adapt to get rpm values?

Thanks for the help everyone!

Re: A very weird digital sidecar problem...
 

Honestly, I don't recall where you were using the encoders, but I guess it really doesn't matter.
Banner makes several sensors that could be adapted. As an example, you could use a Retro sensor and a small piece of the retro tape to create a counter.

Then just use a little math to calculate your RPM.

Re: A very weird digital sidecar problem...
 

Are they fast enough? We were thinking of using the optical sensors (included in the kit last year, and FIRST choice this year) to count either spokes or white/dark coloration, but we don't know if they have a sufficient response time to do the job.

Re: A very weird digital sidecar problem...
 

From the specs page.
It looks like they are able to count, at minimum, up to approx. 888 Hz. Therefore they should be able to track a wheel turning at up to 53.2K RPM. Surely you are not tracking anything that fast! ;)

Re: A very weird digital sidecar problem...
 

1712 uses the QS18 retroreflective sensor for our RPM and a piece of retro tape included in the kit. It has worked great for us after nothing but problems with the USDigital encoders.

Re: A very weird digital sidecar problem...
 

Great to know, guys, thanks! I'll discuss this with my teams on Tuesday.

Re: A very weird digital sidecar problem...
 

Patrick,
I will bet dollars to doughnuts that your have some wayward metallic debris in the DSC. I helped a team with a similar situation locate one single metal flake that had fallen down between pins on the DSC. You need a very bright light. You might have to remove the DSC from the robot, remove the cover and check inside but I am betting that is what you will find. It is likely it is right next to the pins for your encoders.

Re: A very weird digital sidecar problem...
 

Even the brand new, still-in-the-antistatic-bag one we replaced the first one with? Metal shavings was one of our first investigatory culprits; we found none.

Re: A very weird digital sidecar problem...
 

Patrick,
Did you tell us the model number of the encoders you are using? I did a quick check but didn't see a number in your earlier posts.

Re: A very weird digital sidecar problem...
 

We had a very similar problem in Portland. We burned through three DCS's, and no one could figure out the problem. All of a sudden in a match, our robot would quit driving. We checked all of our connections, changed the PDB, swapped DCS's, and checked for metal shavings, yet nothing helped. The only thing that fixed the problem temporarily was a new DCS.

Re: A very weird digital sidecar problem...
 

Sorry, I was out sick yesterday -- feel much better after having spent 20 of the past 24 hours asleep!

We are using these: http://www.andymark.com/product-p/am-0791.htm

Re: A very weird digital sidecar problem...
 

Pat,
These encoders are virtually indestructible but there are a few peculiarities. The first is the encoder base is molded so that the encoder PCB only fits in one way. The other way causes the PCB to be bent due to the through hole electronics not fitting into the recess in the base. When you mount it, and hog down on the screws, it is possible to bend over the through hole leads or damage the PCB itself. Since the PCB does not have conformal coating it is of course easy to get shavings inside. The LEDs internal to the encoder are visible and can easily be seen when the encoder is correctly powered and the cover is off. It is also easy to bend over one or more of the pins in the white connector.

Cierra,
Is it possible you wired the DSC to the 24 volt power on the PD and not from a simple 12 volt output on the PD?

Re: A very weird digital sidecar problem...
 

I'm 99% certain none of that is the case here, but will double-check all of the above when we unbag at Buckeye. Thanks!

Re: A very weird digital sidecar problem...
 

Pat,
I should have also mentioned that while extremely hard to do, it is possible to plug the white connector in backwards. I think you only hope is to recheck everything at this point, including the encoder wheel position relative to the board.

Re: A very weird digital sidecar problem...
 

:eek: I think that would take some serious effort! (I'm certain we didn't.)

That's what I'm thinking, too. Thanks for your help.

Re: A very weird digital sidecar problem...
 

Al -

Is it possible that the mis-assembly you described to short out the DSC's power?


Cierra -

What do you mean by "burned through"? Are you sure you were destroying DSCs?

Re: A very weird digital sidecar problem...
 

Eric,
I am grasping at straws here to understand what might be happening. Pat seems to have checked most of the obvious stuff so now it is time to move onto the other not so obvious. I know one of my students assembled this encoder backwards on the base a few weeks ago. The board cracked in the exercise due to the mounting screws but it is also possible to bend over the tabs on the connector and cause a short between the 5 volts and the adjacent pin. I know another student from last year had managed to push in the connector backwards but the result was almost ripping the connector off the board. I don't have one here at work but as I remember there is only one LED assembly with two photo transistors or diodes and a couple of resistors. There is even the possibility that the mounting screws had been substituted for larger screws which could play havoc as well. I can't wait to hear what Pat finds out.

Re: A very weird digital sidecar problem...
 

Assuming we manage to track it down next Thursday, you can be assured I'll be posting the results here (though likely not until after Buckeye is over.)

Re: A very weird digital sidecar problem...
 

Update: We could not recreate the problem.

The good news is that this means we were shooting in autonomous and even scoring some balls at Buckeye. The bad news is that we have no idea what happened or why.

I've spent the past few days going over the robot with a fine-toothed comb (and oscilloscope) after school, and I can't find a single, solitary issue with the electronics, wiring, or sensors.

Thanks, everyone, for your input into this problem... I wish I had a more satisfactory conclusion in terms of problem identification and resolution, but I don't.