A very weird digital sidecar problem...

[pfreivald]

So after we were eliminated and went to the practice field to diagnose, everything worked fine – we plugged 'em back in to start troubleshooting and it simply worked; indeed, we couldn’t recreate the problem. This is troubling because we didn’t actually diagnose and solve the problem, so it’s far too likely that it will reappear at a particularly inconvenient time (say, Buckeye). I’m happy if it doesn’t, but don’t trust that it won’t.

A hypothesis (based on, as far as I can tell, a hunch and nothing more from one of my students talking with another mentor) is that the encoders spin so fast that they are generating too constant a high voltage signal, and that this is tripping some kind of internal breaker in the sidecar or module. I don’t even know how I’d begin to test that hypothesis, or what we could do about it if it were true – seems fishy to me anyway, as many other teams are using kit encoders on their shooters without the same problem.

Thanks so much, everyone, for taking the time to ask and answer. We’re hoping we can narrow it down fast so practice day results in a fast fix!

I’m answering all questions below; please forgive the terse tone – I’m exhausted and going to bed just as soon as I finish replying to this!

That being said, a few questions (some elementary, but bear with me):

• Which encoders are you using? (I assume the E4P from USDigital, which is what’s standard in the KoP)

Correct.

• Is this behavior consistent no matter which DIO (technically, the term is GPIO ;)) pins are used? That is, it doesn’t matter which pair of pins is connected to the encoder?

Also correct.

• Have you double checked the encoder wiring setup? (i.e. made sure each line is isolated and not shorting to something else, and that you have +5 to +5, A and B to GPIO, and GND to GND)

Yes. We successfully used encoders last year, and they were working through weeks of testing. The wiring was double-triple-quadruple checked for continuity, shorts, and correct pin positions. DSC pins were checked for shorts.

• Do other digital sensors work on the GPIO pins? (If you don’t have any on hand, see if any other teams have spare photoelectric sensors that you could play with).

Yes. We use some digital photoelectric sensors on our ball management system, and they’re working fine (as long as the encoders are not plugged in).

• You said the 5V power fails; I assume this means that the 5V light turns off while the 6V and BAT lights stay on? (note to Eric: p3 of the spec sheet seems to have the 6V and “Power Input good” labels reversed)

Correct – and all Victors start flashing the “no signal” flash.

• What do you mean by bumper? The 9403 module (the 37-pin module that plugs into the cRIO)? Something tells me that you’re not plugging the DB37 into your bumper… :ahh:

Module, sorry – the DIO module has no bumper, and I know that.

• Since the only remaining common at this point is the cRIO port, have you tried using a different slot for the module?

Yes.

• When you replaced parts, did you restore the original after finding that the replacement didn’t fix the problem? (If a replacement has a problem, it introduces another problem to the system that you’re likely to not notice, because fixing the original problem won’t fix the system).

Yes.

Pat,
The power supply in the DSC is capable of several amps so it should handle two encoders. Is there any chance that someone unplugged the encoder connectors and put them back reversed? Some devices are reverse polarity protected and would produce a short if wired backwards and thereby cause the 5 volt supply to shut down.

We repeatedly verified that they are not reversed.

It is also possible that the power feeding the DSC has a loose wire. Does the 6 volt supply also shut down? If multiple power supplies shutdown at the same time, the input power is the likely cause.

Only the 5V shuts down. The 6V and 12V LEDs stay bright and solid – and we can still fire relays (the compressor doesn’t run because of the pressure switch not working, but our LED ring on our camera lights up as normal. Swapping the two caused the compressor to run.).

Also when you tested it in the morning, was there new code deployed, or was it “permanent” code with no new deploying?

“Run at startup” from the problem-free night before.

Also, did you test different power wires/connectors from the Power Distribution Board to the DSC?

Yes.

So I would check any loose connections starting with the battery.

Done and done and done.

Also, test the 12V dedicated supply for the Wifi but use it on the DSC and see if it does that any more.

I’m not sure what you mean here.

It still sounds like a short in the encoder circuit. Whether it’s wired wrong now, it’s shorting somewhere, not sure. The power shouldn’t just go out because of plugging in something unless it is shorting the power supply. Encoders are pretty dumb devices in the terms of sensors. They take in power, and give out a pulse that the cRIO counts. Plugging them into the sidecar should yield no changes at all, ever.

Yup, we know all that. There are no shorts anywhere that we can find, either on the encoders, the sidecar pins, or the cables.

I’d recommend doing the wiring again. I’d recommend YOU doing the wiring again. Only then do you know if it was a wiring fault.

Done and done; same problem.

Could you check the LEDs on the DSC (BAT, 5V, 6V) with the encoders attached and removed?

My current bet is either that the 5V supply is being shorted by the encoder, or that the 12V power isn’t attached correctly and the DSC is being phantom powered by the cRIO.

Wouldn’t the latter cause a loss of functionality altogether, and not just when the encoders are plugged in? We had full pwm functionality when the encoders weren’t plugged in, and none when they were.

As to the former: that was our bet, too, but we cannot find any evidence of a short on any component.

Bingo! Something happened during that match. A hard hit or bounce, etc.

Perhaps, though not that we recall – either way, it’s odd that the short (if it is indeed a short) seems to be undetectable by mortal (that is, 1551) means.

I have seen this behavior, now that Eric mentioned the “phantom power”, I remember this during our build season.

I saw that last year and looked for it. Our LEDs are not weak, they’re bright and strong as normal. And as I said before, we can fire Spike relays.

---

Thanks and goodnight!

[wireties]

There are a max number of counter channels of different types, is it possible you you exceeded them?

Monitor the crio console as it boots. Any error or diagnostic messages?

Try a new digital module?

Good luck!

[EricVanWyk]

From your description of the diagnostic LEDs, this is an electrical problem in either the DSC or your encoder: Your 5V rail is getting shorted out. You can eliminate any theories related to software, the PD, or the cRIO.

I’m betting you have an intermittent short in your encoder wiring. This would explain why it magically fixes itself.

[slijin]

So after we were eliminated and went to the practice field to diagnose, everything worked fine – we plugged 'em back in to start troubleshooting and it simply worked; indeed, we couldn’t recreate the problem. This is troubling because we didn’t actually diagnose and solve the problem, so it’s far too likely that it will reappear at a particularly inconvenient time (say, Buckeye). I’m happy if it doesn’t, but don’t trust that it won’t.

A hypothesis (based on, as far as I can tell, a hunch and nothing more from one of my students talking with another mentor) is that the encoders spin so fast that they are generating too constant a high voltage signal, and that this is tripping some kind of internal breaker in the sidecar or module. I don’t even know how I’d begin to test that hypothesis, or what we could do about it if it were true – seems fishy to me anyway, as many other teams are using kit encoders on their shooters without the same problem.

Perhaps accumulated static electricity? It’s a ridiculous hypothesis, but it’s all I can think of.

The 9403 can withstand ±30V, and since the encoders are just a digital signal, I can’t see how this would cause a problem.

The only suggestion that I have is to prepare a secondary system with Jaguars on CAN (the black ones, not the tan ones) for the explicit purpose of encoder input (and if so desired, to use them to control your shooter as well) in the event that this problem recurs.

[Brandon_L]

Maybe try replacing the encoders? (Didn’t see it suggested here, sorry if it was). It could be something inside the encoder shorting that may have broken from a hard hit. Try borrowing one or just plugging an extra one in using the same wiring to your current encoders and see what happens?

[slijin]

Maybe try replacing the encoders? (Didn’t see it suggested here, sorry if it was). It could be something inside the encoder shorting that may have broken from a hard hit. Try borrowing one or just plugging an extra one in using the same wiring to your current encoders and see what happens?

See the OP.

We tried replacing the encoders, the wires from the encoders to the sidecar, the sidecar, the cable from the sidecar to the bumper, and the bumper… Even replacing ALL of them yielded the same result.

[EricVanWyk]

Does the issue persist when the encoders are plugged in but not mounted to the robot?

[pfreivald]

Does the issue persist when the encoders are plugged in but not mounted to the robot?

Good question – we don’t know. We did test to see if the encoders (and everything else) were electrically isolated from the frame, and they are. If the problem pops up again as I suspect it will, that’s something new we can check.

It looks very much like, as you said, a short in either the encoder wiring or the DSC – which is why it’s frustrating that we replaced both encoders, their wiring, and the digital sidecar and yet still had the same problem.

We’ll try another set of brand new encoders with brand new wiring when we get to Buckeye and can take the bot out of the bag.

Meantime, does anyone have any suggestions on alternate sensors we could easily adapt to get rpm values?

Thanks for the help everyone!

[billbo911]

Meantime, does anyone have any suggestions on alternate sensors we could easily adapt to get rpm values?

Thanks for the help everyone!

Honestly, I don’t recall where you were using the encoders, but I guess it really doesn’t matter.
Banner makes several sensors that could be adapted. As an example, you could use a Retro sensor and a small piece of the retro tape to create a counter.

Then just use a little math to calculate your RPM.

[pfreivald]

Honestly, I don’t recall where you were using the encoders, but I guess it really doesn’t matter.
Banner makes several sensors that could be adapted. As an example, you could use a Retro sensor and a small piece of the retro tape to create a counter.

Then just use a little math to calculate your RPM.

Are they fast enough? We were thinking of using the optical sensors (included in the kit last year, and FIRST choice this year) to count either spokes or white/dark coloration, but we don’t know if they have a sufficient response time to do the job.

[billbo911]

From the specs page.

Output Response Time Opposed:750 microseconds ON; 375 microseconds OFF
All others:600 microseconds ON/OFF
Delay at Power-up 100 milliseconds; outputs do not conduct during this time.

It looks like they are able to count, at minimum, up to approx. 888 Hz. Therefore they should be able to track a wheel turning at up to 53.2K RPM. Surely you are not tracking anything that fast!

[Deetman]

Are they fast enough? We were thinking of using the optical sensors (included in the kit last year, and FIRST choice this year) to count either spokes or white/dark coloration, but we don’t know if they have a sufficient response time to do the job.

1712 uses the QS18 retroreflective sensor for our RPM and a piece of retro tape included in the kit. It has worked great for us after nothing but problems with the USDigital encoders.

[pfreivald]

Great to know, guys, thanks! I’ll discuss this with my teams on Tuesday.

[Al_Skierkiewicz]

Patrick,
I will bet dollars to doughnuts that your have some wayward metallic debris in the DSC. I helped a team with a similar situation locate one single metal flake that had fallen down between pins on the DSC. You need a very bright light. You might have to remove the DSC from the robot, remove the cover and check inside but I am betting that is what you will find. It is likely it is right next to the pins for your encoders.

[pfreivald]

Patrick,
I will bet dollars to doughnuts that your have some wayward metallic debris in the DSC. I helped a team with a similar situation locate one single metal flake that had fallen down between pins on the DSC. You need a very bright light. You might have to remove the DSC from the robot, remove the cover and check inside but I am betting that is what you will find. It is likely it is right next to the pins for your encoders.

Even the brand new, still-in-the-antistatic-bag one we replaced the first one with? Metal shavings was one of our first investigatory culprits; we found none.

[Al_Skierkiewicz]

Patrick,
Did you tell us the model number of the encoders you are using? I did a quick check but didn’t see a number in your earlier posts.

[cierra_shawe]

We had a very similar problem in Portland. We burned through three DCS’s, and no one could figure out the problem. All of a sudden in a match, our robot would quit driving. We checked all of our connections, changed the PDB, swapped DCS’s, and checked for metal shavings, yet nothing helped. The only thing that fixed the problem temporarily was a new DCS.

[pfreivald]

Patrick,
Did you tell us the model number of the encoders you are using? I did a quick check but didn’t see a number in your earlier posts.

Sorry, I was out sick yesterday – feel much better after having spent 20 of the past 24 hours asleep!

We are using these: http://www.andymark.com/product-p/am-0791.htm

[Al_Skierkiewicz]

Pat,
These encoders are virtually indestructible but there are a few peculiarities. The first is the encoder base is molded so that the encoder PCB only fits in one way. The other way causes the PCB to be bent due to the through hole electronics not fitting into the recess in the base. When you mount it, and hog down on the screws, it is possible to bend over the through hole leads or damage the PCB itself. Since the PCB does not have conformal coating it is of course easy to get shavings inside. The LEDs internal to the encoder are visible and can easily be seen when the encoder is correctly powered and the cover is off. It is also easy to bend over one or more of the pins in the white connector.

Cierra,
Is it possible you wired the DSC to the 24 volt power on the PD and not from a simple 12 volt output on the PD?

[pfreivald]

Pat,
These encoders are virtually indestructible but there are a few peculiarities. The first is the encoder base is molded so that the encoder PCB only fits in one way. The other way causes the PCB to be bent due to the through hole electronics not fitting into the recess in the base. When you mount it, and hog down on the screws, it is possible to bend over the through hole leads or damage the PCB itself. Since the PCB does not have conformal coating it is of course easy to get shavings inside. The LEDs internal to the encoder are visible and can easily be seen when the encoder is correctly powered and the cover is off. It is also easy to bend over one or more of the pins in the white connector.

I’m 99% certain none of that is the case here, but will double-check all of the above when we unbag at Buckeye. Thanks!