Go to Post ...when you see someone post about something they don’t like, please don’t post another “I’m sick of all this whining” type of post. If you disagree with them, try to offer an intelligent argument back. - Phil 33 [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 19-04-2004, 21:34
mtrawls's Avatar
mtrawls mtrawls is offline
I am JVN! (John von Neumann)
#0122 (NASA Knights)
Team Role: Programmer
 
Join Date: Mar 2003
Location: Hampton, VA
Posts: 295
mtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to behold
Send a message via AIM to mtrawls
Detecting and handling sensor failure gracefully

On our robot this year we had 3 different pots each responsible for some control necessary for the robot's operation. Two of them helped to resist backdrive (from gravity) and otherwise aid in control, and the remaining was necessary to keep a chain from breaking. That being said, it was fairly important that our pots were working -- otherwise we would lose our competitive edge. And, unfortunately, on Thursday and Friday we had problem after problem with the pots. On Thursday we managed to break 3 or 4 pots (hasn't happened before), mostly unexaplainably -- by break, either the shaft itself came out, the plastic backing ripped apart or the pot mysteriously stopped turning! Well, by Friday, after we had lost our supply of spare pots (many thanks to team 16 and the others who helped us out), the problems started changing. The series of PWM wires connecting the pots would become slightly unconnected, causing the pot to become useless. Then, responsibly for some strange intermittent control issues, I tracked another problem down to a bad PWM cable ... and then another bad cable.

To make a long story short, on Friday our controls didn't work correctly in over the majority of the matches (although we only had a few minor issues at our 3 other regionals). Obviously this is not a good thing! Now, I could have hit sensor override to stop the erratic behaivor (the various appendages would try to hit a setpoint, but since the pot was increasing constantly, they'd obviously never reach it!) -- but this would cause undesirable, and potentially devestating effects with the other elements of the robot. So I started thinking about how the team can get around this next year (alas, I'll be in college) ... and I wondered if I couldn't detect a sensor failure in software and find a way to handle it on an individual basis, rather than all or nothing.

I'm interested to hear if anyone else has done this, or has any thoughts on the issue. The "symptoms" of the two major problems are these: if the pot is unconnected somewhere, the value in software will increase by one each loop; if a PWM cable is bad, the resistance between "red" and "black" will be significantly lower than the rated value (100k in my case, and it was reading about 7k). For the first problem, I imagine simply keeping track of the last N values (say N = 20) so long as the motor output is neutral -- and if the values are progressively rising, it would tend to indicate a bad pot. Then, for the second problem, I thought perhaps I could hook up a current sensor, and using the knowledge of the voltage, deduce resistance and make sure it is within +- 5% of 100k (or whatever it happens to be). Does this sound reasonable? Anyone done this before?
  #2   Spotlight this post!  
Unread 19-04-2004, 22:09
KenWittlief KenWittlief is offline
.
no team
Team Role: Engineer
 
Join Date: Mar 2003
Location: Rochester, NY
Posts: 4,213
KenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond repute
Re: Detecting and handling sensor failure gracefully

I think you are trying to overengineer a solution that is not really the solution you want.

the real problem is your sensors (for whatever reason) were unreliable. You need to find more robust devices, and find ways to attach your cables (tiewraps and doublesidedstickytape holddowns) so they stay attached.

In engineering if a component is critical the common solution is to have more than one - problem is, to be able to tell which one has failed, you need at least 3 - then when one sensor no longer agrees with the other two you ignore it (triple/redunant sensors)

for flight control systems they use quad redundant systems in aircraft

isnt that what you really want, for the bot to work and the loop to stay closed all the time? anything else would be a kludge - if the kludge is good enough, then why use the sensor in the first place?

but I think if you solved the reliability problems, the root cause of your failures, we would not be having this discussion at all - the best solution would be more robust pots attached in a manner that wont destroy them (couplers that can flex a bit maybe?)

Last edited by KenWittlief : 19-04-2004 at 22:12.
  #3   Spotlight this post!  
Unread 19-04-2004, 22:45
mtrawls's Avatar
mtrawls mtrawls is offline
I am JVN! (John von Neumann)
#0122 (NASA Knights)
Team Role: Programmer
 
Join Date: Mar 2003
Location: Hampton, VA
Posts: 295
mtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to beholdmtrawls is a splendid one to behold
Send a message via AIM to mtrawls
Re: Detecting and handling sensor failure gracefully

Oh, don't get me wrong ... I'm not trying to fix an engineering problem in software (although I'm often told to!). I've already located more robust pots. Also, the issue was that we stringed PWM cables together; and our robot's omni-wheel design was "knobbed" causing some pretty serious shaking; this all combined to let the cables come undone, even though they were wire tied, and the connectors taped for insulation. Next year, though, my team knows to just use plain old wire and only attach to the PWM connector at the very end of travel. This should allay most of the problem (though for 3 regionals we didn't have much of a problem at all).

The software detection is just an added step in the process, an extra redundancy (speaking of redundant ...). The issue isn't to come up with a kludge, but to detect failure in a specific system, and let that system revert to manual control individually, instead of relying on a general sensor override switch (aslo, in auto mode it is useful to know if the sensor isn't working, to go to a plan b, or at least to sit still so that the robot doesn't ram into the wall with pieces incorrectly deployed, causing us to get DQ'd, which did happen).

Quote:
but I think if you solved the reliability problems, the root cause of your failures, we would not be having this discussion at all - the best solution would be more robust pots attached in a manner that wont destroy them (couplers that can flex a bit maybe?)
Oh, I agree with you that the reliability is the root cause and needs to be addressed first. (But as a programmer I can't sit idly by and rely on engineers, of all people, to fix the problem! ) Incidentally, one of the pots was attached to a spring type device that gave way and flexed as needed ... but the force became too great apparently.

I did think of adding limit switches, where the pots are used for detecting the end points of travel, to act as redundancies ... though, as you mention, I'd need a third sensor to know which one has failed. I don't know, perhaps if I fix the robustness problem all this won't be necessary at all -- did anyone else using pots experience any problems about failure? The issue that bugs me is two separate pwm cables went bad, which were kept safely away ... and that doesn't have to do with the strength of the pot's mounting or the pot itself.
  #4   Spotlight this post!  
Unread 19-04-2004, 22:55
Solace's Avatar
Solace Solace is offline
Head Hurts. Coffee. More. Now!
AKA: Jake
#0571 (Team Paragon)
Team Role: Alumni
 
Join Date: Feb 2002
Rookie Year: 2001
Location: Windsor, CT
Posts: 569
Solace is a splendid one to beholdSolace is a splendid one to beholdSolace is a splendid one to beholdSolace is a splendid one to beholdSolace is a splendid one to beholdSolace is a splendid one to beholdSolace is a splendid one to behold
Send a message via AIM to Solace
Re: Detecting and handling sensor failure gracefully

we counted out wheel rotations using the banner sensors that FIRST gave us. not extremely accurate (it counted in 1.5" increments), but as were not directly connected to the wheels they were never damaged.
__________________
...What is a man,
If the chief good and market of his time
Be but to sleep, and tool? A nerd, no more.

2004 UTC New England #2 seed
2004 UTC New England Champions with 716 & 230
2004 Archimedes #2 seed, undeafeated in Qualifiers (for what its worth)


Jake
Team Paragon #571
  #5   Spotlight this post!  
Unread 20-04-2004, 00:19
Alan Anderson's Avatar
Alan Anderson Alan Anderson is offline
Software Architect
FRC #0045 (TechnoKats)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: Kokomo, Indiana
Posts: 9,113
Alan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond repute
Re: Detecting and handling sensor failure gracefully

If I end up lead programmer for our team next year, I'm definitely going to put some fault-detection in the sensor system. During autonomous mode testing in our pit at Nationals, a stray fleck of metal shorted the 5v output of the RC, and the arm immediately went out of control. I did have a two-second "stall" timer in place at the time, but that was much too long to deal with the situation. (We managed to disable the 'bot before anything -- or anyone -- was damaged, though we needed to physically lift it to safely disengage the ball grabber from our pit structure. It could have been much worse.)

One standard way to detect sensor failure is to put a resistor in the line from power to the pot, and measure the voltage at their connection. If it goes high, a wire has likely broken. If it goes low, something is likely shorted to ground. The available range of sensor output voltage is reduced slightly, but the easy diagnostic check can be worth it.
  #6   Spotlight this post!  
Unread 20-04-2004, 01:27
Jay Lundy Jay Lundy is offline
Programmer/Driver 2001-2004
FRC #0254 (The Cheesy Poofs)
Team Role: Alumni
 
Join Date: Jun 2001
Rookie Year: 2001
Location: Berkeley, CA
Posts: 320
Jay Lundy is a name known to allJay Lundy is a name known to allJay Lundy is a name known to allJay Lundy is a name known to allJay Lundy is a name known to allJay Lundy is a name known to all
Re: Detecting and handling sensor failure gracefully

We relied heavily on 3 sets of sensors this year: pots for our two arm joints, prox sensors to count wheel rotations (actually sprocket teeth), and a light sensor to detect the lines perpendicular to the ball tees. This was the first year we relied so heavily on sensors and we learned a few things as we tested.

We also had problems with sensor failure. When a connector for the pot came out whatever joint it was attatched to would go crazy, which was really dangerous since our arm was so long and powerful. We ended up just putting in a manual override in case the pots died. When the pots were disabled it made the arms much more difficult to drive and we lost all preset positions, but at least we weren't completely disabled. Luckily we only had 1 pot fail in at least 35 matches.

We relied heavily on the prox sensors to keep us going straight during normal operation and to track our distance in autonomous mode. At first, we had the prox sensors reading the sprocket connected to the wheel shaft. However that sprocket moved every time we tensioned the chain, and one time 60 forgot to make sure one of the prox sensors was reading before a match and they slammed into the platform at 15 ft/s in autonomous mode and a pneumatic fitting on one of their tanks cracked. They lost all pressure and basically all function for the entire match. We ended up moving the prox sensor to a shaft on the transmission, where we got higher resolution and much more reliability (never failed again). We still had a manual override for the prox sensors, but we never needed to use it.

We relied on the light sensor to tell us when we crossed the ball tees and the center of the field so we could do various things depending on which automode we were using (stop / lift the whip / turn). The problem was, if the sensor didn't work the robot would never stop and would continue into the other alliances ball corrall. We solved that by simply putting in a timer failsafe, if 3.5 seconds into autonomous we hadn't seen the lines, then jump into the next state anyway.
  #7   Spotlight this post!  
Unread 20-04-2004, 08:45
KenWittlief KenWittlief is offline
.
no team
Team Role: Engineer
 
Join Date: Mar 2003
Location: Rochester, NY
Posts: 4,213
KenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond reputeKenWittlief has a reputation beyond repute
Re: Detecting and handling sensor failure gracefully

I like the idea of using limit switches in addition to the pot sensor. This is standard practice for many motion control systems. A limit switch is pretty bulletproof, so it not only covers pot/sensor failures, it covers your whole control system.

if the switch is closed you know your arm is at the end of its travel, no matter what the rest of your control system is telling you - in fact many systems have two at each end - an end of travel switch, and a hard cutoff (kill) switch

We used a pot at the base of our arm and we did have one fail for no apparent reason.

OK to get back to your original question then - one simple way to check your sensors is to have a jog function - you tell the arm to move a bit, and watch the sensor. If the sensor reading doenst change, then its failed.

You can do this on powerup, or under driver command (used in the pits) or under a periodic self test, but when a bot only runs for 2 minutes putting periodic tests into it SW isnt very useful.

the timeout test is another option - if you tell the arm to move read the pot first, then if it doenst change within a few SW cycles its probabally gone south (in my mind Im gone to Carolina, cant you see the sunshine, can you just feel the.. HEY WAKE UP! WERE IN THE MIDDLE OF THE FINALS - Oops, sorry, 58, 58, 58, 62, 65, 73). <- this is not very realistic - once a sensor goes south it almost never comes back.

BTW - when you loose your sensor input in a closed loop PID control system, its called 'going open loop' - its usually very funny to watch when it happens - kinda like putting a blindfold on a boxer during a prize fight, and watching him flail about in the ring trying to hit his opponent when he cant see what hes swinging at.

sometimes engineers go open-loop too, which is also amusing to watch :^)

Last edited by KenWittlief : 20-04-2004 at 08:51.
  #8   Spotlight this post!  
Unread 20-04-2004, 14:03
Astronouth7303's Avatar
Astronouth7303 Astronouth7303 is offline
Why did I come back?
AKA: Jamie Bliss
FRC #4967 (That ONE Team)
Team Role: Mentor
 
Join Date: Jan 2004
Rookie Year: 2004
Location: Grand Rapids, MI
Posts: 2,071
Astronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud ofAstronouth7303 has much to be proud of
Re: Detecting and handling sensor failure gracefully

I like this, but it also sounds incredibly complex. The current sensor we used on our hand just made it complicated. That was just a pot and a current sensor. If the pot failed, the driver can work around it. if the current sensor fails, it's screwed. (it's based off a limit-trip. after 5 loops of tripped, it won't move until you reverse). However, i did not include anything for detecting failure.

And what about telling the drivers about failure? OI LEDs or dashboard? what about 3rd party transmitter/reciever setups? I mean, a tablet PC in the driver's station would be cool, but I can think of a thousand other things to spend that money on (Like a second bot!).

One other thing: is there a reliable way to setup so that a computer can send stuff to the RC through the OI? Does the dashboard port take input, too? My thought is you have a computer that monitors the system, and if something goes wrong, you can tell the computer what to do, and it rellays it to the RC.
  #9   Spotlight this post!  
Unread 20-04-2004, 15:19
Ryan M. Ryan M. is offline
Programming User
FRC #1317 (Digital Fusion)
Team Role: Programmer
 
Join Date: Jan 2004
Rookie Year: 2004
Location: Ohio
Posts: 1,508
Ryan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud ofRyan M. has much to be proud of
Re: Detecting and handling sensor failure gracefully

Quote:
Originally Posted by Astronouth7303
One other thing: is there a reliable way to setup so that a computer can send stuff to the RC through the OI? Does the dashboard port take input, too?
The dashboard is one way. (output, just in case anyway gets confused... )
__________________

  #10   Spotlight this post!  
Unread 20-04-2004, 17:19
Alan Anderson's Avatar
Alan Anderson Alan Anderson is offline
Software Architect
FRC #0045 (TechnoKats)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: Kokomo, Indiana
Posts: 9,113
Alan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond repute
Re: Detecting and handling sensor failure gracefully

Quote:
Originally Posted by Astronouth7303
...And what about telling the drivers about failure? OI LEDs or dashboard? what about 3rd party transmitter/reciever setups? I mean, a tablet PC in the driver's station would be cool, but I can think of a thousand other things to spend that money on (Like a second bot!).
I'm considering trying to design a simple "annunciator panel" to plug in to the dashboard port on a more-or-less permanent basis for a fancy OI. It'll just read the RC packets and control highly visible lights based on the user bytes, probably along with a digital display or two. Maybe it could even have a beeper or something to really get the operator's attention when necessary.

Quote:
One other thing: is there a reliable way to setup so that a computer can send stuff to the RC through the OI? Does the dashboard port take input, too? My thought is you have a computer that monitors the system, and if something goes wrong, you can tell the computer what to do, and it rellays it to the RC.
Nope, you can't really do that. The OI just sends the analog (joystick, wheel, "hat") and digital (trigger, thumb, aux switches) data for the RC to interpret. All the interpretation has to be done by the RC.
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 03:07.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi