OCCRA
Go to Post You can make unfounded guesses, but chances are you'll come off looking like a moron. Let's try and avoid that. - Karthik [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 03-12-2018, 05:32 AM
jdao jdao is offline
Registered User
AKA: Jonathan Dao
FRC #4201 (Vitruvian Bots)
Team Role: Mentor
 
Join Date: Jan 2015
Rookie Year: 2012
Location: United States
Posts: 24
jdao will become famous soon enough
How do you find and fix a problem when everything goes wrong?

Hello Chief Delphi,
After a tough weekend at San Diego, I thought I’d like to share my team, team 4201 the Vitruvian Bots, story from my perspective at what we went through over the entire event. I think its an interesting story to share to Chief Delphi not only so that we can learn how to avoid something like this in the future, but to share with everyone how crazy things can get when your robot doesn’t work. This story is a bit long, but trust me, it's worth the read.

Background before San Diego:
On stop build day, we bagged the robot where everything was together mechanically. However, the wiring of electronics and pneumatics to our wrist/intake was left incomplete due to time constraints. In the weeks between stop build day and San Diego, we got everything up and running on our practice bot. Our robot requires the extensive use of potentiometers on both its arm and wrist because the arm and wrist/intake can both exceed the 16” extension limit if they aren’t controlled properly. We solved this by using a look-up table to limit the wrist/intake’s maximum forward angle based on the arm’s current angle and based on our tests before San Diego, we were confident that this would work on our Competition Bot.

Day 1:
Our Controls team got the first chance to turn on the robot and verify its functionality around 1-2 P.M. after replacing our intake with an improved version, doing some mechanical modifications to our arm/elevator mechanisms and completing the wiring/pneumatics we left incomplete. Turning on the robot, we begin to go through an entire systems check to make sure everything works on the robot. All of our mechanisms operated fine in our manual mode (direct power to motors), so we go through inspection and pass.

After completing inspection, we continue to our systems check. At this point, we notice our sensor values aren’t working (sensor voltages don’t change, values [Angle/height] exhibit the same behavior). We try replacing a potentiometer on our wrist. This doesn’t’ change anything, which starts some panic as the sensors are integral to the entire robot’s functionality. Using Shuffleboard’s replay functionality, we notice that we were getting values from our gyro while the robot was moving on our robot cart while moving through the pits for inspection, but now the gyro doesn’t put out a value so we assume the Spartanboard we’re using broke sometime in the last 15 minutes. We then try to look for our spare, only to find that it was left at the hotel, so we send a few mentors and a student to go retrieve it.

They come back about an hour later and we replace the Spartanboard. We turn on the robot and the sensors still don’t work. At this point, we’re beginning to panic a bit since these sensors are critical to the full functionality of the robot. We try taking off the Spartanboard and plug our sensors directly to the roboRIO. We power it on again and still, nothing. We do a check on our sensors with a voltmeter to measure the resistance of our potentiometer to make sure that it works, and verify its functionality.
Our last practice match comes up and we put everything on hold to at least try to participate in it. We move the robot to the field, power it on and… it can’t connect to the field. After a few minutes of debugging, David the FTA (? Can’t remember if this was his actual role) tells us to get off the field and let another team in queue to participate, so we do. Off the field, David helps us troubleshoot our robot and we discover our Ethernet cable isn’t working after it had previously worked before. We go back to the pit and discover that one of the ends got damaged while we were probably swapping out the Spartanboards, so we replace it.
We go back to resolving our sensor issues at this point. We decide to ask pit admin for a replacement roboRIO as if its not a sensor issue or a Spartanboard issue, it must be the roboRIO. We take off our roboRIO and prepare to trade it in. At pit admin, I run into Joe Ross from 330 who was the CSA for the event and talk with him about our issues. In talking with him, he off-handedly mentioned that the DIO ports and Analog Input ports share a common bus. This sparks a thought in my mind that we may be short circuiting our ports due to using DIO ports connected to a custom transistor circuit to control our LED colors (We ran our drive train for a bit with LEDs before stop build day to test some of our autos without our elevator/arm/wrist/intake mechanisms attached, meaning we also didn’t use/attach any analog input sensors to the bot beforehand). We go back to our pit and attach the roboRIO and a single sensor (our wrist pot) and it seemed to work at the time.

Thinking we found the root cause, we go back onto the practice field just to verify that we can connect to the field. We try to connect and… nothing? We check our robot looking for any problems until a volunteer (Sorry I forgot your name) points out that our roboRIO lights aren’t lighting up. We power off the robot and pull the power connectors and see that the red wire was loose, we put it on more securely and power the robot on and… it doesn’t power up? We check everything and the volunteer looks at our PDP and notes that the roboRIO fuse has blown. He goes to get a replacement and we replace it, verified that we can connect to the field and go back to our pits with 15 minutes before the pits close. We decided to swap the old Spartanboard back onto the robot, secure all of our sensor wires and call it a day.

Day 2:
We come into the venue in the morning, power on the robot and find that the sensor values weren’t reporting correctly. We knew the roboRIO was confirmed to work, so we decide to swap the Spartanboard again. We do this and do a systems check in the pits and everything worked, so thinking the problem was resolved, I go to the stands to watch our next match while our drive team queues up. I’m up in the stands for a few minutes before I notice some of our students frantically signaling at our team in the stands, so I go back down to see what’s wrong. I go back down to discover that we’re getting a No robot code error. This boggles my mind because we just deployed and ran our code not even 5 minutes before. I know what the problem is, because in working with the robot, the #1 cause of this is due to the robot not detecting the gyro on the Spartanboard on initialization, so it kills the code. Since the gyro is critical to the code and the problem is usually due to debris on the robot causing a short or not powering the Spartanboard, we don’t usually take this into account. I tell the drive team to repower the bot, nothing. So the choice is made for me to swap with our drive coach to try to debug the issue on the field. I try repowering the robot again, nothing. I disconnect and reconnect the ethernet on the radio, still nothing. I then try to redeploy the code from my laptop. At this point, David comes over at tells me to get off the field and that I was not allowed to tether to the robot on the field. I apologize and leave and swap out with our drive coach and qualification match 8 begins.

Qualification Match 8:
We’re dead on the field due to no robot code, so we just get carried by our great alliance partners.

After the match, we get back to our put and blow compressed air into the Spartanboard. It clears and when we power on the robot, it works fine. I then tell our driveteam what to do in this situation and then go back to figuring out our sensor issue. Another mentor suggested we do a continuity check on our wires, so we do and discover that we had wiring issues so those needed to be replaced. We start work on this, but we immediately need to go queue for our next match.

Qualification Match 13:
We’re running full manual mode on our robot, which is not ideal since this wasn’t what our drivers practiced with. Because of this, we only place one cube in the scale and the rest of the match we go for the exchange. Partway through the match, our drive team stops being able to get cubes due a malfunction in our intake. We end up winning the match, but weren’t performing to the best of our abilities, even in manual mode.
Back in the pits, we discover one of the intake motor wire connectors got clipped while it was going in the exchange, so we move its placement and secure it. We also finally create and replace the sensor wire (it’s a really long wire). However, our next match is coming up and we decide to stay in manual mode for our next match.

Qualification Match 23:
We play the match in manual mode since we didn’t have time to recalibrate the sensors. We do fine, but at the end of the match, we were extended past the 16” extension limit and were rightfully given a yellow card for it. This sucks because we’re right on the edge of having all of our sensors up and running so we can avoid this, but we clearly broke a rule and accept the consequences for doing so.

We get the robot back to the pits to begin recalibrating the sensors. At this point, we notice the voltages for the sensors are way below what they should be (Giving values between 0-1.5v instead of 0-5v). I make the decision to calibrate everything to these values, since we’re on a clock due to the competition and we do all of this in an hour. We go through the rest of the systems check to verify everything works in controlled mode (Our elevator/arm/wrist are controlled by PIDControllers with setpoints for scoring positions, wrist will limit itself if it tries to extend past the extension limit based on the arm’s angle). Everything checks out okay, our drive team goes over the controls and are okay with it and we go into our next match.

Qualification Match 43:
This is the first match we have full robot capabilities and we preformed great with some great alliance members to boot. Unfortunately, this is also the last match at the San Diego regional where we perform with no issues at all.
After this match, a few robot inspectors are sent to our pit by the head ref just to verify that all of our moves the last match we legal, since we were flagged with a yellow card previously. We show our robot software limits the wrist to the 16” extension limit, which works except for a few points, so we’re forced to adjust our limits. I don’t blame the inspectors for imposing this, they’re just doing their jobs and enforcing the rules and I would make the same call in their position. To adjust the limits though, we use a 2D sketch of the robot’s CAD model and check what angle the intake needs to be depending on the arm’s current angle and put it all into a look-up table (This is a bit more complex since the intake has an odd shape due to our backstop, meaning that the angle limits end up being a weird non-linear relation). We do this in an hour and get the okay from the robot inspectors.

After this, we go back to the pits and talk with our alliance members. We decide to go for a center auto since we feel we have the highest chance of doing it out of our alliance members. We run the sequence in our pit to work out the motions of our intake/wrist/arm/elevator so that they perform the motions needed. We finish this and queue up to our next match. On the field, I’m alerted that we’re getting another no robot code error, but its too late and the matches are running behind so

Qualification Match 55:
We’re dead on the field again, and again, our great alliance partners manage to win a 2v3 on their own.

Back at the pits again, Joe Ross comes to our pits and says that we were getting a no robot code due to reading the FMS data before its ready. I’m kicking myself at this point since I did not heed the warnings on Chief Delphi, causing us to lose functionality for a match. Since this was the last match before the end of the day, we just go and perform maintenance on the robot.
At this point, I make the decision to swap the Spartanboard (again!) because in between the matches throughout the day, I’ve noticed the same error where the gyro was not being detected (except this time, it wasn’t crashing the code). So fearing for the worse, I have the team swap it and we call it a day.

At the hotel that night, I go and look up the Chief Delphi thread and fix the FMS problem. Amazingly due to our alliance partners, we’re rank 5 at this point, but we know everyone who’s been watching the matches knows that we’re not reliable at this point, so failures at this point cannot happen. If we want to play the elimination matches.

Day 3:
We head to the pits and power the robot on, verify its functionality because we’re the third match of the day and then I noticed that on power up, the robot was doing something weird. I disable the robot and look at the sensor voltages and see that they are now reporting a proper voltage of 0-5v instead of the 0-1.5v the day before. I quickly tell our team to swap the Spartanboard back to the old one since, we tuned all of our sensor values to voltages between 0-1.5v the day prior, so everything would be off and not work. We swap it and both Spartanboards were working as intended. I’ve never wanted something to be defective more in my life than this at this point. I dejectedly make the decision to have the robot work in manual mode while apologizing to our drive team.

Qualification Match 61:
We play in manual mode for the match. This sucks because this was an important match we noted that we would loose if we could not place cubes on the scale. Since we were flagged with a yellow card prior, controlling the intake/wrist/arm/elevator to score on the scale seemed like a huge risk, so we only played the exchange. This ended up resulting in our first loss.

We head back to the pits and recalibrate our sensors to the correct values. Everything gets check off again to make sure it works, and then we head to our next match. In the mean time a robot inspector comes and warns us that teams are having wires pulled on their robots on the field and to just recheck our wires to be safe. We acknowledge his advice, check all of our wires and then head to our next match. Our strategy is to try to go for the scale in auto if its in front of us, then try for the switch if that is in front of us, otherwise we just drive straight.

Qualification Match 72:
The stars do not align and we end up driving straight, which is a bit disappointing since all of our autos appeared to be just “drive straight” to everyone else. We play through the match only until a field fault is called and the match is reset.

Qualification Match 72 (redo):
We finally get a small break where the scale this time is in front of us and we successfully drive to it. Unfortunately, we decided to ‘shoot’ the cube instead of ‘dropping’ it, leading to us launching the cube past the platform. We start to go through the match again until suddenly the robot stops functioning after 30 seconds into the match. We end up winning the match, but being disabled again is really killing us.
We go back to the pits and the drive team reports that we lost comms due to our ethernet cable being unplugged. Despite the warnings ahead of time, we end up getting hit by the exact same problem. After reviewing our own recording of the match, we see that our own alliance partner just happened to be holding a cube at just the right position and hit our robot in just the exact spot to unhinge the ethernet cable and disconnect us (You can see this here). After a few laughs from the team at such an absurd situation, we recover and relax for a bit before our next match.



Now at this point, you can be thinking, what else can go wrong, right? Oh man, you haven’t seen anything yet until you’ve read the nightmare that is the next few hours.

Right before our next match, our drive team notes that we’re having problems again with the robot. On being enabled the wrist tries to push the intake all the way down. I start trying to redeploy code to try to fix the issue, nothing. Repowering the robot doesn’t seem to work either. We have to start queuing so again I’m given the drive coach badge to try to fix the problem in queue. I repower the robot again, nothing. I check the sensor values and see that the arm’s potentiometer has stuck to 5v, meaning that the arm thinks its at a high angle. When this happens, the code is designed to try to maintain the wrist/intake’s position at 90 degrees perpendicular to the ground, to allow for quick scoring of the cube either forwards or backwards, otherwise it tries to stay parallel to the arm to protect the cube (You can see how this functions in Match 43). But since the arm is physically at a low angle, but the voltage gives us a high angle, the wrist/intake ends up going ‘down’ (which really should be ‘up’), making control of it unpredictable. Again, I try to repower the robot, the voltage still remains at 5v, I try to swap the arm’s potentiometer to a different port and change the address in code to match this. Oops, I switched the wrong sensor, so I redo it and redeploy code. Nope, still 5v. At this point we’re being told to set up for our match, so I’m redeploying the code to put the robot in full manual mode while we’re walking on the field. Additionally, I make the snap decision to put in an untested center auto manual mode since I have not position control of the robot. I apologize to the drive team again and swap out and inform our drive coach that he’s driving in full manual mode.

Qualification Match 81:
Well, the manual auto kind of worked: The robot went to the correct position (albeit not accurately), but the elevator did not move so we couldn’t score. Watching the match live, I can see that we’re starting to play with fire as we’re playing the scale in full auto, knowing full well that breaching that 16” extension limit again is an automatic disqualification. But with some great coaching and patience from our drive team, we’re able to play like this slowly, as when the arm is fully up, we know the intake can’t break the extension limit. We end up taking our second loss after this match, and due to this, end up in 14th place in the rankings at the end of the qualification rounds.

We head back to the pits end also learn that our elevator’s cable had snapped so we start working on it, while I’m still trying to figure out why our arm sensor is sticking at 5v. I look at our sensor values. And what I find is that the arm value is normal. I start to think I’m going crazy because I know it was 5v after seeing it in queue. Looking at the Shuffleboard recording, I verify that I wasn’t insane, but I couldn’t figure out why it was sticking to 5v and more importantly, how do I prevent it. I come up with the pseudo solution of just turning off the robot and waiting a minute, as I had no real understanding of the root cause. I tell the drive team this, give them the voltage values on their main Shuffleboard display so that they can easily identify the issue, and then I go eat lunch while watching the alliance selections.

Out of pure luck, pretty much everyone in the top 8 picked each other in the alliance selections, which ended up shifting our 14th place position all the way up to the 8th alliance captain. Our scouting captain picked our alliance members and we had back to the pits and hope to create the biggest upset in San Diego.

When we go back to the pits and boot up the robot, the arm’s sensor voltage was back at 5v. Not only that but whenever you tried to enable the robot, the arm and wrist/intake would act erratically, both going up and down for a few seconds before we had to e-stop the robot to prevent damage. I try my pseudo solution and it doesn’t work. We try doing everything at this point: verifying that the sensor was working properly, continuity checks, etc., but its too late, we have to queue up for our first match. Before this, we try risking a scale auto which will also go across the field if the goal is across from us, but we’ve never had it fully functional, but going against the #1 seed at this point, we needed to go big or go home.

Quarterfinals Match 1-1:
We lose, badly. Even after a replay of the match due to a field fault. The cross-field scale auto fails to move across the field as the robot stops after hitting the scale. Even if it did move to the correct spot, we had everything in manual mode, so it would not move the intake/wrist/arm/elevator to the correct position. That’s not to say the red alliance didn’t deserve to win (Two hall of fame teams with all of them having great robots that have been performing reliably throughout the entire event. Heck they even won the entire thing, so congratulations!), but its hard to not be frustrated that our own robot is not performing to its full capabilities, especially after everything that has happened up until now.
We go back to the pits. Now we have little time to try to fix our issue before match 2. We go through everything one more time to identify the issue, to no avail. Mentors from other teams come over to help us diagnose our issue, but after going through their suggestions, the sensor still stuck at 5v. We play our timeout card to try to buy us a few more moments to try to find and fix our issue, but as the timer hit zero, the Lead Team Queuer tells us to get on the field. So we do and play our final match of the event.



The drive team comes back to the pits and we congratulate them for doing well in spite of everything. A few mentors, students, and I continue working on the robot, even after we’ve been eliminated, as we’ve got another regional in week 6. I try enabling the robot in controlled mode and it still sputters intermittently. A small, 30 second recording from Shuffleboard then reveals that the arm sensor is not actually sticking at 5v, but will shift form its normal values (~0.8v at its home position) to 5v. This explains the sputtering behavior as if the arm’s sensor suddenly jumps to a value and its using a PID controller, you’ve never changed the setpoint so the arm tries to recompensate to move ‘back’ to its ‘original’ position. Likewise, if the arm is shifting between two values, the wrist will also sputter as it shifts its home position based on where it thinks the arm is. So we identify the issue, but we still haven’t identified its root cause.

After playing around with the robot a bit, we find it: If you move the igus chain, which houses the sensor wire, around it causes the arm’s sensor value to sometimes shift to 5v. We decide to replace the entire wire and resolder the ends to the potentiometer. We finally finish this task, verify that the sensor value remains stable even after shaking the igus chain, and finally get to relax in time to watch the final match 2 of the event.



So there you have it, my entire chronicle of what our team went through in San Diego. In all, we were disabled in 2.75 of our matches, were only partially functional in 6 of our matches, and were only able to be fully functional in 1.25 of our matches. I can’t imagine anyone else going through something like this unscathed, but somehow, we were able to do our best with what were given.

So what could we have done better? It seems like the root cause of all of this was bad wiring and not identifying the issue quickly enough, but from my perspective, it seems really weird to look at the wires as the first thing that may be the cause of your issues. Usually, wiring is a simple thing where you just ‘do it’ and forget about it unless you see that it has clearly melted. After these events, I can’t really be confident that we’ve completely resolved these issues, so I’d like to know if there is something obvious we’re missing in terms of ensuring that our wires work. Some things we’ve learned is to check that the wires were made properly before putting them into the robot by doing a continuity check, but is there really anything else?

For next year at least, I'm definitely going to be pushing to have the robot completely assembled before we bag it. I know our team has been trying to move towards this in the last few years, but any doubts on this should be cleared after what happened this weekend.

I’m extremely proud of all of our students who worked themselves to the bone throughout all of these issues in order to try to get the robot working. I know we were all really close to tears as things kept getting worse and worse, feeling frustrated knowing that the robot works because you've tested the practice version in the weeks before only to see the competition bot fail in its execution, and not being able to show everyone that you bot really does work. All of you guys managed to pull through and managed to show everyone how proud we were of our work so far this season and not let these things dampen our spirits, and that's truly the most inspiring thing out of all of this and what FIRST is all about.

Lastly, I’d like to thank all of our alliance members for working with us with our half-working robot all competition, the teams that offered us assistance in the heat of the competition, and the entire event staff for putting up with the issues we’ve caused them as a result of this.
Reply With Quote
  #2   Spotlight this post!  
Unread 03-12-2018, 07:10 AM
TedG's Avatar
TedG TedG is offline
CAD, Design & Graphics Support
AKA: Team supporter and enthusiast
FRC #0133 (BERT 133)
Team Role: Mentor
 
Join Date: Jan 2011
Rookie Year: 2009
Location: Standish, Maine
Posts: 190
TedG will become famous soon enough
Re: How do you find and fix a problem when everything goes wrong?

Wow, your team certainly had it's issues at that event.
I didn't read the entire post, but I got the gist.. you had compounded issues.

It sounds like you have the right attitude though, yes be proud of your team and what it has gone through, and it's a way to learn what not to do.

As you mentioned your team can use this as a lesson learned, and work to avoid those issues next time.

It's good to read that you have a practice robot, not all teams do that and it's so valuable for fixing bugs, making upgrades, and practicing for events.

Good luck at your next event!
__________________
Bonny Eagle Robotics Team - BERT 133
2009-2010: Mentor, 2010-2013 Advisor/Mentor, 2013-Present: Mentor/Cad & Graphics Support

2010 - GSR: Excellence in Website Design
2011 - GSR: Motorola Quality Award
2012 - Mainely Spirit: Spirit Award, Human Player Award- GSR: Gracious Professionalism Award, Quarterfinalist- Beantown Blitz: Finalist
2013 - Mainely Spirit: Sportsmanship Award- GSR: Semifinalist, Woodie Flowers Award- PTR: Semifinalist- Beantown Blitz: Semifinalist
2014 - GSD: Spirit Award, 5th seed, Semi Finalists- PTD: 2nd seed, Finalists
2015 - Safety Animation 1st Place Award, PTD: Excellence in Engineering Award, 8th seed, Finalists, UNHD: Spirit Award, 9th seed, 6th seed Finalists Alliance Captain; Event Winner
2016 - NSD: Industrial Design Award, 3rd seed, Semifinalist- PTD: Excellence in Engineering Award, 2nd seed, Event Winner, Championship; Carver Finalists

Opinions expressed here are mine alone, and not necessarily of the team.
Reply With Quote
  #3   Spotlight this post!  
Unread 03-12-2018, 09:17 AM
JR0405's Avatar
JR0405 JR0405 is offline
Drive it like you stole it
AKA: Jack Ross
no team
Team Role: Mechanical
 
Join Date: Jan 2016
Rookie Year: 2011
Location: Chicago, IL
Posts: 254
JR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to beholdJR0405 is a splendid one to behold
Re: How do you find and fix a problem when everything goes wrong?

After reading the entire post it comes down to what you said, bad wiring. Wiring, especially in FRC, is never set it and forget it. Once you wire the robot neatly and robustly with everything labeled you’re able to avoid some failure, I’d say a good portion of what happened to you. The rest of the issues can be solved by doing a full inspection of your wiring between every match. As you learned robots get hit hard and there’s a chance of robots reaching onto other robots or even getting stuck on other robots which can damage your electronics.

You’ve already learned a few things especially making sure wires are robust before going on the robot. A few other things you can check are that all your wires are secured(Igus chain so it can’t wobble, secure and guard the ethernet cord so it can’t get unplugged, etc.). Another thing to keep in mind at competitions and in your shop is to never drill, cut, file, etc. over unprotected electronics.

These will lead to a stronger system but it all starts with getting the robot wired early. We usually wire everything on a piece of pegboard by the end of week 1.5-2 to start then start working on all our custom wiring the next week which will be eventually be rewired on the robot. This leaves a lot of time for us to take our time and make sure everything works. Not only is the robot wired and done before we bag it but like most teams we set our own stop build day or mechanical deadline so we can test code, make repairs, and get driver practice.

Lastly, your positive mentality and your eager to learn is impressive. You should be glad of the way you handled everything in the heat of competition. Hopefully your robot will be fully functional at your next event and that could be helped by doing some tests on thursday. Get as much driver practice as possible before the competition so you can spend Thursday checking every wire and don’t be afraid to simulate some of the hard hits you will see on the field. Bang and shake the robot, kick the bumpers, wiggle all your cable routes, and even run it into a wall.

If you have any questions feel free to post them or PM me.
__________________

Reply With Quote
  #4   Spotlight this post!  
Unread 03-12-2018, 09:29 AM
MrForbes's Avatar
MrForbes MrForbes is offline
Registered User
AKA: Jim
FRC #1726 (N.E.R.D.S.)
Team Role: Mentor
 
Join Date: Feb 2006
Rookie Year: 2006
Location: Sierra Vista AZ
Posts: 6,421
MrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond reputeMrForbes has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

Our electronics mentor spends quite a bit of time teaching students wiring methods...how to solder, how to determine how long to strip a wire, how to strip it carefully, how to check to make sure it is properly secured in the connectors, how to dress the wires so they are neat and out of the way of moving parts, labeling wires, etc. Make sure the bolted connections are TIGHT, but don't strip the fasteners.

We had similar fun at the 2008 Arizona regional, we had a mostly non functional robot because of some loose wiring (the power distribution system was new that year, and had many connections instead of the purpose built PDP which was introduced the following year). Our team learned a lot about wiring that year, and since then we've not had issues like you did.
Reply With Quote
  #5   Spotlight this post!  
Unread 03-12-2018, 09:42 AM
Boltman Boltman is offline
Strategy/Rules/Scouting/Volunteer
AKA: Tom Byrne
FRC #5137 (Iron Kodiaks)
Team Role: Mentor
 
Join Date: Apr 2014
Rookie Year: 2014
Location: San Diego
Posts: 1,366
Boltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant futureBoltman has a brilliant future
Re: How do you find and fix a problem when everything goes wrong?

We saw you as a scout team so we know how you performed...glad to hear the details.

I feel for you and others that had a bad week 2 experience with relaibility.... its hard to not have iffy systems when a robot is built by committee. This year game with its unique auto requirement its code , in years past it was durability, or vision.

Just FYI my lead scouts did not have you on our 28 deep pick list we make regardless of where we rank. Hope you do better next time especially with relaibility. They watched every game. Don't feel bad my scouts said we were an iffy 28.

We had software issues too as in dead after auto in practice, found out our auto code never cut off if we did not go the distance, got it sorted at tail end of practice day. Also found out a fully charged battery for some reason threw out auto off weird stuff like that.

From a scouting/strategy perspective simplicity and consistency win the day.
Last year we were #3 captain in Ventura with a $2 part added to our auto, the gear had a small lip that we could hook it on front of our ball intake (that was worthless in game) that got us a gear auto nearly every time. RP generator. Then it turned iffy in elims not the hook the H drive base driving straight. In 2017 we went from 55 rank in SD to 3 in Ventura, we are 51 this year in SD going to rank ?? in OC

This year we wasted two weeks on an H Drive (I tried to convince them west coast) and caused all sorts of headaches in build, in game 5 in San Diego we ripped out the whole center wheel suspension and dropped 30 lbs (We weighed in at 119.5) now we have room and weight to bring and add stuff in OC in our weight allowance limit) ...we gave our new Freshman driver ZERO practice with the actual bot going into SD (after our 4 year driver aged out)

Sounds like most of your stuff was self inflicted, if I were to guess the root cause your robot was too complex to give you consistency. As the team strategy mentor I preach KISS and RELIBILITY , if its iffy you will loose in eliminations (Strong hold-stuck boulder should have been fixed in build).
Stronghold again- Broke three times our shaft tour our shooter until we finally used steel had to totally rebuild it in under 20 minutes between matches, stronghold again we went with a dual layer circuit board all serviceable items on bottom layer seemed like that was a complete joke to troubleshoot....wastes time.

Even with all that I liked that robot becuse when on it performed RP collecting in an amazing way and it was mosty the field itself breaking many a bot (and vice versa), but we learned a lesson just like we did this year.

My suggestion to your team is try to make sure its Simple, Performant and Reliable in future years , try to break your bot in build and find anything iffy. The goal is to have several 10/10 systems not many less than that ones. In OC we hope to have 10/10 Elevator (we were running it at 70% in SD due to iffy intake) 10/10 Climber any direction (new) 10/10 Intake (new) 10/10 portal (new) 8/10 Auto (improved) 8/10 drivebase (same) Driver 7+/10. Well see.

When I myself can relax is when we have a "Capable Bot" (not there yet, hopefully in OC next week they will be one) in SD that intake would not let me relax.

We'll try again in OC hopefully we fix/redo our lame intake in SD and have a few more surprises to bring. Our new driver did great. So he's more confident.

EDIT: i watched your match 43 again to see how you did, just so you know I would have you as decent in your first go at full strength ,I believe you scored 5 cubes in all platforms and crossed almost parked...thats not bad. I like your tall reach. So if you can fix your issues i think you will do fine in CV . Congrats on 8th captain in SD


you are heading to Central Valley the competition up there is brutal, that was Stronghold for us and we were captain 2 with that bot we picked 973... then the stuck boulder issue and a #3 that was iffy in elims. Our iffy issue was known in build and never fixed as it was "too sporadic" and will be "ok", in competition it was an exit in elims and expected unfortunately after seeing it the day prior .

I hated our first iteration completely untested intake we had in SD before we went, rank 51

A DAY 1 BUILD MUST from an engineering perspective...CAN IT CROSS THE AUTO LINE EVERYTIME in auto? Build the base test the code THEN add more stuff without breaking that one rule and retest that once a week. Another few rules STUFF adds weight. In code STUFF adds
errors. If it may break it WILL BREAK.
__________________


Iron Kodiaks Team #5137 San Marcos, CA
2018 Orange County Elimination Alliance (5477.5209) QF
2017 Ventura Captain 3 (8, 3882) QF
2016 Central Valley Captain 2 (973, 2135) SF
2015 Ventura Elimination Alliances (696, 1836) SF
2014 San Diego Rookie All-Star Galileo Division
San Diego (Home regional/practice days) : Not chosen '18 Elimination Alliances:'17 (399, 968) SF ,'16 (1159, 812) SF , '15 (3021, 1772) QF

Volunteer @ Orange County 2016, 2017, 2018 Ventura 2018

Last edited by Boltman : 03-12-2018 at 02:53 PM.
Reply With Quote
  #6   Spotlight this post!  
Unread 03-12-2018, 09:45 AM
mikeb3173's Avatar
mikeb3173 mikeb3173 is offline
Registered User
AKA: Mike Buell
FRC #3173 (IgKnighters)
Team Role: Mentor
 
Join Date: Mar 2013
Rookie Year: 2011
Location: United States
Posts: 45
mikeb3173 will become famous soon enough
Re: How do you find and fix a problem when everything goes wrong?

Days like these can suck the fun right out of FRC.

Your San Diego experience sounds just like our time at the Central New York Regional. Although our issues were less wiring and more about a mechanically fragile acquisition system. It remains to be seen if we have it fixed for Finger Lakes this weekend.
Reply With Quote
  #7   Spotlight this post!  
Unread 03-12-2018, 09:49 AM
Rombus's Avatar
Rombus Rombus is offline
Registered User
AKA: Rick Kosbab
FRC #4188 (Columbus Space Program)
Team Role: Mentor
 
Join Date: Feb 2005
Rookie Year: 2005
Location: Opelika, AL
Posts: 354
Rombus is a name known to allRombus is a name known to allRombus is a name known to allRombus is a name known to allRombus is a name known to allRombus is a name known to all
Send a message via AIM to Rombus
Re: How do you find and fix a problem when everything goes wrong?

This is a great write up! If you don't mind, let me share some of what went though my mind as I read your post.

“Let’s work the problem, people. Let’s not make things worse by guessing.” For those who don't know, this quote is from Gene Kranz, NASA Flight Director for Apollo 13. Its simple and concise, and quite frankly something I always keep in my mind when I troubleshoot. What is the problem telling you? Focus on one problem at a time and work it!

Another famous quote I love to share is from Doug Adams' "Hitchhikers Guide to the Galaxy". Quite simply its "Don't Panic". It's easy to panic, but if you can take a step back and breath, sometimes you can solve problems quicker. Even if everyone is panicking around you, being that one person not in a panic state can calm everyone down.

In my industry (Cable TV/ISP) we have terms for the troubleshooting methods you used. The first is "Shotgun" troubleshooting and the second is Part Swap troubleshooting.

Shotgun troubleshooting is fixing things at random, usually multiple things at once, until you get a desired result. If your lucky and actually solve your problem, then you have no idea what exactly was wrong. If your unlucky, you are likely to cause other issues due to loose cabling, or swapping out a good part for one that may be defective.

Part Swap Troubleshooting is swapping out major components until you get the desired results. You said "we assume the Spartanboard we’re using broke sometime in the last 15 minutes." Did you check it in any way? Is there a reason it would have broken in those 15 min? Also you kept swapping it throughout the competition, so you did not know for sure it was bad. This goes back to the "shotgun" method of troubleshooting, you don't know for sure the problem, your just trying everything to fix it.

Now don't get me wrong, Its very easy to panic, its very easy to fall back into those methods of troubleshooting. I've been working in my industry for 10+ years now, and I still find myself doing it at times. However what typically works best for me is something we call "Halving the Problem". Think of everything on your robot as a chain, here is an example going off your arm sensor value: (For my discussion, I'm going to assume you used a potentiometer as the sensor)

Battery > 120Amp Breaker > PDP > RIO Fuse > Rio > Spartanboard > Potentiometer
(The > indicates wire, or a circuit path)

So in this case, lets say your not getting valid data from the pot. Is the rest of the robot working? The RIO seems to be OK, so we can limit our troubleshooting to this half of the robot:
Rio> Spartanboard > Potentiometer

So my next step would be to bust out a DMM and check the pot at the pot itself. are you seeing a resistance change? Anything acting funny? Something shorted that shouldn't be? If the Pot itself is OK, work backwards. Do the same checks at the input to the Spartanboard coming from that POT. If your seeing the Pot as OK on both ends of the wire, then start checking the board itself. Are you getting proper voltage? Do other sensors work? If you grab a known good sensor and plug it in, does it work on that port?

The main point is to split the problem as much as possible at known good points, then focus in on the cause when you cant split it any more. This way you KNOW you found the source of the problem and fix it.

Good training during the off-season is to have someone break the robot (Remove a wire, fuse, reverse wiring, ect) then try and fix it. FTAs actually do this every year as part of their training. The point is to have the troubleshooting process become second nature so that you can just do it and not think about it.

Also after the season, reread your post a few times, especially going into next season to keep it fresh in your mind, its too easy to forget valuable lessons and basics when the next new game comes out!
__________________

Reply With Quote
  #8   Spotlight this post!  
Unread 03-12-2018, 09:53 AM
fargus111111111's Avatar
fargus111111111 fargus111111111 is offline
ME who pretends to code
AKA: Tim W
FRC #0343 (Metal in Motion)
Team Role: Mentor
 
Join Date: Nov 2014
Rookie Year: 2010
Location: Clemson, that football-y place
Posts: 174
fargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud offargus111111111 has much to be proud of
Re: How do you find and fix a problem when everything goes wrong?

Wow, wow, wow.

Well, I can't say 343 has ever encountered issues so major as these, and I hope we never do but yes, wiring is a huge aspect of the robot running properly and if we are getting odd sensor readings typically one of our first checks is, is everything plugged in correctly and if so check continuity of the wires.

One other thing you might check is if the casing of your sensors are grounded. I would think this would have shown up during inspection as a ground to frame, but it may be worth checking at least.

Also 343 has recently moved away from potentiometers to encoders as we do find them to be a little more stable, though a slightly larger hassle due to needing to "set" them each time the robot powers on.

One thing you might be able to do in the future to mitigate this type of issue is to redo the wiring (time permitting) during build season at least once for the express purpose of cleaning it up. The year we had the most electrical problems recently was the year we did not have time to do this. I understand this task can (and rightfully should) make its way quickly to the bottom of the priority list during build season however the benefits of doing this, both on the trouble shooting and reliability fronts are massive.
__________________
I didn't break it... this time.

play is the highest form of research-Einstein
Reply With Quote
  #9   Spotlight this post!  
Unread 03-12-2018, 09:57 AM
gerthworm's Avatar
gerthworm gerthworm is offline
Making the 1's and 0's
FRC #1736 (Robot Casserole)
Team Role: Mentor
 
Join Date: Jan 2015
Rookie Year: 2015
Location: Peoria, IL
Posts: 744
gerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond reputegerthworm has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

Awesome job on maintaining attitude! Situations like these are immensely frustrating in a shop, and downright terrifying on the field. You and your team have learned some valuable lessons already, and this is truly what is important! But, having a working robot is also rewarding. With that in mind...

The "stacked failure" is by far one of the worst possible things to have to debug. Multiple broken parts makes identification extraordinarily hard. The best technique I've learned over the years is to isolate parts and test them independently to confirm if each is behaving on its own properly. Unit test first, then integration test.

In the particular case of a potentiometer, we had some similar woes earlier this year, ended up swapping it with limit switches. However, some debugging steps we took:

1) Totally unsolder the pot from any wiring. Just the sensor with 3 leads sticking out. Verify that pins 1/3 have the specified resistance with an ohmmeter. Verify pin 1/2 and 2/3 resistance changes with position in a reasonable matter. If this doesn't pass, the pot itself is at fault. Replace/repair until it works.

2) Re-connect the wiring up to the roboRIO, but don't plug it in. With the robot powered off, repeat the resistance test at the connector that would be plugged into the RIO. If this doesn't pass, there's an issue with the wiring between the pot and the RIO. Work through this until it works.

3) Power-on the robot, and repeat the resistance test. This should produce the same results. If not, look for unexpected voltages between robot ground and any of the pins. (expect open circuits between all). This will reveal further problems with wiring. Work on the wiring until this passes.

4) Plug the sensor into the port on the roboRIO. Verify the RIO power light stays green (red indicates issues). With a multimeter, Check for 5V between pins 1/3 on the connector right at the RIO. Check that the signal pin changes voltage with arm position as expected. Debug wiring until this is working properly.

5) Set up your software to display exactly the voltage that the RIO sees at the pin. Verify this voltage matches the multimeter within only a few percent error, and is stable. Work on your software until this is the case.

6) Shake the robot around, wiggle wires, sweep the arm position, etc. Verify all these will always return a stable, correct voltage value in software. If any of these start to cause faulty values, work your way back up this debugging chain to find the cause(es) of the intermittent signal.


This is a specific example, but the key starting from 1 is that the sensor itself is totally isolated from the system, and tested independently (with a multimeter). Only after confirming the sensor is good and robust do you go on to step 2, where you add one more layer of complexity - some of the wiring. If behavior is still good, you know the wiring isn't at fault either. If something starts to go wrong, you can conclude the issue must lie between the output of the sensor and the measurement point. This greatly reduces the amount of searching you have to do. You mentioned swapping the spartan boards - I may have missed something, but I would have done all the sensor debugging first before attempting to swap these boards - changing a variable without knowing how it's impacting the system exactly tends to add chaos, rather than resolve it (at least as far as I've seen).

The other gyro or software issues you mentioned are hopefully independent of this . This is a key mindset to keep, both with yourself and with your team lead pressing you for results right now. . Keep focus, and solve your issues systematically, one at a time. Worrying about all of them at once won't suddenly make them all go away. Does this realistically mean you won't be able to solve all your problems in the alotted time? Perhaps. This is when you set priorities, choose your most-important issue to solve, and start there.
Reply With Quote
  #10   Spotlight this post!  
Unread 03-12-2018, 12:16 PM
Basel A's Avatar
Basel A Basel A is online now
It's pronounced Basl with a soft s
AKA: @BaselThe2nd
FRC #3504 (Girls of Steel)
Team Role: College Student
 
Join Date: Mar 2009
Rookie Year: 2009
Location: Pittsburgh, PA
Posts: 2,294
Basel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

Lots of good troubleshooting tips and processes in the earlier replies. One particular decision you made is the one that gets me: recalibrating to 0-1.5v. "Fixed" is the only possible stable state. Not sure what that problem was, but I would never expect it to stay at a particular range that isn't the correct range.

Ironically, it was that decision which allowed you to perform at full functionality for at least one match. Still, it's a duct tape solution that you had to know would break.
__________________
UMich '17, Carnegie Mellon '18
Team 2337 | 2009-2012 | Student
Team 3322 | 2014-2017 | College Student
Team 3504 | 2018-Present | Grad Student / Engineer
Reply With Quote
  #11   Spotlight this post!  
Unread 03-12-2018, 01:41 PM
philso philso is offline
Mentor
FRC #3103 (Iron Plaid)
Team Role: Mentor
 
Join Date: Jan 2011
Rookie Year: 2009
Location: Houston, Tx
Posts: 1,731
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

Quote:
Originally Posted by Rombus View Post
This is a great write up! If you don't mind, let me share some of what went though my mind as I read your post...

Good training during the off-season is to have someone break the robot (Remove a wire, fuse, reverse wiring, ect) then try and fix it. FTAs actually do this every year as part of their training. The point is to have the troubleshooting process become second nature so that you can just do it and not think about it.

Also after the season, reread your post a few times, especially going into next season to keep it fresh in your mind, its too easy to forget valuable lessons and basics when the next new game comes out!
Yes. Do this. At my previous job, I had to take the class for the Field Service Technicians for some of the equipment we manufactured. Knowing that you will have to fix it made people take the training more seriously. When we did the trouble shooting exercise, we found 5 faults instead of 3. I suspect that some of the other Trainers found out there were R&D staff in the course and gave us "bonuses" during the lunch break.

In the future, your team would also benefit from labeling all of your wiring and pneumatics. Two or three letter acronyms are sufficient. For an example, we put "LF" on the left front motor, all the connectors for that motor, on the motor controller, on the connectors between the motor controller and the PDP and on the PDP for that channel.

Last edited by philso : 03-12-2018 at 01:44 PM.
Reply With Quote
  #12   Spotlight this post!  
Unread 03-12-2018, 02:36 PM
FrankJ's Avatar
FrankJ FrankJ is online now
Robot Mentor
AKA: The Bearded ONe
FRC #2974 (WALT)
Team Role: Mentor
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Marietta GA
Posts: 2,549
FrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

A side story about pots. We had a pot that tested fine with a DVM multiple in ways. It gave bingo numbers to the roborio. Replacing the pot and wiring fixed the problem. We got sidetracked and never did a root cause analysis.

Have all your IO fed to smart dashboard. (The closer to raw the better) Makes it easy to check to see if it is working as expected.
__________________
If you don't know what you should hook up then you should read a data sheet
Reply With Quote
  #13   Spotlight this post!  
Unread 03-12-2018, 03:59 PM
Grim Tuesday's Avatar
Grim Tuesday Grim Tuesday is offline
Registered User
AKA: Simon Bohn
FRC #0639 (Code Red)
Team Role: Alumni
 
Join Date: Jan 2010
Rookie Year: 2010
Location: Rockville, MD
Posts: 1,635
Grim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

Everyone is giving great troubleshooting tips here but I think the real lesson is to take these experiences into account when designing next year's robot. Don't make it rely on sensors and a lookup table to keep a wrist in bounds. I went there in my time in FRC and it is hell. Make it mechanically impossible to get a penalty, even if you have to sacrifice something else from the design. If it makes your design impossible, so be it. Make a new design.

Sensors are wonderful, but the less complex type of sensor you can use and the more sensor-insensitive (ha ha) you can make your design the better off you will be when your sensors inevitably fail.
Reply With Quote
  #14   Spotlight this post!  
Unread 03-12-2018, 04:36 PM
SamCarlberg's Avatar
SamCarlberg SamCarlberg is offline
GRIP/Shuffleboard/WPILib. 2084 alum
FRC #2084
Team Role: Mentor
 
Join Date: Nov 2015
Rookie Year: 2010
Location: MA
Posts: 358
SamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant futureSamCarlberg has a brilliant future
Re: How do you find and fix a problem when everything goes wrong?

Quote:
Originally Posted by FrankJ View Post
Have all your IO fed to smart dashboard. (The closer to raw the better) Makes it easy to check to see if it is working as expected.
In Java and C++ (not sure about LabVIEW), every sensor and actuator you instantiate automatically gets logged to the LiveWindow and can be viewed in Shuffleboard or OutlineViewer.
__________________
WPILib developer
GRIP, Shuffleboard, RobotBuilder, OutlineViewer
Reply With Quote
  #15   Spotlight this post!  
Unread 03-12-2018, 05:25 PM
s-neff's Avatar
s-neff s-neff is offline
Registered User
AKA: Sam
FRC #0841 (Biomechs)
Team Role: Mentor
 
Join Date: Apr 2016
Rookie Year: 2009
Location: California
Posts: 256
s-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond reputes-neff has a reputation beyond repute
Re: How do you find and fix a problem when everything goes wrong?

That sounds pretty maddening. We had a similar "fun time" at SFR last year after bagging a... hastily assembled... base. Ended up redoing the entire wiring harness in the pits trying to find an intermittent short that was killing our radio.

Assembly quality is everything.

This year, everything is crimped to either a ferrule or connector, we have strain relief within 2" of every connector, electrical tape across each connector, and additional wireties along each path in the drivebase "skateboard" so we don't get cables jumping around during competition getting places they shouldn't.

There's a layer of manipulator wiring that doesn't have quite as clean a set of wire paths, but it's all fully strain relieved & taped up.

All our on-field issues at Utah were due to not letting the programmers get on the 'bot until the end of Practice Day this year... zero electrical faults.
__________________
Mentor - Team 841 (2015 - ? )
Mentor - Team 4 (2013 - 2014)
Student - Team 192 (2009 - 2010)
"Remember why you're doing this." -mrnoble
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 10:22 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin®
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi