Go to Post FIRST is hard. It's supposed to be. We can't keep giving people a check and expecting them to succeed. We need to stop this model. It takes a village to raise a child but it takes a community to build a FIRST team. - Andrew Schreiber [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
 
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 16-03-2012, 23:06
RyanN's Avatar
RyanN RyanN is offline
RyanN
AKA: Ryan Nazaretian
FRC #4901 (Garnet Squadron)
Team Role: Mentor
 
Join Date: Jun 2006
Rookie Year: 2005
Location: Columbia, SC
Posts: 1,126
RyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond repute
Team Fusion #364, Bayou Regional, FMS Woes

I'm not sure how many people have had issues at the Bayou Regional, but we haven't been able to have a single good match at the Bayou where we haven't disconnected from FMS. They verified that our router remains connected, but they are saying our cRIO keeps dropping off.

We have replaced everything, and I mean everything regarding the control system. Electrically we're good. Our best guess is that the cRIO is crashing mid match.

Here's the thing we cannot get past. Our code works perfectly tethered and on the practice field using wireless. We can beat the heck out of our robot, and it remains connected. CPU usage hovers around 40%, memory around 40%, we have tons of flash space free. When we put our robot on the field, we're always the last to connect to the FMS, we've had it drop and reconnect multiple times while the field is being setup. The field management bypassed us one match and told us we couldn't play because they couldn't keep a connection.

My question is why? I know we weren't the only ones with the issue today. Kryptonite had the same problem during the same match we were in one time. Another team came to talk to FTA after the award ceremony about the same issue. FTA still claims that they don't see any issues with their system. Maybe they don't see any, but if our code runs perfectly everywhere but the field, that makes me believe there is a problem on the side of the code we cannot touch (cRIO Image, and FMS).

We can sometimes make it through autonomous, and during the match, our robot will come in and out of communication. We finally remained connected for at least half match in our last match of the day, and we did great.

I hate to see our kids' work that they spent 6 hard weeks working on go to waste because of a dumb bug or glitch with the cRIO image.

I know there was a team update on 'lightening' code, but we're not running anything more complicated that what we ran in 2008 on a PIC18 microcontroller, which is way less powerful than the cRIO we're given today. We tried lightening the code during the day between matches, and we finally finished right at 7PM today. We have one match left, and it's the only match we can try to redeem ourselves. We have no closed loop control, and I'm calling it our dumb code, because all it does is take joystick output and apply it to motors. Nothing neat, no safety interlocks, no special features. All of the hard work we worked on is useless now because of a problem that we cannot diagnose outside of the FMS system.

Honestly, I'm exhausted. We've spent 2 days now working on this problem. Our CTA, MJ, looked through all of our code, witnessed it working in the pit, and agreed with us that our code works. We have sat in every match so far. We have a very capable robot, and we know it. It's so hard to accept that we basically missed every match this year for a 12 year old team.

Thanks,
Ryan N

I would also like to add that I have over a year of professional LabVIEW experience. Our code can be found here: http://www.chiefdelphi.com/forums/sh...d.php?t=103902
__________________
Garnet Squadron
FRC 4901
Controls Mentor
@rnazaretian

Previous mentor and student from Team Fusion, FRC 364

Last edited by RyanN : 16-03-2012 at 23:17.
  #2   Spotlight this post!  
Unread 16-03-2012, 23:27
Mr. Lim Mr. Lim is offline
Registered User
AKA: Mr. Lim
no team
Team Role: Leadership
 
Join Date: Jan 2004
Rookie Year: 1998
Location: Toronto, Ontario
Posts: 1,125
Mr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond reputeMr. Lim has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?

Is your radio mounted near motors, and anything else that could cause interference?

Is it buried deep within the robot, possibly shielded from the wireless signal?

Was the router used a lot last year? A bit beaten up? Could any foreign material have fallen into it? Could it just be dirty inside?

Is the power connector to the radio nice and tight? Hot glue, rubber bands or velcro straps could help here.

These are all a bit of a reach, but at this point, I'm sure you're willing to try anything!
__________________
In life, what you give, you keep. What you fail to give, you lose forever...

Last edited by Mr. Lim : 16-03-2012 at 23:37.
  #3   Spotlight this post!  
Unread 16-03-2012, 23:32
Mk.32's Avatar
Mk.32 Mk.32 is offline
Registered User
AKA: Mark
FRC #2485 (W.A.R. Lords)
Team Role: Engineer
 
Join Date: Jan 2011
Rookie Year: 2011
Location: San Diego
Posts: 770
Mk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud ofMk.32 has much to be proud of
Re: Team Fusion #364, Bayou Regional, FMS Woes

Is the router clear of motors and things that might interfere with it? We had to move ours because FMS didn't like the fact we put it low, near the drive motors.
Try a new cRIO? Image it?
Not really sure what it could be just throwing out some ideas. Issues like this always suck, I remember FMS costing us a match or two last year.
Good luck to your team, and I hope you guys get it sorted.
__________________
Engineering mentor: Team 2485: WARLords 2013-

Team President: Team 3647 2010-2013
  #4   Spotlight this post!  
Unread 16-03-2012, 23:39
RyanN's Avatar
RyanN RyanN is offline
RyanN
AKA: Ryan Nazaretian
FRC #4901 (Garnet Squadron)
Team Role: Mentor
 
Join Date: Jun 2006
Rookie Year: 2005
Location: Columbia, SC
Posts: 1,126
RyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

Quote:
Originally Posted by Mr. Lim View Post
I know this is probably obvious, but is there any chance your radio is in Auto or AP mode?

Is your radio mounted near motors, and anything else that could cause interference?

Is it buried deep within the robot, possibly shielded from the wireless signal?
Our radio is connected below our shooter motors, but we're losing connection while disabled as well, so this is not caused by EMF.

Quote:
Originally Posted by Chinmay View Post
Dear Ryan

First off, I'm really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.

I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.

We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. (Wireless has been off.)We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven't tried it, although I'm sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year!

Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I'll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn't something to do with the crio rebooting. We did notice that we were dropping packets.

The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we'll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.

Here's what we have replaced:
  • cRIO
  • Digital Sidecar
  • Dlink Router
  • Dlink Power Cable
  • Ethernet Cables

I would strongly guess that the problem was a connection, but we're losing communication while disabled and everything. We can run wireless in the practice field and it runs perfectly.


Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.

Once again, I'm really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!
I will update tomorrow after our match if we run or not. I have a bad feeling that it has been the code the entire time. No reason the code is failing, just that it's too complicated for the FMS to handle for some dumb reason. I know it doesn't make sense, but there's nothing else I can think off as to why it works great everywhere except the field.
__________________
Garnet Squadron
FRC 4901
Controls Mentor
@rnazaretian

Previous mentor and student from Team Fusion, FRC 364
  #5   Spotlight this post!  
Unread 17-03-2012, 03:30
GrimmReaper's Avatar
GrimmReaper GrimmReaper is offline
Registered User
FRC #2046 (Bear Metal)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2011
Location: Maple Valley, WA
Posts: 24
GrimmReaper is on a distinguished road
Re: Team Fusion #364, Bayou Regional, FMS Woes

I'm also at a loss for what could be the cause of your communications issue. I'd be curoious though if you're able to replicate the same issue in the pits while tethered and running FMS light?

you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx

direct link to the download: http://208.82.132.46/download/fmslight.setup.msi

i've had good luck with this version on windows 7 but not some of the others... i don't know if it will exhibit the same issues when tethered and with this version of FMS, but it might be a worthwhile test... (quick setup = install, set your ether ip to 10.0.0.5, open FMS, click the big reset button(seems to need it every time you open it fresh), then put your team number on one spot and bypass the others, DS should show FMS control and you should be able to start the match from the FMS panel)

if by chance you ARE able to replicate with FMS tethered in your pit, then immediately try a simple example code if that has issues then it should be something electrical/physical still... if that code is fine but your own code is not - you know where the problem must lie...

hope that helps - good luck and keep us posted.
- Jon
  #6   Spotlight this post!  
Unread 17-03-2012, 05:05
techtiger1's Avatar
techtiger1 techtiger1 is offline
Coach Drew?!?!
AKA: Drew Disbury
FRC #1251 (TechTigers)
Team Role: Mentor
 
Join Date: Jul 2004
Rookie Year: 2004
Location: coconut creek ,fl
Posts: 629
techtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond reputetechtiger1 has a reputation beyond repute
Send a message via AIM to techtiger1
Re: Team Fusion #364, Bayou Regional, FMS Woes

Make sure your not getting something called a watchdog error. There is a way to disable it in the code. I am not by any means a programmer but this has happened to my team before. The symptoms you describe are pretty accurate to what was happening with us. Good luck.
__________________
Team 1251 The TechTigers
"Inspiring future innovators, one stripe at a time"
2004 Rookie All Star Orlando Regional
2006 Palmetto Regional Winner
2007 Orlando Regional Winner
2008 Orlando Regional Winner
2010 Orlando Regional Winner
2013 South Florida Regional Winner
  #7   Spotlight this post!  
Unread 17-03-2012, 05:58
Jack Jones Jack Jones is offline
Retired
no team
 
Join Date: Jan 2003
Rookie Year: 2003
Location: Waterford, MI
Posts: 964
Jack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond reputeJack Jones has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

I've been seeing these symptoms alot, each comes with a smorgasbord of possible causes, and any number of eventual outcomes. There were many when the system was new to us in '09, but there seems to be an increase this year.

One thing I've seen in common is that the electronics were mounted to polycarb sheet. Both '09 and this year the robots are like vandergraph generators (fast moving belts within an isolated shell). Is it possible for a polycarb surface to build up static until the conditions are ripe for it to discharge?
__________________
This message is hidden because Jack Jones is on your ignore list.
  #8   Spotlight this post!  
Unread 17-03-2012, 09:00
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,559
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

It sounds like we might be having a similar issue. We've extensively replaced components and code, without a solution (yet). http://www.chiefdelphi.com/forums/sh...hreadid=104713
  #9   Spotlight this post!  
Unread 17-03-2012, 09:13
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,748
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

Can I get DS log files from each of you emailed to me. Also can you include a description of the cRIO, the DLink radio particulars including the firmware version. I have your code Ryan. I'm not sure if that is having any impact.

I was at a week one and two regional and haven't seen this. Comms issues have typically been firewall, IP, or electrical issues. I've also seen the issues go away when the radio was reprogrammed or swapped out. I wish I knew what was causing those.

PM me for my gmail account, though you can probably guess what it is.

Greg McKaskle
  #10   Spotlight this post!  
Unread 17-03-2012, 09:55
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,559
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

We've tried 3 radios, all rev A. Two with firmware version 1.21 (one was ours and one from spare parts) and one with firmware version 1.4 (purchased last month). We've tried 2 cRios, both have been 4 slot cRios (no room for 8 slot).
  #11   Spotlight this post!  
Unread 17-03-2012, 11:03
DonRotolo's Avatar
DonRotolo DonRotolo is offline
Back to humble
FRC #0832
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2005
Location: Atlanta GA
Posts: 6,979
DonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

At Rutgers our team, along with 75 and 41 all had a similar problem to that described above, except that none of us could connect to the field at all.

Long story short, the automatic radio configuration system was not configuring the radio properly. When we configured it manually (with the explicit approval of the FTA) we were good.

Perhaps checking the radio configuration for oddnesses might help?
__________________

I am N2IRZ - What's your callsign?
  #12   Spotlight this post!  
Unread 17-03-2012, 13:42
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,748
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: Team Fusion #364, Bayou Regional, FMS Woes

Do you know what setting was not good?

Greg McKaskle
  #13   Spotlight this post!  
Unread 17-03-2012, 21:07
AllenGregoryIV's Avatar
AllenGregoryIV AllenGregoryIV is offline
Engineering Coach
AKA: Allen "JAG" Gregory
FRC #3847 (Spectrum)
Team Role: Coach
 
Join Date: Jul 2008
Rookie Year: 2003
Location: Texas
Posts: 2,549
AllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond reputeAllenGregoryIV has a reputation beyond repute
Send a message via AIM to AllenGregoryIV
Re: Team Fusion #364, Bayou Regional, FMS Woes

Quote:
Originally Posted by GrimmReaper View Post
I'm also at a loss for what could be the cause of your communications issue. I'd be curoious though if you're able to replicate the same issue in the pits while tethered and running FMS light?

you can find a working fms light installer and info here http://208.82.132.46:81/2009PostSeasonReleaseNotes.ashx

direct link to the download: http://208.82.132.46/download/fmslight.setup.msi

- Jon
The links to FMSlight don't seem to be working does anyone have new ones? I would love to test our control system before our event in two weeks.
__________________

Team 647 | Cyber Wolf Corps | Alumni | 2003-2006 | Shoemaker HS
Team 2587 | DiscoBots | Mentor | 2008-2011 | Rice University / Houston Food Bank
Team 3847 | Spectrum | Coach | 2012-20... | St Agnes Academy
LRI | Alamo Regional | 2014-20...
"Competition has been shown to be useful up to a certain point and no further, but cooperation, which is the thing we must strive for today, begins where competition leaves off." - Franklin D. Roosevelt
  #14   Spotlight this post!  
Unread 16-03-2012, 23:29
Chinmay's Avatar
Chinmay Chinmay is offline
incurable FIRSTaholic
FRC #0846 (Funky Monkeys)
Team Role: Mentor
 
Join Date: Dec 2008
Rookie Year: 2008
Location: San Jose Ca
Posts: 164
Chinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant futureChinmay has a brilliant future
Re: Team Fusion #364, Bayou Regional, FMS Woes

Dear Ryan

First off, I'm really sorry to hear that you and your team had so many difficulties this week at the regional. I sincerely hope that you can have a strong showing in your last match and get the chance to compete in the eliminations.

I would love to hear more about the technical problems you experienced, and hopefully offer a solution. I know you may have already tried what I will suggest below, but I hope you can check this message before tomorrow and that something in my post may solve your problems.

We had this issue last year at chesapeake regional, and it was ultimately fixed by turning off the wireless on our classmate pc. We had intermittent connectivity, much like what you are describing, and then when the wireless was turned off, the robot worked fine. Our symptoms were much like yours, bad connection, and losing coms for almost the entire match. That may be a quick fix if you haven't tried it, although I'm sure you guys have gone through the motions and thoroughly checked everything, and it seems like your router is staying connected, but I wanted to throw this out there. It took us a whole day of trying to work this out last year!

Today, at NYC regional, we had a similar problem which we now have solved. To begin with, I'll describe the problem. We would have coms (albeit after much longer than it normally takes to connect to the field) and then promptly lose communication as soon as we began to drive our robot in tele-op. We were not doing anything fancy that would drop our voltage low enough to reboot the crio ( which would result in about a 45s delay) and we would get erratic times for the loss of communication ranging from 15 seconds, to longer than a minute. This suggested that the problem wasn't something to do with the crio rebooting. We did notice that we were dropping packets.

The FTA, after we spent the day trying different things, and swapping out components, suggested we replace all the wiring to/from the wireless bridge, in our case, specifically an ethernet cable from the bridge to the 2Can. I believe doing this solved our problem, I guess we'll see for sure tomorrow if it worked, but we were finally able to connect successfully. Like you guys, both years we were perfectly fine running tethered.

Let me know if this works for you tomorrow. I can type up a more detailed report, including specifics on what we experienced if you send me a PM.

Once again, I'm really sorry to hear that the communications issues caused you guys to not be able to show what your robot can really do. Best of luck!
__________________
Member 846 Funky Monkeys 2007-2011
Mentor 558 Robosquad 2012
Mentor 846 Funky Monkeys 2011-Present

http://www.becauserobots.org
Man is a robot with Defects
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 16:08.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi