OCCRA
Go to Post Ooohh!! Fire!! - dlavery [more]
Home
Go Back   Chief Delphi > Technical > Electrical > CAN
CD-Events   CD-Media   CD-Spy   FRC-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 04-29-2012, 10:42 AM
steelerborn's Avatar
steelerborn steelerborn is offline
Creo Mentor, Treasurer Tau Beta Pi
AKA: Jonathan Stokes
FRC #1671 (Buchanan Bird Brains)
Team Role: Mentor
 
Join Date: Jan 2009
Rookie Year: 2007
Location: Clovis
Posts: 191
steelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud ofsteelerborn has much to be proud of
CAN on the entire 2012 robot

FIRST team 1671 the Buchanan Bird Brains used CAN on the entire robot, I know our electrical mentor went around in the pits at nationals and was surprised to see the lack of CAN and the Jaguars in general (We used the Black Jags). Just wondering if other teams that used all CAN could comment on their 2012 CAN experience, and why did some teams not run CAN and the Jaguars. From what I understand running CAN helps by letting the Jags run some of the loops and not in the C-rio which helps with processing power, maybe this could be related to some of the FMS and Communications issues on the field. Just a thought.
__________________
Good enough is the enemy of anything great!
Reply With Quote
  #2   Spotlight this post!  
Unread 04-29-2012, 11:13 AM
duggerd duggerd is offline
Registered User
AKA: Daniel Dugger
FRC #0180 (SPAM Robotics)
Team Role: Programmer
 
Join Date: Apr 2012
Rookie Year: 2011
Location: Palm City, Florida
Posts: 1
duggerd is an unknown quantity at this point
Re: CAN on the entire 2012 robot

The problem with using Jaguars is that if you put too much load on it, or change motion, or even change direction quickly, they reset and take a while to reboot and do not re init with the cRIO. This has killed our robot in the past and, even we had problems with it this year. In our Gallileo division we has a team run over our flipper witch ran on a jag, and when it was going up and applied too much force and reset our jag and killed the flipper for the remainder of the match. Using a jag does not affect your field connection issues. That is your bridge and their FMS.
Reply With Quote
  #3   Spotlight this post!  
Unread 04-29-2012, 11:55 AM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 146
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: CAN on the entire 2012 robot

We at team 585 used CAN exclusively, at first. At our first regional in Seattle, we had some connectivity issues, and a few mentors from other teams were adamant that we had too many jaguars on our CAN bus. We use a 2CAN, and these mentors ignorantly (and belligerently) believed that the hard limit of CAN jaguars was only four nodes. They also suggested that using CAN was pushing our CPU utilization to 100%. Our small onsite team was pushed by them into changing four over to PWM.

In fairness, the problem in reality was the lack of robustness in the RJ-11/12 on the jaguars. Losing a PWM connection only takes out that single motor. Breaking the daisy chain on CAN takes away the termination, and usually loses everything. When this happens, acknoledgements don't come back to the cRIO. This may hang up the networking code on the cRIO. I'm only speculating on this last part.

Between Seattle and Las Vegas, our students were split on whether to go back to all CAN. The student who did the most CAN specific work decided to go back to all CAN. I supported her reasoning. We went through our LabView project, and combed everything. We enabled reentrant code, disabled debug code, and enabled in-lining on every sub-vi we could. This greatly optimized our code.

We also put together a backbone for the CAN bus, that had high quality RJ-12 modular connectors attached to continuous wires, similar to this. This changes the topology to a star, instead of a daisy chain. Our backbone had three modules with four ports each, rather than the two rows of six.

In Las Vegas, we were back on all CAN. We had found that one of our older beige jaguars had changed its address after a few matches (thoroughly befuddling). By the end, everything was working great. All 10 of our jaguars were working without issue on CAN. We did have a few communication drops, but I'm unconvinced that CAN had anything to do with that. Bad luck had our two final qualification matches against the thoroughly impressive one-robot alliance that is the 987 High Rollers.

Our robot shipped to the championship with everything working. When our team unpacked it, our shooter no longer worked. We found (again) that it was the RJ-11/12 connection on the Jaguar. They got it working pretty quick, as they had seen this problem enough, by now. Had our best performance all season at the championship, balancing every match, I'm told.

-- Len

Last edited by Levansic : 04-29-2012 at 12:01 PM.
Reply With Quote
  #4   Spotlight this post!  
Unread 04-29-2012, 04:17 PM
gracie.'s Avatar
gracie. gracie. is offline
forgot the heat shrink...
FRC #2399 (The Fighting Unicorns)
Team Role: Alumni
 
Join Date: Jan 2012
Rookie Year: 2011
Location: Cleveland, OH
Posts: 92
gracie. is an unknown quantity at this point
Re: CAN on the entire 2012 robot

We had a lot of success with CAN this year. At first, it took a while for the programmers to sort out the needed modifications to the code, but after that it made things a lot easier from a wiring standpoint. We had 11 motor controllers this year, but our team only had 6 jags that were fit to put on the robot. So we ended up using PWM anyways for the 5 victors needed to fill the gap. However, CAN made my life a lot easier when putting the board together, and made it so we didn't have to worry about cables falling out on the field. We didn't have any connection issues from it. It was great!
__________________
2399: The Fighting Unicorns! - Alumna | Dean's List Finalist 2013
Reply With Quote
  #5   Spotlight this post!  
Unread 04-29-2012, 05:20 PM
mikets's Avatar
mikets mikets is offline
Software Engineer
FRC #0492 (Titan Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Bellevue, WA
Posts: 596
mikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of light
Re: CAN on the entire 2012 robot

We have 9 motor controllers on our robot and 7 of them are Black Jags. The other two are Victors just because we ran low on space. We don't have any problems with them at both regional and championship. There are a lot of posts on this topic. For example, one of them is:
http://www.chiefdelphi.com/forums/sh...d.php?t=103977
__________________
Reply With Quote
  #6   Spotlight this post!  
Unread 04-29-2012, 07:09 PM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 146
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: CAN on the entire 2012 robot

One thing that did frustrate us, was the ease of going into a protected stop condition. Not really sure what to call the state, but we hit it often when trying to optimize our bridge tipper, while using closed loop current control.

Stall currents generally follow Ohm's law, since the motor armatures are not moving, at least when in steady state. We had several trips of the jaguar, before hitting 20 Amps (24 A stall current at 12V), and never saw voltages approaching the battery voltage. At lower currents, performance was reliable, but not enough torque for what we needed.

Now, we dont have an oscilloscope, so the back current spikes due to rapid voltage increases may have been huge, but the jaguar would sometimes failsafe out before there was any discernable movement in the attached motor. If we decreased our PID gains, the jaguar would not source enough power to get the motor moving. Not sure this is clearing much up, but we found the tuning to be finicky, and not robust when the temperature of the attached motor changed. If there was filtering on the feedback signal, or if a voltage output ramp rate could be enforced along with the PID, then this problem could be minimized or eliminated.

There was an earlier thread on PID speed control here. We thought we could tame that beast, but the fast sampling rate on the encoder, coupled with lots of backlash in our tough box nano gear boxes, made the feedback too noisy, and the control loop far to unstable to use. Not sure if there is any filtering on the feedback signal, in the jaguar PID code, but there should be.

We intend to do a lot of testing and characterization work this summer.

-- Len
Reply With Quote
  #7   Spotlight this post!  
Unread 04-30-2012, 12:07 AM
taichichuan's Avatar
taichichuan taichichuan is offline
Software Mentor
AKA: Mike Anderson
FRC #0116 (Epsilon Delta)
Team Role: Mentor
 
Join Date: Feb 2010
Rookie Year: 2010
Location: Herndon, VA
Posts: 252
taichichuan is a glorious beacon of lighttaichichuan is a glorious beacon of lighttaichichuan is a glorious beacon of lighttaichichuan is a glorious beacon of lighttaichichuan is a glorious beacon of light
Send a message via AIM to taichichuan
Re: CAN on the entire 2012 robot

Team 116 had a mixture of CAN and PWM on this year's robot. Jaguars on the ball guide and shooter motors (one on each side) and Victors to run the wheels due to size constraints and the annoying habit of the Jaguars resetting when the drivers do a full forward to full reverse thing. You could address that with motor ramping, but we wanted to give our drivers the ability to do whatever they felt was necessary to drive the robot on the field.

2010 and 2011 were pretty much all CAN except where there wasn't enough room to mount a Jaguar and a Victor was used. We had a couple of issues in 2011 based on a bad Jaguar firmware release. But, those problems appear to have been addressed with firmware 101.

However, to many, the Jaguars are damaged goods. Problems with aluminum falling into the Jags and shorting them out, problems with resets because of rapid full forward/full backward control movements, their sheer size, the relative ease with which students (and mentors) could reverse wire the input power and toast them and problems with the RJ-12 connectors have all contributed to the perception that the Jaguars are not a good choice. The fact that CAN can only be used for control with a Jaguar means that CAN gets a bad rap as "guilt by association".

I think that TI/FIRST putting the Jaguars out for re-competition is a good thing. The RFP is open enough to support some crucial modifications that have been long needed in the Jaguars. But, given the perception that the Jag are bad, I'm not sure that you'll find any manufacturer who'd be willing to take that project on with a design they didn't control the IP for with special firmware specifically designed for FIRST and in a really limited market like our robotics applications.

The Stellaris micro-controller is an ARM Cortex M3 -- a very stable chip that is similar to the controller used in the VEX control unit. The Stellaris line is very successful for TI with sales going up by a factor of 2x this past year. But, they just can't make any money on the Jaguars. So, this RFP is a way for TI to dump the Jaguar and move on to more lucrative ventures.

For folks that say CAN is unreliable, I ask if you drive a car made in the past 10 years. If so, you've likely got CAN bus. CAN is *very* reliable. But, the implementation that we see in the Jaguar can be prone to problems because of all of the failure modes of the Jaguars themselves.

However, the ability to get better low speed control and the ease of wiring the robot really speak well for the Jaguars in CAN mode. Add to it the closed loop support modes, and I think that CAN is ultimately a winner. We just need a better implementation, so we can put the CAN is unreliable issue to rest. So, I really hope that the RFP process will be successful and we can get a manufacturer for these new motor controllers that will stay with us while they work out the issues with CAN control and we work out the learning curve to be better able to take advantage of the features that CAN brings to the table.

Mike
Reply With Quote
  #8   Spotlight this post!  
Unread 04-30-2012, 03:22 PM
mjcoss mjcoss is offline
Registered User
FRC #0303
 
Join Date: Jan 2009
Location: Bridgewater,NJ
Posts: 69
mjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the rough
Re: CAN on the entire 2012 robot

Team 303 used all CAN as well. We had 8 motor controllers, 6 Black Jaguars, and 2 Tan ones. There are some real issues with using CAN, as others have mentioned. Lost configuration on brown out, not stellar PID controls, and timeouts of messages on the bus - as reported by the cRIO. In addition, you have a new single point of failure in either the 2CAN, or the serial cable. Lost of that device or cable will render your robot dead in the water.

We also had a case where one of the Jaguars on the bus went crazy and flooded the bus, causing us to loose comm with all of the Jaguars.

One failure case that we had (and is the reason for the 2 Tan ones) is on our shooter motor. If you rapidly changed the setpoint on our shooter motors, the Black Jaguars would eventual stop working completely, and required a reset, and reflash of the firmware. The Tan ones would reset and this triggered code which reloaded the configuration and start working again.

Lots of people don't like the high current shutdown of the Jaguars, and I'd like to have it be tunable but should you really be running the motors at such a high current? The victors would reboot as well but they rebooted quickly, and had no configuration, and would keep trying to do what you told them until they burned out either the motor or themselves. Seems better to save the equipment, and fix the high current issues.
Reply With Quote
  #9   Spotlight this post!  
Unread 04-30-2012, 03:52 PM
Alan Anderson's Avatar
Alan Anderson Alan Anderson is offline
Software Architect
FRC #0045 (TechnoKats)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: Kokomo, Indiana
Posts: 7,894
Alan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond repute
Send a message via AIM to Alan Anderson
Re: CAN on the entire 2012 robot

Quote:
Originally Posted by mjcoss View Post
Lots of people don't like the high current shutdown of the Jaguars, and I'd like to have it be tunable but should you really be running the motors at such a high current?
There's already a circuit breaker to protect the wiring from overcurrent situations. The Jaguar's shutdown is there only to protect the Jaguar itself, and I think it's too aggressive.
Reply With Quote
  #10   Spotlight this post!  
Unread 04-30-2012, 04:14 PM
FrankJ's Avatar
FrankJ FrankJ is offline
Robot Mentor
FRC #2974 (WALT)
Team Role: Mentor
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Marietta GA
Posts: 966
FrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond reputeFrankJ has a reputation beyond repute
Re: CAN on the entire 2012 robot

We had 7 jags on our canbus without issues. 4 were on the mechanum drive cims, 2 on the fisherprice shooter motor, & one on a banebot 500 conveyor. Only the mechanum drives would be what I consider to be heavily loaded. We used the serial port on the Crio, Anal about the canbus wiring, and careful not to overload the canbus network with traffic. Had zero problems once other issues were sorted.

I did find out that there is an issue with the First implementation of canbus. If you write to an non existent node (Read resetting as well) You will crash the canbus. As I understand it the canbus "driver" on the Crio waits for an acknowledgement that never comes.
Reply With Quote
  #11   Spotlight this post!  
Unread 04-30-2012, 04:35 PM
mswalker mswalker is offline
Mentor
FRC #2583 (Westwood Robotics)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 26
mswalker is on a distinguished road
Re: CAN on the entire 2012 robot

We used CAN in 2011 for four motors without any problem. In 2012 we linked 9 Jaguars (about half each black/tan) with CAN. We had quite a few problems with loss of motor function and increased latency. I believe that the connections and wire integrity were the main issues. We spent several hours one night testing all the cables and connectors to make sure we had good wires and good connections. That worked for a while but then the latency came back. The latency showed up in a lack of responsiveness to the controls - e.g. stuttering instead of driving ahead smoothly. In each case it would take a lot of time to trace the entire daisy chain to try to figure out where we were having a problem. When we watched the network when we had problems we would see a large number of communications errors, e.g. can't find the address. We could usually fix the problems by replacing a cable or working on a jack but it all took time.

After our practice sessions at regionals we replaced the CAN connections with PWM. We didn't have any issues with control latency after that.

CAN seems to be the way to go but doesn't seem sturdy or reliable enough at this point.
__________________
http://team2583.org
Reply With Quote
  #12   Spotlight this post!  
Unread 05-01-2012, 09:12 AM
Brian Selle's Avatar
Brian Selle Brian Selle is online now
Mentor
FRC #3310 (Black Hawk Robotics)
Team Role: Engineer
 
Join Date: Jan 2012
Rookie Year: 2012
Location: Texas
Posts: 143
Brian Selle has a spectacular aura aboutBrian Selle has a spectacular aura about
Re: CAN on the entire 2012 robot

We had 13 motors on our 2012 robot, 9 were controlled with Jaguars/CAN using the 2CAN. The biggest issue we experienced early on was the connector pins not sprung out far enough to make good contact. After fixing that, everything worked flawlessly all the way through the championships. I suspect the latency issues mentioned earlier may have been due to using the serial bus instead of a 2CAN. We experienced none. We ran one closed loop PID loop with a high quality potentiometer and it was actually much less sensitive to the gains than running it through the cRio probably because the 1000hz update rate is more forgiving. Having some options to filter the sensor inputs would be very nice.

We had 2 alliance partners fail during our regional finals due to PWM cables coming loose (one was glued in place). I like hearing the RJ-11 "click" in place.
Reply With Quote
  #13   Spotlight this post!  
Unread 05-14-2012, 04:24 PM
Woodie Flowers Award
JDNovak JDNovak is offline
Mentor
AKA: John Novak
FRC #0016 (Bomb Squad)
Team Role: Engineer
 
Join Date: Jan 2005
Rookie Year: 1996
Location: Mountain Home, AR
Posts: 44
JDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond reputeJDNovak has a reputation beyond repute
Re: CAN on the entire 2012 robot

Team 16 has used CAN since 2010. We started with the 2CAN from the beginning and ran 10 Jaguars the first year and 11 Jaguars in 2011 and this year. The biggest issue the first two years was an initialization problem at boot up that was hard to detect but would render the robot motionless at the start of the match. This risk caused us to convert to PWMs at the end of last season.

This year we started again with CAN intent on finding the issue and solving it rather than avoiding it. We haven't had the problem with this year's firmware and CAN driver. We did have problems with the original 2CAN due to static but we were able to overcome that until we could get the redesigned 2CAN just before the start of the season.

There are tradeoffs with every system. Every time we build another robot we have cable issues until all marginal ones are found and replaced. After that we have had very little trouble. As several have alluded to, a cable problem usually takes the whole system down due to daisy chaning and failed termination. I intend to create a bus system that reduces this weakness similar to the ones referred to above. The 2CAN web page is invaluable in finding problems quickly. It can even be monitored through the router while driving to see motor loading and intermittant problems.

One side issue I would like point out is the Jaguar overcurrent protection vs Victors. I have had a few motors fail in the last year and a couple with very low resistance. The Jaguars shut down fast enough to keep the battery voltage above the reset point of the Crio and D-Link. I have heard of shorted motors on victors in at least a couple of cases that left robots dead while the system rebooted. Sometimes the quick overcurrent protection is worth something. We have had no problem with overcurrent shutdown on properly geared systems.

One of the reasons we continue to come back to CAN is the inherent current monitoring on every motor. One use is to protect motors in parallel. Our winch this year uses worm gearboxes in parallel to eliminate backdriving. if one motor doesn't drive the other is stalled and fails quickly. current monitoring allowed shutdown fast enough to avoid this problem.

As for PID control on board. the testing we have done has worked well but we always wind up back in the Crio because of lack of features in the Jaguar. Syncing two motors is a big issue. If there were some filtering options and the configuration was non-volitile I think we would be using it.

It will be interesting to see what transpires with TI backing out of manufacturing Jaguars but if we have to go back to PWM I'll be dissapointed.

My opinion is that a lot of the challenge of robot design is in finding how much technology you can incorporate into your design without making it the weak link. This about science and technology, isn't it?
Reply With Quote
  #14   Spotlight this post!  
Unread 05-14-2012, 05:01 PM
mikets's Avatar
mikets mikets is offline
Software Engineer
FRC #0492 (Titan Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Bellevue, WA
Posts: 596
mikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of lightmikets is a glorious beacon of light
Re: CAN on the entire 2012 robot

Quote:
Originally Posted by JDNovak View Post
As several have alluded to, a cable problem usually takes the whole system down due to daisy chaning and failed termination. I intend to create a bus system that reduces this weakness similar to the ones referred to above.
Our team ran 7 Jags on a T-tap bus topology like the following:
http://www.chiefdelphi.com/forums/sh...t=99554&page=3
We never had any problem with it. We had a couple of bad CAN cables but they were quickly identified and replaced. Since they weren't daisy chained, only the bad ones showed up with problems, the others kept on working.
__________________
Reply With Quote
  #15   Spotlight this post!  
Unread 10-04-2012, 02:31 PM
enrique's Avatar
enrique enrique is offline
Registered User
FRC #1251 (Tech Tigers)
Team Role: Electrical
 
Join Date: Jan 2010
Rookie Year: 2010
Location: Florida
Posts: 87
enrique will become famous soon enough
Send a message via Yahoo to enrique
Re: CAN on the entire 2012 robot

Hello all. This year team 1251 started off 100% CAN this year at the Orlando regional. Our robot worked in the pits, but would not work on the field. We switched to PWM, for the mean time and it started working. Later we found out our serial adapter was missing the resistor that it was supposed to have. My guess is that the missing resistor opened the CAN network to interference from the field. We then put the CAN network back on and half of the motors on it. 2 motors on the drive system where PWM and the other 2 where CAN. Our shooter motors where also 100% CAN. Everything worked fine, except we had 1 PWM pop out and one of the PWM drive motors stopped, even with glue on the connector. When we went to the regional in Fort Lauderdale, we switched 2CAN. I really was impressed with 2CAN. Plus overtheroad electronics provided good support. We did have some of the CAN network go out 1 time in Fort Lauderdale. I found it was a 12 pin connector we used from radio shack, built up corrosion and would cut power to a couple of the jaguars. The connector pins where black (Char). Removing this connector fixed the problem. I hear the current protection is being changed. I have to say, this year there where motors plugged in backwards (opposing). Each time this happened, the jaguars went to protect mode and we fixed it. In the past I have had victors blow, because of this. Of course, you should not wire the motor backwards, but its nice to know it will not cost you money if it happens. I have read some threads with people causing the jaguars to go into protect mode with correct wiring. I would venture to say the motor in some cases are working to hard and probably do not have a proper gearbox for the job. But as it stands, we now feel comfortable enough and experienced enough to go 100% CAN.

Last edited by enrique : 10-04-2012 at 02:36 PM.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 01:33 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi