Go to Post There's too much whining lately.. we need a hug forum - Gadget470 [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rating: Thread Rating: 7 votes, 5.00 average. Display Modes
  #1   Spotlight this post!  
Unread 13-04-2014, 18:08
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 8,125
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
using interrupts in FRC


I'd like continue a discussion that started in another thread. I made a new thread here as a courtesy to the OP of the original thread who expressed his desire not to discuss the larger issues involved.

Here are the things I'd like to have a discussion about. I'd like to hear from FRC programming gurus.

Scenario1:

A team has code which uses 100ms and 20ms periodic tasks, and they are running low on throughput margin (i.e. CPU usage is too high). They think that they can reduce CPU usage by using interrupts instead of periodic tasks. What say you?

Scenario2:

A team has subsystems that need to respond to sensor data changes within 1ms. Should they: a) use Interrupt Service Routines to control these subsystems, or b) carefully investigate other approaches (such as a hardware limit switch directly connect to a motor controller for example).


  #2   Spotlight this post!  
Unread 13-04-2014, 18:13
Chris_Elston's Avatar
Chris_Elston Chris_Elston is offline
Controls Engineer
AKA: chakorules
FRC #1501 (Team THRUST)
Team Role: Engineer
 
Join Date: Feb 2004
Rookie Year: 2001
Location: Huntington, Indiana
Posts: 751
Chris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond repute
Re: using interrupts in FRC

The timing of this thread is impeccable. We actually have a situation where we are going to add a sensor, that we think will only be active for 20-24 msecs. I was worried that we need an interrupt to make sure we capture the input condition of the sensor and was asking around so I can advise my software team what to expect. So we have a situation that we want to run some code based on a quick input condition and trigger based on an interrupt is how we would like to trigger a task or loop as well.
__________________
Team T.H.R.U.S.T. 1501
Download all of our past robot's source code here:Repository

Favorite CD quote:
"That can't be their 'bot. not nearly enough (if any) rivets to be a 1501 machine." ~RogerR: Team #1369
  #3   Spotlight this post!  
Unread 13-04-2014, 18:55
efoote868 efoote868 is offline
foote stepped in
AKA: E. Foote
FRC #0868
Team Role: Mentor
 
Join Date: Mar 2006
Rookie Year: 2005
Location: Noblesville, IN
Posts: 1,425
efoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond reputeefoote868 has a reputation beyond repute
Re: using interrupts in FRC

If I recall correctly the "best practice" was to use the interrupt to set a flag in the timed routine. That way the interrupt took very few clock cycles and the context switching wasn't too bad. Also it minimized the possibility of getting interrupted during an interrupt.

Obviously this wouldn't work if the sensor had to respond to a condition in an amount of time less than the timer cycle. And from experience, debugging any interrupt routine can be a real pain.
__________________

Be Healthy. Never Stop Learning. Say It Like It Is. Own It. Like our values? Flexware Innovation is hiring!. We're looking for Senior Automation, Software, and System Engineers. Check us out!
  #4   Spotlight this post!  
Unread 13-04-2014, 19:46
The Doctor's Avatar
The Doctor The Doctor is offline
Robotics is life
AKA: Hackson
FRC #3216 (MRT)
Team Role: Programmer
 
Join Date: Mar 2014
Rookie Year: 2013
Location: United States
Posts: 158
The Doctor is on a distinguished road
Re: using interrupts in FRC

Quote:
Originally Posted by Chris_Elston View Post
The timing of this thread is impeccable.
Haha... Ether must be an ISR.

Why cant you just thread?:

If you had a timed task that took as long as in scenario 1, you may want to consider making it into a separate thread (using the Task class in C++ or similar). If any task doesn't require much communication with the main control loop, you could put it in a separate Task.

Now, thoughts on Scenario 1:

If you offload these timed actions into ISRs, they would take up as much (or more) CPU time as if they were running in the main process. The main disadvantage to this is if it happens inside a time-critical operation (like sending data to the host), it could cause problems.

Scenario 2:

If the team in question had code that could respond within 1ms, ISRs are a good choice. A hardwired solution would be very difficult to make, debug, and change, compared to an ISR.
  #5   Spotlight this post!  
Unread 13-04-2014, 21:07
Alan Anderson's Avatar
Alan Anderson Alan Anderson is offline
Software Architect
FRC #0045 (TechnoKats)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: Kokomo, Indiana
Posts: 9,113
Alan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond repute
Re: using interrupts in FRC

Quote:
Originally Posted by Chris_Elston View Post
We actually have a situation where we are going to add a sensor, that we think will only be active for 20-24 msecs. I was worried that we need an interrupt to make sure we capture the input condition of the sensor...
What kind of signal does the sensor provide? If it's a digital input, consider using a counter to capture the occurrence. The FPGA will detect the signal as quickly as it can do anything, and the robot code can watch for the counter to change value on whatever schedule is appropriate for responding to the signal.
  #6   Spotlight this post!  
Unread 13-04-2014, 21:13
Chris_Elston's Avatar
Chris_Elston Chris_Elston is offline
Controls Engineer
AKA: chakorules
FRC #1501 (Team THRUST)
Team Role: Engineer
 
Join Date: Feb 2004
Rookie Year: 2001
Location: Huntington, Indiana
Posts: 751
Chris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond reputeChris_Elston has a reputation beyond repute
Re: using interrupts in FRC

Quote:
Originally Posted by Alan Anderson View Post
What kind of signal does the sensor provide? If it's a digital input, consider using a counter to capture the occurrence. The FPGA will detect the signal as quickly as it can do anything, and the robot code can watch for the counter to change value on whatever schedule is appropriate for responding to the signal.
Digital Input.

Nice suggestion. We will give that a shot on Monday. You might see it work next Saturday...lol....sounds like we are coming to Kokomo on Saturday to the TechnoKats field at 1pm.
__________________
Team T.H.R.U.S.T. 1501
Download all of our past robot's source code here:Repository

Favorite CD quote:
"That can't be their 'bot. not nearly enough (if any) rivets to be a 1501 machine." ~RogerR: Team #1369
  #7   Spotlight this post!  
Unread 14-04-2014, 07:15
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,756
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: using interrupts in FRC

For 100ms and 20ms loops, I would first profile to see what is taking CPU. I would probably decide to lighten the processing of the 20ms and 100ms loops by removing Smart Dashboard access, refnum by name access, and perhaps some of the scaling and wrappers of WPILib. If using LV, I'd serialize some of the items in the loop to avoid context switches that I really don't need/want. Unless I have tons of these loops, I suspect that would be good enough.

If I'm trying to respond to something in less than 1ms, I'm probably in trouble anyway since the motor controllers only pay attention to their PWM power value every 5 or 10 ms depending on the model. Other actuators seem slow as well, but I don't know the values. The limit switch and other Jag inputs are polled at 1ms by comparison. CAN messages bridged over serial or enet might be fast enough.

Anyway, I'd have to compare the timed loop with very little inside it versus the interrupt.

If I was relatively certain the code wouldn't change, and had the FPGA tools, I'd move the polling into the FPGA. Depending on the complexity of the response, it may be possible to put it in the FPGA too. But again, if the code is going to change, you will shave many times before you complete the project.

Doing the polling in the FPGA, and the response in RT is essentially what FRC interrupts are. If the polling fits the envelope of what is available, it allows high speed monitoring and flexible response for FRC.

Greg McKaskle
  #8   Spotlight this post!  
Unread 14-04-2014, 08:05
MrRoboSteve MrRoboSteve is offline
Mentor
AKA: Steve Peterson
FRC #3081 (Kennedy RoboEagles)
Team Role: Mentor
 
Join Date: Mar 2012
Rookie Year: 2011
Location: Bloomington, MN
Posts: 582
MrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond reputeMrRoboSteve has a reputation beyond repute
Re: using interrupts in FRC

We haven't used the interrupt handling abilities of WPILib (C++) on our team -- is there any guidance around as to what works/doesn't work from inside an ISR? For example, I don't remember seeing any special logic in WPILib to allow concurrent access to the actuator or sensor classes.
__________________
2016-17 events: 10000 Lakes Regional, Northern Lights Regional, FTC Burnsville Qualifying Tournament

2011 - present · FRC 3081 Kennedy RoboEagles mentor
2013 - present · event volunteer at 10000 Lakes Regional, Northern Lights Regional, North Star Regional, Lake Superior Regional, Minnesota State Tournament, PNW District 4 Glacier Peak, MN FTC, CMP
http://twitter.com/MrRoboSteve · www.linkedin.com/in/speterson
  #9   Spotlight this post!  
Unread 14-04-2014, 09:01
wireties's Avatar
wireties wireties is offline
Principal Engineer
AKA: Keith Buchanan
FRC #1296 (Full Metal Jackets)
Team Role: Mentor
 
Join Date: Jan 2006
Rookie Year: 2004
Location: Rockwall, TX
Posts: 1,171
wireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond repute
Send a message via AIM to wireties
Re: using interrupts in FRC

Quote:
Originally Posted by Ether View Post
A team has code which uses 100ms and 20ms periodic tasks, and they are running low on throughput margin (i.e. CPU usage is too high). They think that they can reduce CPU usage by using interrupts instead of periodic tasks. What say you?
The period of the tasks is well within the bandwidth of the OS and hardware we use in FRC. The real question is the run time in each period. If we have 3 tasks that run t1, t2, t3 periods and require c1, c2, c3 computation time in each period then you calculate c1/t1+c2/t2+c3/t3 and have some feeling for how much bandwidth is being used. Of course there are other things going on. The OS consumes bandwidth for its clock and scheduler. In FRC robots some bandwidth is used by the task receiving DS messages plus one might have PID callbacks etc. If the sum of all terms for the OS, app and WPI tasks gets anywhere near 1 then you are in trouble (actually lower than 1, see http://en.wikipedia.org/wiki/Rate-monotonic_scheduling).

If the events that generate the interrupts are periodic and you are running the same-ish code to handle the event then you are not saving much time. However if the events are aperiodic using interrupts may save considerable bandwidth. If they use interrupts, the best practice is to quickly 1) determine if your hardware caused the interrupt 2) clear/reset the interrupt cause 3) set a flag, give a semaphore etc to synchronize with code running in a task to react to the event.

Quote:
Originally Posted by Ether View Post
A team has subsystems that need to respond to sensor data changes within 1ms. Should they: a) use Interrupt Service Routines to control these subsystems, or b) carefully investigate other approaches (such as a hardware limit switch directly connect to a motor controller for example).
1ms interrupts can be handled by the hardware and OS we use in FRC. The interrupt latency of our system is measured in single microseconds. Again the real question is how long does it take to "handle" the interrupt. If the interrupt is periodic or the worst case is nearly periodic you can analyze the effect on the system just as you would a task.

BTW, handling a mechanical limit switch using interrupts is a little tricky if the hardware does not debounce the input. I don't know that our DIO inputs do this.

HTH
__________________
Fast, cheap or working - pick any two!

Last edited by wireties : 14-04-2014 at 18:49.
  #10   Spotlight this post!  
Unread 14-04-2014, 13:39
AustinSchuh AustinSchuh is offline
Registered User
FRC #0971 (Spartan Robotics) #254 (The Cheesy Poofs)
Team Role: Engineer
 
Join Date: Feb 2005
Rookie Year: 1999
Location: Los Altos, CA
Posts: 803
AustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond repute
Re: using interrupts in FRC

Quote:
Originally Posted by Ether View Post
A team has code which uses 100ms and 20ms periodic tasks, and they are running low on throughput margin (i.e. CPU usage is too high). They think that they can reduce CPU usage by using interrupts instead of periodic tasks. What say you?
We have used interrupts on the cRIO before in previous years, and my advice is to run away screaming. Most of the times, they work quite well, but occasionally, we would get a full cRIO reboot without any debug information coming back from the cRIO about why. Our programmers very carefully read the docs on what instructions were and weren't allowed in the interrupt, and were handling the interrupt by incrementing a semaphore which would then wake up a high priority task to do the actual work. The code was looked over with a fine tooth comb. Maybe we were still managing to do something wrong, but given the number of matches that it cost us where we either sat dead or were stuck in an infinite reboot loop, we won't be doing it again.

In our case, we were using the interrupts to capture an encoder value when a magnet would pass over a zeroing hall effect. In my opinion, this is a perfect use case for using an interrupt where you want a very fast response time.
  #11   Spotlight this post!  
Unread 14-04-2014, 14:11
Alan Anderson's Avatar
Alan Anderson Alan Anderson is offline
Software Architect
FRC #0045 (TechnoKats)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: Kokomo, Indiana
Posts: 9,113
Alan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond reputeAlan Anderson has a reputation beyond repute
Re: using interrupts in FRC

Quote:
Originally Posted by AustinSchuh View Post
In our case, we were using the interrupts to capture an encoder value when a magnet would pass over a zeroing hall effect.
Did you actually care about the encoder value? Or, as the "zeroing" description implies, were you just reading it in order to use as a reference for later readings of the delta? If that's all you were using it for, you wouldn't have needed to do any special programming. The FPGA lets you define a digital input pin as an encoder reset signal.
  #12   Spotlight this post!  
Unread 14-04-2014, 19:02
wireties's Avatar
wireties wireties is offline
Principal Engineer
AKA: Keith Buchanan
FRC #1296 (Full Metal Jackets)
Team Role: Mentor
 
Join Date: Jan 2006
Rookie Year: 2004
Location: Rockwall, TX
Posts: 1,171
wireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond repute
Send a message via AIM to wireties
Re: using interrupts in FRC

Quote:
Originally Posted by AustinSchuh View Post
We have used interrupts on the cRIO before in previous years, and my advice is to run away screaming.
I tried to use interrupts this year and concur - something is flaky. But it is somewhere in the library or the FPGA. Interrupts are used on FRC robots for the main timer, for the arrival of network data and many other purposes - so they do work. In my case interrupts would fire for a while and then stop firing. The rest of the code continued to run (not crash as in Austin's case) but the limit switch I tied to the interrupt pin just stopped generating interrupts.

There is a list of OS functions (not to call) in the VxWorks Programmer's Guide. Basically do not call anything that only makes sense from a task context (no taking semaphores, no delays, no memory allocation, no formatted I/O, no mutexes, doing anything that might block etc) plus no floating point math (unless you save/restore the floating point context).

In Austin's example my (wild) guess is that the source of the interrupts (occasionally) did not get turned off. If more interrupts occur than the VxWorks can handle its internal queue will overflow (something like a Windoze BSoD).
__________________
Fast, cheap or working - pick any two!
  #13   Spotlight this post!  
Unread 14-04-2014, 19:34
virtuald's Avatar
virtuald virtuald is offline
RobotPy Guy
AKA: Dustin Spicuzza
FRC #1418 (), FRC #1973, FRC #4796, FRC #6367 ()
Team Role: Mentor
 
Join Date: Dec 2008
Rookie Year: 2003
Location: Boston, MA
Posts: 1,102
virtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant future
Re: using interrupts in FRC

Quote:
Originally Posted by Ether View Post
A team has code which uses 100ms and 20ms periodic tasks, and they are running low on throughput margin (i.e. CPU usage is too high). They think that they can reduce CPU usage by using interrupts instead of periodic tasks. What say you?
I used interrupts in the first year that the cRio was used, and didn't have any problems. YMMV though…

I would probably recommend restructuring the code so that everything can run in a single thread. In an FRC context, my opinion is that typically threads cause more trouble than they're worth.

Quote:
A team has subsystems that need to respond to sensor data changes within 1ms. Should they: a) use Interrupt Service Routines to control these subsystems, or b) carefully investigate other approaches (such as a hardware limit switch directly connect to a motor controller for example).
First, I would ask if they *really* need 1ms response time. If possible, I would try to reframe the problem so that the response time isn't required.

I like Alan's suggestion to use a counter, that sounds like a real winner.

As a sidenote, when possible, I connect my limit switches to Jaguars.
__________________
Maintainer of RobotPy - Python for FRC
Creator of pyfrc (Robot Simulator + utilities for Python) and pynetworktables/pynetworktables2js (NetworkTables for Python & Javascript)

2017 Season: Teams #1973, #4796, #6369
Team #1418 (remote mentor): Newton Quarterfinalists, 2016 Chesapeake District Champion, 2x Innovation in Control award, 2x district event winner
Team #1418: 2015 DC Regional Innovation In Control Award, #2 seed; 2014 VA Industrial Design Award; 2014 Finalists in DC & VA
Team #2423: 2012 & 2013 Boston Regional Innovation in Control Award


Resources: FIRSTWiki (relaunched!) | My Software Stuff
  #14   Spotlight this post!  
Unread 14-04-2014, 19:39
AustinSchuh AustinSchuh is offline
Registered User
FRC #0971 (Spartan Robotics) #254 (The Cheesy Poofs)
Team Role: Engineer
 
Join Date: Feb 2005
Rookie Year: 1999
Location: Los Altos, CA
Posts: 803
AustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond reputeAustinSchuh has a reputation beyond repute
Re: using interrupts in FRC

Quote:
Originally Posted by Alan Anderson View Post
Did you actually care about the encoder value? Or, as the "zeroing" description implies, were you just reading it in order to use as a reference for later readings of the delta? If that's all you were using it for, you wouldn't have needed to do any special programming. The FPGA lets you define a digital input pin as an encoder reset signal.
In this case (we were locating discs in our indexer), we really did need the encoder value. That is a handy feature, though. Thanks for sharing!
  #15   Spotlight this post!  
Unread 14-04-2014, 20:07
wireties's Avatar
wireties wireties is offline
Principal Engineer
AKA: Keith Buchanan
FRC #1296 (Full Metal Jackets)
Team Role: Mentor
 
Join Date: Jan 2006
Rookie Year: 2004
Location: Rockwall, TX
Posts: 1,171
wireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond reputewireties has a reputation beyond repute
Send a message via AIM to wireties
Re: using interrupts in FRC

Quote:
Originally Posted by virtuald View Post
I would probably recommend restructuring the code so that everything can run in a single thread. In an FRC context, my opinion is that typically threads cause more trouble than they're worth.
Yikes - a little conservative maybe?. We have a a world-class real-time multi-tasking OS at our disposal and we don't use it? In years where our students don't "get" the multi-tasking from the start I create templates that prevent them from messing up the timing. WPILib hides synchronization and mutual exclusion pretty well and the messaging API is straight forward. By the end of build season the students always seem to "get it".
__________________
Fast, cheap or working - pick any two!
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 02:38.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi