Go to Post Due to risk of unforseeable consequences, teams may not initiate a wormhole for the purpose of scoring ringers from past or future matches. - Richard Wallace [more]
Home
Go Back   Chief Delphi > Technical > Electrical
CD-Events   CD-Media   CD-Spy   FRC-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rate Thread Display Modes
  #166   Spotlight this post!  
Unread 04-23-2012, 05:23 PM
mjcoss mjcoss is offline
Registered User
FRC #0303
 
Join Date: Jan 2009
Location: Bridgewater,NJ
Posts: 69
mjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the rough
Re: Intermittent connection on field only

Well let me add some more hay And this thread seems like a good place to report some information.

I've been concerned about the field network for the past few years because it *seemed* that *which* robots were on the field was a contributing factor to the overall responsiveness, jitter and lag of the network. For example, the number of robots on the field that had onboard ip cameras seemed to be important. So this year, after our first district event, I started asking questions, and then at the MAR championship I brought a hardware packet sniffer to take a look at the field network, and I feel I should share what I discovered.

I originally believed that each robot was going to be assigned a separate channel and that they were using 802.11n @ 5 Ghz. The network at the MAR championship was indeed using 802.11n @ 5Ghz, and they are using a wide channel, however all robots were sharing the same channel and as such were sharing a total theoretically bandwidth of 300 Mbits.

At the Mt. Olive competition, I was told that the robots were running on channel 6. If true, this would have meant that they were running in the 2.4 Ghz range, with over a dozen other 802.11g networks, this would have considerably reduced the theoretical bandwidth.

In general, due to a number of different factors, if you get half the theoretically bandwidth on a wireless network, you're doing well. So let's assume that 150 Mbits is our expected available bandwidth on the field, if you're using a wide 802.11n channel @ 5 Ghz. Much less if you're using 802.11n @ 2.4 Ghz as the interference will be awful.

I've asked but haven't heard whether or not they have established any QoS or bandwidth limitations on each SSID in the Cisco access point that they are using. Without any controls, it will be a free-for-all for the available bandwidth.

This bandwidth is used by the robots in a number of different ways, and here I’m just talking about communication between the robot and the driver station laptop, as I’m not sure what the FMS is using:
VIDEO streaming
If you have an onboard camera, and send a video stream from robot to driver station at 640x480 in 24 bit color @ 30 frames a second, that's ~200 Mbits uncompressed raw bits. MotionJPEG will compress that down on average to around 10 - 15 Mbits. H.264 will do better. I've heard that some teams have 2 cameras onboard. The Axis camera supports a number of options to reduce the bits but there is no rule about how you configure the cameras, and they would work fine in your local build environment and even in the practice fields at competitions where you’re the only user of that wireless channel.
Dashboard data
There is the "normal" dashboard that is part of the default code, and the default dashboard sends data, if I remember correctly, at about 10 times a second. For reasons that I can't remember at this point, we are actually sending the data at 40 times a second from our robot. This is a relatively small amount of data, but it doesn't have to be. With the addition of Smart Dashboards, and other custom dashboards, and no guiding principle on the volume of data this could be a significant amount of data or just a dribble. In our case, we're sending ~1 Kbits per update, or 40 Kbits per second.
Driver station data
This is data packaged by the driver station application provided by FIRST and sends the values for the input devices attached to your driver station to the robot. I've never looked into how much data is being sent, or the frequency with which it's sent but it's not a lot of data probably on the order of 40 - 50 Kbits per second.
Other network traffic
There are several network ports that are open for teams to use for communication between the robot and the driver station. In our case, we ran a UDP server on our robot to collect the results of vision processing performed by our driver station. We sent the results of the calculations back to the robot at the rate of 10 times a second. The data is small (72 bits) so we're sending only 720 bits per second.

So for us, our network utilizations was small ~1 Mbits for the camera - we're using gray scale, 320x240 and 30 frames a second with MotionJPEG compression, and at best another 1 Mbit for the remaining traffic. But that is due to choices that we made. I could easily imagine making other choices, and given that I was operating under the belief that we had a full channel to ourselves, I might have gone down totally different path.

The thing about this is that while there is the catastrophic failure mode where the field network crashes. There are many other situations where the latency and jitter can spike, and dip badly. VxWork’s IP stack is not particularly robust, and for some teams that stack has to handle all of the time sensitive CAN bus traffic, as well as driver station, dashboard and custom traffic.

Further, unless you change your default Iterative robot code (at least in C++), you're periodic functions are synchronized with the arrival rate of the packets from the driver station. Now it a well behaved network, the arrival rate should be pretty stable. But if your code assumes stable packet arrivals, you can run into all sorts of timing issues.

In addition, both the camera traffic, and the driver station packets are using TCP which can be very unfair when it comes to sharing bandwidth. A greedy application can ramp up its utilization of the bandwidth, causing starvation of others. And then there's retransmissions, etc.

Is it possible to saturate the network? You betcha. Is it service impacting? Yes to everyone, including you. Is there anything that can be done? Yes.

When I examined the network at the MAR championship, I saw a number teams that were having problems associating with the field. There were repeated attempts by the robot's DLINK to associate with the field access point. I also saw many corrupt frames.

Our DLINK in Mt. Olive simply gave up completely, rebooting during several of our matches. It had been fine, and then just started reboot when we hit another robot or field element Of course to our drive team it looked like we lost communication to the field (which we did) but it was the DLINK that was rebooting. And no there weren’t any loose wires except maybe inside the DLINK housing. I heard from a team that they had a DLINK that only worked when it was standing on edge. Lay it flat and it didn't work. I think they are cheaply made, and are really not meant for the hostile environment of a FIRST robotics competition. A ruggedized access point/bridge would be a beautiful thing.

I don't know why FIRST chose not to have 6 separate access points to provide a channel for each robot. Maybe they just figured that 150 Mbits/6 = 25, and who’d need more than that. I don't know if they are configuring QoS to ensure a fair share of the network. I will be looking at the network in our next off season competition, and try to come to some conclusion about what exactly is really going on.
  #167   Spotlight this post!  
Unread 04-23-2012, 08:12 PM
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 7,717
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: Intermittent connection on field only

Quote:
Originally Posted by mjcoss View Post
I originally believed that each robot was going to be assigned a separate channel and that they were using 802.11n @ 5 Ghz. The network at the MAR championship was indeed using 802.11n @ 5Ghz, and they are using a wide channel, however all robots were sharing the same channel and as such were sharing a total theoretically bandwidth of 300 Mbits.
Here's the answer from the GDC on bandwidth.

Quote:
Game - The Arena » The Arena » The Player Stations
Q. Are there any bandwidth limits enforced by the field/FMS on the ports listed in section 2.2.9? Is each team able to access the full 802.11n bandwidth, or is it divided by the 6 teams on the field? FRC0330 2012-02-23
A. There are currently no bandwidth limits in place in the field network. In theory, each team has 50Mbits/second (300Mbits/6) available, but that’s not actually realistic. In reality, each team is likely to have ~10-12Mbits/s available. This rate will vary depending on the location of the radio on the Robot and the amount of wireless traffic present in the venue at 5GHz. While this information may help give teams an idea of what to expect, note that there is no guaranteed level of bandwidth on the playing field.
  #168   Spotlight this post!  
Unread 04-24-2012, 08:11 AM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

I mentioned about the networks at MAR Mount Olive district event appearing all to be on channel 6 (course I couldn't look at the 5GHz networks with my phone) back on Page 3. RufflesRidge assured me they were not shortly after.

I never got an answer to my question about whether the 2.4GHz sections of the D-Link are turned off when the fields configure the D-Link APs (post #48 on page 4). If they are turned off, then in theory the channel overlap in 2.4GHz should not be a problem. However, if the second band remains on the radio will attempt to interact with the 2.4GHz networks as well as the 5GHz+ networks.

I got confirmation of my information from people who helped set up the field at the MARs Mount Olive District, not sure where RufflesRidge got his information.

Last edited by techhelpbb : 04-24-2012 at 08:36 AM.
  #169   Spotlight this post!  
Unread 04-24-2012, 10:03 AM
mjcoss mjcoss is offline
Registered User
FRC #0303
 
Join Date: Jan 2009
Location: Bridgewater,NJ
Posts: 69
mjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the rough
Re: Intermittent connection on field only

I had a separate source who assured me that at Mt Olive, they were on channel 6. I did not check to ensure whether that was true or not. But to me having no controls on the apportioning of bandwidth and having robots sharing bandwidth means that the idea that which robots are on the field can impact the communications between the robots and their driver stations is possible. And steps could be taken to mitigate these issues so that there's a level playing field.
  #170   Spotlight this post!  
Unread 04-25-2012, 10:05 AM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

I wrote a several page document on how to address what I suspect is Al's concern regarding my previous troubleshooting flow described on page 10, post 149 of this topic.

This is about checking the power quality to the D-Link AP.

I suspect Al is concerned about a 'blind spot' (see my post #144, page 10) in my troubleshooting flow which I can resolve with:

A. At least 1 troubleshooting D-Link APs provided in spare parts at a competition with the barrel connectors removed and tail of stranded wire soldered to the PCB. That wire should have a good mechanical strain relief and come with a ferrite RF choke to remove noise. This has been asked for at past events and in past years as well.

B. At least 1 low temperature coefficient resistive load (physically large resistor probably with fan/heatsink) that could be provided in spare parts at a competition with a maximum and minimum current monitor in series with it. It's value would draw the same current as the D-Link AP can draw maximum at all times. The current monitor limits are set using Ohm's Law (E = I x R) and the maximum and minimum voltage limits we need to characterize from the DC-DC converters we are using to power the D-Link AP.

I've asked 2 times to have this topic split (because I can't) so that this matter would not distract from the other valuable observations and suggestions but since it hasn't been done I'll post the detailed version elsewhere.

Suffice it to say I think this resolves Al's concerns as best as I understand his communications at this time. I'll post the link later when I have time. I think anyone that's really interested will figure my solution out from the minimum of detail I just provided.

Last edited by techhelpbb : 04-25-2012 at 04:26 PM.
  #171   Spotlight this post!  
Unread 04-26-2012, 08:09 AM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

Interesting observation, as of right now the questions asked after the one about monitoring the D-Link have been answered in the official Q&A forum but this question, now days old, has not.

So much effort to avoid checking a simple voltage with already approved hardware. The amount of energy expended to avoid removing all doubt now greatly exceeds that of just doing it.
  #172   Spotlight this post!  
Unread 04-26-2012, 08:06 PM
tickspe15's Avatar
tickspe15 tickspe15 is offline
Registered User
AKA: Spencer Tickman
FRC #1318 (Issaquah Robotics Society)
Team Role: Engineer
 
Join Date: Nov 2011
Rookie Year: 2009
Location: Issaquah,Washington
Posts: 106
tickspe15 is a glorious beacon of lighttickspe15 is a glorious beacon of lighttickspe15 is a glorious beacon of lighttickspe15 is a glorious beacon of lighttickspe15 is a glorious beacon of lighttickspe15 is a glorious beacon of light
Re: Intermittent connection on field only

We had the same problem in spokane...try setting your camera to 6fps
  #173   Spotlight this post!  
Unread 04-27-2012, 08:46 PM
DonRotolo's Avatar
DonRotolo DonRotolo is offline
Is winter over yet?
FRC #1676 (Pascack PI Oneers)
Team Role: Mentor
 
Join Date: Jan 2005
Rookie Year: 2005
Location: Montvale NJ
Posts: 6,251
DonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond reputeDonRotolo has a reputation beyond repute
Re: Intermittent connection on field only

Quote:
Originally Posted by techhelpbb View Post
I think anyone that's really interested will figure my solution out from the minimum of detail I just provided.
Before we move to "solutions" the step of identifying the problem has yet to be completed. PLEASE stop chasing ghosts publicly, the symptoms so far do not support a power issue with the radios. If you want to test your robot at home, feel free, but stop throwing assertions and opinions out there as facts because those of us who are trying to find a real problem are not happy with waters getting muddied. Bad info is worse than no info, since unsuspecting teams may rely on bad info.

That being said, our comms issue went away in our last 2 matches at MAR CMP and in 8 matches here in STL has not returned. Nothing was changed, so we don' t know why. Yet.
__________________

I am N2IRZ - What's your callsign?
  #174   Spotlight this post!  
Unread 04-28-2012, 04:13 PM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

Quote:
Originally Posted by DonRotolo View Post
Before we move to "solutions" the step of identifying the problem has yet to be completed. PLEASE stop chasing ghosts publicly, the symptoms so far do not support a power issue with the radios. If you want to test your robot at home, feel free, but stop throwing assertions and opinions out there as facts because those of us who are trying to find a real problem are not happy with waters getting muddied. Bad info is worse than no info, since unsuspecting teams may rely on bad info.

That being said, our comms issue went away in our last 2 matches at MAR CMP and in 8 matches here in STL has not returned. Nothing was changed, so we don' t know why. Yet.
I haven't posted to this topic in a bit and yesterday when you posted this I was on my way to FEDEX to deliver a backup shipment of boards to Linuxboy. Sorry I didn't see this sooner but I'm recovering from pnuemonia.

First off, I offered merely a solution to remove power quality issues for the D-Link AP not a guaranteed solution to everything that effects the communications. Please quote me in context.

Secondly:

Team 11 certainly did have power issues with the radio systems.
Other teams certainly did have power issues with the radio systems.

Telling me there is no power issue with the radio denies Team 11 had a power issue with the radio subsystem and so did others. It denies that I have here a bad DC-DC converter that took 3 matches worth of dysfunction on the field in Philly to find, and others have seen the problem go away when the PDB was replaced.

I'm sure it's a real problem if our team is stuck on the field for 3 matches just like anyone who has problems besides the power quality issues. Just as I am sure it's a real problem if your team is stuck because of a software problem.

The fact is there is obviously more than one thing not quite right with the radios and the systems that power them. That was my entire point. Outright denying a power problem can and sometimes does exist...either because of the parts or because of the wiring refutes the evidence that even those who do so present. It also misdirects the people that do actually have power problems effecting them.

A loose connector on the D-Link is a power quality issue. Why are we intent on saying that doesn't happen when in point of fact even Al notes it often does? Not all power quality issues are about the components sometimes it's just the wiring in the power system.

So far as how I troubleshoot the problem: how would you like me to troubleshoot a problem that per the posts in this topic seems to exist most often only on a competition field...when i don't have a competition field to work with? Even when I worked at spare parts at MAR Mount Olive it's not like I can put things on that competition field that FIRST won't allow. Furthermore thanks to my efforts we no longer can dispute that FIRST will not allow it, they officially said no (as I originally stated they would before everyone led me to believe I was incorrect...again why is this my problem when I was correct?).

We do have a field but we certainly don't have the competition field parts.

If the issue was about me distracting this topic I openly offered to take this conversation out of this topic. Such an argument is a straw man.

I do respect you and the others Don. However, expending so much effort on scapegoating me to cover for a power quality issues that sometimes do exist is not a reasonable thing to do. The reasonable thing to do if you all feel it distracts is offer me another place to discuss this issue like another topic (I can't create topics myself).

If you tell people to troubleshoot for a software issue and they have a power quality issue then like Team 11 they'll spin their wheels rifling through the software looking for what might just be bad power supply to the D-Link. Then what? Who's fault is it then? The guy offering the solution to help rule power quality issues out quickly? I think not.

Furthermore, I asked repeatedly in this topic if anyone load tested the D-Link power supply and no one replied. So if you wonder how the bad wiring or bad component (no matter the rarity) might slip through you need look no further than that.

Additionally, it's clear that despite the general assurance we'll find all the problems if I just stop communicating, Championships are over and obviously these issues remain. So what have we solved? How does the situation today differ from the situation at the start of March?

Last edited by techhelpbb : 04-28-2012 at 06:24 PM.
  #175   Spotlight this post!  
Unread 04-28-2012, 05:15 PM
RyanN's Avatar
RyanN RyanN is offline
RyanN
AKA: Ryan Nazaretian
FRC #0364 (Team Fusion)
Team Role: Mentor
 
Join Date: Jun 2006
Rookie Year: 2005
Location: Gulfport, MS
Posts: 994
RyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond reputeRyanN has a reputation beyond repute
Send a message via Skype™ to RyanN
Re: Intermittent connection on field only

Quote:
Originally Posted by techhelpbb View Post
I haven't posted to this topic in a bit and yesterday when you posted this I was on my way to FEDEX to deliver a backup shipment of boards to Linuxboy. Sorry I didn't see this sooner by I'm recovering from pnuemonia.

First off, I offered merely a solution to remove power quality issues for the D-Link AP not a guaranteed solution to everything that effects the communications.

Secondly:

Team 11 certainly did have power issues with the radio systems.
Other teams certainly did have power issues with the radio systems.

Telling me there is no power issue with the radio denies Team 11 had a power issue with the radio subsystem and so did others. It denies that I have here a bad DC-DC converter that took 3 matches worth of dysfunction on the field in Philly to find, and others have seen the problem go away when the PDB was replaced.

I'm sure it's a real problem if our team is stuck on the field for 3 matches just like anyone who has problems besides the power quality issues. Just as I am sure it's a real problem if your team is stuck because of a software problem.

The fact is there is obviously more than one thing not quite right with the radios and the systems that power them. That was my entire point. Outright denying a power problem can and sometimes does exist...either because of the parts or because of the wiring refutes the evidence that even those who do so present. It also misdirects the people that do actually have power problems effecting them.

A loose connector on the D-Link is a power quality issue. Why are we intent on saying that doesn't happen when in point of fact even Al notes it often does? Not all power quality issues are about the components sometimes it's just the wiring in the power system.

So far as how I troubleshoot the problem: how would you like me to troubleshoot a problem that per the posts in this topic seems to exist most often only on a competition field...when i don't have a competition field to work with? Even when I worked at spare parts at MAR Mount Olive it's not like I can put things on that competition field that FIRST won't allow.

We do have a field but we certainly don't have the competition field parts.

If the issue was about me distracting this topic I openly offered to take this conversation out of this topic.

I do respect you and the others Don. However, expending so much effort on scapegoating me to cover for a power quality issues that sometimes do exist is not a reasonable thing to do.

If you tell people to troubleshoot for a software issue and they have a power quality issue then like Team 11 they'll spin their wheels rifling through the software looking for what might just be a bad power supply to the D-Link.
Our issue at the Bayou regional was most certainly not a power issue. We ruled that out early enough... why is it that a team goes from one regional and doesn't run a single match without dropping field communication, goes to a different regional, with a different field, different staff, different environment, run perfectly the entire time with no fixes applied. Literally packing up a 'broken' robot (as FTA deemed ours to be at Bayou), and pulling it out at the LSR, having a highly competitive robot that works 'out-of-the-box'.

There's more going on here than just power issue, but that's not saying that everyone that has communication problems isn't due to power related issues. I'll admit that we did have power issues early on at Bayou, or at least it seemed to be, when we smacked the Dlink, the power would be reset. We corrected it by using the OEM power cable and hot gluing the outside of the plug to the radio to avoid it moving during vibration and hits.

Bandwidth is another thing I don't believe in. While we had a couple of advanced teams at Bayou, it was nothing compared to what we saw at the LSR. Only a handful of robots at the Bayou used live streaming video from the robot to the DS, at LSR, I would say that number doubled or tripled.

Interference? Maybe or maybe not. I'm not an expert in radio interference, so I cannot say whether or not that was the issue. I did notice at least half a dozen APs at Bayou and the LSR, so yea... not sure...

Driver station overloading? Yes, it will cause issues if you are having saturating the CPU on the DS or the cRIO, but that wasn't our issue. We tried 3 different driver stations (Classmate, Dell M5040, and 2010 MacBook Pro), all plenty capable of running the DS software. Our cRIO ran at around 65% usage on average, and for a real time system, that's good.

So yea, my point is that no one can say "This is the problem everyone is having." unless NI or FIRST comes out with a statement saying what the issue is. We have an open ticket with FIRST and they will be investigating the logs from Bayou and LSR to determine what was going on, but as I understand it, the logging features of the field software, the driver station software, and the cRIO software is still pretty underdeveloped, so the cause may never be determined.

There are things that we...
should have done,
could have done,
wish we would have done,
etc...

but we didn't.

These issues for us were just coming up on CD from the Florida regional, where the Bayou field was the week earlier. I wish I would have checked for things such as other routers with our ID running in the pits, and I wish Team Fusion would have kept on an engineering approach and changed one thing at a time when we were trying to solve the issues.

The issues with our field connection put all of our volunteering mentors on edge. Emails were shot out to FIRST with infuriation over the handling of our issues at Bayou from our mentors.

I feel that we proved to the FTA that the issue was not us. While the same FTA that kept blaming us for the issues was packing up the practice field, Team Fusion was running on it, wirelessly, using the hardware they provided to us. We ran our robot for a full 20 minutes without a single drop of communication. I talked with the FTA and asked what he thought the issue was since it runs perfectly everywhere but on his field. He was straight up rude (I could call him a different name...) and said that the issue is not the field, but is still with our robot. If he KNOWS that the problem was with our robot, he should have told us what the problem was, because I spent the three days of the regional ruling out everything but their system. But I believe he said that to cover his butt. That's what he's supposed to do, right? We packed the same broken robot up into the bag and brought it to LSR, and experienced no issues whatsoever. How is that an issue with our robot and not the field?

And the extent of our debugging went down to taking everything off our robot. We put back the given arcade drive code to just drive our base, with the default dashboard software on the classmate PC, with no camera feedback, and still couldn't run. CJ, our CTA at the event spent a ton of time in our pit going over things with us, and I believe he came to the same conclusion as us that the issue was not related to us.

I know that some of this is offending to the FTAs from the Bayou, but I have said nothing that was untrue from my point of view. I'm extremely upset with FIRST over this problem, and so are all of the other mentors for Team Fusion. For many of the students on the team, this was their first event to attend. I can tell you that none of them were impressed with what they saw.

-Ryan Nazaretian
__________________
Controls Mentor
firstfusion364.com

Last edited by RyanN : 04-28-2012 at 05:18 PM.
  #176   Spotlight this post!  
Unread 04-28-2012, 07:10 PM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

As long as the methods used to troubleshoot are anecdotal not quantifiable there's going to be increasingly tension between everyone.

There's too many possible issues, combinations of issues, and things we can't test off a competition field, and that makes sense because the robots are all different and there are different fields. It also makes sense because as we move the robots and the fields around we risk disturbing things.

The field guys are told it's not the field it's the robots.

The robot guys have exhausted all the tools they usually have, including extensive testing, and the problems continue.

This problem seems to continue into situations impacting the performance of the best of the best by elimination. It's no longer about team reputation, veteran status and whose word can be trusted (in a level playing field it should not be about that at all anyway).

There still seems to me only one fix to this problem. To find ways to test and get quantifiable evidence about each system. The field. The robots. Each and every time the problems appear while they appear. FIRST has made great strides in field and robot monitoring since Team 11 first saw communications issues last year at an off-season event and I sincerely do appreciate their efforts.

I hope that FIRST will continue to seek quanitifable information of all kinds to insure that these events move quickly, move cleanly and move with the sort of direction that can only do credit to everyone. It serves no purpose to point fingers at anyone. This is the world of science and engineering it's about the numbers and the evidence.

I still intend to let FIRST evaluate my voltage monitors and even if they only fix a small percentage of the problems by volume...that's a small percentage closer to the goal.
  #177   Spotlight this post!  
Unread 04-29-2012, 09:59 AM
mjcoss mjcoss is offline
Registered User
FRC #0303
 
Join Date: Jan 2009
Location: Bridgewater,NJ
Posts: 69
mjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the roughmjcoss is a jewel in the rough
Re: Intermittent connection on field only

I agree. There are a lot of pieces of the system that can go wrong, and the only way to get a stable competition environment is to have a way to get solid data on a real field. I intend to bring some network tools to view the field at the off season competitions that my team is going to, so that I can take some measurements, and get data. I *believe* the network configuration is part of the problem with the field. Stories of robots turning off their cameras, and have connectivity issues go away isn't proof, just more anecdotal evidence. I also believe that the DLINK is a weak link.

It would be nice to get to the point where the doubts about the field network can be laid to rest. It *may* just be a robot issue, but as long as we don't have data to back that assertion, it's just opinion.
  #178   Spotlight this post!  
Unread 04-30-2012, 07:19 AM
Unsung FIRST Hero
Al Skierkiewicz Al Skierkiewicz is offline
Broadcast Eng/Chief Robot Inspector
AKA: Big Al
FRC #0111 (WildStang)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1996
Location: Wheeling, IL
Posts: 9,825
Al Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond reputeAl Skierkiewicz has a reputation beyond repute
Re: Intermittent connection on field only

Quote:
Originally Posted by techhelpbb View Post
The fact is there is obviously more than one thing not quite right with the radios and the systems that power them.

A loose connector on the D-Link is a power quality issue. Why are we intent on saying that doesn't happen when in point of fact even Al notes it often does?

We do have a field but we certainly don't have the competition field parts.

Championships are over and obviously these issues remain. So what have we solved? How does the situation today differ from the situation at the start of March?
Brian,
We still seem to have a communication issue so I will repeat in as clear a series of sentences as I can.

There is no "obvious" power problem with the radios. There are issues that teams should be aware of that compromise the power delivered to a radio used on an FRC robot. Namely, the power connector can become noisey or intermittent due to team handling. There are rare occasions where the 5 volt regulator is defective, again in most cases due to team handling. And on still rarer occasions (less than ten that I know of in the whole time we have used the PD) the PD power supply may not function correctly. With the 3500 teams, 60+ events, hundreds of matches, thousands of practice runs at both events and home fields, these radios have worked as intended, powered as designed. I personally witnessed failures over this past weekend that were not radio related even though many observers may think so. They were attributable to faults on the robot either in software or hardware that while intermittent were eventually corrected.

We have a good and cost effective competition system that works. There are occasional issues that are as yet unexplained but the FIRST staff is working hard on solutions. They are as committed as I am to insure everyone has the ability to run when they are scheduled to play. Again, there is no indication these issues had anything related to power on the robot radio.

You can continue to look at your own robot and spend your time chasing a power issue that you believe is the only problem that exists. I am convinced (more now that I observed from the scorer's table thousands of matches) that radio power is not the panacea that you believe it is. While I do not wish to stifle teams who are trying to expand our understanding I am firmly committed to preventing misleading lines of thought so that teams will keep their minds open to other possible failure modes.
__________________
Good Luck All. Learn something new, everyday!
Al
WB9UVJ
www.wildstang.org
________________________
All is better now, NOS parts are working fine. Why does this year's game remind me of Violet in Willie Wonka? Hmmmm, I see blueberries!
  #179   Spotlight this post!  
Unread 04-30-2012, 07:55 AM
Gdeaver Gdeaver is offline
Registered User
FRC #1640
Team Role: Mentor
 
Join Date: Mar 2004
Rookie Year: 2001
Location: West Chester, Pa.
Posts: 1,137
Gdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond reputeGdeaver has a reputation beyond repute
Re: Intermittent connection on field only

The CMP proved that there is a problem. Robots go dead during a match. The big problem as I see it is that there are many points of failure. Power and the power connector are just one. Isn't FIRST about the engineering mind set? Let's Give the students a real world example of how to kill a technical problem.
I would like to suggest the people working on this problem need allot more info than they have now. First they need a list of robots and matches were there was a failure. They need specific info about the robots on the field. Everything from radio mounting, language, video stream details, DS configuration, DS laptop details. Failure description (did the robot come back to life) and many more robot specific configuration details. Any solutions they tried. This could be sent out to teams as a questionnaire and the responses tabulated. An analysis may yield points to explore. The dropped coms issue has been around for some time. It needs to be attacked and killed.
  #180   Spotlight this post!  
Unread 04-30-2012, 08:36 AM
techhelpbb's Avatar
techhelpbb techhelpbb is offline
Registered User
FRC #0011 (MORT - Team 11)
Team Role: Mentor
 
Join Date: Nov 2010
Rookie Year: 1997
Location: New Jersey
Posts: 869
techhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond reputetechhelpbb has a reputation beyond repute
Re: Intermittent connection on field only

Quote:
Originally Posted by Al Skierkiewicz View Post
You can continue to look at your own robot and spend your time chasing a power issue that you believe is the only problem that exists.
Again with the straw man arguments. I *never said it was the only problem* and everyone can go back and read my posts to prove it.

Your argument is the equivalent of you have it handled and there's nothing to see here. I think a national broadcast of that was something to see there.

No disrespect to you intended you are in point of fact overwhelmed. It was clearly apparent on the faces of each and every senior person on that field.

It was the direct result of doing everything that could be done to sandbox any argument that differs from your own. In point of fact you continue to do exactly that. It is literally unbelievable. From your own post:

Quote:
We have a good and cost effective competition system that works. There are occasional issues that are as yet unexplained but the FIRST staff is working hard on solutions. They are as committed as I am to insure everyone has the ability to run when they are scheduled to play. Again, there is no indication these issues had anything related to power on the robot radio.
Except on the robots where it was a problem. Now clearly it's unlikely these were the robots during Einstein but your argument ignores your own points about the issues you know exist with the D-Link (namely that power connector). Worse it was you who argued about large relay contacts making noise and having variable resistance in the topic about reverse voltage protecting the Jaguars. The power connector on the D-Link is the functional equivalent of a giant normally closed spring tension relay contact.

Quote:
I am convinced (more now that I observed from the scorer's table thousands of matches) that radio power is not the panacea that you believe it is. While I do not wish to stifle teams who are trying to expand our understanding I am firmly committed to preventing misleading lines of thought so that teams will keep their minds open to other possible failure modes.
Every troubleshooting process for TCP/IP (in fact most things) starts at the bottom. At the electronics and therefore power level. Each and every time you fail to start at the bottom and work your way up you undermine your troubleshooting process.

Again, even if a small number of teams are impacted by power quality issues who is anyone to claim a fair and level playing field when you deny them the opportunity to find their problems quickly?

Last edited by techhelpbb : 04-30-2012 at 01:52 PM.
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 09:50 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi