Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Electrical (http://www.chiefdelphi.com/forums/forumdisplay.php?f=53)
-   -   Intermittent connection on field only (http://www.chiefdelphi.com/forums/showthread.php?t=104713)

RoboSquad 30-04-2012 09:49

Re: Intermittent connection on field only
 
Good Morning All,

Being new to FRC this year I heard about the Bayou issue from a lead NI engineer. So FIRST did send them to LSR I think for free? Knowing this then there is an issue somewhere that needs attention.

I agree we shouldn’t chase some things, however, I would have figured that those running the contests would have some type of monitoring software. If not maybe this is where it should start. Have "HARD" evidence for the FTA staff showing the connection rates etc. and radio strength to prove or support technical difficulties occured during matches (document! Document! DOCUMENT!). As a mentor/coach I would have a hard time selling to my kiddos about their mistakes if I didn’t have something to back it up with when it seems to work fine at home and not at the conest.

Also, I watched the World championship matches and was really wondering what happen with 1114 & 2056 last matches. They averaged 40+ pts. And then to see a sudden drop makes me think of another issue and then the e-mail from headquarters this morning.

I know we try to instill GP across the board and this should be a lesson for all parties involved on how to improve.

techhelpbb 30-04-2012 09:50

Re: Intermittent connection on field only
 
When a team shows up at an event and the only people welcome to troubleshoot on the competition field are your and your group. When your word is gospel. When it can't be questioned.

When representatives of your organization tell a team over and over to go find problems in their robot, and it's clear that your organization field personnel are comprised of people with relationships to certain teams. Worse it's getting increasingly clear that the field personnel don't have any more resolutions than you as a team.

Exactly how do you think your argument does not lead directly to the feeling that certain teams have an advantage by having an 'in' with your organization?

Marginalizing teams and people with troubleshooting knowledge is entirely incompatible with the mission statements of FIRST.

Furthermore Alan spent a decent portion of this topic harping on essentially the idea that Team 11 doesn't know how to measure a battery or build a properly functional robot. It didn't matter what I wrote. The evidence is there for all to see.

So you tell me. Why should I not interpret the continual actions to silence me and the continual actions to down play our team's ability as a direct example of anything other than a very unlevel playing field?

Just how far is FIRST as an organization willing to go to avoid troubleshooting from the bottom up?

Are you going demand next I be removed as a mentor because I stood up and pointed out that this troubleshooting process is tragically flawed when the tragedy has already claimed it's victims?

How far is FIRST willing to go to protect this failure?

I will not go back in my little box Al. Expending all this effort to put me back in that box only demonstrates that there's more wrong in FIRST than a mere failure to troubleshoot some WiFi and field issues.

techhelpbb 30-04-2012 10:28

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Gdeaver (Post 1164366)
The CMP proved that there is a problem. Robots go dead during a match. The big problem as I see it is that there are many points of failure. Power and the power connector are just one. Isn't FIRST about the engineering mind set? Let's Give the students a real world example of how to kill a technical problem.
I would like to suggest the people working on this problem need allot more info than they have now. First they need a list of robots and matches were there was a failure. They need specific info about the robots on the field. Everything from radio mounting, language, video stream details, DS configuration, DS laptop details. Failure description (did the robot come back to life) and many more robot specific configuration details. Any solutions they tried. This could be sent out to teams as a questionnaire and the responses tabulated. An analysis may yield points to explore. The dropped coms issue has been around for some time. It needs to be attacked and killed.

I tried to start a community effort towards this goal already. I was told to slow down.

It does not appear that FIRST values the concept of a community effort. They want a few loud voices to shout over the rest.

All the excuses about my contributions to this topic distracting are also straw man arguments because I asked to be given another place to discuss the matter. They refused.

They don't want to prevent distraction. They want my message to troubleshoot from the foundation up, and myself as the messenger silenced publicly.

Al Skierkiewicz 30-04-2012 10:31

Re: Intermittent connection on field only
 
Okay Brian,
Since you don't know me or my skill set, let me explain. My real job is troubleshooting electronic failures down to the component level. A position at which I excel. I know that power can be a definitive issue with many problems that exist and I know both how to diagnose those issues, and how to correct them when they fail or are designed improperly. Power supply noise in digital systems is nothing compared to the noise generated in analog audio systems where microphone levels are in the -60 to -90 dBm (that's millwatt for those who are wondering) range, can require as little 4 microamps from the power supply and where noise is considered bad when it is only 20 db above the theoretical noise floor. I have been telling you for weeks that power is not the issue you have deemed it to be but you have failed to believe me or the others in this forum that are trying to get you to accept that fact. While there are problems (and they have yet to be diagnosed), power is not the greatest of these. How can you even think that those people who are researching every possible failure point have not considered looking at the power supply first? So to borrow from others, stop muddying the waters, perform your tests and bring us real data that can be duplicated in the lab or field and actually correlates to failures of the robot wireless link. Until that time I will only respond to others seeking real answers.

Edit:
Now that I have have seen your most recent post, let me assure you that Team 11 is a long time friend and a team I certainly respect. When at competitions, I would work as hard helping them compete as my own team and I expect all my inspectors to also do the same.

techhelpbb 30-04-2012 10:39

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1164410)
Okay Brian,
Since you don't know me or my skill set, let me explain. My real job is troubleshooting electronic failures down to the component level. A position at which I excel. I know that power can be a definitive issue with many problems that exist and I know both how to diagnose those issues, and how to correct them when they fail or are designed improperly. Power supply noise in digital systems is nothing compared to the noise generated in analog audio systems where microphone levels are in the -60 to -90 dBm (that's millwatt for those who are wondering) range, can require as little 4 microamps from the power supply and where noise is considered bad when it is only 20 db above the theoretical noise floor. I have been telling you for weeks that power is not the issue you have deemed it to be but you have failed to believe me or the others in this forum that are trying to get you to accept that fact. While there are problems (and they have yet to be diagnosed), power is not the greatest of these. How can you even think that those people who are researching every possible failure point have not considered looking at the power supply first? So to borrow from others, stop muddying the waters, perform your tests and bring us real data that can be duplicated in the lab or field and actually correlates to failures of the robot wireless link. Until that time I will only respond to others seeking real answers.

Your professional credentials have nothing to do with this Al. Nothing at all. You are not testing each and every robot personally.

You know very well that I can't test a competition field environment without a competition field. So you're telling people that I should basically test something you are insuring can not be tested.

Furthermore, each and every robot on that field is wired differently. Your argument that you've tested this out fully simply can not be true. You have only tested the ones you've tested. So your lots are smaller and the results are always rushed because you most often do those tests on team's robots when you access them during a competition and you only test the ones that do not work. Worse you don't have a complete tool set to test on the field during the matches.

The point again, for all to read, is not that all issues are power quality issues. The point is that when you do have power quality issues you are effectively putting those teams at a disadvantage. I fully admit...again and again in this topic there are more problems than merely the power quality.

The only thing I offered was a way to help identify and remove those power quality issues when they exist I left the rest to you to figure out.

Note: this next part was posted before Al edited his post.

So far as this not being a real issue...you prove once again that Team 11 and other teams finding bad power supply components feeding the D-Link AP is not considered a real problem. It's not a real problem why? Perhaps because Team 11 and these other teams weren't on the Einstein field? Going to be hard to get to that field when our not so 'real' problems get in the way.

RyanN 30-04-2012 11:19

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1164410)
Okay Brian,
Since you don't know me or my skill set, let me explain. My real job is troubleshooting electronic failures down to the component level. A position at which I excel. I know that power can be a definitive issue with many problems that exist and I know both how to diagnose those issues, and how to correct them when they fail or are designed improperly. Power supply noise in digital systems is nothing compared to the noise generated in analog audio systems where microphone levels are in the -60 to -90 dBm (that's millwatt for those who are wondering) range, can require as little 4 microamps from the power supply and where noise is considered bad when it is only 20 db above the theoretical noise floor. I have been telling you for weeks that power is not the issue you have deemed it to be but you have failed to believe me or the others in this forum that are trying to get you to accept that fact. While there are problems (and they have yet to be diagnosed), power is not the greatest of these. How can you even think that those people who are researching every possible failure point have not considered looking at the power supply first? So to borrow from others, stop muddying the waters, perform your tests and bring us real data that can be duplicated in the lab or field and actually correlates to failures of the robot wireless link. Until that time I will only respond to others seeking real answers.

Edit:
Now that I have have seen your most recent post, let me assure you that Team 11 is a long time friend and a team I certainly respect. When at competitions, I would work as hard helping them compete as my own team and I expect all my inspectors to also do the same.

I agree with Al here, but I also agree with power being a contributing factor to some teams, but again, as Al pointed out, most teams do start from the ground up in the debugging process.

As I've mentioned many times before...
Team Fusion, at the Bayou Regional, could not run a single full match due to communication problems. Our first suspect was power, and sure enough, we did have a power issue.

The student that purchased the connector for the router purchased one that 'fit' but didn't fit properly, leaving us to have a potentially bad connection. We could pick up and drop, shake, etc... our robot, but it wouldn't drop, but if we put focus to abuse on the router, and picked up, dropped, shook, slammed the router, then we would cause a reset.

That was our first step. We had a spare router, with a spare power cable, so we took the OEM plug and used that. After replacing and gluing the plug in place, we could not brown out the router. We did stress testing to other power components of the router as well, including the 12-to-5 regulator, and the PD board. We replaced the 12-to-5 regulator for comfort, and we hit on the PD board a bit, but could not replicate the issue.

And again, as the title of this thread mentions, the teams with this issue had a "Intermittent connection on (the) field only." We worked great in the pits, on the practice field, during kickoff, practicing at home with our router, everywhere, but the dang Bayou field.

The only thing the FTA would tell us is that it was a robot problem. No specifics on what is failing, or why, just that it was an individual problem with us.

Here are the two things I cannot get around:
  1. We were the only one at Bayou with this issue.

    CRyptonite had a match with us where they dropped out, but after Bayou, I was told that they were experiencing the CAN Autonomous-to-Teleop transition bug.

    Prometheus had an issue late Friday, but I think they determined it was an issue with their camera or something...

    Combustion had lag, but that was related to high CPU usage on the cRIO.

  2. We packed up our broken robot, kept the same code, etc... to the Lone Star Regional, and pulled out a fully working, functional robot.

    The only thing we changed was reconfiguring our router to work with the LSR field. Greg McKaskle worked with us at LSR, but unfortunately for him, but fortunately for us, we never experienced a communication problem at Lone Star.

There's something external to the robot with the issue we experienced. With what I saw on Einstein, the issues experienced by all of those teams was the same issue we experienced at the Bayou Regional, a Week 3 regional.

As far as router placement, our original location isn't the best place, and is within a foot of some noisy shooter motors, but we relocated it during Bayou to a better location, obviously not fixing our issue.

As far as code goes... besides the fact that our code didn't change from Bayou and LSR, we went back to the basic framework code, no camera (unplugged), no CAN, no PID, nothing. We simply had solenoid and motor outputs, no sensor inputs at all.

Then comes the driver station. We had been using a Dell purchased this season as the DS, but during all the debugging, we switched to my laptop, a 2010 MacBook Pro (Core i5, 8GB RAM, SSD... a pretty powerful machine), and we kept on having the issues. Finally, when trying out the basic framework code, we switched to the Classmate PC, configured for this year and updated with the latest FRC updates, and no Windows Updates.

Basically, at the end of Bayou, we had a kitbot control system, but still couldn't maintain a connection, even sitting still.

All of this proving, in our case, to me, that this isn't an issue with Team Fusion 364's 2012 Robot, Aiminite. Our issue was not a robot issue. That's all I have concluded from the work I have done diagnosing the issues.

I'm not sure if we were the only one during the regionals to pack up a broken robot and pull out a working robot, but FIRST should look at our case for some good information. Team Fusion has concluded that the issue is not our robot.

Now to the big question that everyone would like to know... what is the issue? No one knows at this point. Everyone is just speculating.

Interference
I've been speculating that the issue is interference from the audience carrying hundreds of smart phones with WiFi enabled... but that even has errors in the hypothesis.

Lone Star had a much larger crowd than was at Bayou. It's much larger event. The Lone Star regional is also held in a bigger city (and actually in the city). Bayou was right next to the river, with just a small road leading up to it, not part of a major city arterie. It seems like LSR had the potential to be worse as far as interference goes. So interference seems to be a non-issue. And why would interference just target us? Things don't add up for this case. What about the WiFi channel used? What does happen with the Dlink is the 2.4GHz is left enabled?

Field Hardware Issues
Again, if it truly was FMS messing it, at Bayou, it would have affected everyone, not just Team Fusion. We had been placed in multiple spots on the field, but we had problems in all of them.

I'll be down on the Coast next week, and can test some things, but I need a list of things to test.

First off, I want to break communication. What can we do to break communication? We'll need to have two routers and use the same system FIRST uses.

I'm curious as to what will happen is we power up two of our robots, with the same IP address and everything. How will the system handle this? If it works, how do our logs look?

Then what happens if we run some noisy equipment next to the 'field' router. Can we run a Skil saw next to it and still maintain connection? How about running a FP motor at full speed next to it? What happens? How do the logs look with this?

Did Bayou have wireless lighting controls? Were their frequencies within the range of our channel provided? I can't tell.

Anyway, I want to help figure out this issue because it really messed us up in Bayou, and we have no explanation as to what happened.

techhelpbb 30-04-2012 11:28

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1164410)
Now that I have have seen your most recent post, let me assure you that Team 11 is a long time friend and a team I certainly respect. When at competitions, I would work as hard helping them compete as my own team and I expect all my inspectors to also do the same.

Again Al I do respect you. However, you are not personally at all the competitions. Team 11 being friends with you doesn't reduce my concerns.

A level playing field is just that. It should not matter if Team 11 is friends with you. It should not matter if Team xxxx who just walked on that field you've never met is friends with you.

Troubleshooting is troubleshooting. We must insure that guidance for it is uniform. We also must insure that if a relatively unknown team does in fact eliminate all other problems we respect the effort and investigate.

efoote868 30-04-2012 11:33

Re: Intermittent connection on field only
 
Quote:

Originally Posted by mjcoss (Post 1162062)
I originally believed that each robot was going to be assigned a separate channel and that they were using 802.11n @ 5 Ghz. The network at the MAR championship was indeed using 802.11n @ 5Ghz, and they are using a wide channel, however all robots were sharing the same channel and as such were sharing a total theoretically bandwidth of 300 Mbits.

I don't know of any wireless product that comes close to it's theoretical rating.

This year, I was running throughput testing for a demo board. At 10-30 feet in a quiet environment (as shown on a spectrum analyzer), we were achieving about 120 Mb/s out of a theoretical 300 Mb/s using IXChariot software.

If all robots are sharing one channel, that is pretty unnerving, especially given the throughput requirements for a single video feed.

techhelpbb 30-04-2012 11:39

Re: Intermittent connection on field only
 
RyanN:

I deeply respect that power quality issues are not the only issue on the field. That's why I asked to separate the issue repeatedly.

Obviously when component in the power supply to the D-Link AP goes bad it can do so at anytime.

Team 11 drove our robots extensively before the competition in Philly even at other competitions. All of a sudden the DC-DC converter decided to be a problem. We, like apparently most people were not instructed to load test the D-Link AP supply, no process was provided nor even basic specifications. I did ask repeatedly in this topic whom else load tested that supply with no responses.

This was a power quality issue that was intermittent on the field only seemingly, but just because the timing of the failure coincided with the punishing match schedule.

Not all power quality issues are wiring problems. Not all power quality issues are component issues. As others have said, in the absence of procedure and tools it's very hard to walk a troubleshooting process to work that out especially under pressure and especially when there are real WiFi communications issues that keep undermining the basic troubleshooting process.

It's entirely conceivable in the current test environment for a team to check their wiring, replace the D-Link AP. Fail on the field. Replace their DC-DC converter and fail again on the field. Replace their PDB (which can easily consume all the time between matches) and fail on the field.

Then after all of that diligence, they fail on the field for other reasons which may, or may not have been there all along as well.

So in effect that's about 4 matches of effort right there where a robot might be totally or mostly non-functional. That's a great deal of ground to loose. Compound that with the fact that your robot's mechanisms might not be top tier and those 4 or 5 matches might put you out of the competition entirely.

I know we are here to above all have fun. How much fun is it for a rookie team to possibly loose 4 matches due to an unclear troubleshooting path, have to go home and find more sponsorship to come back next year, and while they are at it very likely be entirely unable to repair the issue once they leave that environment? Doesn't sound like fun to me and please be aware I was one of the people that founded Team 11.

Given the opportunity to treat this issue separately that's what should be done. I respect that your team expended all possible tools and efforts to eliminate your concerns. However, until someone helps me to separate these 2 topics properly I am left little choice to respond within the limited venue I've been offered. If people were trying to respect this basic foundational concern they'd help me move out of the way instead of demand my silence.

Perhaps more importantly, demanding my silence here is symptomatic of the fact that we have had communications issues with this system for years and in the past the finger was pointed at the robots. The teams had no reason to believe that the problem could be anything but them because the authorities on the subject insisted that it could not be the field. People should stop insisting, open the floor, divide the topics and if you want me to shut up and test myself let me get onto a competition field and do what I need to do.

Dad1279 30-04-2012 15:19

Re: Intermittent connection on field only
 
Quote:

Originally Posted by RyanN (Post 1164450)
..........
CRyptonite had a match with us where they dropped out, but after Bayou, I was told that they were experiencing the CAN Autonomous-to-Teleop transition bug.
........

What is the CAN Autonomous-to-Teleop transition bug?

I believe we were hit with this last year, but have not seen it documented or mentioned elsewhere. We had 2 unexplained no-coms at NYC regional last year, FTA said it was a bad battery. On a whim we removed CAN and switched to PWM for the next regional, and did not have a problem.

Has anyone else been hit with the 'Joysticks disconnect' problem this year? It would happen mid-match, and the fix was swapping the joysticks in the DS and swapping them back...

Mark McLeod 30-04-2012 15:37

Re: Intermittent connection on field only
 
FRC 2012 - Known Issues List

Quote:

Issue: If CAN is used in the Autonmous VI, the robot cannot transition to the Disable or Teleop VIs.

Workaround: Terminate the autonomous routine just before the VI would be aborted, to stop calling CAN updates before the end of the autonomous period
Move the CAN updates to Periodic tasks and have autonomous simply update setpoints

mjcoss 30-04-2012 15:45

Re: Intermittent connection on field only
 
Unfortunately, which robots are on the field is part of the "environment", and so working in one regional and not in another could simply be the mix of robots. I believe that the DLINK is certainly one point of failure, from the battery to the power connection on the box, and internal to the DLINK. I also believe that there are issues with how the network is configured.

Maybe at one of the off season competitions - our first one is at Monty Madness, we can get a group together and poke and prod the field setup, and network with a mix of different robots. If we get enough robots together we might be able to map out some of these issues, on a real field.

I think that we would all like to have a level playing field for all robots. And to do that we need data on real fields, and I would like to be involved in the analysis, not simply told that it's "fixed" because I'll be back to the "there doesn't seem to be any issue with my robot, in the pit, or at our build site, and there is at competition". And once again, left to tell the kids that I can only theorize about what is wrong because the conversation is closed when it comes to discussing details about the field with FIRST

MaxMax161 30-04-2012 16:07

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Dad1279 (Post 1164629)
Has anyone else been hit with the 'Joysticks disconnect' problem this year? It would happen mid-match, and the fix was swapping the joysticks in the DS and swapping them back...

Yup, we've had that three times this year and a few times last year. Still no ideas on how to prevent it.

Dad1279 30-04-2012 16:42

Re: Intermittent connection on field only
 
@Mark - So that's an issue with Labview and not Java? (We were using Java)

Dad1279 30-04-2012 16:45

Re: Intermittent connection on field only
 
Quote:

Originally Posted by MaxMax161 (Post 1164662)
Yup, we've had that three times this year and a few times last year. Still no ideas on how to prevent it.

Ah, a common factor.... It only hits teams from NJ! That should make it easier to find.....


All times are GMT -5. The time now is 17:53.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi