ARENA Fault

[Jon236]

Ray, thank you for your insightful analysis. As I indicated in the posts about the problems surrounding the Israeli Regional (http://www.chiefdelphi.com/forums/sh...793#post938793), our FTA and support staff worked tirelessly to resolve issues. We could always move backwards to a simpler system; but that system cannot offer the processing power available in the cRio. Challenges are what FIRST is all about. We all have to accept the challenge and move forward; we will be better for it.

We do a disservice as mentors if we simply complain. We must inspire our students by showing them how we rise to the challenge and solve the problems we face.

[kylelanman]

Originally Posted by Al Skierkiewicz

Eric and Kyle,
I believe I was the inspector. The team (and many other teams) brought me an adapter that they couldn't ping. Neither could I, even after two manufacturer resets. I could not connect, I could not see the web interface although it appeared to be handshaking to the computer. A power reset at one point, sent the device into never never land and it never completed. The team took it from me when it would not boot. I merely confirmed what they already thought.
In the final match in Wisconsin, three robots failed. I was informed that two of them lost the Classmate due to dead batteries and the third was a shutdown of the USB hub powered from the Classmate, also a battery issue.
I wish I had a nickel for every match I have missed because I was helping a team. I could buy a nice lunch.
I believe the alliance station also flashes during autonomous if a robot is not running autonomous code.

Al, We thank you for trying to help us with our radio last second. We were in the web interface and configuring it fine. We simply had a configuration question. From what I understand from other threads the program where you simply enter the team number does not work with the newer radios. Would have been nice to know that then. Oh well. After it was reset completely we did not have the time that it would take to configure it from scratch because after applying nearly every setting you have to wait for the device to restart which takes nearly a minute. The robot worked in the next match so the drivers told us to forget about messing with it.

Al, Do you by chance recall which teams were because of dead classmate batteries? We didn't have a USB hub so I would assume one of them was us but I also know that team 1736 let us use there fully charge classmate battery because First did not provide a way (that we were aware of) to charge the classmates during the final matches. I recently found out about the inverter option with a spare robot battery.

The reason I want to know how the comm status is determined is because there are multiple network connection between the classmate, FMS, and the robot. I fully understand a robot radio loosing comms is our fault. There are a countless number of things that could cause this. If the classmate battery is fully charged, ethernet cable securely connected and the classmate looses comms to the FMS I would have to blame the field/arena. The classmate is an unmodified (software and hardware) product from First so nothing the team does aside from the previously mentioned network cable being plugged in properly would influence the comms between the classmate and the FMS. If I am wrong here correct me.

[Radical Pi]

Originally Posted by kylelanman

Al, Do you by chance recall which teams were because of dead classmate batteries? We didn't have a USB hub so I would assume one of them was us but I also know that team 1736 let us use there fully charge classmate battery because First did not provide a way (that we were aware of) to charge the classmates during the final matches. I recently found out about the inverter option with a spare robot battery.

as an FYI the frc director's blog got a post today that said power strips are now available in the queue for charging the classmate while waiting for the match.

Originally Posted by kylelanman

The reason I want to know how the comm status is determined is because there are multiple network connection between the classmate, FMS, and the robot. I fully understand a robot radio loosing comms is our fault. There are a countless number of things that could cause this. If the classmate battery is fully charged, ethernet cable securely connected and the classmate looses comms to the FMS I would have to blame the field/arena. The classmate is an unmodified (software and hardware) product from First so nothing the team does aside from the previously mentioned network cable being plugged in properly would influence the comms between the classmate and the FMS. If I am wrong here correct me.

According to rspurlin the FMS indicators are pulled from the classmate. I'd assume that those are used to determine communication, as well as an extra indicator for classmate communication. Also you should remember that the FMS is a fairly big thing. There are multiple fault points, however most of them would cause widespread failures.

One more thing out of curiosity: In the event that a ball is scored and properly returned on the ball return, however the return sensor fails and doesn't detect the ball, does the scorekeeper have a way to manually override the DOGMA penalties to keep the game from killing an alliance for a field error? Also, what happens if there is the reverse, a ball is missed by the ball counter and is caught by the ball return. Any idea how the system handles this?

[ChrisH]

I'm not sure what all the paths are but by looking at your Dashboard under the Diagnostics tab I can tell whether:
the DS is talking to FMS
what mode FMS/DS is in (auto/teleop/enabled/disabled)
the bridge is talking to FMS
the bridge is talking to the cRio
the cRio is running code
you have decent battery voltage

BTW this is more detailed information than we have ever had before, even in the rosy IFI days some people remember so fondly. I might not be able to tell you what the source of your problem is, but I can pretty well tell you where to look.

[Radical Pi]

On the topic of FMS-classmate-robot or classmate-FMS-robot, I do see one big security issue in the former. We all know about the FMS Locked state, but few seem to know that FMS Locked can be bypassed with F1 for enable and Space bar for disable (those also work in regular mode. Much more convenient than messing with the mouse). If the FMS somehow loses connection with the classmate, but the classmate-robot communications can continue, if the classmate activates the bypass would the robot start operating without being under FMS control?

[kylelanman]

Originally Posted by Radical Pi

as an FYI the frc director's blog got a post today that said power strips are now available in the queue for charging the classmate while waiting for the match.

That helps all the teams from the previous regionals.....It will be nice in the future though. Btw does anyone know why First has never and from what we can tell never will provide power at the driver station. It seems like a simple thing that would make everyone's lives easier and the question has been asked from multiple people on our team. Quite frankly we don't have a darn clue.

Originally Posted by Radical Pi

One more thing out of curiosity: In the event that a ball is scored and properly returned on the ball return, however the return sensor fails and doesn't detect the ball, does the scorekeeper have a way to manually override the DOGMA penalties to keep the game from killing an alliance for a field error? Also, what happens if there is the reverse, a ball is missed by the ball counter and is caught by the ball return. Any idea how the system handles this?

The blue ball counters rarely worked at WI. It took the refs a few matches to realize that the teams were not as slow as they thought. We originally lost a match because it said we got three penalties for not getting balls back in time. When we looked the match results the next day we saw the penalties were gone. The majority of the matches the refs / others kept score manually.

I just confirmed that our classmate battery was 75% charged. If this causes a problem then from the sounds of it maybe the classmate is not the computer to be using a driver station.....or as previously stated it should be able to be powered during competition.

[Radical Pi]

Originally Posted by kylelanman

That helps all the teams from the previous regionals.....It will be nice in the future though. Btw does anyone know why First has never and from what we can tell never will provide power at the driver station. It seems like a simple thing that would make everyone's lives easier and the question has been asked from multiple people on our team. Quite frankly we don't have a darn clue.

Someone in another thread said that the power spots for last year's DS have been taken up by the scoring systems now. 3 units (2 goals and the return) to replace the 3 DS spots. FIRST probably just didn't want to add more power strips to the field.

Originally Posted by kylelanman

I just confirmed that our classmate battery was 75% charged. If this causes a problem then from the sounds of it maybe the classmate is not the computer to be using a driver station.....or as previously stated it should be able to be powered during competition.

75% seemed fine for us. We ran matches below 20% by the end of the day Friday and we didn't have any battery-related issues

[Al Skierkiewicz]

Kyle,
I didn't even see the final match as I was helping out with some other issues. I was told afterward by field volunteer who spoke with the FTA. No teams were identified.
If you were able to open the web interface than I was thinking of someone else. On the device I was on I couldn't do anything.
In Wisconsin, volunteers were counting balls as they passed through the goal. I don't know if or how the Dogma was working. I only know that the ref asked for a count from each volunteer, one at each goal.

[rspurlin]

@kylelanman

I wish I could definitively answer your questions, but there is much more that I don't know than I do. Here are my best guesses:

I think Eric H. is correct above in his description of the alliance station light. This light is different this year than in previous years. You want this solid. I'm not sure what controls it specifically, but it has a different flash pattern than the Robot signal light.

The robot signal light is discussed here. Once again you want this on solid. In finding this document I was reminded that there needs to be a jumper installed for the amber light to accurately reflect the LED status on the digital sidecar.

The classmate is able to exchange packets with the robot prior to match begin. FMS will not allow a start with robots not ready. If a robot-DS connection cannot be made, your robot can be bypassed to enable the match to begin, but at events where I have been, you would know that was the case before the MC begins the countdown. So what we need to know is why your connection dropped.

I'm guessing based on your screen name, but assume that you know something about what a non-routable network is. This is essentially what an FRC field is. This is why if robot communications fails to just one robot, the culprit is most likely either on the robot itself or the classmate and its individual ethernet cable. A failure in the rest of the wiring loop would most likely affect multiple robots.

I do not know how the classmate is controlled by FMS. I think it is most likely detecting the presence of the FMS and then behaving slightly differently, but I'm not certain. At first glance, I'd try to do it this way since once the match is started, packets to and from the robot are then handled only by the communications hardware (router, switch, access point) and not through any upper level software program (which could reduce bandwidth). It would be simple enough to have the driving program on the classmate poll the FMS every 100 ms or so and enable/disable as required. Of course, that might not be how it is really implemented.

I'm not a big fan of Windows*, especially on underpowered devices, so I'm not happy with the boot time and other idiosyncrasies of the classmate. However, it's much better than last year's blue box of death. I have a laptop that forgets it has a DVD drive every time someone closes the lid, so I'm not sure I'd recommend your logout/login regimen. We've already heard reports that USB connections can get wonky in certain circumstances. I think there are issues we are still uncovering as our first season with this device unfolds.

I'm pretty sure you can't connect with the field if your classmate is logged in as developer. You may have been able to have the same effect by simply logging out and logging back in. I've heard that it is necessary to do this after each match to clear the FMSlock.

At your next event, please check with the FTA early on practice day and let him.her know what you have experienced so far. Perhaps by that time more answers will be available.

At my next scorekeeping event, I will be taking even better notes about what is happening and what our best guess is regarding the causes. Hopefully this will help improve gameplay for all teams.


* I am an IT guy IRL, so please spare me the flame war. I haven't met the perfect computer/OS/programming language/DBMS/app/etc in over thirty years of working with them, so let's just focus on making what we have better, ok?

[rspurlin]

Originally Posted by kylelanman

The blue ball counters rarely worked at WI. It took the refs a few matches to realize that the teams were not as slow as they thought. We originally lost a match because it said we got three penalties for not getting balls back in time. When we looked the match results the next day we saw the penalties were gone. The majority of the matches the refs / others kept score manually.

When you say you looked at match results do you mean match review in FMS?

I just confirmed that our classmate battery was 75% charged. If this causes a problem then from the sounds of it maybe the classmate is not the computer to be using a driver station.....or as previously stated it should be able to be powered during competition.

I can't say about the classmate specifically, but some laptops do treat attached peripherals differently based on power profile and will select different power profiles based on the amount of perceived charge.

I've already ordered a power cable to connect an inverter to the standard robot battery. I don't want to go on the field with a classmate that is less than 100%. Especially considering the potentially long queue waits at championships.

[kylelanman]

Originally Posted by rspurlin

When you say you looked at match results do you mean match review in FMS?

http://www2.usfirst.org/2010comp/eve...chresults.html

Match 2

[sparrowkc]

Some information on how the FMS works can be gleaned from the FMS lite package that was released for the 2009 game. Here's the user's manual:
http://usfirst.org/uploadedFiles/Com...Guide_RevC.pdf


I would add that I find it extremely frustrating that not only are we not allowed to look at the source code for the FMS or DS, we're expected to deal with their imperfections without even a general idea of how the field works (or doesn't).

[rspurlin]

Originally Posted by kylelanman

http://www2.usfirst.org/2010comp/eve...chresults.html

Match 2

The posted results do not list components of the score, just the final result. So I'm confused. Do you mean that after the match, red was announced as the winner, yet sometime later some penalties disappeared and blue became the winner?

As a hypothetical situation, had such a situation happened at our event (discovery after the match results were announced that an error needed to be corrected affecting the outcome) the head ref would go to each of the teams involved in the match to explain what had happened and what the new ruling was. After all teams are notified, a general announcement would be made by the MC and/or head ref to the crowd, explaining what happened and what the new ruling was. Because the revised score would be committed to FMS as soon as the new ruling is reached, rankings would reflect the fact first.

I'm surprised about the ball counters at your event. At ours, we think they worked flawlessly. We tested them Wed night, and Thursday, Friday and Saturday mornings. I'm not aware of any complaints that a score was incorrect relative to the ball counters at our event.

[rspurlin]

Originally Posted by sparrowkc

I would add that I find it extremely frustrating that not only are we not allowed to look at the source code for the FMS or DS, we're expected to deal with their imperfections without even a general idea of how the field works (or doesn't).

Dealing with imperfections is a theme FIRST recognizes in areas other than programming. I wonder if the source code to Windows would be more helpful to troubleshooting DS problems than the source to FMS.

Seriously, I'd be more worried if the source were freely available that a large well funded team could exploit some weakness.

While I'm not dumb enough to assert that FMS or the DS software is bug free, it seems pretty solid from my perspective and will get better if we help make it so.

[kylelanman]

Originally Posted by rspurlin

The posted results do not list components of the score, just the final result. So I'm confused. Do you mean that after the match, red was announced as the winner, yet sometime later some penalties disappeared and blue became the winner?

Yes. It was a nice surprise when we saw it that night seeing as how it was our originally our only loss.

Originally Posted by rspurlin

As a hypothetical situation, had such a situation happened at our event (discovery after the match results were announced that an error needed to be corrected affecting the outcome) the head ref would go to each of the teams involved in the match to explain what had happened and what the new ruling was. After all teams are notified, a general announcement would be made by the MC and/or head ref to the crowd, explaining what happened and what the new ruling was. Because the revised score would be committed to FMS as soon as the new ruling is reached, rankings would reflect the fact first.

I know the ref did not talk to myself personally and everyone on the teams was shocked when they saw that we ended up going undefeated so I do not believe the head ref told anyone.

Originally Posted by rspurlin

I'm surprised about the ball counters at your event. At ours, we think they worked flawlessly. We tested them Wed night, and Thursday, Friday and Saturday mornings. I'm not aware of any complaints that a score was incorrect relative to the ball counters at our event.

It worked during practice and possibly for Q1. From then on out the live score on the board was never correct for the blue alliance and you had to wait till the end to see it. It made it nearly impossible to try and keep a 1 point margin between the winners and losers not knowing the score. So we altered our strategy and went with score as many points as possible at all times.