Cypress / First touch I/O module recognition by DS after suspend

Technical Control System
1

Looking for feedback from teams who use the Cypress First Touch I/O module and have it successfully operating with the Driver’s Station and their robot…

Does the Drivers Station software recognize the I/O module for you after you suspend it and then wake it up again?(without restarting the DS software or the Classmate itself)

The DS recognizes the Cypress I/O module fine for us off a fresh boot of the laptop, and everything operates properly.
However when suspended (lid close) and the re-awoken the DS software will not recognize the module. Even restarting the DS software will not result in it being recognized.
Rebooting the Classmate is the only way we found to be able to get the Cypress to be recognized again following a suspend.

2

This happened to us during a qualification match at FLR on Friday. We closed our laptop, for transporting the robot, and when we opened it up, we though it was working. Little did we know (but quicklly figured out) that none of our controls worked, except for the joysticks, which were plugged direct to the laptop. This is really quite a serious problem, and FRC should make an update telling teams no to do this.

3

Is this reproducible? I wasn’t able to provoke it.

How are devices connected to the USB, what is right, left, and what is on the hub?

Finally, is it possible that the FT is drawing too much current or is shorting out?

Greg McKaskle

4

We had this same problem in DC. If we shut the lid to the classmate upon reawakening it the cypress board was not recognized. We found that it required a reboot of the classmate to reestablish connection

Our solution was to just carry the classmate to the field open and never shut the lid.

~Allison

5

If you turn the laptop off and then boot it up while it is connected to the FMS, then it will not find the Cypress board unless you reboot it twice without the FMS. This killed us twice. We also found that once it looses comm to the Cypress board, it keeps sending old data to the cRio, so there is no simple way to detect this loss. We had one Logitech gamepad and the Cypress board connected to the KOP hub, with both ethernet cables on the hub connected to the Classmate. No problems at all during testing at home. Talking to other teams at Kettering, most of the teams that used the Cypress board had a problem with it of some sort. As a side note, it is really annoying how the Cypress board operates on 3.3v while everything else in the system is 5v, they could have picked a better IO board.

6

Yes, 100% reproducible for us:

Attach all devices (see below), boot up, DS opens in Driver. Once stable, close lid. Wake system up. DS software will not recognize I/O board at this point. Restarting DS software (Driver Log-out, then back in, or even going to developer and running there) does not matter; it still will not find it.

How are devices connected to the USB, what is right, left, and what is on the hub?

The cypress, the two 2009 Kit joysticks, and the STOP button are all plugged into the hub, which is plugged into the right port. Nothing is in the left port (due to physical constraints).

Finally, is it possible that the FT is drawing too much current or is shorting out?

Given this configuration, too much current is possible I suppose. But Windows / Classmate USB subsystem is not shutting the port down as would be typical for over-current. The joysticks and e-stop on the same port continue to function and be recognized by the DS.

But, if we never suspend the system it works fine (throughout an entire day of competition). In addition, the board has LEDs attached to it which do light (“randomly” as they don’t get robot-originated commands) even in this condition.

It is not shorting out.

How does the DS do recognition of the Cypress? Could this be some sort of race where the board may not be fully re-initialized/woken up sufficiently when the DS looks for its presence again? Does it look for it when you do the F1 re-enumeration? Pretty sure we tried that, but would try it again if this is the case.
We can also try changing the hook-up scenario and report back.

7

Thank you both for reporting similar issues.

We’d gone a similar route to Allison’s team; we’ve configured our Classmate to not go to suspend when the lid is closed. But this has severe drawbacks in elimination rounds where you’d like to sleep it between matches to preserve battery (and speed the charging process when on charger between match).

8

You can get the I/O board back by terminating the Cypress service and restarting it under Windows Start -> Administrative Tools -> Services. Takes a few seconds. It’s just annoying to have to do, and it’s unfortunately one of those “secrets”.

I seems to be a Cypress issue. They know about it.

9

I don’t have a classmate to experiment with. Does this happen if you plug the Cypress directly into the laptop, then close and reawaken?

I’m confused why we cannot reproduce this. Can you report the version of the driver? Mark, you indicated that Cypress was aware of this. Can you provide a link?

Greg McKaskle

10

This is a 1% problem.
If it’s any consolation, I can’t reproduce it either, but I also haven’t spent much time examining the issue, so maybe I just haven’t tried hard enough.
Work with a statistically significant number of teams and you’ll run across it.
We’d see it more if more teams used the IO board.

My experience with it has only been a single team at NJ in triage for more serious issues, so not much time was spent on this one problem. A work around was good enough.

I believe the problem was narrowed down to the Cypress Board (although I didn’t test the USB cable connecting it to the Classmate).

Classmates were swapped out and it followed the Cypress board.
Cypress board plugged directly into the right Classmate USB port.
It could be produced by sleep and by repeatedly plugging and unplugging the USB connector.
It only happened on the Enhanced IO for the one I saw.
I did not have time to try a different cable or IO board from Spare Parts.

Cypress tech. support was consulted by phone by the team before the Regional, so take my comment about Cypress knowing about it with a grain of salt. The mentor is quite experienced in LabVIEW real-time processing however (and expressed the typical complaints about some of the low-level non real-time calls employed:) ).

11

Thanks. When I said I don’t have a Classmate, I meant at my house, BTW.

Can someone with a failure on awaken try this without the I/O board connected? With the board connected, but without any outputs connected, with outputs and no inputs? Also, if you awaken and the board doesn’t work, can you verify whether the Control Panel>>Admin Tools>>Services shows the PSOC service running or not?

Assuming that the service stops running on awaken, does a reinstall of the PSOC programming tool change things?

Greg McKaskle

12

I can contribute answers to some of those questions from my tests with the team at the event. I just can’t try it again now.

The service was listed as running after the failure occurs. Stopping and Starting the service cleared it up as you would expect.

With the Classmate change:

  • The first (team) Classmate exhibited the problem during build season, but for other reasons we restored it from scratch and reinstalled the Updates. After the Restore, I had them uninstall the Cypress software, then install DSUpdate1.0, followed by DSUpdate1.1
  • The 2nd Classmate was from Spare Parts and as a week 1 regional it got updated when we came in on the first morning.
    While I did look at the PSoC version number, I cannot confirm that because I don’t have a firm memory of it. I can only remember that I was happy with it at the time.

The Classmates were generally booted without the IO board connected.
The IO board was plugged in afterwards.

13

I had access to our team’s school (and, hence, our OI) for some time last night to experiment with this in a more controlled environment than competition.

While it was happening 100% of the time at competition, I could not reproduce with 100% success rate at our home facility. Significant here is that I did not have the same robot from which to send commands to the board (I had linked it to our 2009 cRIO, but the code on such does not use the extended io as we do in 2010). I could, however, still reproduce it about 60-75% of the time.

Happened more often than not when on battery power than plugged in. Still same configuration I mentioned above (right classmate port plugged into hub, which has 2 kit joysticks and the Cypress on it).

We tried this scenario last night; it seemed to alleviate the issue, but we didn’t have time to put in enough tests to consider it fixed. We modified our physical setup to allow us to keep this configuration and use at our next competition in a few weeks.

Can you report the version of the driver?

I don’t have access to the board until next week, but I’ll be glad to report it when I can.

Can someone with a failure on awaken try this without the I/O board connected? With the board connected, but without any outputs connected, with outputs and no inputs?

We did not try any of these scenarios yet - I can try them next week if no one else reports results.

Also, if you awaken and the board doesn’t work, can you verify whether the Control Panel>>Admin Tools>>Services shows the PSOC service running or not?

The service is running (I have never seen it not running after telling Windows to start it automatically). Confirmed in Services CP and also in task manager.

This did work for us in testing last night, too. Thanks for that tip.

14

A little bit more info about our version of the error

Cypress was connected to hub, which was connected to the classmate through only the black cable. Not sure if classmate was suspended when the field cable was plugged in, but I can confirm that the only thing that fixed the issue was to reboot the classmate. I promptly disabled suspend on lid close. Didn’t see the problem again.

At eliminations, I re-enabled sleep on close for power conservation, and before every match I checked the I/O before connecting to the field and everything worked.

Our alliance partners did get a cypress failure though, rebooting also fixed it for them, but their joysticks weren’t working either so it might have been s different problem.

P.S. This has never happened off the field for us

15

For what it’s worth, this happened to us as well. We closed the laptop cover before a practice match and it decided the First touch no longer existed. Luckily, it was end of day and another mentor saw this thread that night. No harm done. And this turned out to be a blessing in disguise because:

Or from the other side, FRC should not tell teams to do this! Between matches 2 and 3 of the finals (NYC regional) over the weekend, there was a field problem. They announced that all 6 teams should shut down the cover of the laptop while they reset the field. Thanks to this experience, we knew not to do that! (I didn’t see what they did, but I imagine they shut down the laptop instead.)

If we lost finals because of this problem, we would have been really mad!

So thanks for posting about the problem and saving us!

16

Thank you, too, for posting that it happened to you as well.

Can you share your configuration at the time to help NI track this down (see Greg’s requests above)? Be very specific. What devices were plugged into what ports? If using the hub, describe what was plugged into it and if you were running it from the single USB connection or both USB connections on the “Y” cable.

17

I wanted to give an update on the Cypress identification issues.

First, some diagnostics.

If the board doesn’t have power, no LEDs are on – fix the cable.

If the board has one green LED, press the button on it. If no red LEDs turn on, your board is fine and working.

If one red LED, in position 2 I think, comes on, the board has not been recognized by the computer and reenumerated.

For testing, we placed a small clamp on the button and repeatedly cycled through different startup procedures.

If the Cypress is plugged into an external hub, it will fail to enumerate about half the time – red light still on. Usually, moving the Cypress to be plugged directly into the Classmate immediately fixed the problem – red light goes off – with no need for reboot or DS restart.

*** The 2010FRCControl%20System-Getting%20Started-Rev-0.7.pdf is a bit weak in its directions. The intent was to have the I/O board plugged directly into a computer port due to power limitations. It will be updated to state this more strongly.

If the Cypress is plugged directly into the Classmate port, the I/O board would occasionally fail to be recognized on a cold boot – approximately 1 in 8 would fail. When this failed, the CyMiniProg Service was always running, but restarting it always fixed the issue. Since there is no automated way to restart the service – yet – it may be easier to reboot when this occurs. We are still investigating this issue, and it may be timing related. This is not directly caused by the FMS field connection, but since the presence of an FMS could change timing, it could make the failure more or less likely to occur.

If the Cypress is plugged directly into the Classmate and in a recognized state, we saw no failures when suspended and reawakened.

To test/identify a joystick, press a button and watch for the LED to turn blue on either the Setup or Diagnostics tab.

My recommendations.

For teams using the I/O board, test it before each use by pressing the onboard button, especially after a cold boot.
1 Red LED = BAD
0 LEDs = GOOD

Do not plug it into an external hub even if using special pigtails that let the hub draw more current. Plug it directly into a Classmate port.

Test joysticks before each match. I have never seen failures except in over-current conditions, oh – and then there is the cable not connected condition, but think like a pilot. Run through your checklist.

When not in use, suspend the Classmate by closing the lid. Briefly press the power button to wake it up. If the Classmate has slept for more than six hours, it may take thirty seconds to restore from disk, but for shorter sleeps it should resume from memory and take six seconds. I know of no issues with network recovery.

If I notice additional issues at the event I’m attending this weekend, I’ll update this thread. Feel free to offer your own advice, but please explain why you believe it is good advice.

Greg McKaskle

18

Greg - was able to do some additional testing here on Tuesday, but wasn’t able to post until now. Glad to see you guys were able to reproduce the issues in some respects. My findings below in context of yours.

This is in line with our observations as well, more or less.

If the Cypress is plugged directly into the Classmate port, the I/O board would occasionally fail to be recognized on a cold boot – approximately 1 in 8 would fail.
When this failed, the CyMiniProg Service was always running, but restarting it always fixed the issue.

While we were unable to run a significantly large number of tests due to time constraints, I can report that moving the device to being directly plugged in also has greatly increased success rate here - we didn’t have any failures in the handful of cold boots that we did.

If the Cypress is plugged directly into the Classmate and in a recognized state, we saw no failures when suspended and reawakened.

To further this… we did a lot of mixed testing of suspend (as this is the primary use case of concern for us). That is, we did not cold boot when changing the configuration. Simply moved where the device was plugged in, waited for it to be recognize, and then started doing suspend/wake tests.
When directly plugged into the left-side classmate port we didn’t see any failures suspending/awakening. When plugged into the hub, we saw intermittent failures.

We’ve since modified our physical enclosure to permit direct connection.

Unfortunately I was unable to gather any data on behavior with no loads connected to the Cypress because unhooking all the devices would’ve taken more time than we had to test.

19

I don’t remember specifically. Conceptually, we had three joysticks and the First Touch. I know that’s not detailed enough to be useful.

I’m guessing the hub and all are in a crate traveling to our next match. I’ll check with the team though and see if anyone remembers more specifically.

20

We had three joysticks plugged into the USB hub, which was only connected with the black USB cable. We sometimes plugged the Cypress into the hub, and sometimes into a separate port - I don’t remember where it was when the Classmate was put to sleep, but afterward the Cypress was not recognized in any port.

We’ve had occasional problems with the Cypress in the past. Sometimes we’ll be running the bot, and all the buttons connected through the Cypress will work fine, and then we suddenly get a single “enhancedIO not found” exception. However, the buttons continue to work properly. I suppose this could be a momentary power issue if the Cypress was connected through the hub, especially since the Cypress has to power several LEDs.

I hope some of this information can be helpful in the debugging process! Just to be safe, we mounted a little piece of plastic on our board to prevent the Classmate’s lid from closing :slight_smile: