Go to Post Over Weight. What else is to be expected? There is still time for the diet team to perform miracles. Meanwhile I still keep adding parts. - Steve W [more]
Home
Go Back   Chief Delphi > Technical > Programming > NI LabVIEW
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #16   Spotlight this post!  
Unread 09-02-2011, 11:31
Mark McLeod's Avatar
Mark McLeod Mark McLeod is offline
Just Itinerant
AKA: Hey dad...Father...MARK
FRC #0358 (Robotic Eagles)
Team Role: Engineer
 
Join Date: Mar 2003
Rookie Year: 2002
Location: Hauppauge, Long Island, NY
Posts: 8,940
Mark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond repute
Re: CRio Connection: Buggy

I don't think the logs I captured are particularly useful, and they weren't comprehensive.
I had to bring in several other laptops after the problem occurred to start the monitoring.

This particular log shows two boots, but unfortunately the problem may have been fixed by this point.
The first boot generated the 4 flashes, and caused an automatic reboot.
The second boot was a successful recovery from the same.

I have had several other cRIOs reimaged without any issues. Maybe one 4-flash that caused a then successful reboot.

In the case I saw, the v27 reformatting enabled NetConsole, and had None for CAN.
Cycling the power had no affect and the 4-flashes continued unabated.

Changing DIP switches, e.g., Safe Mode, didn't seem to affect the 4-flashes and the cRIO remained unresponsive.
The flashes began very quickly, so it may be before any image gets loaded or early in the load process.

Reimaging (from same Classmate) with v27 completed successfully, but had no affect.

Reimaging to v25 worked, then a repeat of v27 was successful.

--------
On a side note, Classmates that installed the LabVIEW Update, but didn't install the LabVIEW DVD, look normal from a distance.
I had to be there to notice all the missing software, but the version information was right up to date...
Attached Files
File Type: txt cRIO_bootup_V27.TXT (2.9 KB, 11 views)
__________________
"Rationality is our distinguishing characteristic - it's what sets us apart from the beasts." - Aristotle

Last edited by Mark McLeod : 09-02-2011 at 14:11.
Reply With Quote
  #17   Spotlight this post!  
Unread 09-02-2011, 12:39
IanG IanG is offline
Registered User
FRC #1535 (The Knights of Alloy)
Team Role: Mentor
 
Join Date: Feb 2011
Rookie Year: 2007
Location: Sault Ste. Marie, ON
Posts: 2
IanG is an unknown quantity at this point
Re: CRio Connection: Buggy

We are having the exact same problem on our robot. I didn't check for the blinking LED, but once we updated to version 27 we could still reimage and connect through ftp, however LabVIEW was unable to connect. Once we reimaged with v25, all was good. One of our tests included not imaging with the black jaguar plugin and we still had issues.
Reply With Quote
  #18   Spotlight this post!  
Unread 09-02-2011, 21:51
mswalker mswalker is offline
Mentor
FRC #2583 (Westwood Robotics)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 26
mswalker is on a distinguished road
Re: CRio Connection: Buggy

I re-imaged our cRio to v27 first with just NetConsole and then with both NetConsole and CAN Jaguar Black Serial plugin. Of course, now it seems to work. I ran LabView code that was essentially the default FRC cRIO Robot Project modified to drive a single motor using CAN.

From the console log, I did see two troubling messages when the cRIO boots:
interrupt: PCI Error: initiator aborted due to timeout
interrupt: PCI Error: initiator aborted due to timeout

I'm not sure what they mean but the system appeared to work. I was able to deploy and run CAN code. The system crashed once, I think when I was using the System Monitor. In the several times I ran code from LabView I received the Unable to connect to real-time target only once, accompanied by four flashes in the status LED. I didn't see anything in the netConsole at that time.

I'll do more testing tomorrow. The NetConsole log I captured is attached.
Attached Files
File Type: txt NetConsole2.txt (7.4 KB, 8 views)
Reply With Quote
  #19   Spotlight this post!  
Unread 09-02-2011, 22:46
otherguy's Avatar
otherguy otherguy is offline
sparkE
AKA: James
FRC #2168 (The Aluminum Falcons)
Team Role: Mentor
 
Join Date: Feb 2010
Rookie Year: 2009
Location: CT
Posts: 446
otherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to behold
Re: CRio Connection: Buggy

We afre seeing the same issue here. Spent a few days trying to figure out what was wrong with the rio before finding this.

We flashed with net console and CAN enabled for the black jags (didn't try with can disabled).

Flashed to v27 innumerable times (flashing always worked fine), labview would not deploy code. I was able to ping and ftp into the rio without issue. The driver station would see robot comms on bootup momentarily, then it would lose it. I was able to successfully deploy from labview maybe 10% of the time. After a sucessful deploy, control through the driver station worked reliably.

This is pretty easy to replicate on our end.

The only fix that seemed to work reliably was to flah back to v25.

We seem to be hitting bugs this year every step of the way. Last week it was problems with the encoders. I don't mean to complain, but these types of issues don't help when your trying to teach the kids how something "should work"

What gives?
Reply With Quote
  #20   Spotlight this post!  
Unread 10-02-2011, 07:44
Chris Hibner's Avatar Unsung FIRST Hero
Chris Hibner Chris Hibner is offline
Eschewing Obfuscation Since 1990
AKA: Lars Kamen's Roadie
FRC #0051 (Wings of Fire (on sabbatical))
Team Role: Engineer
 
Join Date: May 2001
Rookie Year: 1997
Location: Canton, MI
Posts: 1,489
Chris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond reputeChris Hibner has a reputation beyond repute
Re: CRio Connection: Buggy

I have an update from last night.

We added case structures to timed tasks to be sure that none of our code in timed tasks runs unless the robot is enabled. It took about 4 tries to deploy this software (using Run As Startup). After getting this software on the cRIO, we never had another problem.

It makes me think that something in v27 is taking up a lot of CPU overhead causing issues with the communication. I seem to remember issues last year when the user software started taking up too much CPU time, it became difficult to deploy code. Perhaps this is similar.
__________________
-
An ounce of perception is worth a pound of obscure.
Reply With Quote
  #21   Spotlight this post!  
Unread 10-02-2011, 08:28
Mark McLeod's Avatar
Mark McLeod Mark McLeod is offline
Just Itinerant
AKA: Hey dad...Father...MARK
FRC #0358 (Robotic Eagles)
Team Role: Engineer
 
Join Date: Mar 2003
Rookie Year: 2002
Location: Hauppauge, Long Island, NY
Posts: 8,940
Mark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond reputeMark McLeod has a reputation beyond repute
Re: CRio Connection: Buggy

It does have the look and feel of something that doesn't finish booting due to some race condition and locks up specific cRIO resources. Once you break free of it, everything is fine.

Some services are blocked or slowed while others remain unimpaired.
Probably, due to specific resource conflicts.

I would hazard that it isn't cRIO CPU overhead though, since some low-priority services (ftp) sometimes work. Those are the first to fail when there's a CPU shortage.
The 4-flashes might indicate that memory isn't being released by something that sucked it up, then doesn't complete and just sits there. Maybe two boot tasks asking for a combined 110% of memory at the same time, then waiting for the other to give up first. Sort of an OS hydra staring contest.

One thing to test, if it happens to any of us again, is to run the LabVIEW System Manager and monitor the CPU and Memory usage for clues. Do it before attempting to download or do anything else, while the 4 flashes are occurring.
If anything significant happens take a screen shot and post it for the rest of us.
__________________
"Rationality is our distinguishing characteristic - it's what sets us apart from the beasts." - Aristotle

Last edited by Mark McLeod : 10-02-2011 at 23:17.
Reply With Quote
  #22   Spotlight this post!  
Unread 10-02-2011, 22:58
mswalker mswalker is offline
Mentor
FRC #2583 (Westwood Robotics)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 26
mswalker is on a distinguished road
Re: CRio Connection: Buggy

Well, I spoke too soon. When our programmers started up the cRio this afternoon we again got the four flash status light and inability to deploy code to the cRio. They reimaged the cRio with v27 without NetConsole and everything seemed to work.

To debug this further, I reimaged with the v27 with NetConsole and CAN Black Jag. The attached log shows the results of two instances of failed boots. Both resulted in four flash status lights. I tested several more times and got the same results - we had no successful boots. I re-imaged without the NetConsole and the system works. CAN is enabled and working.

It appears from the logs that we are getting two crashes without a reboot which is one of the interpretations of the four flash status. All appear to be an 0x00300 exception.

So it may not be memory related. Though removing the NetConsole code seems to help. The other day when the system was working with the full image including the NetConsole, System Manager reported that I still had about 18900K of free memory.
Attached Files
File Type: txt netConsole0210.txt (17.4 KB, 8 views)
Reply With Quote
  #23   Spotlight this post!  
Unread 11-02-2011, 01:27
jhersh jhersh is offline
National Instruments
AKA: Joe Hershberger
FRC #2468 (Appreciate)
Team Role: Mentor
 
Join Date: May 2008
Rookie Year: 1997
Location: Austin, TX
Posts: 1,006
jhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond repute
Re: CRio Connection: Buggy

Since some of you have narrowed it down to what appears to be NetConsole.out, but the log that captured the exceptions all showed lvrt.out on the call stack, I'd like you to try pulling the NetConsole.out binary from the v25 image an placing it in the v27 image and attempt to reproduce the issue. It's possible that something the NetConsole is doing on startup is causing lvrt.out to crash, but I'd like to make sure the v27 version of NetConsole is to blame and not the v27 environment.

Thanks,
-Joe

Last edited by jhersh : 11-02-2011 at 01:30.
Reply With Quote
  #24   Spotlight this post!  
Unread 11-02-2011, 08:53
mswalker mswalker is offline
Mentor
FRC #2583 (Westwood Robotics)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 26
mswalker is on a distinguished road
Re: CRio Connection: Buggy

I'll give that a try this evening.
Reply With Quote
  #25   Spotlight this post!  
Unread 11-02-2011, 17:45
jhersh jhersh is offline
National Instruments
AKA: Joe Hershberger
FRC #2468 (Appreciate)
Team Role: Mentor
 
Join Date: May 2008
Rookie Year: 1997
Location: Austin, TX
Posts: 1,006
jhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond reputejhersh has a reputation beyond repute
Re: CRio Connection: Buggy

We were able to reproduce the issue on one of the cRIOs here and believe we have fixed the issue causing the 4 blinks and the "Exception code: 0x00000300"

Please download the new image from FIRST Forge here: http://firstforge.wpi.edu/sf/go/proj...crio_image_v28

Let me know if this resolves the issue for you.

Thanks,
-Joe
Reply With Quote
  #26   Spotlight this post!  
Unread 11-02-2011, 23:23
mswalker mswalker is offline
Mentor
FRC #2583 (Westwood Robotics)
Team Role: Mentor
 
Join Date: Jan 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 26
mswalker is on a distinguished road
Re: CRio Connection: Buggy

Thanks Joe! We'll try it tomorrow.

As you expected, replacing the NetConsole.out file in v27 with the v25 file removed the error.
Reply With Quote
  #27   Spotlight this post!  
Unread 12-02-2011, 20:50
otherguy's Avatar
otherguy otherguy is offline
sparkE
AKA: James
FRC #2168 (The Aluminum Falcons)
Team Role: Mentor
 
Join Date: Feb 2010
Rookie Year: 2009
Location: CT
Posts: 446
otherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to beholdotherguy is a splendid one to behold
Re: CRio Connection: Buggy

We flashed to v28 today from v25.
Everything appears to be functioning as expected. We'll report back if we see any issues.

Thanks!
Reply With Quote
  #28   Spotlight this post!  
Unread 12-02-2011, 21:45
Paul Gehman Paul Gehman is offline
Dr. G
AKA: Dr. G
FRC #1712 (Dawgma)
Team Role: Mentor
 
Join Date: Mar 2009
Rookie Year: 2008
Location: King of Prussia, PA
Posts: 3
Paul Gehman is an unknown quantity at this point
Re: CRio Connection: Buggy

Our experience seems to indicate something going on with the "Enable NetConsole - CAN Driver Plugin - Black Jaguar Serial Bridge" option that we chose to install with our v25 and v27 imaging.

At 1:30PM, we installed the FRC_2011_v27.zip image that came with the latest Labview v3.1 update, via the tether. The dialog box indicated that the installation was successful. I rebooted/cycled the cRIO while the main programming crew was at lunch.

At 1:50PM - when the main programming crew came back - they simply tried to install our working baseline code. The download just about immediately stalled and threw an error. Seems that an entry in the ni-rt.ini file under the "Startup Dlls" entry section on the cRIO was incompatible with the newer image. We ftp'd into the cRIO, pulled the ni-rt.ini file to the Developer laptop and deleted the entry - and placed the ni-rt.ini file back on the cRIO. At least that got us back to an earlier version of the ini file.

At 2:20PM -- after finding a copy of the v25 image on a Mentor's laptop - we imaged the cRIO with the older v25 image. We checked the "Enable NetConsole - CAN Driver Plugin - Black Jaguar Serial Bridge" options. This imaging was also successful. AND our CAN/Black Jag code now worked. We figured we lost about 1 1/2 of prime Saturday programming time dealing with this anomoly.

Our experience with v27 and our basic CAN/Black Jag-centric baseline code indicates that there is a serious incompatibility. We figure that until v27 is "fixed" - we will just have to use the v25 image.
__________________

Last edited by Paul Gehman : 12-02-2011 at 21:48.
Reply With Quote
  #29   Spotlight this post!  
Unread 12-02-2011, 21:58
Greg McKaskle Greg McKaskle is offline
Registered User
FRC #2468 (Team NI & Appreciate)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 4,766
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: CRio Connection: Buggy

v27 had changes to a number of components including NetConsole. If NetConsole was enabled, the changes were overflowing a thread stack and in some situations caused the crash, reboot, and blink. v28 is identical to v27 except it contains a NetConsole with about 16KB more stack for the thread.

Obviously you can continue with v25 for the remainder of the build, but v28 has been posted. You can try it if it fits your schedule. FIRST should officially announce it in the next update -- the bug was reproduced and fixed on Friday.

Greg McKaskle
Reply With Quote
  #30   Spotlight this post!  
Unread 12-02-2011, 22:14
Paul Gehman Paul Gehman is offline
Dr. G
AKA: Dr. G
FRC #1712 (Dawgma)
Team Role: Mentor
 
Join Date: Mar 2009
Rookie Year: 2008
Location: King of Prussia, PA
Posts: 3
Paul Gehman is an unknown quantity at this point
Re: CRio Connection: Buggy

Thx Greg, Thx Joe - we'll try the new v28 image when we get back to our Developer laptop.
__________________
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 14:11.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi