Human Vision Camera compatiable with FMS & Java ?

[techhelpbb]

Originally Posted by de_

If you could send me some links, I will verify everything with the lead programming mentor.

Everyone: Please remember the 3000 worked fine via wifi and tethered at all times outside of a regional. It only did not work when connected to FMS.

I am going to direct quote my directions from an e-mail I sent to my students on FRC11:

This was tested this evening (2/28/2016) with the Logitech 920C on my robot configured as Team 11:

DriversStation: 16.0.2
RoboRIO: FRC_roboRIO_2016_v19 aka Update 1 (2nd of the 2016 season)
Oracle Java: 2016 Java Release 4
Pnuematics Control Module: 1.65
Power Distribution Panel: 1.40
Talon SRX: 2.0

I setup the NI SmartDashboard like this:
1. Setup button, 3rd down on the left of the bottom DS panel.
2. Dashboard Type: Default
3. DS Protocol 2015
4. Top panel bottom right of video pane, select 'Camera Off'.
5. Select: 'USB Camera HW'

Upload the WPILib Simple Vision example from Eclipse setup for JAVA:
1. File -> New -> Project
2. Select: 'Example Robot Java Project'
3. Scroll down to: 'Complete List'
4. Select: 'Simple Vision' 5th item up from bottom of that section.
5. Click: 'Finished' at the bottom right.

Open 'Simple Vision' code:
1. Expand in left project column:
'Simple Vision' -> 'src' -> 'org.usfirst.frc.team11.robot' -> 'Robot.java' -> 'Robot'
2. Double click 'Robot to open code for it.

Check USB camera configuration:
1. Connect USB tether to robot.
2. Open web browser and go to: 172.22.11.2
3. Scroll down in second column from the left of the browser till you find:
USB symbol labeled: 'HD Pro Webcam C920'
4. Note the name it's usually 'cam0'.
5. If that's not 'cam0' find it in the code and change the line that looks like:
server.startAutomaticCapture("cam0");

Upload the code to the robot:
1. Right click in the project column (left most): 'Simple Vision'.
2. Scroll down to: 'Run as'
3. Click: 'WPILib Java Deploy'

Code should upload to RoboRIO.
Display should start on NI Smart Dashboard.
See attached picture as evidence.

Do you want me to upload the sample code?
I would like to see your code, as I am sure Greg would as well.

[Greg McKaskle]

Everyone: Please remember

I didn't forget this information, and I don't think others have either.

I think the primary mechanism for the FMS to break your USB camera is by changing the timing of when the client and server apps start and finish their connection. I saw very similar issues with a team at the Bayou regional and who developed a process to work around the issues.

There is, by the way, no official statement about what USB cameras are compatible with or incompatible with the FMS. The FMS should have no impact on whether a particular USB camera works. To the FMS, the images are simply data in a TCP stream.

I'd also like for you and all teams to have a reliable camera for championships -- and all the events. In order to look into this better, can you summarize which dashboard you are using and how the camera is being served on the roboRIO?

Greg McKaskle

[de_]

Greg and techhelpbb: Thanks for the input. Sorry I was away. Our team has a quite high level of technical expertise. We have been at it for over a decade. We did do (AXIS) camera based target tracking a few year ago. We are pretty good at following installation instructions and the MS 3000 worked fine everywhere except on the field. We program in java and have a fairly advanced smart dashboard with the low res camera view in a window. We have heard of a number of other teams also having usb camera issues when connected to the field but then we have heard of teams with 2 cameras working fine. We attempted to contacted FIRST a week ago but have not heard anything back. We have decided to focus on a supported AXIS IP camera for the worlds as we just cannot afford to risk what happened at our two regional s.

For anyone, if your USB camera worked at a regional, it would be nice to know
1) did you program in Java
2) do you use the smart dashboard
3) what version of windows did your laptop use
4) what resolution you used for the camera

[plnyyanks]

Since this thread touches on FMS networking, I'll post a link to the most recent FMS Whitepaper that explains how all of it works:

https://wpilib.screenstepslive.com/s...fms-whitepaper

The main point is that FMS is "just a network" and only communicates with your DS (not the robot).

[techhelpbb]

Originally Posted by de_

Greg and techhelpbb: Thanks for the input. Sorry I was away. Our team has a quite high level of technical expertise. We have been at it for over a decade. We did do (AXIS) camera based target tracking a few year ago. We are pretty good at following installation instructions and the MS 3000 worked fine everywhere except on the field. We program in java and have a fairly advanced smart dashboard with the low res camera view in a window. We have heard of a number of other teams also having usb camera issues when connected to the field but then we have heard of teams with 2 cameras working fine. We attempted to contacted FIRST a week ago but have not heard anything back. We have decided to focus on a supported AXIS IP camera for the worlds as we just cannot afford to risk what happened at our two regional s.

For anyone, if your USB camera worked at a regional, it would be nice to know
1) did you program in Java
2) do you use the smart dashboard
3) what version of windows did your laptop use
4) what resolution you used for the camera

Can your team provide the robot code? Private message would be fine.
If you send it to me I will only forward to Greg or FIRST.

It is no skin off my back if you do not want to use a USB camera at all.
I would, however, like to find a root cause for the issue.

[Greg McKaskle]

I'm not doubting your level of expertise. I was helping a team just this weekend doing C++ with a mentor who does video processing for a living. But reliable client/server video serving is not easy. There have been many bugs in the video stuff in WPILib too, and I suspect that is what we are talking about here.

My main point is that FIST didn't declare that the MS Lifecam is not supported, but if your code's timing has uncovered a bug in the camera server, you can likely reproduce it by changing timing between connections or similar modification in how the client and server start up. Once you can reproduce the bug, you can likely work around it and/or ensure that the Java folks can make progress on it.

Axis cameras have a built-in server and are another good way to get a reliable transmission. If you can't get the USB to work, Axis is a good backup plan.

Greg McKaskle

[de_]

re "I'm not doubting your level of expertise."
Sorry just wanted to give you guys some additional background info.

re "My main point is that FIRST didn't declare that the MS Lifecam is not supported"
Just a reminder that the CSA and I believe it was the FTA at Northbay told us specifically that FIRST knows of an issue with the MS 3000 and FMS. It might have been Java related. I am now trying to contact FIRST HQ to find out exactly what the issue is.

I really appreciate the high level of interest in resolving this both you guys are showing. I need a few more days to replicate our environment in my own test environment and hopefully hear what FIRST has to say re what the CSA said.

[Greg McKaskle]

that FIRST knows of an issue with the MS 3000 and FMS.

As a member of the FRC Control Systems Team, the author of some of the camera support code, and a frequent CSA, I'm trying to squash this statement. The FMS has no involvement with the DS/DB/camera comms. The field network is involved. It blocks some ports and affects connection timings, just as almost any ethernet or wifi network will. But it has no idea what is on your encrypted TCP/UDP stream between your devices. Waiting for FIRST to fix the FMS so that this camera works is not a good use of your time.

I'm happy to help you identify your issue, but pointing at the FMS is not productive. Instead, we need to look at which feature of the Java libraries you are using, which server, which client, etc. We can either look at the specifics of your USB solution, to help identify what is likely a timing issue in WPILib or how it was being used, or we can look at your proposed IP camera to help ensure that it works for you at your next event.

Greg McKaskle

[robert1356]

We have successfully used TWO LifeCam HD 3000. One is connected directly to the roboRIO (programmed in Java) and streamed to the smart dashboard using the CameraServer object. The other is connected to a raspberryPi and piped through mjpeg-streamer to grip. We did not feed the second camera stream to the driver station, but the first camera went to the driver station. We had this configuration at both Palmetto and Smoky Mountains regionals with no issues.

[plnyyanks]

As an FTAA and CSA myself, like Greg said,

Originally Posted by Greg McKaskle

The FMS has no involvement with the DS/DB/camera comms. The field network is involved. It blocks some ports and affects connection timings, just as almost any ethernet or wifi network will. But it has no idea what is on your encrypted TCP/UDP stream between your devices.

After another quick skim through this thread, I also haven't seen you clarify exactly what "doesn't work" on the field. Does the stream not appear at all on the Dashboard? Is it choppy or frozen on an image? Do you see any logs? What kind of things did you try and troubleshoot with the FTA/CSA at your event? All of those things can help us understand the problems you're facing and give better ideas on what to try next.

That being said, I'll spitball a few ideas

• Not all network ports are open on the field, see R60 this year
  

  Originally Posted by R60

  Communication between the ROBOT and the OPERATOR CONSOLE is restricted as follows:
  A. Network Ports:
  i. TCP 1180: Camera data from the roboRIO to the Driver Station (DS) when the camera is connected the roboRIO via USB, bi-directional.
  ii. TCP 1735: SmartDashboard, bi-directional
  iii. UDP 1130: Dashboard-to-ROBOT control data, uni-directional
  iv. UDP 1140: ROBOT-to-Dashboard status data, uni-directional
  v. HTTP 80: Camera connected via switch on the ROBOT, bi-directional
  vi. HTTP 443: Camera connected via switch on the ROBOT, bi-directional
  vii. UDP/TCP 554: Real-Time Streaming Protocol for h.264 camera streaming, bi-directional
  viii. UDP/TCP 5800-5810: Team Use, bi-directional
  
  Teams may use these ports as they wish if they do not employ them as outlined above (i.e. TCP 1180 can be used to pass data back and forth between the ROBOT and the DS if the Team chooses not to use the camera on USB).

• QoS limitations. The radio will limit your connection to 7Mbit/sec (see R60B). If you have the camera configured to stream super high quality video at 60fps, you may run into trouble. This document explains how you can test your bandwidth usage at home. Additionally, the 7Mbit limit is a theoretical ceiling, wireless conditions at your specific event may preclude such speeds. In any case, robot control packets are prioritized over video streaming, so packets from your camera may get dropped, causing issues with your stream.
  
• This one is a real long shot, but I vaguely remember a discussion on a CSA call about a Java WPILib bug with the CameraServer code, I believe it's this on the tracker. I don't know enough about this to recommend further troubleshooting to see if this is what you're experiencing though, I'd have to defer to someone else on the subject.

[de_]

Thanks Phil.

We did notice our cpu loading drop from 90% to I believe 50% when we removed the usb camera. We had been concerned about our 90% cpu usage since our first regional.

We are pretty certain we are in compliance with all the port and bandwidth limitations but I will be rechecking that personally this week (I am not the programming mentor who is busy with enhancements for champs). And the CSA was looking at it at the regional.

I am pretty sure we were running 320x200 resolution but certainly no more than twice that.

[de_]

Hi Greg - thanks for the clarification re your involvement. I'll pm you shortly with more info and code (it may take a day to get to the code that used the USB camera which has since been updated).

[simpsonboy77]

Originally Posted by de_

We did notice our cpu loading drop from 90% to I believe 50% when we removed the usb camera. We had been concerned about our 90% cpu usage since our first regional.

Sounds like this bug.

You can either recompile wpilibj or you can copy cameraserver.java into your project and make the change there.

I saw this bug on 2 teams this past weekend and it sounds identical to your issue.

[JamesTerm]

Originally Posted by Greg McKaskle

The default dashboard will open a TCP connection on port 1180 to the roboRIO's IP address.
Greg McKaskle

This may be what we should have tried...
What I can tell you... same code worked flawlessly in 2013-2014 and worked using 1180... using 554 only worked in one match at Lubbock. The code is essentially ffmpeg... ffplay that has been modified to pass frames... so all connection code is deep within ffmpeg itself. My point is that its a well defined solution, worked flawlessly in the pit... can withstand cold cycles, and passes a simulated fms environment.

I'm replying here, because there may be a chance that someone may have similar issues, and someone may have some insight.

There is one other thing... our URL is
rtsp://FRC:FRC@10.34.81.11:554/axis-media/media.amp

Not sure if the login and password are an issue...

void FFPlay_Controller::FailSafe::operator() ( const void* )
{
	bool EventResult=m_Heartbeat.wait(10000);
	if (!EventResult)
	{
		FrameWork::DebugOutput("***No heartbeat... resetting\n");
		m_pParent->SwitchFilename(m_FileName.c_str());
	}
	else
		Sleep(2000);  //This does not need to be so active
}

I've used this to mitigate the race condition of the connection and this has worked fine in outside FMS environment, and previous years (using 1180).

The thing I find most frustrating with this problem is that I cannot pursue it, and that makes it very frustrating. I'm hoping we can simulate FMS enough to reproduce it, or find some insight as to what it can be. We can only speculate, and wait until the last moment to test again.