Using Processing for custom dashboard

[ekapalka]

Hello! I'm asking this question on behalf of a few of my colleagues whose area of expertise is in the language Processing. They're fantastic programmers, and can make any GUI you can imagine. I believe it's a highly object-oriented dialect of Java (I don't program with it myself, so I'm not sure), and apparently there are libraries you can import to compile it as regular Java. Our actual robot is programmed in C++, and we've never used network tables before. Is it possible to construct a dashboard that can send data to and from the robot using Processing? Thanks!

[joelg236]

Why not just use Smartdashboard (http://firstforge.wpi.edu/sf/projects/smartdashboard)?

[virtuald]

If you want to use Java, just extract the NetworkTables bindings from SmartDashboard. They have a separate jar file available that has them. The JAR is also distributed with the C++ stuff, look in C:\WindRiver\WPILib\desktop-lib

We did our custom dashboard (http://www.chiefdelphi.com/forums/showthread.php?t=115765) in python, also using NetworkTables (http://firstforge.wpi.edu/sf/frs/do/listReleases/projects.robotpy/frs.pynetworktables).

As long as both sides are using a common communication mechanism (like NetworkTables), it doesn't matter what language either side is written in.

[ekapalka]

Why not just use Smartdashboard (http://firstforge.wpi.edu/sf/projects/smartdashboard)?Like I said, we're not sure what we're doing, but if we could use Processing, everything would be perfect.We did our custom dashboard in python, also using NetworkTables. You're kind of the team that inspired us to make a custom dashboard.... Would we simply extract the NetworkTables Jar file into an eclipse workspace and add "some code" to send and receive values on the dashboard and robot-side programs?

[virtuald]

Would we simply extract the NetworkTables Jar file into an eclipse workspace and add "some code" to send and receive values on the dashboard and robot-side programs?

Yup.

[ekapalka]

Can you inform me of somewhere I could go to learn? Thanks, by the way

[Joe Ross]

This document covers getting the client (ie dashboard) side of network tables running in netbeans. http://wpilib.screenstepslive.com/s/3120/m/7912/l/80205-writing-a-simple-networktables-program-in-c-and-java-with-a-java-client-pc-side

[ekapalka]

Network tables seem fairly straightforward from the example on that page. I was wondering if there is much lag/delay between sending and receiving information. We would like to send information from the keyboard to a Motor controller, which I believe has to be updated once every 10ms or so... so that's a lot of information. Do we need to add delay (such as a wait statement or state machine) to both sides of the code or should it be okay to just attempt to send data back and forth in real-time? Also, should we create separate network tables for input and output communication? Thanks!

[virtuald]

We would like to send information from the keyboard to a Motor controller, which I believe has to be updated once every 10ms or so... so that's a lot of information. Do we need to add delay (such as a wait statement or state machine) to both sides of the code or should it be okay to just attempt to send data back and forth in real-time?


Even the joysticks for the driver station aren't updated enough to be used in latencies less than 100ms if that's what you care about. Chances are, that isn't really what you care about, and so the latency is low enough that you won't notice it. But the NetworkTables lag could easily be in the 100ms range or higher.

Also, should we create separate network tables for input and output communication? Thanks!

You should use different elements in the table, but no need to create separate tables. What we've traditionally done is just used the 'SmartDashboard' table, that way we can use all of the SmartDashboard infrastructure on the robot side and use the SmartDashboard itself.

[JesseK]

We use raw Java on the display and custom network code to get custom packets from the cRIO. The data structure is 200 bytes, and contains every input, sensor, motor output, pneumatic state, and vision targeting parameter we have. Then we use Java reflection to open up the data structure and print it on the screen in a panel for quick debugging -- reflection avoids the need to update the data printer every time the software kids need the data structure change to change. We found that using custom packets is less processing and bandwidth-intensive than the Network Tables since we can control the data rate directly. The display also stores the data to file in a .CSV format for quick telemetry analysis post-match. Avoiding Network Tables also avoids "heavy weight" library dependencies, which has bitten us in the past when we've swapped laptops around.

The kids wrote some code around the data structure for a GUI. We may do touch-screen next year, but not at the expense of our current goals (plus a physical button board is better for the offseason stuff we do).

Our software goals for the display are more along the lines of quick integration debugging, sensor tuning, & match replay rather than live match control. Our drivers rarely take their eyes off the field.

I've never heard of "Processing" until now. Will have to look that one up.

[ekapalka]

Thanks everyone! Hypothetically, we now have all the knowledge to make a custom GUI than except the camera display (just the display, not vision processing. Eventually, we'll do that on-board). I believe there's a lot of stuff in WPI for that. Would we have to set up an entire robot project (configured and everything as if we were actually controlling the robot with it) to utilize that or just import it like we did with NetworkTables?

[mman1506]

If you do this please document it. I really like the processing language.

[ekapalka]

So can images be transferred via network tables, too?

[Greg McKaskle]

I wouldn't transfer images in network tables. You could technically do some small ones, but why not do it more efficiently. As for the latency of network tables. The variables are written into the local table rather quickly, like in a millisecond or so. The update rate is used to determine how often modified values are replicated to other clients. This value varies by language, and the LV default is 100ms. If you wire up a different number, it will have a bit more overhead, but less latency. The other option is to call flush yourself, to force the update when something interesting happened, like you finished processing an image.

Other than the delayed updates, there shouldn't be much more latency than straight TCP communications.

Greg McKaskle

[ekapalka]

How exactly does that work? I've been looking everywhere on socket communication (which might be what you're talking about) to transmit mid-high resolution images at a high frame rate in real time. As of yet, we're not planning on doing any vision processing; just a GUI.

[Greg McKaskle]

The cameras run an http server used for configuration and for image requests. There is a CGI request on port 80 for requesting single bmp, jpeg, or an mjpeg stream. Some of the newer cameras support other compression types as well.

You can request the images from a dashboard written in any of the FRC languages, from a web browser, or from RoboRealm.

Greg McKaskle

[JesseK]

Ditto Greg.

The Kinect is similar; the main difference is that a specific process has to start for the Kinect image web server.

The Kinect image stream has a line in each packet header that says "Image Size: ", with an Integer and Carriage Return + Line Feed following. After that is the image, which is exactly the size of what the Integer was. We read in that many bytes, then call ImageIO.read() to create a BufferedImage. Then we wrote a custom generic LayeredPanel class that does some layering, with the Buffered Image on bottom and some text overlays on top. Additionally, we pass the BufferedImage off to ImageIO.write() (in its own single threaded Executor) with a file name (based upon timestamp from System.currentTimeMillis() to save the image off. This allows us to do frame-by-frame analysis immediately following a match. At 7fps the image decoding & writing takes ~40% of 1 core on an i5 (sorry, don't know the clock speed).

I suspect the Axis camera is similar, if you don't want to use a mjpeg stream reader.

[Hypnotoad]

Sorry if I seem a bit new to this, but how would you "extract the networkTables dependencies". Would I have to get the networktables files and decompile them? If so, where would I put them to use them in processing? or any general java.

[Iaquinto.Joe]

Honestly, as with robot design, keep programming simple too. Smart dashboard is easy, effective, and simple to do what you need. The networktable will allow you to send any data you want to and from your dashboards, and it's a helluva lot easier than sockets. Drivers will not want to use a wonky debug driverstation, they want quick data, and smartdashboard will do that without custom dash headaches.

[Hypnotoad]

I simply had trouble figuring out how to overlay things onto the image in smartdashboard. That's literally all we use the dashboard for. That's why we stuck with the labview dashboard. It not only had good vision processing, but overlaying was already built in. I was thinking of going custom because processing would allow me to make a "minimap" much more easily compared to the labview dash.

[ekapalka]

The Kinect is similar; the main difference is that a specific process has to start for the Kinect image web server.

That sounds interesting. So is there robot-side code as well as computer-side code for transferring the data, or is like an Axis camera (broadcasting an MJPEG stream regardless of if you want it)? How is this done with the Kinect (and how do you manage the USB)? Is there a universal way of dealing with the images (for instance, if you're doing vision processing, is it an ordeal to transfer and interpret it)? Thanks!

[Hypnotoad]

So is anyone able to answer my question? Has anyone made a processing library for processing? If not, what would the process be to doing so?

[ekapalka]

Hey! I haven't had access to the robot in a while so I have no way to test this.... Based on what I've read I think that the following code will work for NetworkTables in Processing. I began with the code from WPIScreenStepsLive and whittled it down until most of the errors were gone. I have no idea if it will work. It doesn't compile, but I'm pretty sure that's because there's no robot-side broadcasting the network tables. I imported the NetworkTables JAR file and the Core files from Processing (see Processing in Eclipse (http://processing.org/learning/eclipse/)). Most of the advancements I've made so far are with the camera. Let me know what you think! import edu.wpi.first.wpilibj.networktables.NetworkTable;
import processing.core.*;

public class NetworkTablesTest1 extends PApplet {
public static void main(String args[]) {
PApplet.main(new String[] { "--present", "NetworkTablesTest1" });
}

double x, y;
NetworkTable table = NetworkTable.getTable("datatable");

public void setup() {
NetworkTable.setClientMode();
NetworkTable.setIPAddress("10.32.16.2");
size(displayWidth, displayHeight);
}
public void draw(){
x = table.getNumber("X",0.0);
y = table.getNumber("Y", 0.0);
System.out.println("X: " + x + "Y: " + y);
background(0);
}
}

[Hypnotoad]

getting this error as i try to run it:

Exception in thread "Animation Thread" java.lang.IllegalStateException: Network tables has already been initialized
at edu.wpi.first.wpilibj.networktables.NetworkTable.c heckInit(Unknown Source)
at edu.wpi.first.wpilibj.networktables.NetworkTable.s etClientMode(Unknown Source)
at networktablestest1.NetworkTablesTest1.setup(Networ kTablesTest1.java:23)
at processing.core.PApplet.handleDraw(PApplet.java:22 41)
at processing.core.PGraphicsJava2D.requestDraw(PGraph icsJava2D.java:243)
at processing.core.PApplet.run(PApplet.java:2140)
at java.lang.Thread.run(Thread.java:722)

[Hypnotoad]

Well, it runs without any bugs. It just never connects to the robot when I run it. Is there a special function that needs to be called to initialize a connection?

I used processing in netbeans for mine.

[Hypnotoad]

I got it to work. It seems a certain letter in the table name was capitalized improperly. Thanks ekepalka for the tip on using the networktables jar as a library.

[ekapalka]

Hey! I thought this thread was closed :P I stopped trying to mess around with NetworkTables for quite some time and moved on to neural networks (probably not a great idea). Now that my team is picking up working on robots, I have to get this working, but all of my revised code seems to have disappeared when I reformatted my computer. The thing I was hung up on was related to the timing when sending and receiving values. Specifically try { Thread.sleep(1000); }
catch (InterruptedException ex) {
Logger.getLogger(NetworkTablesDesktopClient.class. getName()).log(Level.SEVERE, null, ex);
}

Which I kept having errors with. I ended up getting some help from a programmer far more skilled than myself and got it to compile (...although I never actually tested it on a robot), but he's currently vacationing in Guam right now, and hasn't been able to help me much thus far. Have any of the people who've been experimenting with this gotten successful communication with the robot? Any ideas? Thanks!