The DevServer TCP-based data server

[yash101]

I have been working on a project to make communications between multiple devices on the field quite simplistic. I present to you, the DevServer. It is an HTTP (TCP) based server, where the Java or C++ API can send and retrieve values, and at the same time, have them available on other devices on the network.

Features:
Mostly STL C++, some DLib, with most headers commented out.
Amazing web interface, which is clean, fast and works well
AJAX for most web loading!
The web pages are loaded into RAM on runtime, so the load times are quite minimal
C++, Java, and possibly JavaScript(browser) APIs will be released soon

Most of you will be wondering, what is this, and why I made this. This is a server, in which you can store and access data through. It supports multiple clients at the same time, so it can be used for passing data between a coprocessor and the cRIO. This nifty server comes bundled with a powerful web interface which has documentation, a way to send values, and a way to monitor all the values in real time. Based off vanilla JavaScript, the web interface requires no client setup. However, it is designed for webkit, so it may not display properly in FireFox and IE.

The code is available on GitHub, HERE]

If you want to automatically download and install the server in Debian/GNU Linux, I have written an installer, and a simple one-line command to install:
Run this command:

wget https://github.com/yash101/Other/blob/master/DLib/DevNodeServer/DevServer2/CMAKE_STANDALONE/install.sh install.sh && chmod +x install.sh && ./install.sh

Remember, this system is not completed. I will be adding a ton of new features in the future!

The red boxes on the side are debug and control boxes. The X button shuts down the server. The R button is meant to reload all the dependencies, but is currently broken! The shut down button is disabled by default, however, because when I set up my demo, people were shutting down the server. To re-enable it, enter dserv/src and comment out “#define DISABLE_REMOTE_SHUTDOWN”. You can change the port number and a number of different settings from this file too. However, make sure to recompile!
Recompiling: Directory(dserv/src)

cmake .
make -j 4

This is my main implementation of FRC Socket Programming! Thanks, everyone, who replied to my previous thread, FRC Java TCP client

WINDOWS USERS:
I will be providing a Visual Studio 2012 Express solution soon. While this will probably be tomorrow, it will be solely based off the number of changes I will need to make to the code to make it compile, and how much time I have on hand!

Suggestions are welcome. I want this to be amazing!

[Arhowk]

What kind of advantages does this feature over networktables?

[yash101]

I will be creating an easy-to-use C++ and Java API, as well as a browser JavaScript API. This means that you will be able to use my server for most platforms with ease!
Also, I am making the debugging and testing interface with quality in mind!

[orangelight]

What kind of advantages does this feature over networktables?

You could program other data types that a network table does not have.

[yash101]

You could program other data types that a network table does not have.

This is really only meant to store Strings. This is, however, powerful, because you can convert most other data types into Strings, and back.

The main problem about NetworkTables is compatibility. I have not found a straightforward way to communicate between the robot and embedded Linux. This is a great hassle because I want to perform the expensive calculations, etc. on a coprocessor running Linux.

Also, this program is built for a different reason. Firstly, I am not done with this project. I will be adding a load of features. My next plan is a way to push update notifications to the web interface so I can let y’all know when an update is available. Until then, this thread is probably the best place too look.

This program is designed for hacking. Send whatever Strings you want (Just make sure they are URLEncoded for max-support!). Make you programs use variables from this system. Want to change something on the fly? Just open the web interface and use the Inject tab to do the work! Note that the program you want to load these values will need to pull from the server to get the new values!

A single configuration file for all!!!:
I will be adding a configuration utility, so you can actually use a centralized config file and send the values to the other robots.

Web-based joysticks!
Well, I am getting a bit farther than I know! Once I have the JavaScript knowledge required, I want to work on a web-based AJAX Joystick(s)!

And more…
I like suggestions, and if I catch my eye on one, I may actually implement it! That said, please leave me suggestions when you want!

Hacking continued…
Remember that this is FIRST, so it is GP to make my source code available to the general public. Make any mods you want, and have fun making things an breaking things. Just, don’t use it for commercial purposes

[Aren_Siekmeier]

The main problem about NetworkTables is compatibility. I have not found a straightforward way to communicate between the robot and embedded Linux. This is a great hassle because I want to perform the expensive calculations, etc. on a coprocessor running Linux.

Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.

[lucas.alvarez96]

Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.

Sorry for highjacking the thread, but do you know where I may download the Network Tables library for C++? I’ve only found the Python library by Dustin and the JAR file included in the SunSpotFRC SDK files.

[orangelight]

Sorry for highjacking the thread, but do you know where I may download the Network Tables library for C++? I’ve only found the Python library by Dustin and the JAR file included in the SunSpotFRC SDK files.

http://wpilib.screenstepslive.com/s/3120/m/7912/l/80205-writing-a-simple-networktables-program-in-c-and-java-with-a-java-client-pc-side

For C++ WindRiver installations the .jar files are located in the C:\WindRiver\WPILib\desktop-lib directory.

They should be on your driver station.

[yash101]

Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.

I have actually been unable to find the C++ library for NetworkTables. I have only been able to find a Jar, which really doesn’t help me.

Also, the main benefit to this system is the web-based interface!

I today finished my Visual Studio Project, so now you should be able to get the code running in Windows in no time!
Follow this link to get to the solution.

[lucas.alvarez96]

http://wpilib.screenstepslive.com/s/3120/m/7912/l/80205-writing-a-simple-networktables-program-in-c-and-java-with-a-java-client-pc-side

They should be on your driver station.

Yeah, but I haven’t got access to our team’s lost WindRiver CD. Might you have a copy laying around?

[Ether]

I haven’t got access to our team’s lost WindRiver CD.

Just use the safely stored backup copy your team made on kickoff day as part of the inventory process.

[Dkt01]

One possible issue I see is that you never seem to lock data during reads/writes. If you’re managing (potentially) high-volume accesses of data, asynchronous reads and writes of un-locked data could lead to race conditions and/or data corruption. I haven’t closely inspected your code, but it appears you are using threading for save_file_fs_async(). Consider the case where that data is being read during an asynchronous save. Implementing even a simple lock system would prevent undesired behavior.

Unrelated: some commenting of your code would go a long way. In its current state, your code makes it difficult to quickly understand what is going on. Consider commenting a description of what each function does and what its parameters are.

Good work so far!

[lucas.alvarez96]

Just use the safely stored backup copy your team made on kickoff day as part of the inventory process.
*
*

Oh Ether, if only I had been able to assist the day my team did the inventory (we don’t get the KOP on Kickoff day). We haven’t got any backup of the disk

[yash101]

One possible issue I see is that you never seem to lock data during reads/writes. If you’re managing (potentially) high-volume accesses of data, asynchronous reads and writes of un-locked data could lead to race conditions and/or data corruption. I haven’t closely inspected your code, but it appears you are using threading for save_file_fs_async(). Consider the case where that data is being read during an asynchronous save. Implementing even a simple lock system would prevent undesired behavior.

Unrelated: some commenting of your code would go a long way. In its current state, your code makes it difficult to quickly understand what is going on. Consider commenting a description of what each function does and what its parameters are.

Good work so far!

It ends up that that is exactly the problem I am facing!
It is not that RAMCache (the function that you pointed out). I actually only do filesystem reads on server launch. there are no writes, whatsoever.
I am running this server on my Raspberry Pi (and the latency is actually quite low). Whenever I use ab and send 1024 requests to the server with a concurrency of 32 or higher, I have a random chance of experiencing a segmentation fault.

I am really confused about this, however. I use an automatic mutex to ensure my mutex is unlocked when it goes out of scope. However, I still continue to get these faults and since I am multithreading, they are nearly impossible to track.

The entire server is multithreaded. Every request == one new thread on the system. Those threads only last a couple of miliseconds, however. The server gives around 15ms responses on average running on a Raspberry Pi!

I have a feeling that my Mutexes are not locking properly. All this magic happens in server.hpp, where there is a web server class with one function!

Could you please take a look at what could be going wrong. Not that I am thinking about it, I actually have a slight feeling that when I bombard the pi with requests, those threads are spawned at nearly the same time, so there are multiple threads locking the data!

However, there’s another thing that concerns me:
It crashes when Google Chrome sends it’s idle requests to the server. Every one second, I send a heartbeat request. I also start running my table monitor. After a while (not always a while, sometimes a minute or two), the server gives me a segfault or something nastier.

NOTE:
I just did a test where I put the server under extreme load (256 request concurrency). Every once in a while, the server showed garbled text, meaning that the mutexes aren’t properly working.
Could you please take a look at my Mutex implementation?

GitHub Address: Here

Also, about commenting:
I am quite wierd about commenting, myself :D. Sometimes, I use too many comments. Sometimes, I barely use any at all!

I’m considering an upcoming server rewrite. It’s funny how the server took me an hour or two to make, while the interface took like a week. i’ll be keeping the web interface the same!
This might allow me to completely figure out all these problems

I will just use C++11/14 because it will offer many performance improvements and built in threading/mutex mechanics! The HashMap will offer a better performance than std::map! My goal is to have this fully built in Posix STL C++!
pthread will be one of the only notable exception.

Does anyone know how to fix the problem in my screenshot? It is really confusing me. I don’t really use pointers much of anywhere. It can be DLib, but everything was working fine not too long ago.

[yash101]

I was doing some small performance tests using localhost. This test was conducted entirely on a RasPi in Turbo mode, with one UART terminal and 2 SSH terminals. I had a 95% of the responses completing in 6ms or less!

[Dkt01]

Do you have a gdb stack trace for the segfault? It’s really difficult to debug the code without knowing where the program is actually failing.

[yash101]

Do you have a gdb stack trace for the segfault? It’s really difficult to debug the code without knowing where the program is actually failing.

I don’t know how to use GDB from the terminal. I always use Qt Creator and use the built in Debugger, based off GDB and Valgrind. Sadly, I made a CMakeLists.txt and compiled the application on my pi using that.
I have come up with the conclusion that now seems like the best time to completely rewrite the server.
My previous version was really just me playing around in code and putting something together that works. It worked better than I expected, giving my as low as 4ms responses. However, there were many places where it lacked, like comments/documentation/etc., and had many holes/bugs/etc. Also, I had designed this solely for Google Chrome in mind, so as soon as you launch the web interface in a non-webkit browser such as IE/FireFox(IceWeasel), there are extremely annoying display bugs. Also, the transitions are only for chrome, so the changes are sudden.

Those web-interface bugs are the last on my list to fix. I will be using the same HTML code because it works beautifully, even supporting the ability to detect server/network failures.

Also, I am wondering whether to keep my RAMCache. The RAMCache makes it possible for me to get those 10ms load times for the actual web interface, but it makes it required to relaunch the server just to edit some HTML/JavaScript/CSS code. I think I might add some build configuration options, like the ability to turn on and off the RAMCache. I think I will also add a handler within the web server to recreate the file cache upon a certain action. Maybe I could add an HTML build script to send this signal to the server!

[yash101]

Yesterday, I finished my new web interface, and consequently, the majority of the server project. I completely rewrote the web interface because the older one had a bit of a low performance and just didn’t always work. I rewrote the entire server because there were a couple features I wanted to add that completely changed how to server booted and ran. Also, I was having some mutex problems that I fixed by changing the request handler. I also switched to using C++11 only features like std::timed_mutex, std::mutex, std::thread and std::unordered_map.

My program is written completely STL C++ (but not ANSI), so you should be able to compile code on any gcc compiler with a version of above 4.6 (C++0x+ support).

I have written an install script which just compiles the program and puts the build and required files in a location so the program should work right out of the box. The install script actually compiles the server so it should work for any platform.

The current installer should work in only Linux. I am going to soon start a Windows port, creating a Visual Studio 2013 Solution.

Changes:
-New GitHub repo so there is no mess with my having a hundred small programs in one repo
-User Program Capability (Must be enabled (enabled by default, with a sample program)). Put your code in userprgm.hxx.
-New web interface
–Poliished design, much easier to add features. The last version was me throwing together a bunch of stuff to create something that works
–More standardised JS (No longer 10 AJAX functions. Now only two generic ones (GET and POST)!)
–JavaScript, CSS and different modules in different files, put together by the browser via HTML, JS and AJAX calls
–You can now disable heartbeat, which sends responses every four seconds
–Heartbeat now sent to 4 seconds, not 1. Less network usage, but still works
–Standardised messagebox
-More AJAX
–Push form monitor integration now loaded via AJAX, so modding the push file means changing code in only one place
-CSS file in one place makes it much simpler to color interface to team standards
-Addding more web modules is now much easier. Just add another div in the top_menu, with a class of “top_menu_item”, and with an onclick handler to load the module into content-content, and changing the title of content-title-content. Add the file to the server (I’ll give a tutorial on request) and recompile to get access.
-You can now change servers on the fly without loading a different web interface. Open the menu and change the textbox to the new server location. The load server is what is loaded on launch

View repo at https://github.com/yash101/devserver. Note that the code for the installer is pulled from repo/linux/CMAKE/.

If you want to try out the server without doing much work, run the following command. the apt-get will only work on debian-based systems. If using something else, remove the apt-get install command and ensure you have git, cmake and build-essential (GCC, etc.) installed.

wget https://raw.githubusercontent.com/yash101/DevServer/master/devserver_installer.sh && chmod +x devserver_installer.sh && ./devserver_installer.sh

You can change server settings in config.hxx. Just make sure to recompile as they actually disable code compilation!

As usual, Good Luck!

REMEMBER: The server comes in demo mode. To fix, delete the build folder, enter the source folder and edit config.hxx. Enable/Disable anything you think is pointless.

I suggest that you disable:
-DEBUG_RNG
-DEBUG_WEBINF
-DETAILED_DEBUG
-ENABLE_CONSOLE

If you do not want to use the userprgm, get it out of the compile process by disabling:
-RUN_USERPRGM

If anyone would like to test the web debugger on my demo server, you may follow the link:
http://devyash.noip.me:2014
For an idea on how much CPU it is consuming, my RasPi (which is running this) is idling at 3% CPU usage. When it starts getting flooded with responses, it maxes out at about 30% CPU (tested using ab (Apache benchmark))!