Go to Post I don't think Dean Kamen ever recieved a real degree. That doesn't make him any less of an engineer.:) - Adam Y. [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rating: Thread Rating: 4 votes, 5.00 average. Display Modes
  #1   Spotlight this post!  
Unread 03-09-2014, 01:10
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
The DevServer TCP-based data server

I have been working on a project to make communications between multiple devices on the field quite simplistic. I present to you, the DevServer. It is an HTTP (TCP) based server, where the Java or C++ API can send and retrieve values, and at the same time, have them available on other devices on the network.

Features:
Mostly STL C++, some DLib, with most headers commented out.
Amazing web interface, which is clean, fast and works well
AJAX for most web loading!
The web pages are loaded into RAM on runtime, so the load times are quite minimal
C++, Java, and possibly JavaScript(browser) APIs will be released soon

Most of you will be wondering, what is this, and why I made this. This is a server, in which you can store and access data through. It supports multiple clients at the same time, so it can be used for passing data between a coprocessor and the cRIO. This nifty server comes bundled with a powerful web interface which has documentation, a way to send values, and a way to monitor all the values in real time. Based off vanilla JavaScript, the web interface requires no client setup. However, it is designed for webkit, so it may not display properly in FireFox and IE.

The code is available on GitHub, [HERE]

If you want to automatically download and install the server in Debian/GNU Linux, I have written an installer, and a simple one-line command to install:
Run this command:
PHP Code:
wget https://github.com/yash101/Other/blob/master/DLib/DevNodeServer/DevServer2/CMAKE_STANDALONE/install.sh install.sh && chmod +x install.sh && ./install.sh 
Remember, this system is not completed. I will be adding a ton of new features in the future!

The red boxes on the side are debug and control boxes. The X button shuts down the server. The R button is meant to reload all the dependencies, but is currently broken! The shut down button is disabled by default, however, because when I set up my demo, people were shutting down the server. To re-enable it, enter dserv/src and comment out "#define DISABLE_REMOTE_SHUTDOWN". You can change the port number and a number of different settings from this file too. However, make sure to recompile!
Recompiling: Directory(dserv/src)
PHP Code:
cmake .
make -j 4 
This is my main implementation of FRC Socket Programming! Thanks, everyone, who replied to my previous thread, FRC Java TCP client

WINDOWS USERS:
I will be providing a Visual Studio 2012 Express solution soon. While this will probably be tomorrow, it will be solely based off the number of changes I will need to make to the code to make it compile, and how much time I have on hand!

Suggestions are welcome. I want this to be amazing!

Last edited by yash101 : 03-09-2014 at 01:20.
  #2   Spotlight this post!  
Unread 03-09-2014, 11:06
Arhowk's Avatar
Arhowk Arhowk is offline
FiM CSA
AKA: Jake Niman
FRC #1684 (The Chimeras) (5460 Mentor)
 
Join Date: Jan 2013
Rookie Year: 2013
Location: Lapeer
Posts: 544
Arhowk is a splendid one to beholdArhowk is a splendid one to beholdArhowk is a splendid one to beholdArhowk is a splendid one to beholdArhowk is a splendid one to beholdArhowk is a splendid one to behold
Re: The DevServer TCP-based data server

What kind of advantages does this feature over networktables?
  #3   Spotlight this post!  
Unread 03-09-2014, 11:50
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: The DevServer TCP-based data server

I will be creating an easy-to-use C++ and Java API, as well as a browser JavaScript API. This means that you will be able to use my server for most platforms with ease!
Also, I am making the debugging and testing interface with quality in mind!
  #4   Spotlight this post!  
Unread 03-09-2014, 20:20
orangelight's Avatar
orangelight orangelight is offline
Registered User
AKA: Alex
FRC #0548 (Robostangs)
Team Role: Programmer
 
Join Date: Feb 2014
Rookie Year: 2014
Location: Northville
Posts: 168
orangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to behold
Re: The DevServer TCP-based data server

Quote:
Originally Posted by Arhowk View Post
What kind of advantages does this feature over networktables?
You could program other data types that a network table does not have.
__________________


  #5   Spotlight this post!  
Unread 03-09-2014, 21:22
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: The DevServer TCP-based data server

Quote:
Originally Posted by orangelight View Post
You could program other data types that a network table does not have.
This is really only meant to store Strings. This is, however, powerful, because you can convert most other data types into Strings, and back.

The main problem about NetworkTables is compatibility. I have not found a straightforward way to communicate between the robot and embedded Linux. This is a great hassle because I want to perform the expensive calculations, etc. on a coprocessor running Linux.

Also, this program is built for a different reason. Firstly, I am not done with this project. I will be adding a load of features. My next plan is a way to push update notifications to the web interface so I can let y'all know when an update is available. Until then, this thread is probably the best place too look.

This program is designed for hacking. Send whatever Strings you want (Just make sure they are URLEncoded for max-support!). Make you programs use variables from this system. Want to change something on the fly? Just open the web interface and use the Inject tab to do the work! Note that the program you want to load these values will need to pull from the server to get the new values!

A single configuration file for all!!!:
I will be adding a configuration utility, so you can actually use a centralized config file and send the values to the other robots.

Web-based joysticks!
Well, I am getting a bit farther than I know! Once I have the JavaScript knowledge required, I want to work on a web-based AJAX Joystick(s)!

And more...
I like suggestions, and if I catch my eye on one, I may actually implement it! That said, please leave me suggestions when you want!

Hacking continued...
Remember that this is FIRST, so it is GP to make my source code available to the general public. Make any mods you want, and have fun making things an breaking things. Just, don't use it for commercial purposes
  #6   Spotlight this post!  
Unread 04-09-2014, 01:34
Aren Siekmeier's Avatar
Aren Siekmeier Aren Siekmeier is online now
on walkabout
FRC #2175 (The Fighting Calculators)
Team Role: Mentor
 
Join Date: Apr 2008
Rookie Year: 2008
Location: 대한민국
Posts: 735
Aren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond reputeAren Siekmeier has a reputation beyond repute
Re: The DevServer TCP-based data server

Quote:
Originally Posted by yash101 View Post
The main problem about NetworkTables is compatibility. I have not found a straightforward way to communicate between the robot and embedded Linux. This is a great hassle because I want to perform the expensive calculations, etc. on a coprocessor running Linux.
Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.
  #7   Spotlight this post!  
Unread 05-09-2014, 17:42
lucas.alvarez96's Avatar
lucas.alvarez96 lucas.alvarez96 is offline
Registered User
AKA: Lucas Alvarez
FRC #2576 (Chilean Heart)
 
Join Date: Dec 2013
Rookie Year: 2013
Location: Chile
Posts: 123
lucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to all
Re: The DevServer TCP-based data server

Quote:
Originally Posted by compwiztobe View Post
Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.
Sorry for highjacking the thread, but do you know where I may download the Network Tables library for C++? I've only found the Python library by Dustin and the JAR file included in the SunSpotFRC SDK files.
__________________
FRC 2576 2015-2016: Mentor
FRC 2576 2013-2015: Programmer & Chairman's Presenter

Los Angeles Regional 2014: Regional Chairman's Award
  #8   Spotlight this post!  
Unread 05-09-2014, 18:28
orangelight's Avatar
orangelight orangelight is offline
Registered User
AKA: Alex
FRC #0548 (Robostangs)
Team Role: Programmer
 
Join Date: Feb 2014
Rookie Year: 2014
Location: Northville
Posts: 168
orangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to beholdorangelight is a splendid one to behold
Re: The DevServer TCP-based data server

Quote:
Originally Posted by lucas.alvarez96 View Post
Sorry for highjacking the thread, but do you know where I may download the Network Tables library for C++? I've only found the Python library by Dustin and the JAR file included in the SunSpotFRC SDK files.
http://wpilib.screenstepslive.com/s/...client-pc-side

Quote:
For C++ WindRiver installations the .jar files are located in the C:\WindRiver\WPILib\desktop-lib directory.
They should be on your driver station.
__________________


  #9   Spotlight this post!  
Unread 06-09-2014, 12:19
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: The DevServer TCP-based data server

Quote:
Originally Posted by compwiztobe View Post
Have you had issues with the existing Java, C++, and Python libraries for Network Tables? You can include the source for any of these when building a Linux executable/script, they are not dependent on platform.
I have actually been unable to find the C++ library for NetworkTables. I have only been able to find a Jar, which really doesn't help me.

Also, the main benefit to this system is the web-based interface!

I today finished my Visual Studio Project, so now you should be able to get the code running in Windows in no time!
Follow this link to get to the solution.
  #10   Spotlight this post!  
Unread 13-09-2014, 17:10
lucas.alvarez96's Avatar
lucas.alvarez96 lucas.alvarez96 is offline
Registered User
AKA: Lucas Alvarez
FRC #2576 (Chilean Heart)
 
Join Date: Dec 2013
Rookie Year: 2013
Location: Chile
Posts: 123
lucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to all
Re: The DevServer TCP-based data server

Quote:
Originally Posted by orangelight View Post
http://wpilib.screenstepslive.com/s/...client-pc-side

They should be on your driver station.
Yeah, but I haven't got access to our team's lost WindRiver CD. Might you have a copy laying around?
__________________
FRC 2576 2015-2016: Mentor
FRC 2576 2013-2015: Programmer & Chairman's Presenter

Los Angeles Regional 2014: Regional Chairman's Award
  #11   Spotlight this post!  
Unread 13-09-2014, 17:13
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 8,125
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: The DevServer TCP-based data server

Quote:
Originally Posted by lucas.alvarez96 View Post
I haven't got access to our team's lost WindRiver CD.
Just use the safely stored backup copy your team made on kickoff day as part of the inventory process.


  #12   Spotlight this post!  
Unread 13-09-2014, 17:50
Dkt01's Avatar
Dkt01 Dkt01 is offline
Programming Mentor
AKA: David
FRC #1756 (Argos)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2009
Location: Peoria, Il
Posts: 145
Dkt01 will become famous soon enough
Re: The DevServer TCP-based data server

One possible issue I see is that you never seem to lock data during reads/writes. If you're managing (potentially) high-volume accesses of data, asynchronous reads and writes of un-locked data could lead to race conditions and/or data corruption. I haven't closely inspected your code, but it appears you are using threading for save_file_fs_async(). Consider the case where that data is being read during an asynchronous save. Implementing even a simple lock system would prevent undesired behavior.

Unrelated: some commenting of your code would go a long way. In its current state, your code makes it difficult to quickly understand what is going on. Consider commenting a description of what each function does and what its parameters are.

Good work so far!
  #13   Spotlight this post!  
Unread 13-09-2014, 17:53
lucas.alvarez96's Avatar
lucas.alvarez96 lucas.alvarez96 is offline
Registered User
AKA: Lucas Alvarez
FRC #2576 (Chilean Heart)
 
Join Date: Dec 2013
Rookie Year: 2013
Location: Chile
Posts: 123
lucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to alllucas.alvarez96 is a name known to all
Re: The DevServer TCP-based data server

Quote:
Originally Posted by Ether View Post
Just use the safely stored backup copy your team made on kickoff day as part of the inventory process.

Oh Ether, if only I had been able to assist the day my team did the inventory (we don't get the KOP on Kickoff day). We haven't got any backup of the disk
__________________
FRC 2576 2015-2016: Mentor
FRC 2576 2013-2015: Programmer & Chairman's Presenter

Los Angeles Regional 2014: Regional Chairman's Award
  #14   Spotlight this post!  
Unread 16-09-2014, 22:59
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: The DevServer TCP-based data server

Quote:
Originally Posted by Dkt01 View Post
One possible issue I see is that you never seem to lock data during reads/writes. If you're managing (potentially) high-volume accesses of data, asynchronous reads and writes of un-locked data could lead to race conditions and/or data corruption. I haven't closely inspected your code, but it appears you are using threading for save_file_fs_async(). Consider the case where that data is being read during an asynchronous save. Implementing even a simple lock system would prevent undesired behavior.

Unrelated: some commenting of your code would go a long way. In its current state, your code makes it difficult to quickly understand what is going on. Consider commenting a description of what each function does and what its parameters are.

Good work so far!
It ends up that that is exactly the problem I am facing!
It is not that RAMCache (the function that you pointed out). I actually only do filesystem reads on server launch. there are no writes, whatsoever.
I am running this server on my Raspberry Pi (and the latency is actually quite low). Whenever I use ab and send 1024 requests to the server with a concurrency of 32 or higher, I have a random chance of experiencing a segmentation fault.

I am really confused about this, however. I use an automatic mutex to ensure my mutex is unlocked when it goes out of scope. However, I still continue to get these faults and since I am multithreading, they are nearly impossible to track.

The entire server is multithreaded. Every request == one new thread on the system. Those threads only last a couple of miliseconds, however. The server gives around 15ms responses on average running on a Raspberry Pi!

I have a feeling that my Mutexes are not locking properly. All this magic happens in server.hpp, where there is a web server class with one function!

Could you please take a look at what could be going wrong. Not that I am thinking about it, I actually have a slight feeling that when I bombard the pi with requests, those threads are spawned at nearly the same time, so there are multiple threads locking the data!

However, there's another thing that concerns me:
It crashes when Google Chrome sends it's idle requests to the server. Every one second, I send a heartbeat request. I also start running my table monitor. After a while (not always a while, sometimes a minute or two), the server gives me a segfault or something nastier.

NOTE:
I just did a test where I put the server under extreme load (256 request concurrency). Every once in a while, the server showed garbled text, meaning that the mutexes aren't properly working.
Could you please take a look at my Mutex implementation?

GitHub Address: Here

Also, about commenting:
I am quite wierd about commenting, myself . Sometimes, I use too many comments. Sometimes, I barely use any at all!

I'm considering an upcoming server rewrite. It's funny how the server took me an hour or two to make, while the interface took like a week. i'll be keeping the web interface the same!
This might allow me to completely figure out all these problems

I will just use C++11/14 because it will offer many performance improvements and built in threading/mutex mechanics! The HashMap will offer a better performance than std::map! My goal is to have this fully built in Posix STL C++!
pthread will be one of the only notable exception.

Does anyone know how to fix the problem in my screenshot? It is really confusing me. I don't really use pointers much of anywhere. It can be DLib, but everything was working fine not too long ago.
Attached Thumbnails
Click image for larger version

Name:	error_detected.png
Views:	14
Size:	145.6 KB
ID:	17318  

Last edited by yash101 : 17-09-2014 at 01:21.
  #15   Spotlight this post!  
Unread 18-09-2014, 01:17
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: The DevServer TCP-based data server

I was doing some small performance tests using localhost. This test was conducted entirely on a RasPi in Turbo mode, with one UART terminal and 2 SSH terminals. I had a 95% of the responses completing in 6ms or less!
Attached Thumbnails
Click image for larger version

Name:	the_speed.PNG
Views:	16
Size:	85.2 KB
ID:	17320  
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 03:03.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi