Preamble:
I always have trouble with file I/O. I’m not sure why. It should be easy, right? Bear with me if I’m overlooking things right in front of me (if, for example, I should be using datalogs instead of a tab-delimited text document)
Problem:
I want to log data to a file. There will be new data regularly; the point of the log is to record a history of the data. This means I want to read it as an array when I FTP the file over to my computer. It may be thousands of entries long. I don’t want the data to become corrupt if I quit the program improperly (the robot is shut down, for example, or my project explorer looses comms during a temporary deployment).
I’d prefer human-readable, but I’ll let that go if it’s not practical.
I’d also like it to have a fairly low overhead, so the logging does not slow down the data collection. I could be logging anything from 6 bytes every 20ms to 8 bytes every 1ms.
I barely qualify as experienced, and am certainly not an expert, but here is how I have done it. I wrote directly to the cRio for the purpose of capturing match data, so the file was relatively small. I got this idea from Rob VanHam from FRC67 & FRC 548.
There probably isn’t one pat answer or it would just be built in.
The simplest to use are the single icon blue functions. Easy, but probably not highest performance, and not as much control.
A step up from that is to open the file once, store the refnum or pass it through the looping element. Each iteration adds to the file with a write text or write binary, possibly flushing the file. The flush ensures that the underlying file manager actually writes the info it may be caching into the disk representation of the file and will ensure that a power outage or crash will not lose data. Outside of the loop, you can close the file and zero out any info you may have about the log.
If either of these simple approaches works, stick with it. If not, you may need to introduce your own caches or find ways to sneak the I/O writes in and have less impact on execution. You can keep info in arrays and write the info to a new file all at once. You can alternately put the file I/O into a background priority subVI and use queues or RT FIFOs to send data to the logger and allow the logging to proceed whenever it has enough info and CPU. I’m sure there are lots of other tricks that can be applied as well, but as usual, start simple and determine where that gets you.
Hmm.
I think I could simply use that “write to text file” function (using a refnum), writing each piece of data as it is received, and then just flush the file regularly.
Firstly, however, I need to make a new file store the data.
I made a VI to do that, but it doesn’t completely work – I’m having trouble with the “list folder” function.
Much of the VI you posted seems to deal with a file group with some naming scheme that increments.
The write to spreadsheet string actually opens and closes the file each time. That should be more effective and a bit more expensive than a flush on an open file. If it works, go with that. If it slows down your program too much, open it up to see how the lower level primitives are used.
If you still decide to do the file group, you basically have to step through enough instances that you trust the logic. I try to avoid the file group or keep it simple, as I seem to make a mess for myself that I have to debug whenever I attempt such a thing.
Here’s a simplified example of how I intend to use it:
The “make new file” is only run once, until the program is restarted. I don’t want to erase my data just because I restarted the program, but I do want to separate each session into a different file.
The reason I’m not using a “write to spreadsheet file” is because I would have to use a queue to get the data out of the loop so I could write it as an array. Since I don’t need to read that data - I just need to write it to a file - I thought using the file cache would be an efficient way of doing this.
EDIT:
I fixed the VI. I found that the “list folder” function wasn’t taking the regular expression “[0-9]+”. I think it doesn’t support the braces. Anyways, I worked around that by simply replacing it with “*”. I know that the “number string to integer” function will return 0 if there is no number in the string. Since I’m only using the maximum value in the array, that will be fine.
I’ll try implementing the file logging on the cRIO later.
