Thanks for your post! I’m hoping others will follow. I agree, a presentation or whitepaper on the topic of data (and IT) management for FIRST teams would be very beneficial.
I can reply back with a little bit about our setup. It’s older hardware but it’s been working well for us. And being older hardware, it’s been very affordable for the value we’re getting out of it.
Dell Poweredge 1850 with two 160GB 10K RPM SCSI drives in RAID1, 8GB RAM, and two Xeon dual-core processors. We run Windows Server 2008 R2 and Autodesk Vault 2014 Basic Server on here and use it for CAD only. Either drive dies, we’re ok (in theory). We keep an identical drive in “cold storage”
Rackable Systems 2U, half depth server with four 1TB SATA drives in RAIDZ, running FreeNAS 8.3.2. Dual Xeon Dual Core, 8GB RAM. This is our primary data storage server for all work and archiving other than CAD. We keep photos, videos (mostly finished works, very little raw footage), CAM files, graphics, animations, Matlab files, software installers, content we’ve created, and as of recently, Windows System Image backups here. The Vault server (above) also backs up to here. The Vault Server and FreeNAS server live in separate buildings. FreeNAS lets us control user access and permissions. We’re currently at about 1.4TB used, and 0.98TB remaining. Within the next month, we’re going to upgrade the drives to WD Red 2TB drives to double the capacity to 8TB total, which should give us somewhere around 5.5TB I imagine in ZFS. While we’d much prefer RAIDZ2, RAIDZ is all that’s practical on only 4 drives. The good news is, ZFS is a pretty reliable file system with lots of error checking, and the only time you really need to worry is when you’re in the process of replacing a bad drive (which is a real concern). Also of note, supposedly FreeNAS takes “snapshots” of each dataset on an interval that we set and that can be rolled back at any time, but I’ve never tested the restoration to an earlier snapshot.
Seagate 3TB external drive, formatted NTFS - This is where the FreeNAS server is backed up to. It’s not automated yet. At least once per year, we manually copy all the network shares to this external drive. It’s kept in “cold storage” in a separate building and is not used for anything other than backups. Obviously, this is not a sustainable way to continue, especially after we upgrade the FreeNAS Server. What we may do is set up a Windows desktop with the four older 1TB drives we pull out, plus the 3TB external, and do automated backups to that. Long term, maybe we build another FreeNAS box and rsync to it.
Dropbox We have a very small amount (<2GB) of business-team related data (logos, artwork, etc) for access from home when needed.
Google Drive - Mostly for collaborative editing of spreadsheets and presentations. Our accounting is done here, before being entered more formally into Quickbooks.
Website - We have 10GB of hosted space on a cPanel shared virtual server, that we pay monthly for.
** AVID ISIS Server ** - This does not belong to the robotics team, but we can access it. It has somewhere in the neighborhood of 48TB of drives and serves about 160 users in the Cinematography program at our school. All raw footage is loaded onto the ISIS server and then worked with there during production. I’m not aware of the archival and backup processes and standards for video on the ISIS server.
Very little critical data is ever stored on local machines. Our general policy is not to. The only exception may be files for vinyl cutting, but the artwork exists elsewhere.
All of this is something we’ve implemented just in the past two years. I like to think we’re pretty solid on data integrity (at least for our operation), at least in theory. Nothing we’re doing would ever compare to proper hardware and practices in a business with a proper IT department, but I like to think we do it pretty well just for a school FIRST team. Some of the backup and restoration processes have never been tested, which is a little scary, but in theory we shouldn’t lose much of anything with the way we’re set up.
The big idea is that no data should ever exist for any extended period of time on only one physical storage device, and never only a device (flash drive) that could become easily lost, stolen, or damaged. Also remember, redundancy is not the same as a backup!