So, with Gradle coming to the masses in 2019. I was thinking about implementing a mature build process. Before reinventing the wheel, I thought I would post here to see if other teams might have solved this. Here’s what I’m thinking our build process would look like.
We would have a git server locally at our build space. The build server would go to competitions and be in the pit (wired lan). Developers commit locally, and push to the server. Competition releases are built from the server and deployed. If in an emergency case, deploy from laptop to robot, a developer can use the Wpilib deploy action in VS Code. Git hub would be used when we make our code public.
Anyone else doing this or similar? I would love to see what the pain points are or what works.
Use two laptops (two developers) while in our shop and push periodically to a private GitHub repository during code dev. At competitions, we only brought one laptop with us with the latest “Pre-Competition” commit. During competition, we would make minor changes i.e. adjusting paths, outtake speeds, etc. and deploy those changes to the Robot (using GradleRIO).
The same laptop mentioned above that was used to make quick modifications at competition was also the Driver Station. This is probably why we didn’t bother with setting up servers where multiple developers can push code at competition. After the competition ended, we would then push a “Post-Competition” commit to GitHub.
What problem are you solving by bringing an additional piece of (stationary?) equipment into your pit with students and laptops physically connected to it?
Will you be able to push code updates while on the practice field?
The software engineer in me doesn’t like the idea of developers pushing come to production. There isn’t a guarantee the code has been checked in.
Last year, our developers were working on the practice field to add a new auton. They would make changes and test. Before going back out for the next match, they would comment out code and deploy back to a previous version. Well, suddenly after several iterations our original auto would finish, then our bot would just sit and stutter trying to move for all teleop. After the first occurrence, I worked with the developers, and I thought they fixed the issue. Next match, it happened again. So only safest way to continue was to roll back to a check in before the competition. The iterations of testing weren’t checked in. So we have no idea why it broke.
Always pulling from git, at competitions, ensures any code ran has traceability. So we know we have the code that was on the bot. Doing that on the developer laptop is hard, since a pull would over write the code they are trying to work on at the practice field.
If we had internet access at competitions, I’d just pull for github. Developers can always deploy their code directly to the bot, but at competitions that should be the exception. I get really nervous when we don’t have time and you have to short cut steps.
It is possible to put Git/Gitlab on an Oracle VirtualBox on Windows or even install it with the Windows binaries (if you like doing very strange things, use the Nginx Windows binaries).
Then you can have the entire repository to work with not just the local branch.
You don’t have to do this on your driver’s station obviously.
Just put it on a development computer and use a network switch or cross over cable with static IP, or a wired router with DHCP as a switch (don’t use a wireless router and if you ignore this remove any external antenna from it to ruin the range of the WiFi you really should not be using).
The point here is you don’t need a stationary extra device. Use a Oracle VirtualBox VM. It won’t cost you anything except electricity. This works as well on Mac OSX and Linux. Of course you could also use VMWare, VMWare Fusion, Parallels, KVM, Xen but the licensing/compatibility starts to cost or be less.
Might be an interesting thing to make a Oracle VirtualBox Appliance available to the community. Just download, install Oracle VirtualBox, finish configuration using documentation.
So my gut instinct is that setting up a server is energy better spent elsewhere.
In your scenario, the programmers working on the robot could be committing locally and continuing to iterate and add versions and have the ability to roll back without ever needing to touch another system (read: push to a remote). Keeping code for a specific developer or competition or laptop in a branch provides for those changes to even be merged in later to the main branch - you don’t technically need a branch for it but I think it helps.
Like I said - to each their own. We used rsync for some of our shenanigans this past year and while that didn’t involve a server it was definitely not the cleanest method.
Unlike CVS or SVN, git doesn’t need a server. Each repository is self contained. We have, in the past, had users commit locally, and then push to a master flash drive that could move between computers.
I’m with Marshall on this. It seems that the use case you’re describing is adequately handled by programmers properly managing their local repositories. The situation you’re describing is handled by the programmer creating a new branch for the new autonomous, then switching back to the stable branch for matches.
Even if you have multiple programmers tuning on multiple laptops at the same time (which I really hope you don’t), you can achieve the same functionality by usb tethering the laptops to smartphones and pushing to hotfix branches on your remote repo.
At best, the extra server would work as another backup if you’re worried about bricking programming laptops during a competition. (This is something I have done as a student.)
I’m not even sure what use it would be for building, since most robot code I’ve seen builds on a laptop in under a minute, and you’re not going to take your server to the practice field anyways.
The largest git hurdle for most teams is between the keyboard and the chair. It takes work to drill new students in proper git procedure, especially since many consider it a bureaucratic waste of time at first. This is the area where the greatest benefit is likely to be derived.
Here’s another way to get all of GitLab on-the-go.
While you can run Docker on Windows with Windows 10 Pro/Enterprise you’d be better off putting Docker on LXLE in a VM on Windows such that you are booting Windows, running an Ubuntu Linux kernel in a VM hosting a Docker container.
The only upside here I can see over just GitLab in the VM is the HowTo and community already exists. LXLE still wants 512MB of RAM just for the OS (CrunchBang can run in 256MB but is more Debian than Ubuntu). So putting Docker in LXLE and then web server, DB, and git don’t be surprised if that VM needs 1GB of RAM to even work.
For those that don’t realize GitLab is a web interface over Git. Yes Git is a distributed system for the command line parts. GitLab makes it more like GitHub in that you have the web interface. Git alone can operate a remote repository without GitLab or GitHub. Only you’d loose the web interface, collaboration tools, and git notes are, in my experience, not as handy as issue tracking. Course you can extend git command line with 3rd party tools without resorting to such a full web interface, see products like this: http://bugseverywhere.org
It’s an interesting use case, sort of like operating GitLab instead of using GitHub because you have security and compliance needs like I often have on military and finance projects. It could also be a use case for Internet caching systems for offline Internet content:
There is no need for bringing a server with you (and I suspect would cause more issues than its worth). Git handles this nicely because not only can you push/pull to a server, but (as mentioned) you can also push/pull to a place on your filesystem (such as a USB drive).
The one thing missing from the other comments is how to do it. Here’s how!
First, you need to create a ‘bare’ repo to push/pull to. Insert your USB drive, let’s say you’re on Windows and it’s on E. Open a terminal.
E:
git init --bare repo
Next, you have to point your source checkout to it (do this from each computer that needs it). To do this, you need to create a ‘remote’ in your local checkout. You’ve probably noticed that when using git we often type the word ‘origin’. Turns out, this is the default remote usually created when you initially clone a repo.
cd path o\repo
git remote add usb E:\repo
Finally, you have to push code to it. The idea is 'git push '. So…
git push usb master
Pulling is the same.
git pull usb master
You can do all of your normal git things. Just where you would often type ‘origin’, you can type ‘usb’ now. There are some shortcuts and some limited gotchas, but it’s been in my experience the best way to share code at a competition without using internet.
Probably worth noting that if you create a filesystem repo as documented by Virtuald above with --bare you can still make a mirror repository in GitHub or GitLab with --mirror.
So if your USB drive with Git filesystem repository manages to go missing: you still have all your commits, tags and refs in GitLab or GitHub to go and start again.
Of course you could just use the USB filesystem repository till you get a finished product and cherry pick that commit only and put that in GitLab or GitHub.
Another handy approach is this:
Also take note if you do create a USB filesystem Git repository: make sure your USB flash drive is formatted FAT32. Not all NTFS filesystem or EXT2/3 filesystems work right across Windows/Linux/Mac OSX (see FUSE). It may be less efficient at storing files but FAT32 has wide support across OS and lots of recovery tools if you have a filesystem error.
(Noted because as CSA at Mount Olive District event one year someone couldn’t get to critical files on their USB flash drive with their laptop, but once we got the files off, reformatted the USB flash drive as FAT32, and put the files back on it all was fine.)
So the root issue is running code on the bot that’s not checked in.
This year we’re looking at using gradle to add a config/properties file with some build information, such as branch, version tag, and commit ID of the code. The developers want to have that information at the beginning of log files. Having everything checked in is important for that information to be meaningful.
I could address the issue with gradle, if the machine has uncommitted code, don’t deploy and let student know. There could be an over-ride, with a force flag, where it would ignore the uncommitted code.
We use gitflow as our branching strategy. A student can work on a feature branch at competition. If they checkout master to redeploy, but forgot to commit the feature changes, depending on the change, it might still exists on the master branch. The deploy would warn the student. The student could revert to the feature branch and commit the change, or they could not worry about the change and deploy with the force option.
Am curious: are you simply cherry picking when you commit into GitHub or are you doing development in GitHub and making it work with your GitFlow instead of the more natural GitHub Flow?
Github, we push often. Actually, last year our repo was public all season. Github has master and Dev. Feature branches are on the developer machine, same with release branch.
I want release branches, after created, to be verified by the drive team. If they work, finish release and merge into master. If there needs a tweak, make it in the release branch, and then finish the release.
Master should be the only branch on the bot. At a comp, developers can create a feature branch to add or change something. Once it’s good, finish feature and create a release branch. Drive team approves that it’s good and they want it in a match. Finish release branch and deploy to bot.
There are always exceptions, but they should follow a work flow, and then get exceptions approved when needed. But in any case, the code on the bot should be committed.
Definitely on the marshall/Joe Ross bandwagon here.
If the last four years of my FRC software experience has taught me anything, it’s that you’re only going to get about a 48 hour window to program the robot and tune things before it’s bagged. Because of how precious this time is, I’m pretty uninterested in anything that increases the time it takes to iterate.
Most students I work with have never used git before. Merging, branching, and all of that can be a lot to wrap your head around at first - it was for me, anyway. A complicated workflow increases iteration time.
Most autonomous routines, are, in my experience, being iterated on at competition via practice matches and practice field time. Heck, ask 558 - sometimes robots are being iterated on at competition. Having the concept of “releases” doesn’t really make sense to me in FRC, unless you’re releasing every match.
On 6844 (also true for when I was on 20), the number of people actively writing code that ended up on the robot is countable on one hand. And with n being that low, it is super easy for me to remind everyone to push commits.
Was there code that didn’t get checked in immediately? Totally. You should see some of our commit messages; many read “snapshot of code from event X” (the commit timestamp may or may not be ~1 week after that competition ended).
In my view, while what you’re advocating for is definitely industry-standard, it doesn’t consider the reality of FRC. I’d imagine you’d have more “emergency releases” then not, and that the overhead of having a server build releases and deploy isn’t going to pay dividends.
This is a very fair concern. Different teams will have different standards, but I think it would be universal that all teams would desire to know what source code is presently running on the robot.
It’s fundamentally a people-executing-process issue. If your developer(s) have the discipline to upload code from a single laptop and not touch it till next time you want to make changes, making commits (and build servers and whatever else you have) don’t actually matter.
If the build process you described effectively gets you around the people issues, it’s worth whatever overhead it costs. However, it does involve some overhead, so it may not be worthwhile for all teams. As cool as a potential solution may be, it’s rarely worth the effort to execute unless it provides a clear solution to a well defined problem.
Very fair. We do however enforce the usage of a “blessed repository” - usually github, although it becomes a flash file during competition. In either case, this blessed repo is the canonical one, and developers are not supposed to tag things on it unless the content has been approved for robot usage. This makes sure everyone has the same notion of what the latest reliable software for the robot actually is. In a purely distributed environment, anyone’s version could potentially be the “correct” one, and this does not jive well with our development strategy as a team.
the ability to prohibit deploy with any changed file in the repo (force commit first)
—Wouldn’t always want this for early development, but could turn on for competition.
—Would therefor need to be a build-wide config toggled via a setting in an archived file.
The ability to read SHA1, branch, tag info, and working directory dirty/clean status @ build time in software
–So, as noted here, you could potentially know exactly which version of software is running right now.
Item #1 - One way to handle the toggle, might be to base it on the branch. On develop, it just pushes code, even if changes are staged/unstaged. On master you could have deploy only deploy if everything is checked in. To deploy uncommitted code from master, you’d use a ‘deploy force’. Then developers could go to town when on Dev Branch. While keeping master more policy enforced branch. I’ll have to talk that idea over with our programming students and mentors.
Item #2 - That’s what we’re looking to do. Looking to add Tags to specific commits on Master. The idea is this is a human readable tag, “Version 1.0”, this will be added to a properties file, and published over network tables. So Raspberry Pi and Driverstation can use it for logs. Additionally, it will be displayed on the driverstation for the drive team to read. This way it’s easy for them to know the bot is at the expected version. The commit ID will also be shared in the same way, but this is intended more for developers to use in analyzing competition logs to debug issues. If a deployment isn’t a tagged version, it will have a generic version displayed : Master build 12:24pm (for example)
The programmers also want the technician (which we had a developer do last year) to select a Match Number on the driverstation, pre match, so all logs will have the match number noted. I just asked that Driveteam has enough going on getting setup for a match, and to not add to their list of things to do. So they offered up the Technician as a solution.
There is no need to manually do this at competition events. At competition events, this information is provided by the FMS to the DS to the robot, and is available in software. The DriverStation class has functions getEventName(), getMatchType(), getMatchNumber(), and getReplayNumber() to retrieve it. You can publish this information to NT if desired.