Reducing code build time

[NWChen]

Often our team makes small changes to code (e.g. adding a command to an autonomous sequence) that aren't otherwise adjustable via RobotPreferences/NetworkTables/SmartDashboard (as far as I know...)

What can we do to reduce the ~2 minutes it takes to recompile our code each time? Is RobotPy (being interpreted) a faster alternative?

[Tom Line]

We've made a concerted effort over the past three years to create code that doesn't need to be recompiled.

For instance, any constant (numeric) value that we use on the robot is actually stored in a text file and loaded at code startup.

Likewise, we use a scripting system that reads the commands from text files for autonomous.

In that way, changes to auton or constants are as quick as a change of a number in notepad, and then double clicking a batch file to dump it onto the cRIO. We can write entirely new and complicated autonomous programs in a minute or two.

You can even do the same for your hardware assignments, like DIO or PWM channels, etc, so if hardware breaks you can change which port your using in seconds rather than 5 minutes.

[chris.boyle]

Originally Posted by Tom Line

We've made a concerted effort over the past three years to create code that doesn't need to be recompiled.

For instance, any constant (numeric) value that we use on the robot is actually stored in a text file and loaded at code startup.

Likewise, we use a scripting system that reads the commands from text files for autonomous.

In that way, changes to auton or constants are as quick as a change of a number in notepad, and then double clicking a batch file to dump it onto the cRIO. We can write entirely new and complicated autonomous programs in a minute or two.

We have also done the same thing, using a text file to store constants (we've added strings this year). LabVIEW has some built in Configuration File VI's that allow for a standard Windows config file format:

[Section]
Key = Value

These 'Preferences' are read before any other calls in Begin.vi. To aid in the development, we've added a multiple button press on the two driver joysticks to reload these values. This makes it a lot faster to change some constants and ftp them down to the cRIO and not have to reboot it.

Also, this year we have implemented a record and replay system for autonomous. From the custom dashboard we enter a file name and flip a toggle switch. The code on the cRIO opens a binary file and stores the needed values to it. We store the following every 10 ms:

left joystick Y
right joystick Y (we use tank drive),
start position of the catapult
stop position of the catapult
intake arm position (up / down)
intake roller speed (sign determines direction)
transmission state (high / low gear)
launch button

When the toggle switch is pressed again or a time limit has been reached the file is closed.

To replay this file, the driver selects it from a drop-down list on the custom dashboard (list is published from Begin.vi, which reads the root drive of the cRIO), this file name is passed back to the cRIO and opened. Every 10 ms the next set of values is read from the file and written to their original variables. All control of the robot is done in Periodic Tasks.vi.

[joelg236]

cough cough

Why spend time compiling when you don't have to?

[SoftwareBug2.0]

Have any of you guys heard of makefiles? It's the traditional way to set up a comilation so that it only redoes the steps needed when things change.

[Arhowk]

Originally Posted by Tom Line

For instance, any constant (numeric) value that we use on the robot is actually stored in a text file and loaded at code startup.

Likewise, we use a scripting system that reads the commands from text files for autonomous.

As do we, although we have a custom auton language that compiles into binary and is than stored on the cRIO through NetworkTables.

For OP, heres our FileRead.java

package org.chimeras1684.year2014.iterative.aaroot;

import com.sun.squawk.microedition.io.FileConnection;
import java.io.DataInputStream;
import java.io.PrintStream;
import java.util.Enumeration;
import java.util.Hashtable;
import java.util.Vector;
import javax.microedition.io.Connector;

/**
 * API for the FileRead class
 * 
 *  Usage :
 *      Loading a file to the robot
 *          0) FileZilla is recommended because of its ease of auto uploading changes
 *          1) Connect to the robot's FTP, 10.16.84.2, using FRC/FRC as the ftp password
 *          2) Open the \auton\$target$.txt file
 *          3) Make Changes
 *          4) Reupload
 * 
 *      Adding and deleting values on the file
 *          * indicates a comment
 *          leading and trailing whitespace are discarded
 *          use : to seperate the key from the value
 *          newlines for every new value
 *
 *      Loading a file in java:
 *          1) Construct a new FileRead class, passing the name of the file in the \auton\ folder
 *          2) Call the newly constructed file using file.getDouble("yourKey")
 *          3) Auton commands can also be passed string arguments that are read from \auton\config.tt
 *
 *
 *  TODO :   
 *      Make usage of the BufferedReader class.
 *      Accept directories other than auton
 *
 * @author Arhowk
 */
public class FileRead {
    final String dir = "auton/";
    final static char deviceEscape = 0x0A;
    final static String comment = "*";
    final static String divChar = ":";
    final char space = 20;
    final char tab = 9;
    
    PrintStream out;
    DataInputStream theFile;
    FileConnection fc;
    Hashtable dataMap;
    
    /**
     * what why is this a public function
     * @param s
     * @return
     */
    
    /**
     * Opens a file and enters it to the database
     * @param file url of the file to read
     */
    public FileRead(String file){
        try{
            fc = (FileConnection) Connector.open("file:///"+dir+file, Connector.READ);
            theFile = fc.openDataInputStream();
            dataMap = new Hashtable();
            
            while(true){
                try{
                    String next = readLine(theFile);
                    if(next.length() > 0){
                       dataMap.put(next.substring(0, next.indexOf(divChar)).trim(), next.substring(next.indexOf(divChar) + divChar.length(), next.length()).trim());
                    }else{
                        break;
                    }
                }catch(Exception e){
                    
                }
            }
            
        }catch(Exception e){
     //       e.printStackTrace();
        }        
    }
    
    private String removeWhitespace(String s){
        for(int i = 0; i < s.length(); i++){
            if(s.substring(i,i+1).equalsIgnoreCase(""+space) || s.substring(i,i+1).equalsIgnoreCase(""+tab)){
                s = s.substring(0,i) + s.substring(i+1,s.length());
            }
        }
        
        return s;
    }

    /**
     * reads an int out of the file
     * @param key what key
     * @return return int. default 0
     */
    public int getInt(String key){
        try {
            Enumeration e = dataMap.keys();

            while(e.hasMoreElements()){
                String t = (String)e.nextElement();
                if(t.equals(key)){
                    return Integer.parseInt(removeWhitespace(((String)dataMap.get(t))));
                }
            }
        } catch (Exception e) {
        }
        return 0;
    }
    
    /**
     * reads a double from the file
     * @param key read key
     * @return double
     */
    public double getDouble(String key){
        try{
            Enumeration e = dataMap.keys();

            while(e.hasMoreElements()){
                String t = (String)e.nextElement();
                if(t.equals(key)){
                    return Double.parseDouble(removeWhitespace(((String)dataMap.get(t))));
                }
            }
        }catch(Exception e){
            
        }
        
        return 0;
    }
    
    /**
     * Gets a string from the file
     * @param key key 
     * @return return
     */
    public String getString(String key){
        try {
            Enumeration e = dataMap.keys();

            while(e.hasMoreElements()){
                String t = (String)e.nextElement();
                if(t.equals(key)){
                    return (String)dataMap.get(t);
                }
            }
        } catch (Exception e) {
        }
            return null;
    }
    
    private String readLine(DataInputStream s){
        String ret = "";
        boolean isComment = false;
        try{ 
            while(true){
                char next = (char)s.readByte();
                if(next == deviceEscape){
                    if (ret.length() == 0){
                        return readLine(s);
                    }
                    else{
                        return ret;
                    }
                }
                if(comment.indexOf(next) != -1){
                    isComment = true;
                    ret = "";
                }
                if(!isComment){
                    ret += next;
                }
            }
        }catch(Exception e){
            
        }
        if(!isComment){
            return ret;
        }else{
            return readLine(s);
        }
                    
    }
}

(currently only folder available is /auton/)

to use this, place a text file that looks like this in /auton/yourPath.txt (ext doesnt matter, i use txt for easy editing)

constValue : 105.54
* commentedValue : 930

and call it through

FileRead f = new FileRead("yourPath.txt");
double constValue = f.getDouble("constValue");

Originally Posted by NWChen

Often our team makes small changes to code (e.g. adding a command to an autonomous sequence) that aren't otherwise adjustable via RobotPreferences/NetworkTables/SmartDashboard (as far as I know...)

What can we do to reduce the ~2 minutes it takes to recompile our code each time? Is RobotPy (being interpreted) a faster alternative?

I've never tried python before but I believe the dragons, 1243, used Python this year and loved it. If you're programmers know what they're doing, than I'd recommend trying Python. The only reason our team doesn't use python is because a majority of our programmers aren't experienced enough to use it and we have the auton language system.

Out of curiosity, what language are you using? Compiling and loading java takes about 30 seconds plugged in 1 minute on battery for me ( battery saver settings ).

[NWChen]

Originally Posted by Arhowk

Out of curiosity, what language are you using? Compiling and loading java takes about 30 seconds plugged in 1 minute on battery for me ( battery saver settings ).

Thanks for the detailed reply.

We are using Java, standard WPILibJ and all. Compiling and loading OTA takes between 1.5 minutes to 2 minutes plugged in.

[Arhowk]

Originally Posted by NWChen

Thanks for the detailed reply.

We are using Java, standard WPILibJ and all. Compiling and loading OTA takes between 1.5 minutes to 2 minutes plugged in.

You're laptop seems a bit outdated, if you/your team has the money I'd recommend getting a new laptop in November. Anyway, feel free to use that snippet as you like, our team constantly uses it for things like pid tuning or controls. I'd Aldo give you auton but it's framework isn't ready for public release.

[virtuald]

Originally Posted by NWChen

Often our team makes small changes to code (e.g. adding a command to an autonomous sequence) that aren't otherwise adjustable via RobotPreferences/NetworkTables/SmartDashboard (as far as I know...)

Then you should probably restructure your code so it can be. Some of the scripting suggestions above may help.

What can we do to reduce the ~2 minutes it takes to recompile our code each time? Is RobotPy (being interpreted) a faster alternative?

RobotPy doesn't require any compilation, so if you're using pyfrc for uploads then its maybe ~15 seconds for an upload, and then the ~30 seconds or so to wait for the robot to reboot (currently RobotPy doesn't support hot reload very well, so we just always reboot).

One nice thing about RobotPy/pyfrc is that it has a robot simulator you can run on your PC, so you can try many types of new things out without having to even touch your robot. This makes your recompile time effectively zero.

[MrRoboSteve]

An SSD in your build machine would also be worth trying. $100 gets you a decent one these days.

[yash101]

Originally Posted by MrRoboSteve

An SSD in your build machine would also be worth trying. $100 gets you a decent one these days.

Most teams have a hard time shelling out $100 because funding can be hard to come by!

I have a 32 giger in my laptop and running win7 on it, it boots in 3s. However, it is impractical because of the price, even for just a small one. Thankfully I have a second hard disk bay in my laptop.
I decent hard disk should be able to provide sufficient speed for a lengthy compile. Getting an SSD becomes impractical, because, for example, my laptop has a very low rating because the processor gives me a 3.1 but the ssd gives me 7

Just get a decent machine, with preferable an i3 or i5 (or i7 if you are lucky to find one cheap).

While this is kinda off topic, C++ is known for large build times, and I want new code to update on the system on bootup every match. For that, I wrote a build script that just automatically builds the code. It takes 5 mins so it isn't too bad and the program is automatically launched after the build and if the build fails, the last successful build is run and the errors are logged!

[NWChen]

Originally Posted by virtuald

One nice thing about RobotPy/pyfrc is that it has a robot simulator you can run on your PC, so you can try many types of new things out without having to even touch your robot. This makes your recompile time effectively zero.

I've been looking at pyfrc's simulator, but with the addition of Gazebo to the Java tools (though I haven't been able to get it to work yet) and given what we've done in past years we're staying with Java so far.

Thanks for all the ideas so far. Our mentor Priscilla added that a lot of the rebuild time is due to the cRIO building, not necessarily compilation.

[AndrewD]

Our team uses RoboyPy and what you can do is have a button in your code that sends the command sys.exit() which forces your not to reload the code. Basically reduces change time from 2m to around 30s.