Pixy camera + multiple X values from reflective tape strips + serial communication

This year we’re using a Pixy (CMUCam5) to center on the gear pegs in autonomous. We’ve had pretty inconsistent results centering and we think the issue may be because of the design of the retroreflective tape. Because there is a strip on each side of the peg, it appears as if the X value for the signature is jumping between two values (we had assumed that the Pixy would just return the center of the two, as it would look like an object which was obstructed in the middle).

In the past, we were using the Pixy’s analog output to center, but we have since switched to serial communication. Using code from an older post on here (made by user 2B || !2B IIRC) we have tested it and successfully read the X coordinate from a random colored object we set as the signature. We then pulled out our LED ring and dummy gear peg and taught the Pixy the retroreflective tape.

The value returned now appeared to be alternating between the two strips of tape. Not perfectly alternating… but it was fluctuating by about 40 pixels pretty frequently.

My question is… is there a way to pry the individual X coordinates of each strip of tape from the Pixy? If we could just average the two X values somehow to find the center that would be great. I’m sure there are other teams out there using the Pixy for this… I don’t understand the serial communication code very well, so bear with me. It looks as if there might be a way to scan through the rawData array and check to see if there are multiple packets with the same signature… I would have no idea how to go about implementing this though.

Here’s the code we are using for our Pixy object at the moment (using readPacket(1).X to retrieve X value of signature 1) Please forgive the polymorphic sins I committed in order to condense it into one class:

public class PIDpixy implements PIDSource {
	PIDSourceType m_pidSource;
	SerialPort pixy;
	Port port = Port.kMXP;
	PixyPacket] packets;

	public PIDpixy() {
		pixy = new SerialPort(19200, port);
		packets = new PixyPacket[7];
		m_pidSource = PIDSourceType.kDisplacement;	

	// This method parses raw data from the Pixy into readable integers
	public int cvt(byte upper, byte lower) {
		return (((int) upper & 0xff) << 8) | ((int) lower & 0xff);

	public void pixyReset() {

	// This method gathers data, then parses that data, and assigns the ints to global variables
	public PixyPacket readPacket(int Signature) throws Exception {
		int Checksum;
		int Sig;
		byte] rawData = new byte[32];

		try {
			rawData = pixy.read(32);
		catch (RuntimeException e) {


		if (rawData.length < 32) {
			SmartDashboard.putString("Error","byte array length is broken");
			SmartDashboard.putNumber("Length", rawData.length);
			return null;

		for (int i = 0; i <= 16; i++) {

			int syncWord = cvt(rawData*, rawData*); // Parse first 2  bytes
			if (syncWord == 0xaa55) { // Check if first 2 bytes equal a "sync word", which indicates the start of a packet of valid data

				syncWord = cvt(rawData*, rawData*); // Parse the next 2 bytes
				if (syncWord != 0xaa55) { // Shifts everything in the case that one syncword is sent
					i -= 2;

				// This next block parses the rest of the data
				Checksum = cvt(rawData*, rawData*);
				Sig = cvt(rawData*, rawData*);
				if (Sig <= 0 || Sig > packets.length) {

				packets[Sig - 1] = new PixyPacket();
				packets[Sig - 1].X = cvt(rawData*, rawData*);
				packets[Sig - 1].Y = cvt(rawData*, rawData*);
				packets[Sig - 1].Width = cvt(rawData*, rawData*);
				packets[Sig - 1].Height = cvt(rawData*, rawData*);

				// Checks whether the data is valid using the checksum *This if block should never be entered*
				if (Checksum != Sig + packets[Sig - 1].X + packets[Sig - 1].Y + packets[Sig - 1].Width
						+ packets[Sig - 1].Height) {
					packets[Sig - 1] = null;
					throw new Exception();
		// Assigns our packet to a temp packet, then deletes data so that we don't return old data
		PixyPacket pkt = packets[Signature - 1];
		packets[Signature - 1] = null;
		return pkt;

	static class PixyPacket {
		int X = 0;
		int Y = 0;
		int Width = 0;
		int Height = 0;

	public void setPIDSourceType(PIDSourceType pidSource) {
		m_pidSource = pidSource;

	public PIDSourceType getPIDSourceType() {
		return m_pidSource;
	public double pidGet() {
		try {
			return readPacket(1).X;
		} catch (Exception e) {
			return -1;


Thanks in advance.****************

Fixed image:


1 Like

We also noticed the left and right strips swapping order in the data stream as the Pixy reports objects in largest to smallest order. Our solution was to create a single composite rectangular object from the first two reported signatures. This has worked well for us. I’m sure our programming mentor will elaborate (I’m a hardware guy).

That would be awesome… that sounds like the source of our problem.

I think the biggest thing that would help you is to instead of looking for just one of the strips, take the coordinates of both of the strips from the pixy and get an average x.

To do this, you’ll need to process both objects instead of just one.
The way you’ll probably need to do this is to look for all objects that the pixy describes on a specific frame.
In Serial mode, the pixy sends two sync bytes when starting to describe what it saw in the next frame. What you’ll need to do is capture all of the objects it is describing in the specific frame by breaking on the sync bytes.

See the Pixy Serial protocal information here for additional details.

This is what I want to do… but I have no idea how to do this. What do you mean by “breaking on the sync bytes”?

I’ve read through that second bullet point list about 10 times… still confused :confused:

Thanks for the reply

We use the Pixy camera to locate the peg and place the gear. We configured the pixy to see both targets as individual items. To do that we connected to the pixy using their pixymon software and did the following:

Have the camera and ring light placed and the ring light on. (Note we found that one 100mm ring light is enough. Too much light only over saturates the area and makes detection less accurate.)
In Configuration select the Camera tab check all three boxes on.
In the Signature Tuning tab set the camera brightness down to about 25-26. This will make the entire picture dark except for the reflective tape. The color of the tape reflection will also darken which make it easier for the camera to identify the targets.
Go back to the Camera tab and uncheck all of the boxes and apply.
Close that window and goto the Action pull down. Select Set signature 1 and sample a section of the now darkened colored targets.
Go back to the configuration menu and use the signature 1 range slider to identify and separate the targets. You want them to give nice clean boxes with little to no noise. We had to increase the value to get them to separate nicely.
Now you should have two nice clean targets that appear when the camera can see the targets. We then brought the centerX, centerY, height and width information into the robo. Once you have the location of the center of the two targets you can identify the center of the peg.

We have not tried using the serial interface (we use the I2C pixy interface), but they appear to have very similar data being provided back over the connection. In looking at the code you posted, it looks very similar to some I2C code that we started with for our own interface. We took that code and reorganized and refactored it for our own purposes which included changing the way that data is saved in our class. As Ken mentioned in addition to storing all the targets reported, we create a “composite target” which is essentially the full bounding box that covers all the targets. In our case “all” is only 2 targets since that is all we look at.

Looking at this code I see the potential issue you are dealing with. That code is storing the data block received into an array indexed by the signature (e.g. packetsSig - 1].X = cvt(rawData*, rawData*); ). Since you are looking for a single color, ie. a single signature, then it will overwrite the same block of data with each subsequent target it sees with that same color. You could test this by printing the data as it assigns it to a block and see how many targets it is actually reporting. The change you would want to make is to keep a count the number of blocks being processed and index your packets array with the block number instead of the signature. Thereby saving the data for each target received.

Once that is done you can implement whatever additional processing you like to generate a composite type target, like finding the center of the combination of the 2 targets you are seeing at the peg. As Ken mentioned you need to be aware that targets are reported biggest to smallest [not right to left, or left to right, or top to bottom, etc.], so when doing some operations you may need to be careful of the order you are doing calculations (eg. packets[0].X is not necessarily < or > packets[1].X). We handled this by having saving the index of the leftTarget, rightTarget, topTarget, and bottomTarget which allowed us to simplify the calculations for our bounding box composite target.

I hope this helps, please follow up if you have more questions.**

So the packets array is an array of structures; where each structure holds the following fields: X, Y (the center coordinates), Width, Height (of the target in pixels). The data being received is already being split into blocks by the code you have it is just overwriting the packets with information because they have the same signature. Presuming that you have only one color signature that you are looking for it will overwrite all the targets on the same array index. Note that if you have 2 signatures you could add the signature as another field to the structure.

So instead of this code:

packets[Sig - 1] = new PixyPacket();
packets[Sig - 1].X = cvt(rawData*, rawData*);
packets[Sig - 1].Y = cvt(rawData*, rawData*);
packets[Sig - 1].Width = cvt(rawData*, rawData*);
packets[Sig - 1].Height = cvt(rawData*, rawData*);

You would do this:

packets[blockNum] = new PixyPacket(); 
packets[blockNum].X = cvt(rawData*, rawData*); 
packets[blockNum].Y = cvt(rawData*, rawData*); 
packets[blockNum].Width = cvt(rawData*, rawData*); 
packets[blockNum].Height = cvt(rawData*, rawData*); 

In addition, I would have the ReadPacket function return a block count instead of returning a “random” packet (since based on the way the current code overwrites the data I would expect that it is telling you the smallest target it reported for that signature.

So instead of this at the end:

//Assigns our packet to a temp packet, then deletes data so that we dont return old data
PixyPacket pkt = packets[Signature - 1];
packets[Signature - 1] = null;
return pkt;

You would change the return type to an integer and do this:

 //return the current block count 
return blockNum; 

You will then be able to retrieve the X,Y locations of each target using an interface function like:

public int getX(int index)
    return packets[index].X;     

Does that help??****************

I was a bit janky, but we had an arduino that took the largest two objects from the pixy and took the average of their center x values. If it was outside a threshold distance of the center, it would tell the roboRIO to turn left or right. It worked pretty well, we had 8/8 autonomous gear placement.

What did you use for communication between the Arduino and the roboRIO? Might keep this in mind if we can’t get regular serial communication working

You can use PixyMon to configure how many blocks it will return to you, so that you know you are not going to overrun your array. In addition, it would be smart to test that you are not going beyond the array size when incrementing the blockNum value. The good part of this is that you know that the blocks are sent with the biggest target first, so you would only be “ignoring” the small targets sent at the end.

Yes, you would want to set the blockNum to 0 each time you call your ReadPackets function. This would then always tell you how many “valid” blocks (or packets) you have in your array.
The index parameter in the sample calling method GetX allows you to pick out an individual block to get data from. It would be a good idea to always check that against the current value of blockNum to ensure you are returning currently valid data.

I skimmed this thread pretty quickly, and I think all I can add is some explanation of the initial behavior.

The PIXY has two types of reports it can provide. One is a list of adjacent-colored-pairs. This A/B adjacent pairs is marked as a special block including location and angle.

The second, more common report, is for the lists of specified-color blocks within the image. If you have two blocks of equal size, the sensor noise and edge aliasing can cause the blocks to trade places as “largest”. As mentioned, you probably want to look at more than one block and look for the pair of blocks that make sense together as the target.

Greg McKaskle


Has anyone published labview elements similar to the “getblocks” arduino function? We couldn’t seem to find one, so we connected to a arduino did our aspect ratio processing there and then passed the final result to the rio on a serial interface.

Disclaimer: I’m not a programmer and I don’t play one on TV! I’m asking on behalf of our programming team.

I know this isn’t the prettiest solution, but if you unfocused the pixy so the strips blur together you’ll get a single center point.

We also couldn’t get serial working, so I came up with another solution. In case our vision wasn’t already janky enough, we used GPIO. One pin for left, one for right, one for straight. If both left and center were high, it would go slightly left, if only left was high it would go more left. Same thing for right. Only center pin high was go straight. Again, janky but it worked, since we also couldn’t get serial working.

FYI national instruments posted vi’s to directly read pixy data to the robo rio. Seems like they could easily be ported to c or java.

When was this? We were struggling with reading strings off of I2C and SPI in LabVIEW and just found it easier to have an Arduino mediator.