The following report is an internal postmortem from FRC 7461 Sushi Squad detailing the Power Distribution Hub Failure which was experienced at the offseason PNW Block Party event.
Timeline of Events
Pre - Competition
Summary
July/August — Design of the Robot
Details
Key electrical members are not available during the robot design phase. The electrical lead gives preliminary electrical input in early august and raises concerns regarding the indexing motor passing the belly pan perimeter. Concerns are also raised about the placement of the circuit breaker.
9/10/22 — Initial Control System Wiring
Details
Electrical team wire control system. They experience issues with wiring power into the Power Distribution Panel due to lack of space and its proximity to the Back Left Swerve Module. Other issues experienced include loose mounting wires caused by a lack of a patterned belly pan. Indexing motor concerns are also brought up.
10/4/22 — Circuit breaker mounted
Details
Circuit breaker is mounted to the Delrin compression panels.
10/8 prematch — PNW Girls Generation
Details
The electrical system experienced zero failures. The electrical pit crew conducts electrical systems check on the morning of 10/9 and does not see any issues.
10/20/22 — Climb Wiring
Details
The electrical team conduct an electrical systems check during climb wiring do not see any issues with the battery ground wire.
Week of October 24th 10/23 & 10/21 — Prog Testing and Drive Practice
Details
Programming and Drive Teams test and practice with the robot. Some wiring comes loose but no electrical systems check is conducted
.
In - Competition
Summary
Load-in
Details
- Bellypan is removed to replace the tread on swerve modules. The back left module is worked on close to the PDH
Practice Match 1
Details
- No electrical issues are experienced
Quals Match 1
Details
- No Electrical Issues
Quals Match 8
Details
-
breaker pops -1:10
-
Pit Crew Concludes that the rouge ball popped the breaker
-
Pit Crew Places Duct Tape ring around breaker for protection
Quals Match 12
Details
-
breaker pops -41
-
FTA Recommends Circuit Breaker Swap
-
Not enough time between matches
-
Other teams recommend re-characterizing the current limit
Quals Match 16
Details
-
Breaker pops -35
-
Pit crew replaces circuit breaker
-
No electrical systems check due to time constraint
Quals Match 19
Details
-
Intake malfunctioned during auto
-
Heavy Defense by 4911
-
4911 slotted under 7461
-
Robot disables at 1:30
-
The robot comes online at 2:31
-
The robot combusts at 2:32
-
Post Match Electrical Inspection
-
Ground Wire lose from PDH
-
Power Wire metal shavings causing a short circuit between power and ground
-
Melts motor power wiring around the short
-
Pit Crew Electrical Team splice power cables and replace them as necessary
-
PDH/ports are not fully inspected
-
The ground wire is placed back into the port after the splice
-
Electrical Systems check and subsequent intake testing are successful
Quals Match 23
Details
-
Robot Fails on Startup
-
The robot restarts but shuts down
-
Robot Restarts near the end of the match
-
Climb works
-
The problem is suspected to be a current limit issue
-
No Electrical Systems Check
Quals Match 27
Details
-
Auto is successful
-
The robot fails multiple times during the match
-
FTA forces inspection of PDH
-
The breaker in the on position does not send power through the system
-
PDH has ground wire melted into the system
-
Wago connectors break when removing Ground Wire
-
Further, inspection shows previously molten material inside the wago connectors
Analysis
Summary
To understand the root cause of PDH failure, it is important to understand the electrical system architecture. Power and Ground wires from the battery route into the PDH. The power wire is spliced by the Circuit Breaker while the ground wire is directly routed into the PDH. During battery replacement, both wires are tensioned. The power wire is mounted on the circuit breaker rendering it immune to said tension. However, the ground wire places direct tension onto the PDH port. Previous generation distribution panels (CTRE PDP) use a nut and washer assembly to mount the ground wire which is far more resilient to external forces. The PDH, in contrast, used WAGO connectors which function based on metallic cage clamps exerting force onto the wires. This system does not have a bolt preventing unsanctioned removal. Over multiple battery replacements, the ground wire can be slipped out of the port.
This was the most likely cause. The lack of thorough electrical inspections did not help in any regard. It is likely the case that the robot entered Block Party with loose wires.
Additionally, after the practice match, or quals 1, the battery replacement may have been aggressive causing the wires to further shift out of the WAGO connectors.
By match 8, the robot was experiencing issues with the breaker tripping. The likely cause was a minor short circuit between the power and ground wires. It was small enough to be undetectable but large enough to cause the breaker to trip.
Breakers are designed so that the threshold for tripping lowers each time it trips. For this reason, breakers tripped in subsequent matches.
The pit crew replaced the breaker before match 19 but did not inspect the PDH directly after. The power wire may have slipped out further in the instance. In match 19, the robot was heavily defended by 4911. During this, 4911 slipped under the robot and caused the ground wire to shift more causing a large short circuit. This melted through other power wires causing a fire.
After this match, Pit Crew replaced the wiring but failed to inspect the PDH due to time constraints and the fact that the PDH was buried under wires. The Pit Crew failed to notice molten residue left inside the port and instead mounted the wires back into the WAGO port. This meant that the ground wire did not have a proper connection to the PDH and was not mounted. over the next two matches, the robot experience power loss as the wire shifted from contact.
In match 27, the wire likely caused a short circuit that melted through the plastic casing on the WAGO connections.
Oversights
Design
-
While some electrical considerations were taken (placement of electrical components, and batteries), the wiring was not thoroughly planned out
-
Components on the electrical board were not placed until the design had been finalized.
-
The electrical subteam was not kept in the loop during the design process (members were absent).
-
Bellypan and electrical board were made of flex poly making damage to the electrical system during heavy defense easy.
-
Bellypan was smoked poly so it had to be removed for every inspection leading to hesitation/time delay for full systems check.
Mechanical/Pit Crew
-
During battery mounting, stress is placed on the connections to the PDH.
-
Electrical fixes were not necessarily prioritized in pits.
Electrical
-
Wiring was not planned well (in conjunction with design).
-
Not all electrical cables were tensioned.
-
Power and Ground wiring were not expected to slip — no effort was made to tension them.
-
Lack of cohesive systems check.
-
Lack of frequent systems checks.
-
PDH breaker and CAN layout not recorded or updated.
-
Wire- protection was not used as extensively as needed.
-
Hot glue for mounting was not used extensively.
Organization
-
Electrical was given minimal time during robot wiring.
-
The electrical subteam has only 3 official members.
Drive Team/Programming
-
No cohesive drive or testing logs led to a lack of contextual data.
-
Battery mounting
Business
- Electrical was not necessarily prioritized in the budget leading to a lack of spares.
Recommended Optimizations
Design
-
Components need to be placed on the CAD.
-
The wiring needs to be taken into consideration.
-
The electrical board needs to be machined
-
The electrical system needs to be better protected (Rigid Poly or integrated electrical)
-
if a belly pan is used, it needs to be transparent.
-
Prebuilt electrical board and drivetrain
In Depth Design Optimizations
Battery:
- Ensure easy wiring from the battery to PDH while also routing to the breaker. Note that the battery cable is relatively rigid, which means that you should assume a linear path mount Anderson for PDH on a rigid point to ensure less slip.
- Ensure the battery is mounted in a fixed location that is easy to access machine holes into box tubing if box tubing is in the way. Super duper important that the path is somewhat linear
Motors/Electrical:
-
If using neos, ensure that sparks are mounted in a rigid position and are in cad, ensure that there is a mounting point
-
Check for motor orientation to ensure that wires are able actually to move around and be oriented
-
Have holes/a means to zip tie motor cables around points
-
Have holes for routing wires. Ensure that wires don’t have to cross impossible locations
-
CAD any energy cables that are necessary if using a moving mechanism w/ motors that are moving (elevators, turrets, etc.)
-
Use a metal, pocketed belly pan, and add electrical mounting holes if possible. Review with electrical to ensure everything works.
Communication:
-
After DT and belly pan design, check with electrical to ensure that wiring can be done accordingly
-
Check w/ electrical after each finished assembly that’s mounted to DT
-
If unsure about the electrical’s ability to work around certain constraints, consult and redesign to make it better
-
Have a final review once cad is “complete” and cad any fancy 3dp things for QOL
Electrical
-
Plan electrical in advance in conjunction with the design
-
Tension all cables
-
Make a cohesive electrical inspection checklist
-
Inspect electrical frequently and between every match
-
Protect all cables
-
Hot glue all connections
-
Record circuit breaker, ports, and CAN layout
Pit Crew/Drive Team/Programming
-
Better battery mounting procedure
-
Prioritize Electrical in pits
Conclusion
This report concludes that the direct cause of PDH failure was due to a combination of gradual degradation of ground wire connection integrity, along with heavy shock and stress experienced during PNW Block Party 2022 Match 19. Further failure was caused by the lack of time, organization, and extensive inspection procedures in the pit leading to greater damage in the PDH.
It also concludes that the indirect cause of PDH failure is poor electrical considerations in design, improper battery mounting procedures, and lack of extensive inspection procedures.
This report details recommended optimizations to multiple processes to prevent both the direct and indirect causes of PDH failure.