Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Scouting (http://www.chiefdelphi.com/forums/forumdisplay.php?f=36)
-   -   Best way to estimate the best defense bot (http://www.chiefdelphi.com/forums/showthread.php?t=115634)

dellagd 31-03-2013 20:50

Best way to estimate the best defense bot
 
Im sprucing up our scouting system for our next event and would really like do work defense into it.

As of now, we have 6 scouts (+ a head scout but he/she does different things), and they all watch every match.

I have heard of DPR, and have a crude understanding that its basically the same calc as OPR but using the opponents score, but Ive heard this is inaccurate.

I could give each scout a slider of 1 - 5 in "How well does robot play defense", but this is purely objective.

How does your team do it? Is DPR actually useful?

Hallry 31-03-2013 20:55

Re: Best way to estimate the best defense bot
 
We don't use DPR (or OPR, for that matter), but our experienced scouts do rank defensive teams. During our scouting meeting the night before eliminations, we then use those rankings and look back at our match footage (with every match recorded), to see how those teams perform and how consistent they are.

Grim Tuesday 31-03-2013 20:57

Re: Best way to estimate the best defense bot
 
We have considered implementing the following into our scouting database and haven't but might get around to it for Championships:

For reference, the main goal of our scouting is to record exactly how many points a robot scores in a match. The advantage this has over OPR is that robots can not be 'carried' to a higher position than they deserve, it is literally exactly how many points they individually scored that match.

The concept for a true DPR is to create a marker in this system that notates if a robot plays defense and upon whom they played it. We would then compare the amount a robot 'should score' aka their score in matches without defense to the score in their match where defense was played. A number would then be assigned to the defending robot for how many points or percentage of points a defender shut down.

MrJohnston 31-03-2013 21:59

Re: Best way to estimate the best defense bot
 
We look at several things:
* What sort of drive train does the bot have?
* What sort of traction are on the wheels?
* How fast is the robot?
* How well does it turn?
* Has the robot done well in pushing matches?
* How smart is the drive team? (i.e. do they commit fouls?)

They key is to scout the pits just as well as the games. Often the best defensive bots play no defense during qualifying matches.

themccannman 31-03-2013 22:07

Re: Best way to estimate the best defense bot
 
The idea behind DPR makes sense, however when used in practice it's wildly inaccurate and I would recommend not using it. Our team's defensive ranking system so far has proven to be extremely successful so far. We have 4 purely subjective categories pertaining to a robots ability to play defense and their actual defensive performance, and 2 defense scouts (one per alliance) to rate each individual robot. At the end of qualifications we compile the data into a spreadsheet with predetermined weighting for each category and it spits out a couple of numbers that tell us exactly how good every robot is at doing the things we are interested in. The most important part of our system however is that you need experienced people who know what to look for in a robot to be able to adequately judge their performance otherwise your numbers are worthless.

Edit: the person above me basically covered most of the categories we use to judge robots. We use:
-Drive train speed
-Driver ability
-Height (< 30")
-Blocking performance

Only the last category is actually affected by robots playing defense, the rest are affected by their apparent ability to play defense if we need it.

2789_B_Garcia 31-03-2013 22:17

Re: Best way to estimate the best defense bot
 
themccannman & MrJohnston, I'm the lead mentor and drive team coach for a defense-oriented team. If you have time, I'd be interested in your opinions on some match footage to tell me what you think about our team! I agree that OPR and DPR aren't helpful or accurate, and I've also used a more qualitative system for making defense strategy decisions.

Out of curiosity, why do you think the best defensive robots don't play defense during quals?

fox46 31-03-2013 23:45

Re: Best way to estimate the best defense bot
 
Quote:

Out of curiosity, why do you think the best defensive robots don't play defense during quals?
I'm sure we all can agree that the most important quality to have in a defensive machine is a strong drivetrain. However, in recent years FIRST has been steadily reducing the attractiveness of defensive strategies. In this year's game, a good defense doesn't score you any points. For this reason, most teams who build strong drivetrains do it not for defensive capabilities, but rather because if you can't get to where you need to be, you can't execute your gameplay.

Team 2013's robot uses a 6 CIM motor drive in a 6-wheel skid-steer configuration, rolling on custom aluminum rims wrapped with 2.5" wide roughtop conveyor belting. The machine weighs in at 117.3 lbs.

By all standards, a very defense-capable machine. However, the real reason for having a strong drive this year is that if we can't get to and from the feeder station and pyramid, then it doesn't matter how good the shooter and climber are.

During gameplay, the only times we have played defensively are if one or both of the other systems are down and we have no other way to contribute to the alliance, or if we deem defensive play necessary, such as if the other alliance has us out-gunned.

Hence a very defense-capable team which very rarely uses it.

Chris is me 01-04-2013 00:02

Re: Best way to estimate the best defense bot
 
You can't objectively do it. Don't even try.

The best you can do is a Defense option on your scouting sheet. "None" / "Ineffective" / "Somewhat Effective" / "Strong" / "Complete Shutdown" should be your scale (1-5 is too vague), and that isn't so much to quantify it as much as it is so you can know on Friday night which teams tried to play defense and *roughly* how good they are at it. Pit scout a little and determine which robots are 4 CIMs (minimum) and have traction drives. Also, be sure to scout Saturday to see who shows up to play excellent D.

All that said, more likely than not you'll find 1-3 excellent defenders at a regional, maybe a dozen (if you're lucky) average defenders, and then a bunch of teams limping around, trying to score or something and really just not doing much. You'll probably have to use other offensive features (10 point hang, any autonomous functionality) as your main decision maker. You are going to need smart people sitting there on Saturday morning watching matches and looking for diamonds in the rough. That's how you find them.

EricH 01-04-2013 00:05

Re: Best way to estimate the best defense bot
 
You guys are almost all overlooking one thing (I've seen one mention so far). And this one thing could lose you the event.

Penalties.

Does the potential defense robot in question (whichever robot it happens to be) rack up their opponents' points from fouls or their alliances' points from fouls? If you have a potential defender that is constantly getting penalties, particularly "stupid" penalties, then they should probably not be very high on your list. Back when I was scouting, penalty points were given a weighting of -1 in our metrics to deduct "score" from that robot.

Now, if the robot takes a smart penalty, then you also want to note that. It means that they know that a 20-point foul is better than letting their opponent potentially get 50 points, for example.

MrJohnston 01-04-2013 00:12

Re: Best way to estimate the best defense bot
 
I agree with fox46's answer...

The strong drivetrain and grippy wheels mean that the robot can push most anybody around. Certainly, this is a huge benefit offensively, allowing such tasks as forcing itself to a feeder station or to an ideal place to maek a shot. However, used defensively, it can make it very difficutl for a full-court shooter to get to its feederstation or a little shuttlebot to blitz back-and-forth between the pyramid and feeder.... Or, a really tall blocker can be attached mid-compeititon to it to deal with a full-court shooter as it would be nearly impossible to move.

Often teams with the where-with-all to build such a strong drivetrain, realize that many teams can't score. So, they need to have some scoring ability (a shooter, for instance). If the team is good enough to do this, their shooter will work well. So.. even though they are thinking "defense" they might must be the best offensive robot on the field in a qualification match.

fox46 01-04-2013 00:21

Re: Best way to estimate the best defense bot
 
Quote:

You guys are almost all overlooking one thing (I've seen one mention so far). And this one thing could lose you the event.

Penalties.
All robots can incurr penalties- defensive or offensive. This should be considered for all teams you are scouting. It's far more likely to clip a climbing robot on the way to a feeder station than playing D with half a brain. Running back and forth accross the field you aren't focussing as much on the activities of other machines than if you were covering them playing D.

dellagd 01-04-2013 00:52

Re: Best way to estimate the best defense bot
 
Quote:

You guys are almost all overlooking one thing (I've seen one mention so far). And this one thing could lose you the event.

Penalties.
Thats a main point of our system right now and we are sure to keep track of it.

Quote:

"None" / "Ineffective" / "Somewhat Effective" / "Strong" / "Complete Shutdown"
Quote:

All that said, more likely than not you'll find 1-3 excellent defenders at a regional, maybe a dozen (if you're lucky) average defenders, and then a bunch of teams limping around, trying to score or something and really just not doing much. You'll probably have to use other offensive features (10 point hang, any autonomous functionality) as your main decision maker.
I've tried the 1-5 scale of None, Ineffective, ect. but that got shot down by the scouts when we tried that. I'll try for it again. Hopefully working in some pit scouting into that will work out nicely.

Good to know DPR is no good.

Any other Ideas?

fox46 01-04-2013 01:09

Re: Best way to estimate the best defense bot
 
Case in point:

Quote:

The strong drivetrain and grippy wheels mean that the robot can push most anybody around. Certainly, this is a huge benefit offensively, allowing such tasks as forcing itself to a feeder station or to an ideal place to maek a shot. However, used defensively, it can make it very difficutl for a full-court shooter to get to its feederstation or a little shuttlebot to blitz back-and-forth between the pyramid and feeder....
http://watchfirstnow.com/archive3.php?id=62936455

themccannman 01-04-2013 02:28

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by dellagd (Post 1255356)
I've tried the 1-5 scale of None, Ineffective, ect. but that got shot down by the scouts when we tried that. I'll try for it again. Hopefully working in some pit scouting into that will work out nicely.

Good to know DPR is no good.

Any other Ideas?

We do ours with a similar scale, 0 is a average robot, (-) points are if they perform poorly or are a detriment to their team and (+) points are if they perform above average. I would highly recommend a system like that or the 1-5 system you tried before. Tell the scouts to come up with a better system if they don't like it, otherwise just use that.

Quote:

Originally Posted by EricH (Post 1255333)
You guys are almost all overlooking one thing (I've seen one mention so far). And this one thing could lose you the event.

Our "driver ability" category is supposed to cover this, if they are consistently causing penalties we will score them very poorly in that aspect. We've found that there tends to be a fairly close correlation between apparent driver skill level and average penalties incurred.

Quote:

Originally Posted by 2789_B_Garcia (Post 1255284)
themccannman & MrJohnston, I'm the lead mentor and drive team coach for a defense-oriented team. If you have time, I'd be interested in your opinions on some match footage to tell me what you think about our team! I agree that OPR and DPR aren't helpful or accurate, and I've also used a more qualitative system for making defense strategy decisions.

Out of curiosity, why do you think the best defensive robots don't play defense during quals?

I would be happy too look at some of your matches. The reason why most good defensive bots don't play defense in quals is because they are usually some of the better offensive bots too. This years game tends allow good offensive teams to play effective defense very easily, all they need is a fast drive train and a good driver, which good offensive teams almost always have. I feel like our team demonstrated how easily the switch can be made from offense to defense at CVR when we spent all of eliminations playing counter-defense despite being one of the highest scoring teams there. We had a couple matches at other times (including the Sacramento regional) where we had to play defense and our driver was able to almost completely shut down the other teams offense with nothing but a drive train (they scored 0 tele-op points in several matches).

dellagd 02-04-2013 06:19

Re: Best way to estimate the best defense bot
 
Quote:

The concept for a true DPR is to create a marker in this system that notates if a robot plays defense and upon whom they played it. We would then compare the amount a robot 'should score' aka their score in matches without defense to the score in their match where defense was played. A number would then be assigned to the defending robot for how many points or percentage of points a defender shut down.
Im implementing this, we'll see how it works.

Anything else?

goldenglove002 02-04-2013 09:38

Re: Best way to estimate the best defense bot
 
It's probably not the best method, but I had our scouts watching for collisions between robots. Give a point for a collision won, take a point away for a collision lost. Find the ratio between the two.

While this has it's imperfections, it gave us a pretty good idea of which robots had the ability to play valuable enough defense and which would just get pushed out of the way.

It's not easy to quantify defense and this year it's really just best to figure out who has a defensive strategy, then watch to see if they are actually any good at it. IMO, picking a robot to play defense in elims over one that can score is a bad strategy unless it is clearly a strong defender (strong enough where you don't need numbers to realize it can play defense).

arbym 03-04-2013 22:10

Re: Best way to estimate the best defense bot
 
We view defense from an offensive slow down of the opposing alliance. We log all defense events that happen (blocking movement, nudging while shooting, etc.) and correlate back to the offensive team. We then display those statistics as shooting percentage and shooting percentage while defended. We also look at the cycle time from pickup to shoot, to pickup and compare defended vs. undefended. Because we logged the event who was defending we can show who caused the offense to decrease. All ties back to our real-time scouting effort.

themccannman 04-04-2013 01:00

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by arbym (Post 1257021)
We view defense from an offensive slow down of the opposing alliance. We log all defense events that happen (blocking movement, nudging while shooting, etc.) and correlate back to the offensive team. We then display those statistics as shooting percentage and shooting percentage while defended. We also look at the cycle time from pickup to shoot, to pickup and compare defended vs. undefended. Because we logged the event who was defending we can show who caused the offense to decrease. All ties back to our real-time scouting effort.

If you have the resources to do this, it is a great method for tracking defensive capabilities of robots. The only problem is that it only tells you how well robots that play defense do, not how well they could play defense if you asked them to. Some of the best defensive robots are the ones who never play defense until elims.

Tom Line 04-04-2013 01:09

Re: Best way to estimate the best defense bot
 
We've found that generally speaking, scouting for a 'good' defensive team is a factor of a couple things.

Obviously, the first thing you require is a good drive train.

Next in our experience is how well that team has worked with us. We get a chance to be with most teams in the qualification matches, and we know most other teams from previous experience. How receptive their drive-coach and drive team is, and how well they can carry out a plan, is immensely important to us.

This year, modifying any robot to block is fairly easy while keeping a low cg. That plus a smart drive team and coach, plus a nice 10 point hang will get you on just about every defensive pick-list out there.

bduddy 04-04-2013 02:29

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by fox46 (Post 1255321)
However, in recent years FIRST has been steadily reducing the attractiveness of defensive strategies. In this year's game, a good defense doesn't score you any points.

I'm not really sure what you mean by this - in most FIRST games, and indeed most games, period, a good defense doesn't score you any or many points. Good defense keeps the score low, hopefully to an extent where your remaining robots can win the game. And I agree that in the last couple years FIRST has reduced the attractiveness of defense, through not only protected zones but also the ranking system, but lately, and especially this year, this is not the case; having lower scores only hurts you in tiebreakers, and because the first tiebreaker is autonomous (unaffected by defense), it's unlikely to matter at all. Yes, there are still plenty of protected areas, but a good defensive robot has more potential than ever before to rank highly.

Justin Shelley 04-04-2013 07:10

Re: Best way to estimate the best defense bot
 
Our robot played defense these year at STL regional because our climber broke and we couldn't feed :( we never scored a single point in teleop and were ranked 28th, but even still we were the second pick in the whole regional!!! We thought that it might even be the first time a robot that has never scored a teleop point was picked so high. Anyways here's some videos of our robot defending with our 6wd pneumatic, 2 cim, toughbox drivetrain

1) http://www.thebluealliance.com/match/2013mosl_qm75
the full court shooter usally put up around 120pts, they were second seeded team. Also their shooter would shatter lexan at 80% power so we had to have something beffier to defend them with. and yes i did commit a foul by running across field after them :( heat of the moment thing and the only foul we got the whole regional

2) http://www.thebluealliance.com/match/2013mosl_qf3m2
we are robot that starts on left for red alliance

3) http://www.thebluealliance.com/match/2013mosl_qm81
we are robot that starts on left of blue alliance

Rate me please?

2789_DrT 04-04-2013 13:24

Re: Best way to estimate the best defense bot
 
I'm one of the scouting mentors for FRC 2789 and I've been recently brainstorming new ways to analyze our data to pull out the effects of defense so our team can illustrate what we do best. We currently have a system that's good at gathering quantitative data to rate offensive bots and qualitative data to help our drive team play effective defense. We don't have a system in place that can use quantitative data to rate defensive teams because we're looking for our counterparts this year and we primarily playing defense.

It's been a big headache to find methods that work because so many variables impact individual and alliance scoring and because defender bots can play different roles that impact these stats. Depending on the match dynamics, a defender may be expected to play zone defense, man-to-man defense, to set picks for other shooters, or various combos of the above.

I'm so glad that there are teams going to Champs that have systems that are sophisticated enough to take this problem seriously so I can stop worrying about this and trust that you will find us.

However, because it's fun to geek out, here's what I'd do if I really had to find out who were the best defenders on the field with a small scout team. [ Full disclosure: a lot of what I'm going to say is informed by changes our team has implemented that have improved our defensive play from good to great - so if you do these things, you may just find us ;) ]


YOLO WAY TO FIND GREAT DEFENDERS:
  • send scouts into the pits and look for robots with low centers of gravity, chassis with at least 4 cims, and several optional blocking devices that can be added to or removed from the robot for different situations
  • send scouts into the stands to take a peek at other scout systems. Look for teams that gather qualitative data that can give their drive teams the intel they need to play shut down defense. For example - my team's scouts gather things such as robots' favored movement patterns, favorites feeds, favorite shooting spots, the accuracy in those spots, etc.
Tell scouts match scouts to look for the following:
  • drive teams that know what angles / pathways are good for efficient zone defense
  • drive teams that have the skill set to play effective man-to-man defense
  • drive teams that know the rules - these teams play aggressively within the rules and have drivers that head quickly to the question box whenever it's necessary to seek clarification on controversial calls
  • drive teams that know how to set picks and know when to set picks and when to play defense

Citrus Dad 29-04-2013 17:58

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by 2789_B_Garcia (Post 1255284)
themccannman & MrJohnston, I'm the lead mentor and drive team coach for a defense-oriented team. If you have time, I'd be interested in your opinions on some match footage to tell me what you think about our team! I agree that OPR and DPR aren't helpful or accurate, and I've also used a more qualitative system for making defense strategy decisions.

Out of curiosity, why do you think the best defensive robots don't play defense during quals?

Our robot, Citrus Circuits, is a case in point. This year our shooter failed mid match several times. Each time we turned to play defense and completely shut down the opposing alliance. In two matches, we held off 2 other robots and didn't allow them to score more than 3 goals the rest of the match. Yet, we are usually the top scorer in our qualifying round alliances. (We get paired with high scorers in the elimination rounds.)

And it's not just defense, but also counter-defense that we look for. That was key when we selected 295 in Madera. They protected our FCS 840 who emptied the magazine several times. 295 wasn't playing much defense in qualifications, but showed excellent driving skills.

Send the video--it would be fun to look at!

Louisiana Jones 29-04-2013 19:19

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by dellagd (Post 1255240)
Im sprucing up our scouting system for our next event and would really like do work defense into it.

As of now, we have 6 scouts (+ a head scout but he/she does different things), and they all watch every match.

I have heard of DPR, and have a crude understanding that its basically the same calc as OPR but using the opponents score, but Ive heard this is inaccurate.

I could give each scout a slider of 1 - 5 in "How well does robot play defense", but this is purely objective.

How does your team do it? Is DPR actually useful?

I'm a newbie at scouting, but after working this year with a student to create a scouting system for our team to use, we have determined that OPR is the single best indicator of the offensive/defensive capability of a robot. OPR is misnamed. Everything we have tested indicates that it accounts for both the offensive and defensive contributions of a robot to a high degree of accuracy. While there are some weaknesses with OPR the advantages are that it is easy to calculate with available data that doesn't need to be scouted, and when combined with data from your scouts can give you a perfect idea of the offensive and defensive contributions of a robot. DPR is also misnamed and therefore seems inaccurate when used to describe how good a robot is defensively. DPR isn't simply a defense ranking, and if you try to use it that way will always seem inaccurate.

EricH 29-04-2013 19:26

Re: Best way to estimate the best defense bot
 
Quote:

Originally Posted by Louisiana Jones (Post 1269517)
I'm a newbie at scouting, but after working this year with a student to create a scouting system for our team to use, we have determined that OPR is the single best indicator of the offensive/defensive capability of a robot.

For this year, it might be (I remain unconvinced of OPR as an indicator of the best robot, particularly when planning elimination strategy).

For next year, it might not be. There are some years where OPR works well, and other years where it's as bad as the W-L-T seeding system with too few matches (like 8!).

SkittlesCharge 30-04-2013 21:13

Re: Best way to estimate the best defense bot
 
To keep track of defensive bots this year, we used both pit scouting and match scouting. We rated the drive train's speed, pushing power (torque), and manueverability on a 1 to 5 scale. Then we also kept track of the number of times the robot pushed another robot, blocked another robot, pushed a robot while it was shooting, and blocked shots to see what kind of defense it played.

While we didn't get to pick partners at any regionals this year, but it seemed to work fairly well. Except for two easily fixable major problems (the 1 to 5 scale's default was 1, making many average robots look bad and in pushing matches back and forth it was hard to decide the number of pushes to award the robot), the system could tell us how good a robot was at blocking full court shooters, or if it tried to push robots frequently, but was just too slow to catch them, etc.

I wouldn't recommend using a system exactly like ours was this year, but hopefully an idea or two from it would be helpful to you.


All times are GMT -5. The time now is 16:13.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi