Monday, December 05, 2011

Misreading the Journal Sentinel on traffic stops

Both the authors of the Milwaukee Journal Sentinel's article on the percentage of African American and Hispanics stopped by the police and blogging lawyer Ed Garvey make a common error of interpretation.

The Journal Sentinel authors cite the fact that blacks and Hispanics who were searched were no more likely to have contraband than whites who were searched (22% in both cases) as argument against the notion that more frequent stops (and searches) of blacks and Hispanics can be justified on non-discriminatory grounds. They write:

After the stop, Milwaukee police searched the vehicles of black drivers twice as often as whites, or one search for every 12 stops. But police found contraband items in searches involving black drivers at almost the same rate as whites - about 22% of the time.

Garvey, as is his wont, announces that the case is closed.

The first question is whether the black and Hispanic drivers had more contraband. In other words, is there some justification in stopping more Hispanics and blacks. They did not! So, no, there is not.
Both Garvey and the paper have it exactly wrong. The fact that blacks and Hispanics who were stopped and searched  were just as likely as whites who were stopped and searched to have contraband suggests (even if it does not definitively prove) that the stops and searches were not racially motivated.

 Here's why. If the Milwaukee police department were engaged in racial discrimination, i.e., stopping persons for "driving while black and brown" then would expect those searches to yield fewer arrests and seizures than the stops and searches of white drivers who have presumably not been selected for discriminatory reasons. Having stopped blacks and Hispanics for no reason but that they were blacks would be expected to yield fewer seizures of contraband among the population subjected to discrimination because there was no good reason to stop them.

The fact that this was not so - that the likelihood of finding contraband in the cars of black and Hispanic detainees was equal to that of finding it the cars of white detainees - suggests that the criteria used for detention was not discriminatory.

Remember these statistics do not show that blacks and Hispanics in general are no more likely than whites to have contraband in their vehicle. That data might support an inference of discrimination. The comparison here involves only those who were stopped and searched.

This is a fairly common misinterpretation of statistics comparing outcomes among different racial groups. To put it in another context, assume that the UW was discriminating against black applicants. One would expect those blacks who are admitted to have higher credentials than the average white admittee. If the credentials for both groups are the same, an inference of even handed treatment is supported.

I expect the sloppiness from Garvey. The paper should have dug a bit deeper.


Phil Scarr said...

The issue is that of the traffic stops done, the police were twice as likely to search African Americans.

"After the stop, Milwaukee police searched the vehicles of black drivers twice as often as whites, or one search for every 12 stops. But police found contraband items in searches involving black drivers at almost the same rate as whites - about 22% of the time."

Therefore, with double the opportunity to find contraband, they found the same volume of contraband. That means that whites were actually twice as likely to have contraband (based on the sample) than African Americans.

Rick Esenberg said...

No it does not. Whites were as likely to have contraband as blacks. That twice as many blacks were stopped as whites does not change that. Your observation would be correct only if we were measuring the total number (or as you say "volume") of contraband items (as opposed to the percentage of detainees who had contraband.) That's not the case. The percentage of persons with contraband is not a measure of the volume of contraband.

Phil Scarr said...

I don't think you're accounting for the variability in the opportunity for defect (more stops versus fewer stops). Each incident of contraband is a defect. Using basic six-sigma methodology, the DPMO for whites would be twice what it is for African Americans (same volume of defects for 1/2 the number of stops).

Rick Esenberg said...

Phil, I'm sorry. You don't understand basic mathemtical and statistical concepts at work.

The 22% is not a measure of "volume." If we measured that, there would be a much higher "volume" of contraband (treating seizure and non-seizure in a binary way) seized from African Americans and Hispanics because it'd be 22% of a higher number of stops.

I have litgated stuff like this for over 25 years and have discussed this concept with many economists and statisticians. It is simple math and simple logic.

The question here is whether the disparate stops of African Americans and Hispanics reflects discrimination, i.e., stops for no good reason or, as some would have it, "driving while black." If that is the case then one would expect stops of blacks and Hispanics to be less likely to yield contraband. But that didn't happen.

This suggests that whatever criteria were being applied to stop motorists were equally effective for blacks and whites - at least as measured by the liklehood of finding contraband.

Using six sigma jargon doesn't change this. DPMO is a measure of defects per opportunity - in this case a stop and search. In this case - if you view finding contraband as a "defect" the DPMO (putting aside the fact that we don't have a "million" opportunities) is the same for blacks and whites - 22%.

This suggests that the criteria for generating "opportunities" - e.g, stops and searches - were not racially biased.

jimspice said...

"...expected to yield fewer seizures of contraband among the population subjected to discrimination because there was no good reason to stop them."

This assumes there is a real difference in likelihood of holding contraband between those stopped for cause and no cause, and I'm not convinced you can just assume that. In other words, are speeders more or equally likely to carry illegal firearms than a random sample. We'd need a much more detailed look at the nature of the original offenses to reach any reasonable conclusion.

jimspice said...

And it doesn't address the differing rates of stops and searches, the real question here.

Tom said...

Phil, my first reaction was the same as yours until I re-read the article. If the article said that of all the cars STOPPED, 22% of them had contraband across racial lines, then yes, you would be correct. But it said that of the cars SEARCHED, 22% of them had contraband across racial lines. So the percentage of stopped cars that were searched is irrelevant to that statistic.

Phil said...

[NOTE: This is the same person as Phil Scarr above, I'm posting with my google account so I can get followups by e-mail]

Ahh, I see the confusion. The question raised by the statistics is not one of whether or not people were stopped driving "while minority," but how they were treated once they were stopped. Minorities were much more likely to be searched based on the number of opportunities to be searched.

Let's say your normalize the number of stops (as in the case of the Wisconsin State Patrol) so that each ethnic group is proportionally represented in the number of stops to their ratio in the general population. Once the stop is made, the question becomes how is that stop handled. For simplicity, let's try it this way. Assume two populations, evenly divided (50/50) as part of the population.

Group A: 1000 stops / 10 events
Group B: 500 stops / 5 events

You would say that groups A and B have the same "event rate," correct? Statistically speaking (putting aside the question of statistical significance for another day), the likelihood of having an "event" is the same for both populations.

Now, let's try it this way. Same scenario, 50/50 population but with these events.

Group A: 1000 stops / 10 events
Group B: 1000 stops / 5 events

We can say that Group A is twice as likely as group B to have an event. Right? Ok. Next scenario.

Group A: 2000 stops / 10 events
Group B: 1000 stops / 10 events

This same phenomenon has been recorded (with statistical significance) in Major League Baseball. Umpires treat minorities differently.

[H]ome-plate umpires call disproportionately more strikes for pitchers in their same ethnic group. Because most home-plate umpires are white, this has been a big form of racial privilege for white pitchers, who researchers show are, on average, getting disproportionately more of the benefit of the doubt on close calls.

[M]inority pitchers reacted to umpire bias by playing it safe with the pitches they threw in a way that actually harmed their performance and statistics.” Basically, these hurlers adjusted to the white umpires’ artificially narrower strike zone by throwing pitches down the heart of the plate, where they were easier for batters to hit.

[T]he data suggest that racial bias is probably operating at a subconscious level, where the umpire doesn’t even recognize it.

I would argue that the police data are operating through the same subconscious level (for the most part) and do not represent overt racism on the part of individual officers. But to deny that it exists is to perpetuate the problem.

Do I have a solution to this? No, but that doesn't mean we shouldn't be looking for one.

Phil said...

So it's been 24 hours, so are we settled? Do we now agree that, like the case of the MLB umpires, the MPD participated in a system of stops and searches that were influenced, consciously or subconsciously, by racial bias?

Rick Esenberg said...

No, we don't. You offer no evidence of that and your citation to the MLB study cuts against your position. I obviously can't vouch for the study but, taken at face value, it found what the Journal Sentinel did not - a higher error rate with respect to the allegedly disfavored group.

Contrary to the comments of Mr. Spice, the absence of a differing rates of seizure does address, at least, the differential search rates and suggests that it was not motivated by bias - subconscious or otherwise.

Whatever criteria is being used for searches seems equally accurate for blacks, whites and Hispanics.

No, I suppose the stops could be a different story, but it's hard to argue that there is a bias in the rate of stops that somehow disappears when it comes to searches.

I certainly can imagine that a group that expects to be discriminated against might alter their behavior but it that doesn't seem to be in play here. How is that happening? In the MLB study, I assume that the claim is that black pitchers facing white umpires were more likely to place pitches within some zone within the strike zone. What's the equivalent here?

Incidentally, I'd like to see somebody vet the MLB study. Why, for example, is bias only directed toward pitchers and not batters? How do you distinguish between pitchers who are getting more accurate calls and those who are grooving it? Are there subcategories of pitchers that are important. Given the fact that academics love to play with baseball statistics, I suspect there will be plenty of volunteers.

Phil said...

The MLB study was based on data from the Ques-Tec system installed in many MLB parks. Ques-Tec is used to track umpire accuracy. You ask what the equivalent is of an umpire narrowing or widening the strike zone based on the race of the pitcher to a police officer making a decision to search a suspect? Simple. The officer, like the umpire, is using race to base his decision on whether to search (or, in the case of an umpire, whether the pitch is a strike).

And I'm not sure what a "higher error rate" has to do with the discussion. What do you mean by "error rate?"

The paper was published in the American Economic Review (link) so it's in the "system" and will be subjected to the scrutiny that all scientific papers undergo once published. But it did make it through the peer-review process.

The issue of pitchers vs. batters is an interesting one and I wonder how they controlled for that in the study. I assume the normalized for batters, but I don't have access to the source paper (paywall). I will ping the author and see if he can provide me with a copy which I would be happy to share with you.

Thanks for the engaging discussion!

Phil said...

Oh, FWIW, I did find a working paper version of the MLB umpire study here. (PDF)

Nick said...

I think Spice brings up a good point. I think the troubling thing here is that contraband is found only 22% of the time. After all, the police are stopping people for traffic violations here, and then using that as an excuse to do a further search without a warrant.

"Police stopped black and Hispanic drivers about five times as often as white drivers solely for equipment violations". That's basically the old "broken tail light" excuse. Then, because they've been pulled over, they can do a further search, once again, without warrant.

I think Spice's comment that its probably a random sample is accurate. After all, if there were real suspicion of some sort of other crime, then why wouldn't they get a warrant for the extra search to ensure that the contraband they're sure is there is able to be admitted into evidence in court?

Twenty Two Percent! That means that 78% of people who are searched were completely innocent and had their right to be free from warrantless search and seizure violated.

That is a horrible statistic.

Rick Esenberg said...

I'm using the term error rate to equate to your use of the MLB study or defect rates. The irrefutable point here is that if searches of blacks and Hispanics are just as likely to yield contraband as searches of whites then the criteria for a search would seem to be not racially discriminatory.

Nick's point is a different one. He says that they are searching everyone too much. I don't think the JS story has enough info to evaluate that but it is, in any event, a different point than the one I was making. I am not really prepared to express an opinion on that.

Art said...

Anyone who allows a police officer during a traffic stop to inspect any place that is not in plain sight without a warrant needs to have their head examined.

Open up for ICE at the border and near it. Everyone else needs to have either a good 4th amendment exemption or a warrant.

We all have to stand our ground. When asked to look around say no.

If they then make an order of it let them do as they please. You do not want to go to jail over resisting.

If we do not all make the police use the tools available to them to get access to our private papers and belongings we each weaken the next person. If they have a need to see it they will have PC and can wake up the judge.