US vs City Of Erie


Last week, March 7-10, I testified in federal court in the matter of DOJ v. City of Erie. At issue was the job-relatedness of a physical ability test for new-hires for the police department. The test is not being used since the state now mandates a new test at the training academy. There were no grieving parties. You're starting to get the idea here; "why the law suit?" Good question.

Defense Team in the matter of US v. City of Erie.
L-R Ken Zack, Esq., Jerry Villella, Esq., Paul O. Davis, Ph.D.

The Justice Department accepted the validity of the obstacle course, a series of events very similar to our LEOPARD (Law Enforcement Officer Performance And Reaction Drill) course. The big hang up is an expectation to perform 17 push-ups and 9 sit-ups in 30 seconds upon completion of the simulated foot pursuit.

My comments in the rebuttal phase is repeated below;

Rebuttal

First, assembled before you were some very talented and sagacious scientists. I can greatly appreciate how lay personnel can leave this courtroom talking to themselves after listening to three straight days of this stuff. I’m pleased to have been able to share my knowledge and develop a better appreciation of the area of psychology. However, this is a case about physiology. By their own admission, neither Dr. Jones nor Dr. Siskin is an expert in this field and Dr. McArdle claims no expertise in test development for the public safety sector.

It’s not hard to be a Monday morning quarterback and opine about all the things that should have been done. And, I’m the first one to say that we can always do things better, particularly with such an august group of critics telling you how. There have been some excellent suggestions that were made by the plaintiff’s experts and some that I would whole-heartedly agree with. But, what the city did was to create a valid test and I’m going to tell you why.

My rebuttal comments are divided into two categories:
Statements wrongly attributed to me or miss represented
Statements on direct examination that are inconsistent with facts and not addressed in my direct testimony or beyond the rebuttal reports.

First: the Uniform Guidelines are exactly that: Guidelines. The 3rd Circuit threw out the Division 14 Principles in the first SEPTA case. Numerous courts have elected to ignore them. In the Berkman case, the 2nd circuit said that management had wide discretion in using their own experience and judgment in making decisions on effective job performance just by observing employees in the performance of job tasks. There is not one true path to validation and in this case, it’s a straight up physiological model. A numerical coefficient is not needed. I will elaborate.

You have to look not only what a subject matter expert is saying, but why; what is the motivation? If plaintiff’s experts are to be believed, it will cast a pall over the entire law enforcement community for years to come. Basically, what they are saying is that you must hire a professional to design your test. There’s nothing in the law that says you have to park your common sense at the door. In fact, science is nothing more than the logical extension of common sense.

Dr. Jones:
Said that I (meaning me) don’t like the use of incumbents for norming purposes; this is not true. I can tell you categorically that recruitment of incumbents can be difficult and skew data because of a lack of motivation.

For example: I am bound by the canon of ethics of the American College of Sports Medicine for research involving human subjects. Participants must be allowed to de-volunteer. When we did our Marine Corps validation project I was required to inform all the Marines who were randomly selected as a part of a stratified sample that they had the right not to participate. The Navy Medical Department, funding the study, required this to be in the informed consent document executed by every Marine. While Marines, as a whole are a motivated group, there were some Marines who did not like the idea of a day long combat simulation exercise at 9000 feet in the snow. The data were interesting in that clearly some of the subjects did not put out. Later I found out that when these Marines informed their Captain that they did not want to participate and were invoking their right to de-volunteer, the Captain informed them that the last time that they had that chance was just before they raised their hands for swearing in; “now get on the helicopter.”

As Dr. Siskin remarked, motivated people are very helpful. There is clearly nothing wrong with wanting to emulate or recruit people who exceed by a reasonable distance the marginal performers. The danger that he expressed, while understandable, certainly can be controlled for.

What virtually all of the plaintiff’s experts expressed is a search for the best qualified person, yet their suggestions run contrary to that espousal.

First, what I find to be inconsistent with reality is a lack of appreciation of the true consequences of attempting to find these so-called bottom dwellers. Yes, you will ultimately improve the quality of the workforce by removing from hiring those on the bottom. He suggested that 6.7% would be eliminated by lowering the standards using his paradigm of a cutoff that is 1.5 standard deviations from the mean. My preferred approach is to rank people and hire from the top down; the city’s approach was to eliminate half of the applicants by using the mean.

Keep in mind, we cannot gainfully employ all of the applicants. There will probably be 5 or more qualified applicants rejected for each hired. We need to reduce the field and going for the best candidates is a sure-fire way to execute the only shot you have in improving the quality of the work force. Let me explain. Despite testimony by Dr. McArdle- and I agree enthusiastically with everything he said about the plasticity of the human condition, but experience tells me otherwise. For neigh on 39 years, I have been a practitioner of the principles of adult fitness and an evangelist for helping people take control of their personal fitness profile. There are many police officers who share my vision for a fit, trim and productive law enforcement community. But, the practical reality is that without retention standards, things are pretty much status quo. The published literature belies Dr. McArdle’s dream that cops will intrinsically come to appreciate the benefits of a regular program of physical activity. Heart Disease kills more cops than bullets. Obesity, and its concomitant consequences of diabetes, musculo-skeletal injuries and other metabolic disorders takes a huge toll and we haven’t even addressed the job performance aspects of the problem.

In this state, the PSP’s program was brought to its knees by the bargaining unit. I was intimately involved in the creation of one of the most advanced and effective programs anywhere in the nation. We were several years into the mission, with data demonstrating that troopers were improving in all measurable dimensions of fitness. The power of the program in predicting medical problems was almost scary: if you couldn’t pass the simple five-minute step test, you were going to die. Well, not immediately, but many troopers who failed the step test had in close proximity to the test, heart attacks or strokes some resulting in death.

The union grieved the test and the program folded, demonstrating once again that safety takes a back seat to employment security; yet what kind of security is this if you spend your entire life working for that length of service retirement only to fall short of the goal line because of poor health.

I have worked with the FBI, the Secret Service, DEA, US Marshals, and hundreds of other law enforcement agencies in implementing fitness programs. None of them are mandatory. Where they once required mandatory fitness testing, there is now only one organization that I am aware of that requires its members to continue to meet a retention standard and that is the Ohio Highway patrol. I am not aware of any patrol officer that has been terminated yet, but just wait and see what happens when that occurs.

Both Drs. Jones and Siskin took the position that in order for a test to be valid, it had to be correlated to successful job performance in the incumbent population.

Their amorphous and sweeping definition of “successfully performing the job” is a near impossibility to achieve. It is highly subjective and assuredly, the bargaining unit will not cooperate in that they take a contrary position that there are no significant differences among people; everyone is performing the job satisfactorily; I have personally heard this argument from union presidents and shop stewards.

By his own admission, Dr. Jones’ opined that this was a challenge. I submit that a series of content-type tasks is a de facto surrogate for the job. Absent the entire psychological baggage attendant with the abstract definition of “successfully performing the job” there is much clearer agreement on the actual physical tasks that comprise those essential functions of the position under investigation.

Dr. Jones opined that the limited scope of the survey tool might have left out more important elements of the job; one of those he suggested was balance since in the past there had been such a test. Balance is well known to be highly unreliable (i.e., a subject passing once may fail a second time).

Dr. McArdle proffered the opinion that the use of the sit-ups and push-ups in the Erie PAT are a contravention of the standards of the ACSM. The ACSM has no such standards. They do publish recommendations for testing protocols for adult fitness populations. While Erie elected to utilize a callisthenic as a surrogate for a physical task, there is nothing wrong with this use. Characterizing this as though it is was some egregious violation is a complete mischaracterization of the College’s intention. In fact, virtually all of Dr. McArdle’s comments were directed to research designs, where one controls for such factors as weight, sex to make comparisons or inferences and not in an absolute setting such as the occupational setting. We are not interested in divining a pure measure of muscular endurance. We are interested in knowing if an applicant has the ability to minimally perform the job.

Dr. McArdle’s logic is flawed in several of his recommendations. Assuming that any and all deficiencies on entry can be rectified in the police academy is a dangerous precedent. Not everyone can be trained to a job-related standard. The PAT reduces the cost of replacement by ensuring that people who pass have a reasonable chance of graduating. A major problem with his recommendation is the Halo effect: a situation the marginally performing recruit cannot overcome deficiencies and management decides that despite the fact that despite the candidate is deficient a rationale is conjured to retain the person because “we like him”.

The far more sensible approach is to select those candidates who exceed by a wide margin the minimum action limits of the job. At least, they have further to coast before they intersect with that minimally acceptable level that defines unsatisfactory performance.

All of the plaintiff’s experts represent that selecting high performers on the PAT is mutually exclusive with candidates who are also smart or present with an array of psychological benefits. This is clearly not the case.
What could possibly be wrong with, all things being equal, choosing the most outstanding candidate? I know many stellar athletes who are also smart.

Dr. McArdle remarked about the draconian impact the Erie PAT had on women and the overall pass rate. My characterization of the test as easy is a subjective term. More importantly, the pass rates for women as a percentage of the men are precisely what exercise scientists would expect. In fact, it comports with Dr. McArdle’s data as well as the plethora of published research on gender differences. This validates my point that the Uniform Guidelines, as written by psychologists fails to show a basic understanding of physiology. No physiologist is disturbed by what is, the inalterable laws of nature. The women’s score is 1.5 s.d. below the mean of the men. To attempt to “norm”, modify, or outright change nature to fit some preconceived notion of altering nature is political correctness personified.

Dr. McArdle continually made references to the Olympics and other sports as examples of gender norming. He failed to mention that the martial arts recognize that physical size is a huge contribution to predicting the outcome of a bout. This is why they have weight classes. This flies in the face of his statement that you cannot predict the winner in an altercation. But real life is not gender normed. As one of the Erie police officers opined, the perpetrator doesn’t slow down because a 40 year old is chasing him. The real world works on the basis of absolutes, not corrected for research designs or other statistical manipulations. And, while skill and technique are clearly important, there are severe limitations to what small people can do to overcome a size differential of more than a few weight classes. Even with defensive tactics training, as we have heard from the testimony of Erie police officers the gap can likely prevent the officer from affecting an arrest absent help.

Dr. McArdle attempted to make a point about Lincoln Kennedy, the standout lineman for the Oakland Raiders. I agree wholeheartedly with his point that you can’t tell just by looking (a obscure reference to my remarks about “going to the videotapes”); but I believe that he missed my point. First, I never suggested that one simply look at a static presentation. But a “moving” picture is invaluable. While he may have jumped to some conclusion based upon a casual inspection of Kennedy’s body weight, I have a much better basis for an opinion. Lincoln Kennedy was on my ESPN TV show. He, and a number of other NFL Players participated in our Firefighter Combat Challenge in Bethesda, MD. Lincoln and his team members raced against a firefighter team over the course. Lincoln picked up the 175-pound dummy with one arm and dragged it 100 feet. This video, reviewed by any of my peers, or the general public for that matter, is highly instructive. We are looking at an objective performance and rendering an opinion on the time it took to perform the task. The qualitative data that one derives from people who struggle to accomplish what others do with ease is very helpful in understanding why people are having difficulty. In the Erie PAT, it’s very clear that they do not have the metabolic capacity to perform police work.

Dr. McArdle’s unwillingness to proffer an opinion on the value of a combination of push-ups and sit-ups lacks candor. To say that an inability to do even one push-up tells him nothing, is absurd.

Dr. Siskin made an excellent point with regard to the employment of the push-ups and sit-ups and how they are integrated into the overall scoring. He, and Dr. Jones described it as a metric and that is exactly what it is. But first, let me finally address what this test is and is supposed to measure.

The purpose of this test is to select those individuals presenting for employment as Erie police officers who have the capacities (not the skills) to minimally perform the essential physical functions of the job. There is no issue with the elements of the obstacle course. The entire crux of this matter comes down to the use of the sit-ups and push-ups. I have stated that this is a test for Resistance to Fatigue, a factor that we identified in our original research while at the University of Maryland.

Without belaboring the court with the physiological underpinnings that include the citochrome oxidase system and the 18 steps of the Emden-Meyerhoff pathway of liver glycolysis, all one reasonably needs to know is that fatigue is a limiting factor in the performance of any muscular activity.

Chief Bowers opined and the survey data confirm that a foot pursuit and a struggle, or a struggle, loss of contact and a foot pursuit may take as little as 90 seconds. There were expressions of time and distances far greater than those advanced by Chief Bowers in the 76 officers who provided written narratives (this is a job analysis in its most basic format). In the performance of any muscular activity, lactic acid begins to accumulate. As the blood level raises, respiration increases. At that point where metabolism is unable to turnover the lactate, activity ceases. The question that we want to know is simply this: can an applicant negotiate the course and engage in 30 seconds of moderate physical activity approximating the same metabolic demands of a shortened foot pursuit?

Contrary to claims by Dr. McArdle, this is a standardized test. An election to utilize a callisthenic for a shorter duration shorter than repetition to failure does not convolute, invalidate or distort its intended purpose in the instant case. The test is standardized since everyone is doing the same thing. It is not intended to be used as push-ups were in the Anne Arundel County study, contrary to the testimony of Dr. Siskin who completely missed the point. The use of the push-up is a more favorable application than a gross strength test as all subjects are using their own body weight as a resistance and is exactly on point to Dr. McArdle’s first report.

Dr. McArdle states that he is highly experienced in testing people to fatigue, yet fails to recognize the highly fatigued state of individuals who fail the test not because of their gender, but because of their lack of conditioning. Absent sampling for blood lactate levels, an inspection of the physical condition of the applicants at the end of the test speaks volumes for the poor level of physical condition that is exacerbated in many cases by obesity.

The construct for which we are regressing is the performance of some moderate physical task such as the case where a perpetrator is passively resisting arrest through noncompliance. In other words, he is not fighting back. We’re asking in a simulation mode, for the applicant to show that s/he can hold on for 30 seconds until help arrives. It could be representative of other moderate levels of exercise following this foot pursuit including bending and lifting evidence. The muscle does not change the nature of generating energy as a consequence of what activity it is expected to perform. Said another way, the muscle doesn’t care what the object is that it’s moving. Metabolism works the same at the cellular level. Asking for a demonstration of 30 seconds of vigorous activity, against body resistance is neither an unusual nor an unreasonable request.

I opined that with my perspective on law enforcement I would have included a more demanding demonstration of gross body strength. Also, in response to the question posed of the judge regarding a national standard, we have such a tool. The product, called the LEOPARD (Law Enforcement Officer Performance And Reaction Drill) was designed and developed in Quantico, at the FBI Academy and is now being rolled out to law enforcement organization all over the country. It is currently in use at the Ontario Police College.

The validation of the use of the Erie PAT is a combination of a content valid obstacle course, and a physiological construct of resistance to fatigue. The tests must necessarily be linked because that is how the job is performed. Let me site an example of how psychologists, in a failure to understand the principles of physiology created a disaster and would likely do so again.

In an attempt to control the rate of travel in a physical ability test for firefighters, the test designers had the candidates stand in a holding area between events to ensure that there would be no range in time that would have normally been the outcome if the applicants were allowed to self-pace. They wanted a pass/fail test. The result was venous pooling and hospitalization for rhabdomyelysis, a condition caused by high levels of lactate and uric acid that can damage the kidneys.

While there is little danger of this in the Erie PAT, failure to allow subjects to run through the entire event as they would in an emergency scenario would be less realistic than what is the common practice. Dr. Siskin does make the excellent point that there should be no bifurcation between the obstacle course and the push-up/sit-up portion. This maneuver (his suggestion) would, as he said have resulted in the norming group with a more relaxed pass rate. I would have made that change in the test.

Whatever machinations we may employ with manipulating the configuration of the Erie PAT, this much is certain. The perpetrator population can certainly address the physical performance requirements of the test with far greater ease than incumbents. Don’t forget the job-related handicap of almost 20 pounds associated with the duty belt. None of the experts even touched on that.

All experts continued to ignore my position that a very large number of middle school students would have been able to master this test. My seven years as a physical education instructor affords me this perspective.

Dr. Siskin’s proffer of differential validity was clearly wrong. We never intended that the Anne Arundel County P3 be used as he did for his statistical analysis. I have never seen differential validity in any physical performance test and in fact, we specifically postulated that if any such outcome existed in our very comprehensive NYC study, we would include it in the protocol. None resulted.

While the validity of the obstacle course has been accepted, what is interesting is the predictive value of pushups. To say that they have no value after having shown that they do predict wall surmounting success supports their use for other more utilitarian purposes, and invalidates Dr. McArdle’s claim that scaling the wall is all technique.

In summary
1. There is no legal mandate to follow the Uniform Guidelines nor the I/O Principles; they were designed for written tests and fly in the face of the body of published physiological literature
2. There are multiple pathways to validation
3. The survey of Erie police officers clearly establish an objective taxonomy of frequently performed, arduous and highly critical tasks. This is a JTA (job task analysis) limited to physical tasks solely
4. Virtually all of the Erie police officers have or routinely perform arduous physical tasks
5. The Erie PAT tests for Resistance to Fatigue an essential component in the successful performance of law enforcement duties
6. The sample used to norm the test was comprised of volunteers; union contract precluded ordering randomly selected personnel to participate. This would increase the passing score.
7. The passing score is more attuned to the concept of hiring for built-in obsolesce since there is no mechanism to ensure the maintenance of fitness once on the job and in the bargaining unit.
8. The identification of “successfully performing the job” using supervisor ratings as a criteria for validation lacks objectivity and is fraught with problems of unreliability and bias. The test is valid by virtue of its representativeness of job demands, including resistance to fatigue.
9. The demographics of the “threat” are predominately males under the age of 40 as identified through arrest records.
10. The Erie Police Department is challenged with effecting the arrests of this population; not the incumbents.
11. Improvement of the quality of the workforce is a legitimate governmental function
12. Hiring individuals who exceed the minimum standards of the job ensures longevity and reduces the incidence of orthopedic injuries
13. While individuals may “choose not to age” this requires the active participation in a regular program of physical activity. This is called a mandatory fitness program; it is mandatory to work out to earn the benefits.
14. The preponderance of police officers do not engage in a regular program of physical activity and will not do so under any set of circumstances other than retention standards.
15. If criterion-based validation studies using ratings of police physical performance were as easy to perform as the plaintiffs suggest, we should see many published in research journals, backed by hundreds of others in technical reports. The stark absence of such  data attests to the technical difficulties involved in carrying out such research.