In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
88
1 Afternoon Session
2 -- --- --
3 THE COURT: Okay. Next witness.
4 MS. MASSIE: Call Jay Rosner.
5 THE COURT: Mr. Rosner, you're still under oath.
6 MS. MASSIE: Judge, I do apologize. I'm slightly
7 behind the ball here because we got a bit behind on the
8 security check. I need to pass some stuff out to you guys.
9 THE COURT: No problem. Take your time.
10 MS. MASSIE: For everyone's information, these are
11 proposed Exhibits 201 through 210. Those were handed out
12 before, but we thought it would be prudent to make some
13 extra copies. There is also a demonstrative exhibit that
14 we're going to try to get admitted.
15 And I think the people in the gallery, copies should
16 be coming.
17 THE COURT: You have changed attorneys.
18 MS. MASSIE: Yes. Sorry, Judge, and I had raised
19 that with Mr. Payton and Mr. Kolbo.
20 THE COURT: I don't mind. That's fine. I was
21 waiting for Mr. Washington.
22 MR. WASHINGTON: I'm taking it easy today, Judge.
23 MS. MASSIE: I think we have reached the state of
24 resonance, and I apologize to everybody for the holdup.
25 Hello, Mr. Rosner.
89
1 THE WITNESS: Hello.
2 MS. MASSIE: I would like to start, if it's okay
3 with you, Judge, by reading very, very briefly from the tail
4 end of Mr. Rosner's testimony last time just to make it
5 relate, because I think we're going to pick up exactly where
6 we left off.
7 THE COURT: Go on.
8 MS. MASSIE: This is you testifying, Mr. Rosner.
9 "I hear from educators all the
10 time, the student is so wonderful. A
11 minority student who in class --"
12 This is page 208 from Volume 7 of the trial
13 transcript, for everyone.
14 "I hear from educators all the time,
15 the student is so wonderful. A minority
16 student who in class has got the papers,
17 responds really well, and has mediocre
18 test scores. How can that be? The answer
19 is very common and very understandable.
20 Different skills."
21 And then the Court interjected: "For everybody,
22 not just minorities."
23 And you responded to the implied question:
24 "For everybody, although the way the
25 skills are measured is different and
90
1 we will talk about that when we come to
2 do some specific test questions."
3 DIRECT EXAMINATION (Continued)
4 BY MS. MASSIE:
5 Q So I would like to turn now to your testimony on
6 specific test questions and I should start by asking you
7 whether people in different racial groups have different
8 outcomes as racial groups on different standardized test
9 questions.
10 THE COURT: First you better ask him and lay a
11 foundation, because his expertise -- I mean, I let him kind
12 of go on a lot last time, and in fact, I thought about it
13 and thought about it and probably let him go on too much.
14 From now on we're going to answer the questions.
15 And I haven't heard any -- why don't you lay a
16 foundation if he has expertise in that area.
17 MS. MASSIE: I'm happy to do that for this specific
18 purpose.
19 BY MS. MASSIE:
20 Q Have you studied the ways in which SAT's and LSAT's
21 and other similar standardized admissions tests have an
22 impact on minority test takers?
23 A Yes.
24 THE COURT: How have you studied them?
25 THE WITNESS: In a number of ways, Your Honor.
91
1 THE COURT: First of all, have you had any training,
2 statistical training or anything of that nature?
3 THE WITNESS: I don't have any formal statistical
4 training, no.
5 THE COURT: Go on. How have you studied it?
6 THE WITNESS: I have studied this topic from working
7 myself with many dozens of SAT test forms and many dozens of
8 LSAT test forms, looking at those forms, analyzing them, and
9 then working with majority students and minority students on
10 individual test questions in the context of test
11 preparation, and I have done many, many hours of that.
12 I have also had discussions, particularly in recent
13 years, with minority students before they took the test,
14 while they were preparing for the test, and then I have had
15 discussions with minority students after they have their
16 test scores. So it's a continuum of information involving
17 the test questions themselves, the interaction of minority
18 students with the test questions, and then the results that
19 minority students get when they get the test scores.
20 THE COURT: Did you compile this data, did you write
21 it down, did you do a paper on it?
22 THE WITNESS: I have written -- I have written a
23 couple of articles on these topics. I speak regularly to
24 organizations about these topics. In terms of, I think
25 perhaps what you're asking is, have I done a statistical
92
1 analysis of this --
2 THE COURT: That's not what I'm asking.
3 THE WITNESS: Then I misunderstood.
4 THE COURT: Is there a paper I can read that you
5 have done on this? Is there anything other than your
6 anecdotal discussions with people?
7 THE WITNESS: Probably the -- there is a test that
8 we will talk about that I have compiled that, in the context
9 of this testimony, that I just compiled a couple of weeks
10 ago and that's, I think, the only thing.
11 THE COURT: Just compiled a couple weeks ago?
12 THE WITNESS: Yes.
13 THE COURT: Go on. Let see where he goes and see if
14 it helps in the qualification, and I think he is qualified
15 to do certain things, but let's see. Go on.
16 MR. KOLBO: Your Honor, if I can just make it clear,
17 we have an objection on foundation. We don't -- we believe
18 this witness hasn't been qualified as an expert on test
19 design and psychometrics and --
20 THE COURT: I agree with you, and that's what I
21 thought we were getting into.
22 In terms of him teaching the class, in terms of who
23 comes to his class, he is obviously well qualified for that,
24 he has been on campuses, he goes out, part of his job was
25 to start a whole division out in Seattle in terms of getting
93
1 students, but in terms of the other kinds of areas I haven't
2 heard anything other than he has the same degree that I
3 have.
4 MS. MASSIE: But he doesn't have to, as an expert,
5 to be qualified as an expert, he doesn't have to have
6 specialized academic training. I think we agree about that.
7 THE COURT: I absolutely agree, but he has to have
8 more than anecdotal kinds of information if he is going to
9 testify as an expert.
10 Anyhow, let's go. Let me -- we heard, we have heard
11 from experts, so we know what they are, including
12 Dr. Shapiro who testified before, who by training, by every
13 kind of imaginable thing has the expertise to testify in
14 testing areas and statistics and so forth. Go on.
15 The only reason I mention it, I'm looking at your
16 exhibits and --
17 MS. MASSIE: I think Mr. Rosner's testimony will
18 build on Professor Shapiro's.
19 THE COURT: It may build on it, but he's got to have
20 some expertise other than the fact that he is a lawyer and
21 he works for a company that gives classes on testing.
22 MS. MASSIE: Let me try, continue trying to
23 establish that.
24 THE COURT: Sure.
25 BY MS. MASSIE:
94
1 Q You have trained teachers?
2 A I have trained test preparation instructors, yes, to
3 teach SAT courses and LSAT courses and other courses.
4 Q And you have made a study of test items and their
5 impacts on minority students as a part of your job
6 responsibilities?
7 A I have, over the last five years, not in the more
8 formal sense that the Judge was talking about in terms of
9 writing a journal article, but it's my job to try to
10 understand how minorities respond to and react to and can
11 answer better individual test questions. That is a central
12 part of my job.
13 THE COURT: Central part of your job now that you
14 have become the Director of this Institute, and you became
15 the Director of this Institute when?
16 THE WITNESS: In 1995, Your Honor.
17 Not -- actually, it predates that, because it was
18 also part of my job when I was teaching predominantly white
19 students in Seattle in the late '80's to understand how
20 those students in a test preparation context would better
21 answer individual test questions. That's what a test
22 preparation person does.
23 And my focus has been on minority students since
24 '95, in that regard.
25 BY MS. MASSIE:
95
1 Q And Mr. Rosner, you seek through the Foundation to
2 make an intervention in testing bias and the test score gap;
3 correct?
4 A Yes.
5 Q And that means seeking to understand it; correct?
6 A Yes. Understanding the test score gap and the
7 components that go into it and the questions that feed into
8 it is part of trying to reduce that gap in my work with
9 minority students.
10 Q And that means trying to understand what contributing
11 causes are to the test score gap; correct?
12 A Yes, as best I can, yes.
13 Q And that means being able to make clear on the basis
14 of hard information to minority students when you speak with
15 them that the test score gap is not a product of their lack
16 of qualifications or lack of capacity; correct?
17 A Yes, that's right.
18 Q Tell us how you know that there is a test score gap.
19 A Well, anyone who looks at the average results of tests
20 like the SAT or the LSAT sees that there are very disparate
21 impacts on under-represented minority populations,
22 particularly African Americans and Latinos. There is also
23 gaps between men and women, as Professor Shapiro pointed
24 out, but the focus of my work has been primarily on African
25 Americans students, secondarily on Latinos students.
96
1 Q And do you agree with Professor Shapiro that the
2 characteristics of this gap are broadly similar across the
3 different standardized admissions tests widely used in the
4 United States?
5 A Yes, the gaps, if we focus on the SAT and the LSAT,
6 for example, the gaps are persistent. They are consistent
7 and they operate the same way, and have, between whites and
8 blacks, for example, as they have for the last ten or twenty
9 years. They are also consistent on the other most common
10 admissions tests, like the GRE, et cetera.
11 Q I would like to turn your attention to the question of
12 -- to the matter, to avoid the word I was about to use --
13 of question selection, item selection, because I think
14 you'll be able to concretize for the Court some of the
15 points that Professor Shapiro was making earlier today.
16 A Yes. I have been concerned about item selection for
17 years, and have acquired some data recently that I think
18 illuminates concerns that I have had for a long time about
19 how test items are selected.
20 Q And why have you been concerned about it?
21 A I recall an article that I wrote in 1993 that focused
22 on test question selection, and in that article I used the
23 distinction between males and females, but in thinking about
24 these topics after that article it became apparent to me
25 that the much more significant test score gaps are between,
97
1 for example, whites and blacks. I think your question was
2 why, and the answer is, it relates to my work.
3 THE COURT: From now on, please answer the question.
4 THE WITNESS: Yes.
5 BY MS. MASSIE:
6 Q Mr. Rosner, if I could direct you to Exhibit 202.
7 MS. MASSIE: So everyone knows, we expect to
8 introduce Exhibit 201 through a subsequent witness, that's
9 why we're skipping it.
10 BY MS. MASSIE:
11 Q Tell us what's going on with this exhibit, if you
12 would.
13 A Sure. Very briefly, I wrote Black or White Preference
14 Question at the top. That's something that I just raised
15 for an issue. This is an antonym question, an SAT antonym
16 question.
17 You'll notice the instructions, chose the opposite,
18 and what the test taker is supposed to do is to chose the
19 word that is the best answer for a word that is opposite in
20 meaning to the word, reneg, and very briefly, you'll see
21 that the answer C is in italics. That was in italics in the
22 San Francisco Chronicle, which was the source of this
23 particular information.
24 And so the correct answer is C, but that's not what
25 is really significant about this question. What is
98
1 significant is what I have in smaller print on the bottom,
2 and that is the results are that African Americans do
3 significantly better defining the antonym than do whites.
4 That's a quote from the San Francisco Chronicle, citing ETS.
5 And because African Americans do answer this question
6 correctly at higher rates than whites, I call this a black
7 preference question, and a white preference question has the
8 opposite meaning, that --
9 THE COURT: How would you know this, other than you
10 saw it in the San Francisco Chronicle?
11 THE WITNESS: Well, that's the point, Your Honor,
12 you would not.
13 THE COURT: What special knowledge do you have
14 concerning this other than you saw it in the San Francisco
15 Chronicle?
16 THE WITNESS: The special knowledge --
17 THE COURT: Have you done any testing of your own,
18 done anything like that?
19 THE WITNESS: I have talked to minority students.
20 THE COURT: My question is, have you done any
21 testing?
22 THE WITNESS: No.
23 THE COURT: Your source is the San Francisco
24 Chronicle?
25 THE WITNESS: Yes, whose source is Educational
99
1 Testing Services.
2 THE COURT: Did you go to them and talk to them
3 about it?
4 THE WITNESS: No, I didn't.
5 THE COURT: All right.
6 BY MS. MASSIE:
7 Q I see that this question was rejected for use on the
8 SAT?
9 A Yes, this was a question that was rejected for use on
10 the SAT.
11 THE COURT: How do you know that?
12 THE WITNESS: Again, that's -- that was quoted in
13 the San Francisco article citing ETS. The purpose of the
14 article --
15 THE COURT: So your source is the San Francisco
16 Chronicle?
17 THE WITNESS: Yes.
18 THE COURT: You didn't go back to them and say, hey,
19 I have to testify in court. Did you reject it? Why did you
20 reject it? Is there a reason you rejected it? Did you use
21 it later?
22 Did you do any of those things?
23 THE WITNESS: No.
24 THE COURT: Yes or no?
25 THE WITNESS: No, I didn't.
100
1 THE COURT: You may continue.
2 BY MS. MASSIE:
3 Q And does this question -- just looking at this
4 question on the face of it, would you imagine that there
5 would be a significant gap in performance?
6 A This question looks like a typical SAT question. I
7 have seen many hundreds of antonym questions. There would
8 be no way that looking at this question -- I would like to
9 find a way, but there is no way that I could tell that this
10 is a black preference question by looking at the face of the
11 question.
12 Q And Mr. Rosner, what's the source of the gap in
13 performance on this question?
14 A The source of the gap -- well, this question is
15 clearly capturing something about race, because racial
16 groups answer it in disparate fashions. Beyond that,
17 despite my expertise and working with lots of minority
18 students, I can't be more specific as to exactly what about
19 race this is capturing.
20 Q It's capturing something that we don't yet know how to
21 describe or define?
22 A Something that I can't yet describe.
23 THE COURT: The only reason you know that is because
24 you read it in the San Francisco Chronicle?
25 THE WITNESS: That's right, Your Honor.
101
1 BY MS. MASSIE:
2 Q Why is it -- is it easy, Mr. Rosner, to get pretested,
3 rejected questions, questions that are pretested and then
4 not use on a scored test?
5 A No, it isn't.
6 And in fact, Your Honor, in partial answer to an
7 earlier question, I have asked Law Services for these
8 kinds -- this kind of data information, and they say there
9 is none publicly available. I haven't asked ETS about SATs.
10 I asked Law Services about LSAT questions.
11 THE COURT: But you know that Dr. Shapiro testified
12 that he was able to get some of that information through
13 litigation, not through --
14 THE WITNESS: Yes, and he is --
15 THE COURT: That's all.
16 THE WITNESS: Yes.
17 THE COURT: Go ahead.
18 BY MS. MASSIE:
19 Q Let me ask you to look at Exhibit 203.
20 Tell us about that question briefly, if you would.
21 A Okay. Very briefly, this is a sentence completion
22 question. It's another SAT question. The objective here is
23 to pick from among the five answers the pairs of words that
24 best fit into the meaning of the sentence with the two
25 blanks. Again, I note at the bottom the correct answer is
102
1 C, and again that's not what is most significant about this
2 question.
3 Most significant about this question as it pertains
4 to this case is that eight percent more African American
5 than whites answered this question correctly, again, a quote
6 from the Wall Street Journal, which again cites Educational
7 Testing Services.
8 And the disposition of this question, as the last
9 question, was this is a black preference -- again, my label
10 entitled a black preference question that was also rejected
11 for use on the SAT after pretesting. They told us in the
12 article that the pretesting was done in 1998.
13 Q Looking at this question, Mr. Rosner, does it seem
14 like a typical SAT question?
15 A Same as the last question, this looks to me like a
16 typical sentence completion question. I have seen hundreds
17 of these on the SAT. Yes.
18 Q And do you have any greater ability to explain in this
19 instance how the question captures something about race in
20 our society?
21 A No, same answer as to the last question.
22 Q Okay. Let me direct you to Exhibit 204. Tell us what
23 this is about, if you would.
24 A Yes. This is a math question, a math SAT question.
25 It's an algebra question, because it uses X as a variable,
103
1 and the question asks about the square root of a number.
2 The correct answer here is C, and again, that's not what is
3 really important here.
4 What's really important is the results which I
5 quoted from the Wall Street Journal, that seven percent more
6 African Americans than whites answered this question
7 correctly. It's a black preference question. It also is
8 rejected for use on the SAT after pretesting, as I have
9 written on the bottom there.
10 Q Looks like a normal question.
11 A Looks like any one of hundreds, many hundreds of
12 algebraic questions that I have seen on the SAT.
13 Q And do you have any idea what it is about this
14 question that --
15 A Same answer, it's capturing something about race
16 generally, but I can't be specific despite my experience as
17 to what it's capturing.
18 By the way, for students in the room, if I can just
19 say --
20 THE COURT: We're not here -- we're just here to
21 answer the questions.
22 THE WITNESS: Okay. I'm sorry, Your Honor.
23 THE COURT: But since you piqued their interest, you
24 can tell them now. The rule is -- you're a lawyer, and you
25 know the rules, but go on, since you've piqued mine, too.
104
1 THE WITNESS: Okay. One sentence.
2 If X equals one half, you can get the correct
3 answer. If you don't see that, there is a little trick
4 built into the question. You just won't get the correct
5 answer. I didn't want students to be puzzling about this
6 for a couple of hours.
7 THE COURT: I agree with you, but it's --
8 MS. MASSIE: I'm hoping that my inability to
9 understand what you just said doesn't predict failure in my
10 future.
11 THE COURT: You don't have to take the test again,
12 so you don't have to worry about it.
13 BY MS. MASSIE:
14 Q You have just shown us three rejected pretested
15 questions on which black test takers outperformed white test
16 takers?
17 A That's right.
18 Q And in general, those kinds of questions aren't
19 available, that's why you're using data from a newspaper,
20 true?
21 A Very few of these questions are available. None are
22 available for the LSAT, very few are available for the SAT.
23 THE COURT: They must be somewhat available. Two
24 newspapers were able to receive them and write pretty
25 lengthy stories, it looks like, but go on.
105
1 BY MS. MASSIE:
2 Q Have you seen, other than these questions, have you
3 seen a lot of rejected pretested questions?
4 A On the SAT, I have seen approximately twelve pretested
5 questions that were rejected and those were all questions in
6 these two newspaper articles.
7 THE COURT: Did you ever -- you know, Professor
8 Shapiro was kind enough to tell us they put out this paper
9 about every test. Have you ever read one of those?
10 THE WITNESS: No, but what he didn't say about that,
11 Your Honor, is that they don't have the statistics for each
12 individual question. They --
13 THE COURT: Do you know that?
14 THE WITNESS: They have the statistics for the test,
15 as I understood what he was saying.
16 THE COURT: But you have never seen one, you have
17 never asked for one? They obviously were pubic. He was
18 able to get them.
19 THE WITNESS: Yes, I have never seen one, never
20 asked for one.
21 THE COURT: Never knew they existed before you heard
22 his testimony, did you, or did you?
23 THE WITNESS: Not -- I know they make -- they make
24 information available. I don't know that that specific
25 information was available to that extent.
106
1 MS. MASSIE: Well, Professor Shapiro didn't say they
2 had item specific information, right?
3 THE COURT: Well, I heard his testimony. I wanted
4 to know if he heard of them. I don't know what's in there.
5 I just know what Professor Shapiro testified, is all.
6 Go on.
7 BY MS. MASSIE:
8 Q Mr. Rosner, let me ask you this: Is it only black
9 preference questions, to use your phrase, black test takers
10 that outperform white test takers, that are rejected by ETS
11 for use on tests?
12 THE WITNESS: No.
13 MR. KOLBO: Objection again on foundation.
14 THE COURT: He is testifying what he read in the
15 newspaper, and with all due respect to the newspapers, they
16 are my favorite thing to read every morning and every night,
17 and I was just on an airplane and I couldn't get through an
18 airplane trip without four or five newspapers, but his
19 knowledge is limited to what he reads in the newspapers. He
20 said that he has never tried to get the information. I
21 don't -- we're sitting here reading the newspaper.
22 MS. MASSIE: Judge Friedman, he has made a study of
23 the sources, the character, the impact of the selection of
24 these kinds of questions.
25 THE COURT: I'll give you a little more latitude,
107
1 but he even just testified that he doesn't know why the one
2 group or other group would do it other than what he read in
3 the Wall Street Journal or that he read in the Chronicle.
4 MS. MASSIE: But that's because this society hasn't
5 finished understanding race, that's the whole thing.
6 THE COURT: The whole thing is, you're right, except
7 he is here as an expert. I mean, an expert doesn't get
8 their data out of the newspaper.
9 MS. MASSIE: But Judge, these questions look like
10 you, anybody reading these questions, would think there is
11 not a racial bias in that question.
12 THE COURT: I agree.
13 MS. MASSIE: There is no insult there. There is
14 nothing in it that's a reality distractor.
15 THE COURT: Absolutely. If you said, these
16 questions, do you think there is a bias, I would say I had
17 no idea, because, you know, I probably couldn't answer them,
18 but that's not the issue. The issue is that all he is doing
19 is he is telling us what a newspaper article had to say
20 about these and he can't -- he does not know why and he said
21 he doesn't know why. He has been very honest.
22 Hundreds of thousands of others ones, he said --
23 MS. MASSIE: Judge Friedman, there aren't released
24 pretest items that are publicly available.
25 THE COURT: Okay. Go on.
108
1 BY MS. MASSIE:
2 Q Do they reject questions on which white test takers
3 outscored black test takers as well or do they only
4 reject those --
5 THE COURT: Or do you know?
6 THE WITNESS: I do know.
7 THE COURT: Okay.
8 THE WITNESS: And they reject white preference
9 questions, also.
10 BY MS. MASSIE:
11 Q Tell us how you know that and please explain.
12 A Because of the dozen problems in the newspaper
13 articles, there were about seven or eight white preference
14 questions that were also rejected.
15 Q Do you have any other source of information on that
16 proposition?
17 A No, that is the entirety of information that I have,
18 that I have with regard to pretested rejected questions.
19 Q And how about on the scored sections, do they only
20 include white preference questions?
21 A Well, a later exhibit will demonstrate that. The
22 answer is, on the scored sections, the scored sections are
23 virtually entirely comprised of white preference questions.
24 Q Please tell us which exhibit you want us to take a
25 look at.
109
1 A The next one, labeled 205.
2 And this was the study that I did that I referred to
3 earlier, Your Honor.
4 Q Tell us what's going on on this sheet of paper.
5 A Sure. I was able to obtain data on four SAT's,
6 actually, through Professor Shapiro, who had this data.
7 These were four SAT's administered to all students in the
8 State of New York in 1988 and 1989, for test forms at four
9 administrations, and this data had specifics for individual
10 questions, and that's what we're talking about, and that's
11 what I'm looking for.
12 And so I looked at the 580 questions that comprised
13 the four SAT's and I simply counted the number of white
14 preference questions, the number of black preference
15 questions, because I had the data of the percent answering
16 correct, rate for whites and blacks. That was in the data
17 set which is appended to my supplemental report. And I was
18 able to count only one black preference question and 574
19 white preference questions.
20 You'll notice a category, if I may continue, you
21 will notice a category on the bottom, it says, no preference
22 questions. Those were questions that were answered at
23 exactly the same percentage rate by whites and blacks.
24 Q And how many students took the tests that are the
25 basis of this count?
110
1 A The four tests were taken by in excess of 200,000
2 students. This is a very, very substantial data sample.
3 Q 200,000 students each?
4 A No, combined, 200,000 students. Approximately 90,000
5 students took two of the four tests. Another test was taken
6 by 20,000 students, and the final test was taken by 11,000
7 students.
8 Q Did you do any other analyses of the impact of these
9 four test forms on minority students?
10 A Yes. That would be the next exhibit.
11 Q And tell us about this.
12 A Very briefly, this, I mentioned before I had the
13 percent correct. The data set included the percent correct
14 for each question of whites, the percent correct for each
15 question of African American students, and the percent
16 correct for each question for Hispanics.
17 So the last sheet I did a white/black comparison.
18 For this sheet I did a white/Hispanic comparison, and where,
19 as I said before, the white answering percentage was higher,
20 that was a white preference question. Where the Hispanic
21 answering correct percentage was higher, that was a Hispanic
22 preference question.
23 And the breakdown is that of the 580 questions, I
24 found 11 Hispanic preference questions and 566 white
25 preference questions, with only three neutral, which I call
111
1 no preference question. Again, answered rates of whites and
2 Hispanics were at the same percentage for those three
3 questions. And these were all questions selected for use on
4 actual SAT's in the prior exhibit. Those were all -- these
5 are the same 580 questions selected for use on actual SAT's
6 administered to students.
7 Q So in other words, they are not questions that were
8 rejected after pretesting?
9 A No, they were in fact questions that were selected
10 after pretesting.
11 Q Do you have those questions?
12 A I managed to get those questions, yes.
13 Q And you have some here, of course?
14 A That is, I managed to -- I had the answering correct
15 data only and then I managed to get the content of all of
16 those questions.
17 Q And were you able to take a look at that?
18 A Yes. Then I studied the content of those questions
19 with regard to the answering percentages.
20 Q You have some here?
21 A The next exhibit is one of those questions.
22 Q That's the one that says oblivious at the top?
23 A Yes. And it's an antonym question like we discussed
24 before. Would you like me to briefly go over what's on
25 here?
112
1 Q Please do.
2 A Okay. Again, the test taker is to chose the opposite.
3 The correct answer is C. And the results of this question,
4 which were in the data set, is that 27 percent more whites
5 than African Americans answered this question correctly.
6 This was on the SAT form that I note there in the -- toward
7 the bottom right-hand corner.
8 And the disposition here, this was one of the 574
9 white preference questions to appear on these four tests in
10 doing a white/black comparison, and this question happens to
11 be the verbal white preference question with the largest
12 percentage difference. This is a large -- not only a large
13 gap between whites and blacks, but the largest gap for the
14 verbal questions.
15 Q Is this question supposed to be, in the eyes of ETS,
16 an easy or a hard question or a medium question, if you
17 know, and how do you know?
18 A Yes, this was an easy question. I know that, one,
19 from its placement on the test, and two, because the correct
20 answering percentage for whites was 83 percent, which is an
21 easy question.
22 Q So --
23 A At least for whites it was an easy question.
24 Q Say more about the question number, the question
25 placement, I'm sorry.
113
1 A That was question number four out of ten to fifteen
2 antonym questions, and the questions in the one, two, three,
3 four positions on the test are easier questions, typically,
4 answered by 70 -- answered correctly by 75 percent or more
5 of test takers.
6 Q Here, is there anything, looking at this, at this
7 question on its face that let's you -- enables to you
8 explain how it is that it has such a disparate effect on
9 test takers?
10 A No, nothing on its face. I would have to look at
11 statistics.
12 Q Let's turn to --
13 THE COURT: Let me just ask one question.
14 I note that's 1988. If we were to believe Professor
15 Shapiro, he indicated something to the effect that after
16 each test they go over and they analyze each question in
17 terms of different things, one of them being, you know, to
18 improve upon it, I forget exactly his words, but he was very
19 eloquent.
20 Would twelve years make a big difference, or do you
21 know?
22 THE WITNESS: Not in my experience, having worked
23 with students and looked at the results. These things,
24 particularly the SAT is extremely constant from year to
25 year.
114
1 THE COURT: In terms of results, but you have never
2 looked at -- these are the only individual questions you
3 have ever looked at?
4 THE WITNESS: These are the only individual
5 questions, Your Honor, I have looked at where I have the
6 percentage answering rates for whites versus blacks.
7 THE COURT: Go on.
8 BY MS. MASSIE:
9 Q So we're on 208, am I right?
10 A Yes.
11 Q Tell us about this. What is this question about?
12 A This is a math question from the SAT. It's a
13 particular, unusual kind of question called quantitative
14 comparison, that's the question type, and you're given two
15 values.
16 If you look down in the middle of the page, you will
17 see Column A and Column B, and the task for the test taker
18 is to decide whether Column A is greater, Column B is
19 greater, whether the two quantities are equal or whether
20 there is not enough information to answer the question, and
21 the answer choices, A, B, C, D, reflect those different
22 decisions by the test taker.
23 Q Tell us what is of interest about this question for
24 the overall test score gap.
25 A Well, what's interesting, certainly the correct answer
115
1 is D, what's interesting about this question is that as I
2 note below, three percent more African Americans than whites
3 answered this question correctly. This was a difficult math
4 question. It is -- and if you note the question position,
5 the source is noted there, it was question number 25.
6 And the disposition of this question is that this
7 was the -- this was that one black preference question that
8 existed on all four tests, so if you're looking for black
9 preference questions on the test, this was it.
10 Q So this was the one out of 580?
11 A One out of 580, yes.
12 Q And again, for this question and the last one, is
13 there anything in it that would cause you to think that
14 there might be something quirky in the data or there might
15 be a big gap just looking at the question on its face?
16 A On its face, my answer is the same as before. It's
17 capturing something about race, but I couldn't be more
18 specific than that.
19 The answering rate, by the way, I wanted to -- the
20 answering, this is a difficult question. Whites answered
21 this question -- 14 percent of whites answered this question
22 correctly, 16 percent of blacks, and the difference being
23 three percent, it was because of rounding.
24 Q Is that their rounding or yours?
25 A That's their rounding.
116
1 Q ETS?
2 A Yes, this three percent was their data in a table, as
3 was 16 percent and 14 percent.
4 Q Let's look at your final question here. This is
5 Exhibit 209?
6 A Yes. This is a math question, a geometry question,
7 very common on the SAT. Again, this is a question from
8 those four released tests. The correct answer, as I note,
9 is D. The results here are again from the table, 32 percent
10 more whites than African Americans answered this question
11 correctly. It's a white preference question, as were all
12 the questions on the test other than the last one, except
13 for the neutral questions.
14 And at the bottom I note, this happens to be the
15 math white preference question with the largest percentage
16 difference of all the white preference math questions on the
17 four tests selected to appear on the SAT.
18 Q Do you think that the questions -- that the individual
19 questions you have talked about so far this afternoon, that
20 they have any capacity to tell us anything about the overall
21 verbal or math skills of the test takers by race or in
22 general?
23 MR. KOLBO: Object to foundation.
24 THE COURT: Sustained.
25 BY MS. MASSIE:
117
1 Q Do you think that the gaps that you have indicated
2 exist in terms of correct answering rates, gaps by race in
3 terms of correct answering rates to these questions, whether
4 they are the questions on which the black test takers
5 outscored the white test takers or vice versa, indicate
6 something about the overall verbal or mathematical abilities
7 and competencies of the test takers?
8 MR. KOLBO: Same objection.
9 THE COURT: Sustained.
10 BY MS. MASSIE:
11 Q Mr. Rosner, what do these numbers mean?
12 A Which numbers are you referring to?
13 Q The gaps.
14 MR. KOLBO: Well, Your Honor, I don't like to pop up
15 and down, but I think it's the same question and I have the
16 same objection.
17 THE COURT: Same ruling.
18 BY MS. MASSIE:
19 Q Based on your experience working with ranges of
20 students in terms of ranges of test scores, students of
21 different races over a period of years, having some sense of
22 how different people answer different questions and what we
23 can deduce from answers to different questions and whether
24 you get a question right or wrong, do you believe that these
25 gaps mean, for example, in the first several questions, that
118
1 black test takers as a whole have a nine percent better
2 vocabulary than white test takers?
3 MR. KOLBO: Objection on foundation.
4 THE COURT: You know, there is no foundation that he
5 has anything other than his own personal opinion. He is
6 here as an expert. If you want to ask him expert questions
7 within his realm, you can ask him. He is not --
8 MS. MASSIE: He has worked with hundreds of
9 questions with test bias, different impacts and outcomes, he
10 has coached teachers, he has studied the test. I don't
11 understand how he could be more of an expert about the
12 relationship between --
13 THE COURT: I do. I made my ruling.
14 MS. MASSIE: Okay. Well, I need to make my record.
15 We need to be able to develop this kind of question in order
16 to make our case.
17 THE COURT: Your objection is noted.
18 BY MS. MASSIE:
19 Q Is the rejection of questions on which black and other
20 minority test takers outperform white test takers the source
21 of the test score gap?
22 A Selection of questions, yes. The selection of
23 questions obviously from this data, which is consistent --
24 this data confirms my experience over thirteen years that
25 the tests are comprised of white preference questions
119
1 virtually entirely. Actually, three of these four tests
2 were entirely white preference questions with a few neutrals
3 and then one test had one black preference question and
4 that's -- this confirms the experience that I have seen over
5 the years working with students on these questions and
6 seeing their scores at the end of the process.
7 Q And is it that all the questions on which white
8 students are outperformed by other students, is it that
9 those questions are weeded out, is that the only source of
10 the gap?
11 A No, the question selection process is not the only
12 source of the test score gap, but in my opinion, it's a
13 central key, critically important source of the test score
14 gap.
15 Q And in terms of the questions that are selected for
16 inclusion on the test, do they mostly fall in the same range
17 of disparate impact on minority students?
18 A No, and the next exhibit outlines that.
19 Q Which is to say, Exhibit 210; is that right?
20 A Yes, Exhibit 210.
21 Q Okay. Tell us about this exhibit, Mr. Rosner.
22 A Actually, we just said range, I was thinking of a
23 narrow band, and in fact, they do fall in a range. It's a
24 wide range.
25 So to clarify my previous answer, I specifically
120
1 looked at the ranges of differences in percentage answering
2 rates and they were large. This simply lists the ranges of
3 answering of differences in percentage answering rates when
4 I did the black/white comparison and when I did the
5 Hispanic/white comparison, and the first paragraph notes
6 that for the 574 white preference questions on the four
7 tests that I have been talking about, there was a range on
8 the white preference questions from one percent to 32
9 percent, and I note that 108 of the 580 questions had a 20
10 percent or more gap favoring whites.
11 By the same token, I took a look at the white/
12 Hispanic comparison and that was also a large gap ranging
13 from one percent to 28 percent. Again, these are all
14 questions that are selected to appear on the test where
15 those who were selecting the questions know this data ahead
16 of time. That's what the pretesting process is all about.
17 Q And what would happen if you picked different
18 questions?
19 A You would get different results, and I actually did
20 that.
21 Q Tell us about that.
22 A I compiled a test using these 580 questions as my
23 test, as my question pool.
24 THE COURT: The 580 questions, the '88, '89
25 questions?
121
1 THE WITNESS: That's right. For 580 questions from
2 '88, '89, for which I had data for each question.
3 THE COURT: When did you do this?
4 THE WITNESS: I did this approximately two weeks
5 ago. I obtained the questions approximately a month ago,
6 having had the data a little earlier than that, I believe.
7 And so I compiled the test form and was able to
8 alter the results considerably, picking from their
9 questions, picking from selected SAT questions.
10 We have a demonstrative exhibit that outlines what I
11 found when I did that.
12 BY MS. MASSIE:
13 Q Is that the one that does not have a number that was
14 handed out for everybody's information?
15 A That does not have a number in the lower right-hand
16 corner.
17 Q Tell us about -- tell us about the test that you
18 constructed.
19 A Well, I called this, at the top, the Reduced White
20 Preference SAT Form, that's a name that I gave it. Again, I
21 state here it's created from the '88, '89 SAT question pool,
22 which was 580 questions, and then I compared how my new form
23 behaved and the kind of results that it generated with how
24 the prior forms behaved; in other words, the results that
25 they generated.
122
1 Q And what did you -- did you impose any constraints
2 on yourself?
3 A Yes. When I constructed the new SAT form I chose
4 145 questions, because that's what was on the SAT at that
5 time. It's since been reduced by a couple of questions.
6 And when I chose those 145 questions, I paid very
7 close attention to question category, question difficulty,
8 question type, so that I created a test that I was
9 comfortable with that looked -- that looked like any
10 other SAT, that looked in fact like the other four SATs.
11 THE COURT: So you have never had any training in
12 creating a test, putting them together, other than your
13 experience in teaching the SAT's and LSAT's?
14 THE WITNESS: We put tests together when we teach
15 the SATs and the LSATs, but no, not with this kind of
16 orientation.
17 THE COURT: And you realize there are people that
18 are trained in that area?
19 THE WITNESS: Yes, psychometricians are trained to
20 pick white preference questions here.
21 THE COURT: Go on.
22 THE WITNESS: And what I found is, if you look at
23 the -- it's a little clearer to explain if you look at the
24 bottom first here. You notice on the left-hand margin, test
25 form zero, test form one, test form two, and test form
123
1 seven. Those were the four different test forms with the
2 date that they were administered the test form.
3 And then what I did was I compiled for each of
4 the tests the average white correct answering percentage
5 for 145 questions on that test, and that's the first column,
6 and then the average black answering correct percentage for
7 the 145 questions on the test, that's the second column, and
8 then the difference, which is the third column.
9 So for example, test form zero, 48 percent was the
10 average white answering correct percentage for all the
11 questions; 36 percent, the average black answering correct
12 question, and 12 percent the average difference.
13 And then if you look down, you'll see I did the same
14 thing for the three other tests. Then I took an average
15 of those four tests, and that is at the very bottom.
16 The 53 percent was the average white answering
17 correct percentage, 40 percent average black answering
18 correct percentage, and the difference I carried out to one
19 more decimal place, which was 13.2 percent or 13 percent,
20 if that would -- if one deems 13 percent more appropriate.
21 Now, if you look at the top, here's what I found
22 when I compiled my new test form, and I knew that it would
23 be reduced, but I didn't know how much until I compiled it.
24 I had the white answered correct percentage was
25 significantly lower at 47 percent, the black answering
124
1 correct percentage remained about the same at 40 percent,
2 but the difference was significantly lower, and 13.2 percent
3 down to 7.8 percent represents a 40 percent reduction in
4 the black/white gap on the four tests, comparing the tests
5 I compiled and the average of the four tests that were for
6 which I had data.
7 Q And you again used questions in the same difficulty
8 range?
9 A I said before, I paid very close -- I didn't pick --
10 I could have picked the 145 questions with the lowest gaps
11 and that would have produced an even lower percent, but
12 it was -- it was fair to construct what would have the
13 structure and appearance and difficulty level and test
14 question variety that a regular test form did, so that's
15 the process that I used.
16 THE COURT: You ran this by somebody like Professor
17 Shapiro to make sure you got this right and that's how they
18 do it in the industry?
19 THE WITNESS: I did not run it by Professor Shapiro.
20 I used my own experience of looking at dozens and dozens of
21 these tests and knowing how they are structured to match up
22 question type and percentage.
23 BY MS. MASSIE:
24 Q Are you trying to do it like they do it in the
25 industry?
125
1 A I was trying to do -- well, the way they do it in
2 the industry is they chose all white preference questions
3 with very significant ranges. I specifically was trying
4 to reduce those ranges. In any set of choices that I had,
5 I was trying to chose the question with the smallest
6 difference, black/white difference, consistent with general
7 difficulty level, question type, question style.
8 Q And what was your purpose in doing this?
9 A The purpose was to highlight and emphasize what a
10 dramatic difference can be made even if you're choosing all
11 white preference questions or virtually all white preference
12 questions, a dramatic difference can be made by paying very
13 close attention to the question gaps and in fact having as a
14 priority trying to minimize those gaps, even though you're
15 working with selected SAT questions.
16 Q So it was a demonstrative exercise?
17 A It was a demonstrative exercise.
18 Q What role do you detect that randomness plays in the
19 numbers that you have told us about today?
20 MR. KOLBO: Objection to foundation, Your Honor.
21 BY MS. MASSIE:
22 Q Mr. Rosner, do you have an opinion on whether these
23 processes are systematic or whether they are in fact
24 affected with a large level of randomness?
25 MR. KOLBO: Same objection.
126
1 THE COURT: Sustained.
2 BY MS. MASSIE:
3 Q Mr. Rosner, in your work with the tests and with
4 students over the years, your work studying the tests and
5 its effect on students, your work training teachers, your
6 work reading material on the tests, have you come to a
7 conclusion about whether the outcomes on the test are the
8 product of design or chance?
9 A Clearly the product of design. That was clear to
10 me before I had this data and I believe this data simply
11 confirms what I have experienced all along in my test
12 preparation work.
13 MR. KOLBO: Well, Your Honor, I'll just move to
14 strike. That was a yes or no question and I'll object.
15 THE COURT: I'll overrule. I'll give it as much
16 credibility as it deserves.
17 MS. MASSIE: There is no need to insult the witness.
18 THE COURT: I'm not insulting anybody. Credibility
19 is up to the Court and that's why I overruled his objection.
20 MS. MASSIE: You have been insulting the witness
21 the entire time he has been on the stand this afternoon.
22 BY MS. MASSIE:
23 Q Mr. Rosner, could you please summarize for us your
24 view of the relationship between question selection and
25 the overall test score gap by race on the LSAT?
127
1 A Sure.
2 MR. KOLBO: Object to foundation, Your Honor.
3 THE COURT: I'm sorry, I didn't hear the question.
4 BY MS. MASSIE:
5 Q Can you please summarize for us your opinion on the
6 relationship between question selection procedures and the
7 score gap, the test score gap by race on the LSAT?
8 A The question selection procedures on the LSAT are
9 identical to this question selection procedure that I have
10 been talking about for the past hour, and that is, white
11 preference questions are chosen for the LSAT in a great,
12 great predominance.
13 MS. MASSIE: Thank you, Mr. Rosner.
14 THE WITNESS: By the way, Your Honor, I don't feel
15 insulted by you.
16 THE COURT: Thank you. I didn't mean to insult you.
17 MR. PAYTON: Can I have just five minutes?
18 THE COURT: Sure, you can have five minutes. Do you
19 want to take a real break for five minutes?
20 MR. PAYTON: Yes.
21 THE COURT: We will take five minutes. Stand in
22 recess.
23 COURT CLERK: All rise.
24 (Recess taken at 2:42 p.m.)
25 (Back on the record at 2:53 p.m.)
128
1 COURT CLERK: All rise.
2 THE COURT: You may be seated. Okay.
3 CROSS EXAMINATION
4 BY MR. PAYTON:
5 Q Mr. Rosner, I actually just want to ask you some
6 questions about what you testified to when we were last
7 here, which is what your business is.
8 Your business, the Princeton Review, is about
9 improving test scores, SAT scores, LSAT scores for people
10 who are going to take the test, is that it?
11 A My business is the Princeton Review Foundation and the
12 purpose of both the Foundation and the Princeton Review is
13 as you stated, yes.
14 Q Let me just focus on what the Princeton Review then
15 does, because its purpose is just to increase test scores,
16 SAT, LSAT. We will come to the Foundation later.
17 A Okay.
18 Q Okay. And that's also the business of Kaplan; is that
19 right?
20 A That is one of their businesses, yes.
21 Q What's the full name of Kaplan?
22 A Stanley Kaplan Educational Centers, I believe is the
23 full name.
24 Q Okay. And I want to focus on just that part of the
25 business of Kaplan and of the Princeton Review that focuses
129
1 on increasing LSAT scores; okay?
2 A Okay.
3 Q Mr. Washington, I think two weeks ago, referred to
4 that as boosting LSAT scores?
5 A He may have.
6 Q Okay. You seem to react badly to that. Is that --
7 A Well, improving, boosting. One can generate a number
8 of synonyms that would refer to it.
9 Q All right. So is it fair to say that Princeton Review
10 and Kaplan are the two major companies that improve or boost
11 LSAT scores?
12 A Yes.
13 Q And Kaplan is the largest company, Princeton's
14 smaller, but still a large company in this business?
15 A Yes, they are the two -- they are the only two
16 entities of substantial size who do this on a national
17 basis.
18 Q Okay. And am I right that the students that are
19 likely to go to either the Princeton Review or to Kaplan
20 are those students who are actually looking to apply to the
21 more selective law schools, Michigans, Harvards, Yales, not
22 all of them, but are they disproportionately represented
23 among those applicants?
24 MR. KOLBO: I'll object to the foundation, Your
25 Honor.
130
1 THE COURT: If he knows.
2 BY MR. PAYTON:
3 Q If you know. If you know where they apply. Is that
4 speculative?
5 A There is a range and I'm not sure whether -- there are
6 a considerable number and a considerable percentage applying
7 to highly selective institutions. I'm not sure that it's
8 disproportionate.
9 Q Okay. Now, I think you told us that some 70,000
10 students take the LSAT in any given year, that was a rough
11 number?
12 A Right.
13 Q And of the 70,000, some 45,000 are white, is that
14 correct?
15 A That sounds correct to me, yes.
16 Q And I think you said that between Kaplan and the
17 Princeton Review together, you serve about 25,000 of those
18 45,000 white students?
19 A Actually, what I -- the answer is yes, almost, because
20 there are some students who we serve who don't apply.
21 Q Yes, you said some who back out or for whatever
22 reason.
23 A Yes.
24 Q Very small number?
25 A Yes, a relatively small percentage of students who we
131
1 serve don't apply.
2 Q Okay. And I believe you also said that neither the
3 Princeton Review nor Kaplan has more than a token number
4 of black or Hispanic students in their courses?
5 A That's generally true, yes.
6 Q You said you were sad to say that the black or Latino
7 representation was just token numbers?
8 A Yes.
9 Q If any.
10 A That's correct.
11 Q So let me just add that up. These are in essence
12 booster courses for all-white test takers of the LSAT?
13 A Primarily and predominantly, yes.
14 Q And you testified that some studies had been done for
15 both Kaplan and Princeton that indicated that in fact the
16 courses do boost the LSAT scores. I think you said that
17 there was a study of Kaplan and that it boosted LSAT scores
18 by seven points. Do you remember that?
19 A That's correct, and that's appended to my expert
20 report in this matter.
21 Q And I believe you said that there was a similar study
22 done for Princeton, and that it indicated that Princeton
23 boosted LSAT scores by six to seven points?
24 A Yes. I think actually the figure was 6.7 points, but
25 yes.
132
1 Q That's 6.7, so both about seven?
2 A Both about seven.
3 Q And I think you then told us that a seven point boost
4 was quite significant, and I think you said that if it
5 were in the middle range, around 150, that would be a
6 27 percentile difference; is that right?
7 A That's correct, and the 27 percent jump related to a
8 score of 146, I believe, but yes, you get the higher jumps
9 in the middle of the range.
10 Q And I also think you said that this boost, seven point
11 boost in LSAT scores, didn't reflect in any way an increased
12 aptitude for law or law school?
13 A Not at all, as far as I could tell, based on my
14 experience.
15 Q So it's a synthetic boost?
16 A It's a synthetic boost compared to the real world.
17 It's a boost in terms of better understanding and better
18 responding to the test.
19 Q Now, do I then understand this correctly, that given
20 the dramatic boost, say 27 percentile dramatic boost in LSAT
21 scores, on average, for your all white, 25,000 white student
22 clientele in a given year, that you have literally no black
23 or Hispanic participants in your courses?
24 A Token participants.
25 Q You have token participants. Am I right, would you
133
1 agree that the Princeton Review and Kaplan are responsible
2 for much of the test score gap in LSAT scores between
3 African Americans and Hispanics and white students?
4 MR. KOLBO: Object to foundation, Your Honor.
5 THE COURT: Overruled. If he knows.
6 A Kaplan and the Princeton Review are responsible for a
7 portion of the test score gap, which is significant. There
8 was one part of your -- there was an assumption, though,
9 in your question that I need to address. It was a long
10 question.
11 You pointed out the 27 percent boost. You picked
12 the maximum boost --
13 Q I understand that.
14 A -- along the range, and in the context of the question
15 there might have been an assumption that that was a typical
16 boost. Seven points is 27 percent maximum. It's less than
17 that elsewhere.
18 Q As you move up or down?
19 A Yes.
20 Q But if you take it in the middle range, which you
21 directed us to the middle range so you could see what the
22 maximum boost would be --
23 A Right.
24 Q -- if you're in the middle range, it's a 27 percentile
25 boost?
134
1 A No question about that. I think as you made a general
2 set of assumptions in asking your question, though, you
3 rejected one assumption which maximized an effect rather
4 than took a typical effect.
5 MR. PAYTON: I take your point. Thank you very
6 much.
7 THE COURT: Plaintiff, any questions?
8 MR. KOLBO: A few questions, Your Honor.
9 CROSS EXAMINATION
10 BY MR. KOLBO:
11 Q Good afternoon, Mr. Rosner. My name is Kirk Kolbo and
12 I'm one of the lawyers representing the Plaintiff in this
13 case and I just had a few questions for you.
14 First of all, just a clarification. My
15 understanding was, and correct me if I'm wrong, that your
16 estimate is that Kaplan and Princeton Review serve combined
17 about 25,000 students total; is that correct?
18 A Total annually, yes.
19 THE COURT: That includes, I'm sorry, not only LSAT,
20 though.
21 THE WITNESS: No, that's just LSAT, Your Honor,
22 and the breakdowns were approximately 10,000 for Princeton
23 Review, approximately 15,000 for Kaplan, give or take, those
24 were my estimates.
25 BY MR. KOLBO:
135
1 Q And that's 25,000 students including all minority
2 students that attend these courses; correct?
3 A Yes.
4 Q Not just the white students?
5 A That's true. It includes the token number of minority
6 students.
7 Q But you also testified that there are about 45,000
8 white students all together nationally that take the LSAT
9 score; is that correct?
10 A That's the data I have seen from Law Services, yes.
11 Q So there are about 20,000 white students each year
12 that take the LSAT test without taking either the Princeton
13 Review or the Kaplan course; is that correct?
14 A That's correct.
15 Q And then there is another -- over and above that there
16 is another 15,000 of presumably different races that don't
17 take either of those two courses; is that correct?
18 A That's correct. Actually, it's -- if I can amend the
19 prior answer, it's slightly more than 20,000, because there
20 are some who take our course who don't apply, but you're in
21 the ballpark.
22 Q If my math is right, and it's been wrong before up
23 here in front of everybody, there are about twice as many
24 students who don't take either one of these preparation
25 courses as students that could take them; is that correct?
136
1 A Define --
2 Q Total of 70,000, of those about 25,000 students take
3 one or two of these preparation courses?
4 A And would you repeat your question?
5 Q There is roughly, by my count, about 45,000 students
6 that don't take either one of these test preparation
7 courses; correct?
8 A That's right, however, in your hypothetical you're
9 missing the fact that there are regional courses of
10 considerable intensity that I would consider. We were
11 talking about just national courses. There are regional
12 courses of considerable intensity that I would deem to be a
13 fully equivalent or a near equivalent to a Princeton Review
14 or a Kaplan course, so actually, there are more than 25,000
15 students taking intensive courses, with an additional couple
16 of thousand taking a regional course or a local course
17 rather than a national course.
18 Q Okay. Well, still it's then probably no more than
19 30,000 students all together that are taking some kind of
20 a preparation course?
21 A If you're talking about intensive preparation courses,
22 that would be a reasonable estimate, perhaps slightly on the
23 high side.
24 Q And another 45,000 or so who aren't taking any of
25 these type of courses?
137
1 A Slightly more than 45,000, again, because not
2 everybody is applying.
3 Q And I'll let everybody else do the math on that.
4 Is it fair to say it's possible to do well on the
5 LSAT test without taking any of these preparation courses?
6 A Yes.
7 Q There are among that 45,000 plus number of students,
8 there are many of them that do well on the test; correct?
9 A There would be a considerable number who do well
10 on the test without taking an intensive test preparation
11 course, that's correct.
12 Q And one can actually study for the test, the LSAT
13 test, without taking one of these organized preparation
14 training courses; correct?
15 A Certainly.
16 Q You could spend, what is it, 40 or 50 bucks to buy one
17 of the books on the LSAT and study that way?
18 A Yes, you can buy released prior tests, you can buy
19 test preparation books in the bookstore, and one can study
20 on one's own for the test, that's correct.
21 THE COURT: They even have a CD-Rom, don't they?
22 THE WITNESS: CDR's, yes, Your Honor, and there are
23 on-line courses now, so there has been a change in the last
24 couple of years in terms of software and even a greater
25 change in the last year or so with on-line resources.
138
1 BY MR. KOLBO:
2 Q And am I correct that -- correct me if I'm wrong, but
3 is it your understanding that the testing organizations,
4 ETS and ACT, that's the other one?
5 A ACT produces their own high school exam and from time
6 to time has written questions for the LSAT.
7 Q My recollection from your prior testimony at trial
8 here is you're not sure who it is, which of these testing
9 organizations that's writing the LSAT these days; correct?
10 A I lost track of that years ago, yes.
11 Q But is it -- am I correct that it's the position of
12 these testing organizations that test preparation, taking
13 courses like Kaplan's and Princeton Review, does not
14 significantly improve test, actual test performance, is
15 that a fair summary of their position?
16 A That's a fair summary of the position of every test
17 manufacturer. The SAT, ACT, Law Services, the AAMC produces
18 the MCAT, yes.
19 Q And they have taken that position, as you understand
20 it, based on studies that they have had done; correct?
21 A They do their own studies, yes. Are those studies the
22 basis of their position, I think marketing is the basis of
23 their position, but they do have studies to support their
24 position, also.
25 Q Well, do you have any reason to believe that they are
139
1 making this claim in contradiction to the studies that they
2 themselves have produced or sponsored?
3 A Yes, I do.
4 Q And what's that based on?
5 A That's based on reading their studies, which I think
6 are almost humorous in the flaws and fallacies that exist.
7 Q Well, studies themselves at least on their face, they
8 support the ETS and the ACT position?
9 A ETS paid researchers to reach conclusions that support
10 ETS's marketing position, and that's the same for Law
11 Services and the same for every test company, yes. That
12 is -- that is as consistent as the test score gap. That
13 always happens.
14 Q And you disagree with the conclusions that they have
15 reached?
16 A I disagree with the methodology.
17 Q You're not a --
18 A And I'm sorry, and I disagree, yes, and I disagree
19 with the conclusions.
20 Q Just to be clear, and I think this is clear, but
21 you're not formally trained in statistics?
22 A No, I'm not.
23 Q Or psychometrics?
24 A No, I'm not.
25 Q Or psychology?
140
1 A No, I'm not.
2 Q Or education?
3 A I was close to an educational certification to teach,
4 but never got it, so no, I'm not.
5 Q And I think I said this, but if I missed it,
6 psychology, you have no formal --
7 A You didn't miss psychology, and I said that I was not.
8 Q I apologize for repeating.
9 A That's okay.
10 Q And as I understand it from reading your report, it's
11 your position that these testing organizations are motivated
12 in their position that these preparation courses are not
13 significantly helpful because it's profitable for these
14 testing organizations to maintain the perception out there
15 that it doesn't help to prepare for these tests; is that a
16 fair statement?
17 A Not entirely.
18 Q I didn't state it very well, so why don't you tell me
19 what you believe motivates these testing organizations to
20 make these marketing claims that you believe are based on
21 erroneous data.
22 A It's a very simple concept, and the concept is, if
23 you're putting a number out there, a test score which the
24 University of Michigan or any other school is going to rely
25 upon, there has to be a certain sense that that number is
141
1 strong and solid. If there is a sense created that that
2 number can be moved around depending on whether somebody
3 can afford $1,000 for a test preparation course, that number
4 has very little meaning.
5 So there is a very strong marketing and sales
6 imperative, but more personally, an image imperative for
7 the test companies to take a position, and poorly designed
8 studies that conclude that test preparation doesn't work,
9 because the inverse of that that they do work would cut to
10 the heart of reliance on this number, which in fact is not
11 consistent or firm and can be moved pretty dramatically
12 depending on interventions.
13 Q Are these testing organizations, are they for-profit
14 enterprises, as far as you know?
15 A ETS, the largest of the organizations we're talking
16 about, is a non-profit organization.
17 Q Well, part of it, as I understand, though, you're
18 suggesting is that the testing companies in order to
19 justify -- one of the reasons, one of the ways to justify
20 the continued existence and use of these tests is to
21 disparage the value of test preparation courses, is that
22 a fair statement?
23 A That's a fair statement and reflects their behavior
24 over many years.
25 Q Okay. Now, Princeton Review, not the Foundation, but
142
1 Princeton Review, is a for-profit company; is that correct?
2 A Princeton Review is a for-profit company, yes.
3 Q And it makes a lot of money by giving test preparation
4 courses around the country?
5 A I can state emphatically that it doesn't make enough
6 money to satisfy the Princeton Review folks, but it does
7 generate a profit most years in running test preparation
8 courses.
9 Q Okay. And would you agree that the Princeton Review
10 has an interest in persuading and convincing and marketing
11 to people that test preparation courses in fact are valuable
12 in increasing the actual test scores of test takers?
13 A Yes, as does Kaplan.
14 Q And Princeton Review has produced studies showing that
15 use of review courses improves their scores and the testing
16 companies have produced studies saying that in fact they
17 don't improve significantly in performance; correct?
18 A That's correct, but Mr. Kolbo, you have been saying
19 the Princeton Review, and I'm assuming in the last couple
20 of questions you're talking about our for-profit side, you
21 did say that, and the answers to the questions would be
22 different if you talked about the Princeton Review
23 Foundation. I just want to make that point.
24 Q Oh, sure. But who is paying for these studies that
25 support the Princeton Review Foundation on the value of
143
1 these tests -- I'm sorry -- value of test preparation, who
2 is paying for that, the Princeton Review Foundation?
3 A No, you said in your initial question, before you
4 corrected, it was the Princeton Review Foundation. It's
5 the Princeton Review is paying for the study that supports
6 the Princeton Review score gains. The Foundation plays no
7 role in that other -- that I know about it.
8 Q Just to be clear, the Princeton Review certainly has a
9 financial motive for persuading the public that there is a
10 value served by taking these prep, test preparation courses;
11 correct?
12 A Correct.
13 Q Do you have any idea what percentage of the students
14 who apply to the University of Michigan take a test
15 preparation course of some kind, whether it's Kaplan or
16 Princeton Review or one of these regional courses?
17 A No.
18 Q And you have no idea what percentage, I assume, from
19 your answer, in terms of breaking down the percentages of
20 African Americans, Asian Americans, white students, Hispanic
21 students, Native Americans, you have no idea what percentage
22 of those students in those groups applying for Michigan law
23 school take the preparation course; correct?
24 A I don't have specific information as to Michigan, but
25 when you -- as soon as you talk about students overall, as
144
1 soon as you start talking about black students and Latino
2 students, I know the rate at which those students take test
3 preparation courses, I testified several times they take
4 them at token rates, so one can make an inference as to what
5 would happen with those groups applying to Michigan, but I
6 don't have specific information.
7 Q And I just want to -- I think I understand this, but I
8 want to be clear. You have testified several times that
9 only token members of certain minorities take --
10 A Yes.
11 Q -- Princeton Review courses. I assume these courses
12 are open to students of all races and ethnicities?
13 A Students are -- the courses are open to students of
14 all races.
15 If I can amend my last answer, no one, I believe,
16 would know percentages of students in various racial or
17 ethnic groups who have taken courses who are applying to
18 the University of Michigan. I stated in my direct testimony
19 two weeks ago that there are indeed motivations for students
20 not to reveal that information.
21 Q Well, someone can certainly ask, could they not, do a
22 survey?
23 A Oh, that's how -- you have just hit upon something we
24 talked about before. That's how the test companies do their
25 studies. They ask students whether you have taken a course,
145
1 and they assume the response that they are getting is
2 accurate, which is laughable.
3 Q You're not suggesting that it's impossible somehow
4 to commission a survey to find out how many individuals
5 applying to the University of Michigan typically each year
6 have taken a test preparation course?
7 A I assert that that's impossible.
8 Q You assert that it's impossible?
9 A Yes.
10 Q And that's not a question that could, for example, be
11 asked on an application?
12 A Oh, it can be asked. It's just, there is no reason to
13 assume that the answer would be honest or accurate, because
14 the student, as I said, has a motivation not to reveal that
15 information.
16 Q And so because of that --
17 THE COURT: I don't understand. What would be the
18 motivation if they are doing blind surveys? We do blind
19 surveys all the time, send out a blind survey all the time.
20 And when I say blind, you don't have to put your name, no
21 numbers, no nothing. They send them out on Federal Judges
22 at least every, I forget, every four years.
23 THE WITNESS: Understood, Your Honor. The basis of
24 that answer is having discussions with perhaps 50 or 60 or
25 70 students over the last ten years who have said to me,
146
1 there is this question, have I taken a course, what do I
2 say. And I say, you know, you answer the question. I
3 can't tell you to answer it any way but honestly. And the
4 students say, well, they have no business of knowing, I'm
5 not going to tell them.
6 THE COURT: But that's the same thing if you get
7 any kind of survey, some people are honest and some people
8 aren't.
9 THE WITNESS: I think --
10 THE COURT: You think law students are less honest?
11 THE WITNESS: Not to argue with you, Your Honor,
12 but I --
13 THE COURT: I'm not arguing with you. As I say, we
14 all get surveys.
15 THE WITNESS: I think I'm pointing out a difference
16 in this kind of a survey. It would be more -- it would
17 even -- there would even be a greater difference if the
18 University of Michigan asked this.
19 THE COURT: What's so bad about saying, yes, I took
20 a bar review course or I took an LSAT course?
21 I don't want to get into it with you. That's okay.
22 That's okay.
23 THE WITNESS: The short answer to that, Your Honor,
24 is that there are lots of disparagement of these courses
25 coming from the test companies and students can get the
147
1 feeling very easily that I'm not maybe supposed to be doing
2 this.
3 BY MR. KOLBO:
4 Q Do you assume that because a student believes that an
5 answer to a question will be contrary to their interests
6 that they will give a false answer on an application?
7 A As a general principle, no, but in the specific
8 instance, I told you the kinds of experiences that I have
9 had where students indicated to me that they weren't going
10 to give this information, because they had a funny feeling
11 about it and they felt the entity asking the information
12 didn't have any right to know it.
13 Q Another question on preparation, and I'll move on
14 very briefly to something else.
15 We talked about how some students may take one of
16 these formal courses, like Princeton or Kaplan or one of
17 these regional courses and how others students might just
18 buy a book and study that way. Have you done any -- have
19 you conducted any studies to determine or are you aware of
20 any studies that assess the differences in performance
21 between those types of preparation?
22 A I think law -- if we talk about the LSAT, Law Services
23 does a study on the effect of preparation, like all the
24 other companies, and they do this survey data, and I
25 believe -- I believe they do have separate questions on,
148
1 for example, did you prepare using books, did you take a
2 course, so in the context of the methodology that they use,
3 they may have some conclusions as to that distinction, so.
4 Q You don't know what those are?
5 A I don't know what those are.
6 Q You have testified last time and this afternoon about
7 often referring to a number of questions, SAT questions;
8 right?
9 A Yes. Those were all SAT questions. I was unable to
10 obtain LSAT questions.
11 Q And these are SAT questions that are about ten or
12 eleven years old or so?
13 A Yes, twelve or -- in some cases, thirteen years old.
14 Q Now you say that you have testified that you have not
15 had access to LSAT questions; correct?
16 A Let me be clear. LSAT questions, the content of the
17 questions, is available. It always has been. I'm sorry,
18 since 1980 it has been. What is unavailable is the percent
19 answering rate breakdowns disaggregated by ethnicity, and
20 when I talk generally about unavailability, that's what that
21 data is that I'm referring to that is unavailable from Law
22 Services completely, and it's unavailable from ETS except
23 for these four tests.
24 And I checked with Dr. Shapiro. The data that he
25 was talking about, Your Honor, does not include this racial
149
1 breakdown data. So it's entirely unavailable other than
2 what we have talked about in the courtroom here with regard
3 to the SAT and the LSAT.
4 Q But for example, rejected questions, those are a
5 certain form of question that one can take a look at,
6 correct, you have done that with SAT questions?
7 A I have done that with the dozen or so questions that
8 have been made available to me through ETS, through the
9 newspaper.
10 Q And they simply have not been available to you from
11 the LSAT test; correct?
12 A Tried to get them. Was told that they are not
13 publicly available.
14 Q Presumably.
15 A Again, referring "they" to the statistics, not the
16 questions.
17 Q "They" being the testing organization; correct?
18 A Correct.
19 Q And you're not sure which one that is, even, ETS of or
20 its ACT?
21 A No, I'm sure what that is. That's Law Services.
22 Q Law Services, okay.
23 A And they administer the test. The other company
24 subcontracts test writing, test question writing.
25 Q Okay. So Law Services would have -- someone in
150
1 Law Services has this detailed breakdown on performance
2 among racial groups on these tests; correct?
3 A Yes, when they do their DIF analysis, they have them.
4 Q And Law Services has access to these rejected
5 questions that are rejected for various disparate impact
6 reasons; correct?
7 A If the test companies supply them with sets of
8 rejected questions, they have them, and I assume that
9 they have them, or certainly easy access to them.
10 Q And you were here this morning when Professor Shapiro
11 testified?
12 A Yes.
13 Q And explained that he has been able to see, get access
14 to some of these, at least SAT questions in the course of
15 litigation, correct, Illinois litigation, for example?
16 A Yes. Yes, I recall his testimony on that, but Mr. --
17 Professor Shapiro informs me, and again, I checked on this
18 at the break, that other than these four tests, no data
19 like this is available for any SAT question.
20 Q Well, presumably, if a lawsuit was pending and one
21 of the parties served a subpoena on Law Services, there
22 would be somebody there who could answer questions based
23 on this data that's not available to you; correct?
24 A There would be somebody there who would refuse to
25 answer those questions, yes, but who would be capable of
151
1 answering the questions.
2 Q And who would be subject to a subpoena?
3 A Yes.
4 Q You're a lawyer; correct?
5 A Pardon?
6 Q You're a lawyer?
7 A Yes, I am.
8 Q Law Services could be compelled by a Court to make
9 certain disclosures?
10 A They could theoretically be compelled by a Court, yes.
11 Q That just hasn't happened in this case as far as you
12 know?
13 A That hasn't happened in any case with regard to the
14 SAT of which I'm aware or in any case with regard to the
15 LSAT of which I am aware.
16 Q Just to be clear on this, there is -- to your
17 knowledge, there has been no subpoena served on Law Services
18 in connection with this case, correct, for this data?
19 A I could respond to that, but it would be hearsay. I
20 have no direct knowledge.
21 Q Well, I have no knowledge of a subpoena, Mr. Rosner.
22 I have just a couple more questions for you
23 regarding the exhibits that you used. Each one of them,
24 starting with page -- or Exhibit 202 through 208, 209, at
25 the top there is a designation, black or white preference
152
1 question. That's your designation, correct, to call it a
2 black or white preference question?
3 A I'm sorry, which one were you referring to?
4 Q Well, you can take any one of these, starting with
5 Exhibit 202.
6 A Okay. Sure.
7 Q You prepared these exhibits?
8 A I prepared these exhibits, and yes, I tried to be
9 clear in my direct testimony that the label, Black or White
10 Preference Question was my label, and I defined it, I hoped,
11 to the understanding of everyone.
12 Q Okay. These aren't the newspapers that you're relying
13 on that are using this term?
14 A No, again, I wanted to emphasize that this was my
15 terminology. I hope I made that clear.
16 Q And you're responsible for the way these exhibits were
17 prepared?
18 A Yes, I'm responsible for the way -- I'm responsible
19 for the content of these exhibits. 202 through 209 were
20 prepared by me on my computer, with the exception of the
21 figure on 209, which was my handiwork with a ruler and
22 a pen.
23 Q Okay. Could you turn to Exhibit 204, please?
24 A Sure.
25 Q And I'm just going to ask a couple questions more out
153
1 of curiosity than anything else, but I'm just about done,
2 so I don't think I would be wasting much time.
3 I was just reading ing this, taking a look at
4 this question while you were testifying, and the -- as I
5 understand it, the correct answer is supposed to be D; is
6 that right?
7 A Yes, that's right.
8 Q And just to take a look at these questions, there is a
9 value for X that one would need to put in here to determine
10 what the correct answer is; correct?
11 A That's right.
12 Q Isn't the correct answer for X, correct value for
13 X the number three?
14 A Oh, you fell into the trap.
15 Q How's that?
16 A Shall I explain?
17 Q Sure.
18 A That's the way -- that's one of the traps that -- one
19 of the common traps that is built into questions. Both
20 black preference questions and white preference questions
21 have traps. The trap here is that X could be three or
22 negative three and that's why D is the correct answer.
23 Q So I got that question wrong?
24 A You're opening yourself up for a comment which I
25 won't make.
154
1 MR. KOLBO: Your Honor, I have nothing else.
2 THE COURT: Any further questions?
3 MS. MASSIE: Just a couple questions.
4 REDIRECT EXAMINATION
5 BY MS. MASSIE:
6 Q Mr. Rosner, have you in your teaching and your
7 supervision of teachers known people's scores to improve
8 by give or take seven points on the LSAT over the course
9 of a Princeton Review class?
10 A Yes, that confirms with my experience. I'm
11 particularly concerned about the impacts of white preference
12 tests and white preference questions to black students and
13 so I have paid very, very close attention to that over the
14 last several years. Seven points is a typical improvement
15 for black students, although the average improvement would
16 be maybe -- for black students maybe five, five and-a-half
17 points, something in that range, based on my anecdotal
18 observations. I haven't done a formal study on that.
19 Q When you say anecdotal observations, you have taught
20 hundreds of students and you have been attentive to the
21 degree to which their scores improve; is that correct?
22 A Yes. It's a large body of information, though
23 anecdotal.
24 Q And it confirms the more formal studies that have
25 been commissioned, maybe I shouldn't be saying this in the
155
1 plural, at least the one study that was commissioned by the
2 Princeton Review?
3 A There is a general confirmation, yes, with possibly a
4 relatively small difference, in my experience.
5 Q And in your experience, Professor Shapiro has
6 testified about the process of test equating so that a
7 score today means more or less the same as a score several
8 or even more years ago. Have you in the course of your
9 teaching noticed that there are -- that results remain
10 relatively constant?
11 A Yes, and test score gap is consistent. It's
12 consistent, and one of the reasons it is is because these
13 are white preference tests comprised of white preference
14 questions and that remains. It's clear to me that that
15 remains consistent from 1988 to today on both the SAT
16 and the LSAT.
17 Q You clarified that the four SAT's you spoke about
18 earlier in your direct were the only four SAT's that have
19 been made publicly available with percentages, answering
20 correct percentages of people answering correct by race.
21 A Yes.
22 Q There aren't any questions like that available for
23 the LSAT; right?
24 A Base on my discussions with Law Services, they tell me
25 that there are none publicly available, that none of that
156
1 data with regard to the LS -- with regard to LSAT questions
2 is publicly available there.
3 Q And we did try to get that information through legal
4 process in this litigation, didn't we?
5 A We discussed -- you and I discussed serving a subpoena
6 on Law Services, and I remember we were -- we had a
7 discussion on the question, the kinds of questions that
8 would be in that subpoena, and I was told by you that the
9 subpoena was served, although I never saw a copy of it, or
10 that's why I said it was hearsay that that kind of thing
11 happened.
12 Q And I also told you that it got stonewalled; right?
13 A Yes, you told me the response, which was no
14 information.
15 MS. MASSIE: Thanks.
16 THE COURT: You may step down. Thank you.
17 Appreciate you coming back to our nice weather.
18 THE WITNESS: Try to arrange some sunshine next
19 time. Thank you.
20 THE COURT: Because your weather sometimes isn't
21 much better than yours.
22 THE WITNESS: Sometimes.
23 THE COURT: Next witness, please.
24 MS. MASSIE: Jodi Masley will be examining our next
25 witness.
157
1 THE COURT: Before the day is out, let's kind of get
2 an idea of how many more witnesses. We can do it now or
3 later, I don't care, before you leave.
4 I'll tell you why, I have a jury that I have to pick
5 before next Thursday. I don't have to try the case, but I
6 have to pick a jury before -- I think it's next Thursday or
7 Wednesday, we will talk about that.
8 MS. MASSIE: Okay. We can talk about it either now
9 or later. Do you have a preference?
10 THE COURT: Right now, I don't care, since you're
11 standing there.
12 MS. MASSIE: We have two witnesses tomorrow. I have
13 spoken with Mr. Kolbo and Payton. We have two, Crystal
14 James and Walter Allen. Crystal James is a second year
15 student at the UCLA School of Law and Walter Allen is the
16 team leader for the team that conducted our campus climate
17 study.
18 On Thursday we have Eric Foner, Professor of History
19 from Columbia University, designated as a witness by the
20 University. We have Eugene Garcia, who is the Dean of the
21 Graduate School of Education at UC Berkley, and David White,
22 who is an expert on testing, some testing issues.
23 And on Friday, we have Rick Lempert, who you heard
24 from as a fact witness, but he is also an expert studying
25 the -- comparing the career trajectories and career
158
1 success of --
2 THE COURT: I know who he is.
3 MS. MASSIE: You know him. And Frank Woo, Faith
4 Smith, and we are also working out which -- I have discussed
5 with Counsel several students we may call in addition.
6 THE COURT: Okay. Mr. Payton, did you want to say
7 something?
8 MR. PAYTON: At some point we should talk about
9 rebuttal and the rest of it, but we don't have to do it now.
10 MS. MASSIE: Let's do that later.
11 THE COURT: But let's do it before we leave, because
12 I want to pick this jury sometime next week. I wouldn't
13 start the case until we finished with this one, but I have
14 to at least pick a jury. The farthest they are coming from
15 is Milan, so --
16 I'm sorry. Stand for one more second and I will
17 swear you in. Don't look so scared.
18 -- --- --
19 C O N C E P C I O N E S C O B A R,
20 having be