In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
1
1 UNITED STATES DISTRICT COURT
2 FOR THE EASTERN DISTRICT OF MICHIGAN
3 SOUTHERN DIVISION
4
5 BARBARA GRUTTER,
6 For herself and all others
7 Similarly situated --
8 Plaintiff,
9 -v- Case Number: 97-CV-75928
10 LEE BOLLINGER, JEFFREY LEHMAN,
11 DENNIS SHIELDS, and REGENTS OF
12 THE UNIVERSITY OF MICHIGAN,
13 Defendants,
14 And
15 KIMBERLY JAMES, et al.,
16 Intervening Defendants.
17 ------------------------------------/ VOLUME 12
18 BENCH TRIAL
19 BEFORE THE HONORABLE BERNARD A. FRIEDMAN
20 United States District Judge
21 238 U.S. Courthouse & Federal Building
22 231 Lafayette Boulevard West
23 Detroit, Michigan
24 SATURDAY, FEBRUARY 10, 2001
25
2
1 APPEARANCES:
2
3 FOR PLAINTIFF: Kirk O. Kolbo, Esq.
4 R. Lawrence Purdy, Esq.
5
6 FOR DEFENDANTS: John Payton, Esq.
7 Craig Goldblatt, Esq.
8 Stuart Delery, Esq.
9 On behalf of Defendants.
10
11 George B. Washington, Esq.
12 Miranda K.S. Massie, Esq.
13 On behalf of Intervening Defendants.
14
15 COURT REPORTER: Joan L. Morgan, CSR
16 Official Court Reporter
17
18
19
20
21
22
23
24
25
3
1 I N D E X
2
3 WITNESS: PAGE:
4
5 ORAL ARGUMENTS RE: WITNESS, GAIL HERIOT
6 By Mr. Goldblatt 5
7 By Mr. Washington 10
8 By Mr. Kolbo 11
9
10 REBUTTAL WITNESS CALLED ON BEHALF OF PLAINTIFF
11 KINLEY LARNTZ, PhD
12 Direct Examination by Mr. Kolbo 19
13 Cross Examination by Mr. Delery 56
14 Cross Examination by Ms. Massie 101
15
16
17 E X H I B I T S
18 MARKED RECEIVED
19 Trial Exhibit 225 54
20 Trial Exhibit 226 54
21 Trial Exhibit 227 54
22
23
24
25
4
1 Detroit, Michigan
2 Saturday, February 10, 2001
3 (At or about 8:33 a.m.)
4 -- --- --
5 THE COURT: Let's talk first, if we might just take
6 a second, and we have --
7 MR. WASHINGTON: Could we wait just a minute? I
8 think Ms. Massie is having car trouble, and she is going to
9 be handling today, so if we could wait for a minute, I'd
10 appreciate it.
11 THE COURT: Oh, I'm sorry. I didn't notice she
12 wasn't here. Of course, it may go much quicker.
13 MR. WASHINGTON: I understand.
14 THE COURT: I'm kidding. No, we will definitely
15 wait.
16 Can we argue the motion in terms of the witness,
17 Gail Heriot, or is Ms. Massie handling that, also?
18 MR. WASHINGTON: I can argue that, I suppose.
19 THE COURT: It's up to you.
20 MR. WASHINGTON: We are really supporting the
21 University's position, so, yeah, fine. We would be happy
22 to do that.
23 MR. PAYTON: Your Honor, Mr. Goldblatt will be
24 arguing this motion.
25 MR. GOLDBLATT: Good morning, Your Honor.
5
1 THE COURT: Good morning.
2 MR. GOLDBLATT: The Defendants have brought this
3 motion to exclude the proffered expert testimony of Gail
4 Heriot. She is a proposed expert submitted by the
5 Plaintiffs. She has submitted an expert report in this
6 matter. I believe it's Proposed Exhibit 136, Your Honor,
7 and that's a report about diversity.
8 It's a report about the opinions of Professor Heriot
9 that policies like the ones at issue here that involve the
10 consideration of race as a factor in admissions don't bring
11 about benefits, don't improve, in sum, the educational
12 experience.
13 As the Court is aware, the Defendants have prepared
14 a case directed exactly at that issue. We believe it is the
15 most rigorous, comprehensive, thorough-going empirical case
16 proving that students who are brought together on campuses
17 that are diverse in many ways, including with respect to
18 race, receive a better education. They learn better, the
19 schools prepare better citizens, the law schools prepare
20 better lawyers.
21 And that interest, that diversity is compelling, not
22 just as that term is used as a matter of legal doctrine, but
23 in its plain English sense that that interest is compelling.
24 It's a case, Your Honor, that the Defendants and the
25 University are quite proud of. We would be delighted to put
6
1 on that case.
2 And we had, of course, a number of witnesses
3 prepared to speak to that issue, Professor Garin, Darren
4 Bok, Thomas Shagru, Albert Camareo. The Court hasn't seen
5 any of those witnesses, and the reason for that, of course,
6 Your Honor, is that this Court's ruling on December 22nd
7 said that that question was out of this trial.
8 Our testimony, the reports that we put in, are part
9 of the summary judgment record, and everyone agrees, they
10 are going to travel with this case as it makes its way
11 through the Federal Judiciary, and that's the Court's ruling
12 and we will, of course --
13 THE COURT: You mean everyone is not accepting my
14 ruling in this case?
15 I'm just kidding, go on. I thank God for the Court
16 of Appeals every day, so.
17 MR. GOLDBLATT: Your Honor, this is what Mr. Purdy
18 said when we were talking about this issue. He said, "And
19 of course, Your Honor, as everyone has made clear, no one is
20 contesting that there are educational benefits of diversity.
21 It is simply not an issue in this case."
22 Now, what Professor Heriot's report says is that
23 there aren't sufficient benefits to justify policies like
24 these, and in that sense, what the Plaintiff is purporting
25 to do is to put on a witness who is going to say that which
7
1 Mr. Purdy told this Court isn't true. And Your Honor, we
2 just think that the Plaintiffs should be stopped from doing
3 that.
4 But there is another reason, also, Your Honor. It's
5 that Professor Heriot isn't an expert within the meaning of
6 the Federal Rules of Evidence. She shouldn't be allowed to
7 offer opinion testimony in this case. She certainly has
8 opinions, but there are cases, as this Court is of course
9 aware, like Daubert and Cumo Tire that make clear that
10 before those opinions become evidence in a case this Court
11 plays a gatekeeping role, and before she is permitted to
12 offer opinion testimony there are certain standards that
13 need to be satisfied.
14 Professor Heriot doesn't have any degree that's
15 relevant to her testimony at all. She is a lawyer. She has
16 the same degree that many of us have. She teaches court
17 law. She says that she has, quote, "studied the policy and
18 legal aspects of the issue of racial preferences." That's
19 from her deposition, Your Honor. And she said those two
20 things, the policy aspects and the legal aspects, are
21 interconnected. But this isn't a subject that she has
22 even taught.
23 The law is clear, Your Honor, that the testimonial
24 latitude, I think that's a quote from Daubert, that expert
25 witnesses get that allow them to say more than just their
8
1 firsthand factual testimony isn't something a witness earns
2 just by having an opinion. There are standards, and with
3 respect to the specialized knowledge to which an expert
4 witness will testify there needs to be evidence that the
5 testimony is reliable, that it's more than just one person's
6 opinion.
7 This Court mentioned yesterday that there was a copy
8 of the Daubert opinion on the Bench, and it's actually an
9 encapsulation of that opinion and subsequent Supreme Court
10 case that I mentioned, it's the Cumo Tire case, and on
11 pages, I think it's 149 and 150 of that case, the Supreme
12 Court tells us exactly what the questions are that we're
13 supposed to ask here. There are essentially four questions.
14 Is there a theory or a technique that's been tested?
15 THE COURT: Does it say -- I don't have the case
16 here, I have the same thing you have, the four questions
17 right there.
18 MR. GOLDBLATT: Exactly, Your Honor.
19 Is there a theory or a technique that's been
20 tested? Have the opinions been subject to peer review
21 and publication? Are there standards we can look to to
22 determine that the opinions are reliable? Is there some
23 technique or theory that's gained general acceptance within
24 a relevant scientific community?
25 Your Honor, the answer to every single one of those
9
1 questions is no. This isn't a close case under Daubert and
2 Cumo Tire.
3 Another subject that Professor Heriot discusses in
4 her report is Proposition 209 and its effects, but it should
5 be noted, Your Honor, that Professor Heriot doesn't even
6 teach at a public university in the University of California
7 system. She teaches at a private university, the University
8 of San Diego Law School, a university that, by the terms
9 of Prop 209, isn't affected. It's not, Your Honor, the
10 University of California at San Diego that Dean Garcia was
11 discussing, it's a private university not affected by
12 Proposition 209.
13 And in this case, the Daubert qualifications of the
14 witness with respect to that question, the effects of 209,
15 aren't any better than they are with respect to the benefits
16 of diversity. All four questions, again, all four Daubert
17 questions are answered no.
18 Your Honor, this is really just one person's
19 opinion. It's exactly the kind of testimony that cases like
20 Daubert and Cumo Tire say aren't evidence, aren't evidence
21 in the Federal Court under the Federal Rules. I don't doubt
22 that these opinions are genuine, that they are sincerely
23 held, but that doesn't make them competent evidence and
24 Professor Heriot should be precluded from offering them
25 here.
10
1 THE COURT: Thank you.
2 MR. WASHINGTON: Your Honor, we certainly agree with
3 Mr. Goldblatt's remarks on this. I would just add a couple
4 more.
5 I think insofar as Professor Heriot purports to
6 testify as to the effects of 209 on the educational system
7 of the State of California, there is no evidence that she
8 has ever taught at the University of California other than
9 in speeches or forums organized by the proponents of 209.
10 There is not even any indication that she has ever had any
11 connection with the University of California. There is no
12 indication that she has published any studies other than
13 popular articles about the University of California.
14 Her report contains at most some alleged statements
15 about grades at the public university in the City of San
16 Diego, which as I understand is a city of something like
17 three million people, and there is no indication where or
18 how she got that or how she has any particular expertise
19 at all.
20 There is no technique or theory here that this
21 witness has. She has not submitted anything to any peer
22 review. There is no kind of background in education,
23 anything of that nature, and in that regard, I think this
24 is really the rankest of lay testimony by a partisan in the
25 debate brought here to say things which aren't true, but
11
1 more relevant than that, are not competent as testimony.
2 So we certainly think this witness should not be
3 allowed to testify.
4 MR. KOLBO: Your Honor, I guess my response in a
5 few words is, surely they jest. They must be making these
6 arguments, in part, tongue in cheek, and I want to address,
7 first of all, the relevance issue.
8 It's true that we made some objections early in this
9 case to any testimony about the value of diversity, and I
10 suspect if there hadn't been any testimony about the value
11 of diversity, and there has been a lot in this case from
12 both the Defendants and the Intervenors, that we wouldn't
13 be at least putting on Professor Heriot to talk about that
14 issue.
15 And we understand that the Trial Court's issue is
16 limited, but this case, as has been mentioned, is going
17 to be heard on appeal and we think it would just not be
18 appropriate for there to be lots of evidence on the
19 Defendants' side of the case, on the Intervenors' side of
20 the case, about the value of diversity, but none, even
21 from one witness, the one witness that the Plaintiffs have
22 offered in rebuttal on that particular issue. We think that
23 would be just highly inappropriate.
24 And just so I can illustrate my point, Your Honor,
25 when Mr. Goldblatt says that they had a case that they
12
1 wanted to bring with respect to diversity and they didn't
2 bring it, it's true, there are some witnesses that they had
3 on that issue that they haven't called, but they elicited
4 very directly and clearly and over our objection a number, a
5 substantial amount of testimony on the issue of the value of
6 diversity, what critical mass is, and why it's important.
7 They obtained that testimony, Your Honor, not
8 necessarily always from qualified or formally qualified
9 experts, but from people who are educators.
10 President Bollinger testified why he believes
11 diversity is important, why critical mass is important.
12 Dean Lehman testified about why diversity is important, why
13 critical mass is important and how you go about achieving
14 it. Professor Lempert testified on that issue, as well,
15 at length, and of course, they called Professor Severud
16 on that issue, really only on that issue in terms of the
17 importance of diversity and how you get it and so forth.
18 So it just seems absurd to me, Your Honor, to
19 suggest that it's not an issue in the case, and therefore,
20 the Plaintiffs shouldn't be able to call a witness even in
21 rebuttal on those issues. It just doesn't make -- it
22 doesn't pass the smile test, Your Honor, is I guess what
23 I would say about that.
24 With respect to the substance of Professor Heriot's
25 testimony, she is not going to testify just about diversity
13
1 and whether it's important. Her report, Your Honor, is
2 before the Court and I think you have a sense about what
3 she is going to testify about. Her view is, there is
4 nothing wrong with diversity, certainly diversity,
5 intellectual diversity is a good thing. You'd rather have
6 it.
7 But one of the things she is going to testify to is
8 how you go about getting it, and what she says, you don't
9 have to work very hard to get diversity in the classroom,
10 it pretty much comes naturally. If you're going to have a
11 classroom of 30 students, people are individuals, they bring
12 by their very nature different experiences, outlooks and
13 backgrounds, so you don't have to do much to bring it and
14 you certainly don't have to use racial preferences to
15 achieve diversity in the schools.
16 We think that's relevant to the issues in the case
17 and it's certainly, it's certainly relevant rebuttal
18 testimony to the voluminous testimony that the Defendants
19 have brought. And I haven't even mentioned the Intervenors,
20 Your Honor. They have marched student after student up here
21 to talk about their classroom experiences and so forth. We
22 had Professor Garcia yesterday talking about diversity on
23 campuses that he's not even involved with at the University
24 of California system.
25 So I think it's just frankly a little absurd to
14
1 say that the issue isn't in the case, and therefore, the
2 Plaintiffs shouldn't be able to bring a witness to respond
3 to their witnesses. What they are really trying to do,
4 Your Honor, is to suggest that there should only be one side
5 heard on this issue. Their side has been heard and our side
6 should be heard, and from the one witness that we intend to
7 call on this issue.
8 If I may then, Your Honor, address the issue of
9 qualifications, again, I have to say that this brings a
10 little bit of a smile to my face when I hear that Professor
11 Heriot, first of all, is not qualified to testify on the
12 subject of legal education, the very same subject that,
13 again, President Bollinger testified about, Professor and
14 Dean Lehman testified about, Professor Lempert testified
15 about, Professor Severud testified about.
16 Our witness, Your Honor, Gail Heriot, is a professor
17 of law and a full professor of law. She has been teaching
18 since 1989. She was Dean for a year at George Mason
19 University. When we're talking about diversity, as I
20 understand, what I have heard from the witnesses on the
21 other side, we're talking about what goes on in the
22 classroom, what's important, what does diversity bring to
23 the classroom, what does it bring out of the students, how
24 does it affect the teacher's performance and so forth.
25 I don't know how anybody could bring more expertise
15
1 to that subject than someone who is, in fact, a legal
2 educator and I think that's all that's required to pass the
3 expert testimony threshold in this case. That's all it took
4 for their experts on that subject. That's all it took for
5 their lay witnesses to testify on the subject of the value
6 of diversity. So there is just -- it's just not a serious
7 argument to suggest that our witness, a full professor of
8 law and a teacher of education and one who has thought about
9 and researched some of these issues, doesn't have the basic
10 threshold qualifications to present testimony.
11 Now, I understand they think their experts are
12 better qualified. They think more of Professor Severud than
13 they think of Professor Heriot in terms of credentials and
14 they are certainly entitled to think that and they can
15 certainly argue in terms of the weight of the testimony on
16 that. We think Professor Heriot is very highly qualified
17 and her opinions will be or should be well received by the
18 Court, so it's just not a matter of throwing her testimony
19 out.
20 And finally, Your Honor, with respect to the issue
21 of California, again, I have to smile. They are suggesting
22 that a professor in California is not appropriate to testify
23 about the effects of Proposition 209 in California. I
24 didn't hear exactly what Professor or President Bollinger's
25 or Dean Lehman's or Professor Lempert's or Dean Severud's
16
1 qualifications were to testify about California, and they
2 did testify about what they thought would be the dire
3 consequences in Michigan, based on California, if race
4 couldn't be used as a factor, so it's just not a serious
5 argument to suggest that their Michigan professors are
6 better able to tell this Court about California than a
7 California professor.
8 And finally, Your Honor, with respect to that issue,
9 they seem to misunderstand the point that we thought has
10 already been brought out in this case, which is the
11 consequences of Proposition 209 aren't just limited to
12 the Berkeley campus and they aren't just limited to the
13 University of California system, and they are not even just
14 limited to the public schools of California.
15 One of the things that Professor Heriot is going to
16 testify concerning is that Proposition 209 has had positive
17 consequences, and those positive consequences extend to
18 schools that aren't even in the California system, schools
19 like hers, where she has seen improvements in the quality
20 of classroom discussion and so forth as a result of
21 Proposition 209 and what she believes should be attributed
22 to it. So there is a great deal of relevance there and she
23 certainly has foundation to testify in these matters, Your
24 Honor.
25 THE COURT: In this matter, I know, Mr. Goldblatt,
17
1 you would like to say some more, but let me just -- no use
2 taking a lot of time on it.
3 As to qualifications, I will have to listen to what
4 the qualifications are. I hear both sides arguing what the
5 qualifications may or may not be, but I'll tell you in no
6 uncertain terms that I'm going to use the same standard
7 that I have used throughout this case, and there have been
8 witnesses that have testified that have done no studies,
9 there have been witnesses that have testified that they have
10 had no formal teaching and so forth, but that the Court has
11 ruled that they could testify, because the reason being is,
12 that they do have special knowledge because of their
13 position and so forth.
14 For instance, I think Mr. White was very helpful
15 after we heard his testimony and I think he was very
16 credible in terms of what he had to say and the statistics
17 and so forth, though he had no formal training, hadn't done
18 any personal studies or anything of that nature, and I am
19 going to use the same kind of test.
20 Even Dean Severud, obviously, had no formal
21 training, but he had been around a long time and had taught
22 courses, he had done things that perhaps, judicial Daubert,
23 you know, you would say, well, that wouldn't apply.
24 I only give you these examples because I know that
25 this person is coming from a long way and so forth, but I'm
18
1 saying that I can't rule on her qualifications until I hear
2 them, until I have had an opportunity to have the Plaintiff
3 present them and the Defense and the Intervenors voir dire
4 the witness and then I can make that determination. It's
5 hard to do in the abstract, though.
6 As I indicated, I have listened to both sides and
7 I want both sides to know that I'm going to use the same
8 standards that I have used for everybody at this point.
9 My other issue is that certain matters have arisen
10 that were important and I wanted to hear them, and I have
11 heard them, that perhaps the Plaintiff was not anticipating
12 or so forth, and this is rebuttal, and I'm going to treat
13 it as rebuttal and it's going to be limited to -- her
14 testimony, if permitted, is going to be limited to rebut
15 those things that have been put into the record by the
16 Defense and the Intervenors.
17 So with that said, I suspect the Plaintiffs will
18 call and we will have some voir dire as to qualifications
19 and I will make that ruling, and my ruling, should I allow
20 her to testify, will be that it's limited to rebut those
21 issues that have been raised by the Defense or by the
22 Intervenors.
23 Okay. Mr. Goldblatt?
24 MR. GOLDBLATT: No, thank you, Your Honor.
25 MR. KOLBO: Your Honor, we appreciate the Court and
19
1 the Parties' agreement to take our first rebuttal witness
2 out of order. We will recall Professor Kinley Larntz to
3 the stand, Your Honor.
4 THE COURT: Welcome.
5 K I N L E Y L A R N T Z, P h D,
6 having been previously called as a witness herein, and after
7 having been first duly sworn to tell the truth, the whole
8 truth, and nothing but the truth, was examined and testified
9 as follows:
10 DIRECT EXAMINATION
11 BY MR. KOLBO:
12 Q Dr. Larntz, thank you for coming again.
13 In the interim, I think it's been a couple of weeks
14 or so since your testimony, you understand that Professor
15 Steven Raudenbush offered testimony with resect to the same
16 issues, in some respects, in response to your testimony?
17 A Yes, I understand that.
18 Q Now, you weren't here in the courtroom for that
19 testimony, were you?
20 A I was not.
21 Q But have you been able to review a transcript of
22 Dr. Raudenbush's testimony?
23 A I have.
24 Q And do you understand that Dr. Raudenbush took issue
25 with some of -- some aspects of your testimony and report?
20
1 A Yes.
2 Q Have you also reviewed your own transcript, trial
3 transcript testimony in this case?
4 A Yes.
5 Q Were you able to ascertain from reading, perhaps
6 rereading or reading your testimony, as well as
7 Dr. Raudenbush's testimony, were you able to ascertain
8 whether you had already addressed or covered some of the
9 issues that Dr. Raudenbush took issue with in his testimony?
10 A Well, certainly. I mean, we exchanged our reports
11 before and I, in my direct testimony, offered information
12 on a number of criticisms that he talked about in his
13 testimony, yes.
14 Q And did you, in reviewing your testimony -- you
15 understood that you had already responded to some of those
16 or anticipated some of those in your direct testimony?
17 A A good number, I think, yes.
18 Q Did you also come to the conclusion that there were
19 some aspects of Dr. Raudenbush's testimony that you hadn't
20 really addressed or hadn't been explained in your direct or
21 cross examination that occurred?
22 A There are a few points, yes.
23 Q Okay. I want to just limit, obviously, our questions
24 and answers this morning to that, so please keep your
25 answers limited to that respect.
21
1 And I'm going to ask you about some criticisms that
2 Dr. Raudenbush raised and to the extent that you don't
3 believe they were addressed either in your prior direct or
4 in your cross examination, I would like you to respond to
5 some of these criticisms, if you have some opinions.
6 First of all, do you recall that Dr. Raudenbush had
7 a criticism with respect to the fact that there are -- that
8 in determining or ascertaining odds ratios, there are high
9 odds ratios that don't reflect large differences in
10 probability; that is, at higher ranges of probabilities, so
11 they don't reflect large odds ratios?
12 A There were cases presented by Dr. Raudenbush, and also
13 I think in my cross examination relating to odds ratios that
14 may be relatively high where the difference in probabilities
15 is not -- well, depending on how you think, not great.
16 Q Do you understand what the criticism is of your report
17 and testimony in that area?
18 A Well, I think my understanding is that the -- while
19 there may be relatively high odds ratios, there isn't much
20 of a difference in probability, and I think it's important,
21 I think it's very important that the Court understand and
22 that everyone understand the effect of odds ratios on
23 probabilities.
24 And I tried originally to provide some illustrations
25 and they provided some illustrations and I just want to make
22
1 sure that we're very clear on the relationship from a
2 baseline probability how a particular odds ratio would
3 translate into another probability.
4 So for instance, a baseline probability, I think in
5 my direct examination we talked about a baseline probability
6 of 10 percent going to 90 percent from an odds ratio of 81
7 and so I wanted to -- I wanted to make sure we were clear
8 about how that works over the whole range of probabilities.
9 Q And did you read in the reports of Dr. Raudenbush's
10 testimony where he compared relative probabilities of, say,
11 90 versus 99 percent and the effect of going from 90 to 99
12 to 999 and so forth?
13 A Yes, yes, I did read that.
14 Q And what is demonstrated by that?
15 A It's again illustrating for a certain baseline
16 probability how relative odds of a particular value would
17 translate into another probability for another group.
18 Q And I think you have testified, you indicated this
19 morning that you have prepared something to illustrate the
20 range of probabilities?
21 A What I did, what I did is it's in the booklet that I
22 think you have, and the last page of the booklet, what I did
23 was prepare just a table, and we have it as a slide.
24 I prepared a table for a range of probabilities over
25 the whole range, rather than just choose a certain point,
23
1 where I used 10 percent as a base, and they, Dr. Raudenbush,
2 used 90 percent for the whole range of probabilities, going
3 from a base of one percent, a baseline probability of one
4 percent up here in the corner, all the way down to, I think
5 it's 99, down in the bottom.
6 And so what I did is I did it for various relative
7 odds and I chose values that we have seen results of, 5, 10,
8 20, 50, 100, 200. I have chosen -- what I have done is I
9 have said, how does this baseline probabilities translate
10 into the probability of another group that has a relative
11 odds of a given amount.
12 So for instance, for instance, Dr. Raudenbush had
13 90 percent, and I think was looking at 90 versus 99, and
14 that's a relative odds of 11. I don't have 11 on this
15 table, but I have 90. I have 10, and 90 goes to what, 98.9.
16 And then talking about 111, I think, was one of the ones he
17 used, and 90 goes to -- he had 111, I have 100, and 90 goes
18 to 99.89.
19 So these are -- I'm referring to them as fractions,
20 but we often talk in terms of percentages. And other
21 values, well, for instance, 10 percent, I had reported
22 10 percent with an odds ratio of 81 going to 90. Well, we
23 can see 10 percent here going, well, at 100, goes to 91.7.
24 And all I have done on this table is just to make sure we're
25 clear what the effect of a relative odds is on a baseline
24
1 probability. And that's the only purpose of the table, is
2 for clarification.
3 Q Let me ask, Doctor, Dr. Larntz, do you disagree with
4 Dr. Raudenbush's premise that when you start with a high
5 baseline probability like 90 and then have a comparative
6 probability of 99, that odds ratios go up rapidly as
7 those probabilities diverge?
8 A Well, sure, the odds do, the odds ratios do go up
9 mathematically. They are what they are, and that's what
10 this table is trying to just illustrate, what the
11 mathematics is, and I think that's clear.
12 In point of fact, in that particular case, if you
13 look at the percentage denied, obviously they go up or
14 they go from 10 percent to one percent, so that's a big
15 difference in that probability scale.
16 Q So as I understand it, you agree with Dr. Raudenbush
17 on that fundamental principal?
18 A On the calculation, absolutely, there is no
19 disagreement on the calculation. That's why I provided
20 the table, so you can see the whole range of possible
21 calculations.
22 Q And does that mathematical fact, does that change
23 anything with respect to your opinions on the extent to
24 which race is taken into account in the admissions process?
25 A No, no.
25
1 Q Dr. Larntz, were you also -- did you also understand
2 that Professor Raudenbush in his testimony took issue with
3 your -- or criticized you for what he called discarding some
4 of the data in your analysis or selectively attending to the
5 data?
6 A Yes, I did read that, yes.
7 Q Do you have an opinion with respect to the validity of
8 that criticism?
9 A You mean the fact that different parts of the data
10 provide different amounts of information? I certainly have
11 an opinion that that's true, and I'm not quite sure -- in
12 the sense that certain cells in the grid provide more
13 information than other cells, that's true.
14 Q What would be an example of that?
15 A Well, I mean, obviously cells that have no one
16 admitted would provide no information about admission rates,
17 in comparison of admission rates. Cells that have everyone
18 admitted provide no comparative information.
19 And that's what we're talking about here, because
20 we're trying to compare the admissions of minority students
21 to majority students. So the different cell grids in fact
22 do provide different amounts of information.
23 Q Okay. And as I understand it, Dr. Raudenbush has
24 suggested that you did not consider certain cells in the
25 data.
26
1 A Well, in the calculation, every cell entered the
2 calculation and the computer made the judgment of how much
3 information would be -- you know, how much information was
4 there. There are some cells that the computer found that
5 there was no comparative information and other cells where
6 it found comparative information and used all the cells that
7 had comparative information in deriving the summary relative
8 odds ratios that we talked about.
9 Q What would be the kinds of cells that would provide no
10 comparative information, what categories would those fall
11 into, into terms of what you saw?
12 A Well, cells in a grid that would have no one admitted,
13 okay, there is no comparative information, everyone
14 admitted. And there are very few cells like that, but there
15 are a few, okay? Cells, if there are no people in the
16 cells, there is no comparative information.
17 And in fact, the one additional category is if
18 everyone in a cell was a member of one of the ethnic groups,
19 just one, so if everyone in the cell, for instance, if
20 everyone in the cell were caucasian and there were no
21 minority students in that cell, or no member of any other
22 ethnic group, then that wouldn't provide any comparative
23 information, because everyone would be the same.
24 Q What would be an example of a cell that provided some,
25 but not very much information?
27
1 A Oh, there are cells, for instance, a cell where
2 virtually everyone is admitted, but not everyone, then that
3 cell would provide -- may provide comparative information,
4 but the amount of information in how much it contributes to
5 the estimation of the overall relative odds would be small.
6 Q Have you prepared some charts to illustrate what cells
7 provided comparative information in your analysis?
8 A What I have done is for each of the years, and this is
9 also in the book, for each of the years what I have done
10 is -- and we have the first one for 1995. What I have done
11 for each of the years is looked at the original grid, this
12 is for all applicants, and I have highlighted, I have
13 highlighted in yellow the cells that provide comparative
14 information.
15 So all the cells that provide comparative
16 information are highlighted in yellow and the cells that
17 are not, the blank cells, except for the permanent cells,
18 the blank cells in the balance of the table for which there
19 is no comparative information, that is, in the sense that
20 the computer would check and say, well, there isn't anything
21 I can add based on these cells, those are left blank.
22 Q Would there be a way of including the cells that have
23 no applicants, first of all?
24 A Cells that have no applicants?
25 Q Yes.
28
1 A Whatsoever?
2 Q I'm sorry, that have no -- for across racial lines
3 there are no admits, or let's take, for example, no
4 admissions.
5 A Well, there -- if there is no admits, then there is no
6 comparative information, so they -- the computer makes a
7 judgment that there is no comparative information there and
8 so it attributes -- it doesn't actually contribute to the
9 estimation.
10 Q Did Dr. Raudenbush in his analysis use those cells in
11 some fashion, cells in which there are applicants, but no
12 admissions from any racial group?
13 A Oh, I believe that in some of his models he used those
14 cells, yes.
15 Q And does that have any effect, as far as you're
16 concerned, on the validity of the results?
17 A Well, with respect to providing comparative
18 information you would have to have a model that says how
19 they enter, and so based on how his model works, that model,
20 whether that model is true or not depends on how that
21 information would enter. I'll just say it that way,
22 that's true.
23 Q Would it provide a more accurate her less accurate
24 picture of the -- as far as you're concerned -- the extent
25 to which race is taken into account, like including these
29
1 cells in which there are no admissions from any racial
2 group, or would there be a distortion there or what
3 would be --
4 A Well, I actually don't know what the effect is, I
5 mean, as far as that goes. With respect to the statistical
6 principle of looking at cells with comparative information,
7 they don't provide comparative information, so that's what
8 I know.
9 Q And so you have got in front of you what we have up
10 here, 1995?
11 THE COURT: They don't provide any comparative
12 information, in your opinion. If you included them, would
13 there be a downside to including them, even though they have
14 no information or is there just no reason for it or --
15 THE WITNESS: Well, I mean, what you have to do
16 is you have to model the whole grid, okay, and that to me,
17 that's a hard process, okay. I looked at it and took the
18 comparative information and compared cell by cell. To me
19 it's harder to model the whole grid. It's just a more
20 difficult problem and your results would depend upon how
21 you did that modeling.
22 THE COURT: So there would be another level of
23 discretion, because that new level of discretion is now an
24 extended modeling of the whole thing?
25 THE WITNESS: That's the way I looked at it.
30
1 THE COURT: I mean, is that pretty much -- I
2 remember you talked about the less that you have to do in
3 terms of discretion, the better the results sometimes are.
4 THE WITNESS: That's right.
5 THE COURT: That's what you told us.
6 THE WITNESS: Right. The kind of modeling would be,
7 there is some kind of -- you know, I'll try to be as clear
8 as I can. You basically have a surface that you're modeling
9 here. You're trying to model a response surface over this
10 whole grid and you have to make assumptions about how to do
11 that, and what I did is, I didn't do that, and then what
12 you -- I have to use two hands, sorry.
13 THE COURT: That's okay.
14 THE WITNESS: You model, say, the majority students
15 one way and the minority students in another and sort of the
16 difference here becomes the odds ratio, right? So depending
17 on how you do that modeling, depending on how you do that
18 modeling, you could get different answers.
19 THE COURT: And the more modeling you do, the more
20 assumptions you make?
21 THE WITNESS: Yeah, well, sure, yeah.
22 THE COURT: Your position is that you did -- you
23 tried to do as little modeling and as little assumptions as
24 you can to actually get the figures?
25 THE WITNESS: Right. And exactly in this way, what
31
1 I did is I said, let's not try to model the whole surface,
2 let's just look at it one point at a time, and look at the
3 difference one point at a time. And that's basically what
4 I did, was look at the difference one point at a time.
5 THE COURT: Creating a situation where you have to
6 make fewer assumptions?
7 THE WITNESS: I think there are fewer assumptions
8 in that, surely there are fewer assumptions in that, yes.
9 BY MR. KOLBO:
10 Q And you have done this for each one of the years in
11 question?
12 A That's right.
13 Q Can we just take a look at 1996?
14 A I think so.
15 Q Same pattern?
16 A Oh, same pattern. The cells that were no admitted
17 were the predominant cells that were left out and they are
18 the ones that have very low test scores and very low GPA's
19 and those students just, you know, they didn't admit them.
20 So that's true universally. There are a few other cells
21 that don't provide very good information for the other
22 reasons I said, but predominantly, they are the lower grade
23 point averages and the lower test scores.
24 Q And 1997, can we see that same pattern here?
25 A Same pattern.
32
1 THE COURT: And '98, perhaps --
2 THE WITNESS: Same pattern.
3 BY MR. KOLBO:
4 Q And then '99. I wanted to ask you about a cell or a
5 couple of cells for 1999. I don't think these are -- these
6 kinds of cells show up on the earlier versions, but there is
7 a cell, for example, under grade point 3.0 to 3.24 and 170
8 and above in which there are nine applicants and three
9 admissions; correct?
10 A That's correct. That's the cell over on the -- let me
11 see, it's the cell right over here in this corner, 170 and
12 above, and this nine and three here that's not shaded.
13 Q And it's not shaded, meaning that it didn't provide
14 comparative information?
15 A That's right.
16 Q And why in that case was there no comparative
17 information?
18 A Well, I don't recall exactly the group, but all nine
19 of those applicants were from one ethnic group, so they were
20 from one ethnic group.
21 Q That shows no comparative information across racial
22 lines?
23 A That's right.
24 Q Did you -- there has been some testimony, I think,
25 from Dr. Raudenbush about the number of cells or percentage
33
1 of cells or so forth that were used in your analysis or that
2 were -- that, as you have testified, provide comparative
3 information. Can you -- do you have some idea in terms
4 of the number of applicants that contributed comparative
5 information to your analysis across these different years?
6 A Yes, I did calculate the percentage of applicants that
7 are in the shaded areas and I don't remember exactly from
8 each year, but it's between 84 and 88 percent across the
9 six years.
10 Q That all provided comparative information?
11 A That's right.
12 Q Dr. Larntz, then if I can ask you, do you recall
13 that in reading Dr. Raudenbush's testimony that he had a
14 criticism with respect to your reporting of uniform odds for
15 composite odds ratio assumptions?
16 A Yes.
17 Q Do you understand what his criticism is in that
18 regard?
19 A Well, I understand the technical, statistical aspect
20 of the criticism, sure.
21 Q Do you agree or disagree with it?
22 A I disagree on its importance, and I am not sure if
23 that assumption is technically satisfied or not, but in
24 fact, the importance of the assumption is that my composite
25 odds ratios, if in fact that assumption is not perfectly
34
1 satisfied, and no assumptions are ever perfectly satisfied,
2 if that assumption is not perfectly satisfied and there are
3 some cells that have higher odds ratios than others, then
4 the number I gave you would be a composite that would be
5 lower than they would be for the ones that are high and
6 a little bit higher than the ones that are lower.
7 Q Do you have an opinion as to whether Dr. Raudenbush's
8 criticism -- does it change any of your opinions with
9 respect to the extent to which you believe race is
10 considered in the admissions process as reflected by
11 your analysis?
12 A No, it doesn't change my opinion with respect to
13 that at all, no.
14 Q Can you explain why not?
15 A Well, I mean, well, first of all, I think the odds
16 ratios we see are a direct reflection of the cell. We can
17 compare the grid cells and see high odds ratios in the grid
18 cells and the summary numbers that I provided, I think, are
19 a reflection of those, of those values.
20 Q Do you recall Dr. Raudenbush testifying that there are
21 a lot of large odds ratios in the middle ranges of the
22 grids?
23 A Oh, I believe that he said that there are very high --
24 or there are high odds ratios in the middle, yes.
25 Q And do you agree or disagree with that?
35
1 A Yes, if there weren't higher odds ratios in the middle
2 we wouldn't have composite odds ratios. If there weren't
3 high odds ratios someplace, and they're mostly in the middle
4 where there is a difference in admission rates, then we
5 wouldn't -- the composite odds ratios wouldn't be as large
6 as they were.
7 Q Do you understand Dr. Raudenbush's testimony to be
8 that the odds ratios are smaller at the upper end of the
9 grid?
10 A I think I -- I think the implication of what he said
11 was they were smaller in the upper end, that's right.
12 Q Do you agree or disagree with that?
13 A Well, I think in the upper end it's very hard to tell,
14 because there is really -- when almost everyone is admitted,
15 there is -- you can estimate odds ratios, but there is --
16 the amount of comparative information up there is relatively
17 so small, so they don't contribute a great deal to the
18 composite odds ratio, because if almost everyone is admitted
19 the comparative information there isn't great, so I think
20 it's very hard to determine the relative odds up in the
21 corners unless you create some kind of model.
22 Q Where does most of the comparative information come
23 from him?
24 A Well, most of the comparative information is down in
25 this area here where there are clear differences and where
36
1 there is some discretion being made with respect to the
2 admissions process where not everyone is admitted.
3 Q The middle of the, primarily, shaded area?
4 A Well, actually, it's the lower part of the shaded
5 area, if you want to call it, the middle of the grid.
6 Q And then do you recall reading a criticism of
7 Dr. Raudenbush with respect to what he termed, I think,
8 the stability of the estimates over the years?
9 A Yes, I do remember reading that.
10 Q And do you have an opinion as to whether you disagree
11 or agree with his criticisms there?
12 A Well, I certainly think that they are relatively
13 stable over the years and I think that we have to -- I think
14 we have to do -- probably do a little bit of work, a little
15 bit of statistical work in the courtroom, to show that.
16 Q So you disagree with Dr. Raudenbush's conclusions?
17 A I do disagree that there is a -- that there is any
18 kind of major problem with stability over the years, yes,
19 I disagree completely.
20 Q And could you explain the bases, and using the board
21 if the Court will permit?
22 THE COURT: Sure.
23 A Well, a great deal was made of a comparison -- a great
24 deal was made of a comparison between African American
25 relative odds in 1997 and African American relative odds in
37
1 the year 2000, and I think that a great deal was made of
2 that comparison, and I think we need to talk about African
3 American relative odds. I have to say, if you select two
4 years out then you might get a different comparison, so I
5 want to look at all of them across all six years, okay, and
6 so I want -- can I go to the board?
7 Q Sure. Have you prepared some notes?
8 A I have a little note that summarizes things that I had
9 from before.
10 THE COURT: Can I make a suggestion?
11 THE WITNESS: Yes, sure.
12 THE COURT: I didn't know where you were, I thought
13 you were sitting.
14 If we put the board over here just for purposes
15 of this, then everybody can still have a seat and sit and
16 see it.
17 MR. KOLBO: In fact, I think we're done with the
18 projector.
19 MS. MASSIE: Thanks, Judge.
20 (Discussion off the record at 9:15 a.m.)
21 THE WITNESS: What I want to do is, I want to look
22 at all the years, so 1995, 1996, 1997, 1998, 1999 and 2000.
23 And I want to talk about relative odds and I want to talk
24 about comparing relative odds, and I said I have to -- I
25 have to do a little bit of statistics and a little bit of
38
1 math here, so that's just because it's been raised and I
2 want to make sure that we all understand.
3 So the relative odds, and I'll just call it the
4 estimated relative odds that we calculated, the estimated
5 relative odds from the report for 1995 was 257.93. That's
6 1995. This is, we're looking at the relative odds for
7 preference for African Americans, and that's always to a
8 baseline of caucasian.
9 For 1996, it was 313.59. For 1997 it was 53.49.
10 And for 1998 it was 132.16. For 1999, it was 206.45. And
11 for 19 -- whatever it is -- 2000, it was 443.26.
12 So what was talked about, what was talked about was
13 the comparison of this value, the 53, to the 443. Now, I
14 presume, I presume those were chosen because, well, if you
15 look down the list, what's the smallest one? It's 53. And
16 what's the largest one? It's 443. Would there be -- would
17 there always be a largest and a smallest? Well, that's
18 mathematics. There will be a smallest and a largest. And
19 if you're doing comparisons, well, depending, you would
20 typically look at the -- might look at the extreme, that's
21 fine, okay.
22 Now, I also had a number of standard deviations, a
23 Z score, and I need to look at that, and I'll talk about
24 that in a second, but I just want to record the one. These
25 are straight out of the reports and we saw these before, and
39
1 the standard deviation, and I'll need these, so 14.40,
2 13.18, 13.96, 13.46, 12.64, and 12.53 -- or 51, excuse me,
3 51. So this is how this is reported. These were reported
4 in the first report that I did, these first four, and these
5 were in two supplemental reports. And these are the
6 standard deviations which measure the level of statistical
7 significance of how far these are away from one, okay. So
8 this is all summary information.
9 Now, how do we actually do the calculations? Where
10 do they come from? They come from logistic regression. If
11 you recall, that's the technique we used. And logistic
12 regression works in terms, like every regression, works
13 in terms of regression coefficients.
14 And what are the regression coefficients that are
15 behind these numbers, what are the regression coefficients?
16 Well, in fact, these are not the regression coefficients
17 themselves, there is -- in the logistic regression there are
18 coefficients that we can use to calculate these. In fact,
19 the coefficients, the coefficients themselves turn out to
20 be, to be careful, it's the natural logarithm of these
21 values, the natural logarithm, not the log to the base ten,
22 but the natural logarithm. On your calculators, that's the
23 LN function, right?
24 So in fact, the regression coefficients, the
25 coefficients, the coefficients themselves, the coefficients
40
1 themselves are what you use for doing all the inferences.
2 And so what I'm going to do is write down what the
3 coefficients are, and that's what you use for comparison,
4 are the coefficients, because that's the basic underlying
5 calculation mechanism, are these coefficients.
6 And so the coefficients are, and I'll put them in
7 order and say them, the coefficients are, and I'm going
8 to round them to two places, the coefficients are -- for
9 corresponding to 57.93, the natural logarithm is 5.55. For
10 313.59, the natural logarithm is 5.75. For 53.49, the
11 natural logarithm is 3.98. And for 132.16, the natural
12 logarithm is 4.88. And for 206.45 the natural logarithm
13 is 5.33. And for 443.26, the natural logarithm is 6.09.
14 So these are the actual coefficients in the computer
15 output that you would find in the logistic regression
16 computer output.
17 Now, a relative odds of one means no preference. If
18 you take the logarithm of one, you get zero, that's just the
19 way it works, and so these need to be compared to zero. And
20 in fact, the way these standard deviations are calculated
21 is they take the coefficient, this is out of this, and
22 divide it by something called the standard error of the
23 coefficient, and you get the number of standard deviations.
24 Now, I think it's informative to plot these values,
25 because this is the scale for comparison, the comparisons in
41
1 the regression coefficients, so I need to plot these values,
2 and I'll do the best I can, okay. Zero is here, and then we
3 have got one, two, three, four, five, six, and you can see
4 if I did a reasonable job of making those equally spaced. I
5 tried, okay.
6 And then what I would do is plot values, plot the
7 values here, so I better give myself a code, one, two,
8 three, four, five. So 5.55 is here. And 5.75 is here.
9 And 3.98 is here. And 4.88 is here. And 5.33 is here.
10 And 6.09 is here, okay.
11 So this, these are the six years. This is '97 and
12 this is 2000, these are the extremes, but this is where the
13 coefficients fall in terms of the regression coefficients
14 and this is the scale for comparison.
15 Now, statistically what you're doing is comparing
16 each of these values to zero. You're comparing each of
17 these to zero, because that's in the regression coefficient
18 scale and clearly these are clustered relatively far away
19 from zero.
20 In fact, in fact, the standard deviation tells us
21 that for -- well, this standard deviation number tells us
22 how many standard deviations we are away from zero and so,
23 for instance, the year 2000 one is 12.5 standards deviations
24 away relatively. Each one of these has a standard error
25 comparative measurement attached to it, and so we're fairly
42
1 always far away. So when I say that these results are
2 statistically relatively stable, I'm making the statement
3 of how they are in the regression coefficient scale. So
4 that's how to illustrate and say that the variation from
5 year to year is within reasonable statistical standards.
6 Now, there are a couple of more things we could do
7 with this, and I'm not sure how far we should go, but one
8 more thing we could do is we could, for instance, provide
9 confidence limits for these relative odds if we wanted to.
10 I was asked about, could you provide confidence limits, and
11 I think I was criticized for not providing confidence
12 limits, I'm not quite sure. It's not hard to do.
13 So I could provide confidence limits, and if I do
14 one of them, if I do one of them and write down the results
15 of the -- for the others, we can see how that goes and I'm
16 going to give -- so what I need to do to provide confidence
17 limits for the relative odds is I need to determine the
18 standard error of the coefficient, because the way I would
19 provide confidence intervals is, I have to provide a
20 confidence interval for the coefficient and then I have to
21 retranslate it into relative odds.
22 Remember, the coefficient is the logarithm, so
23 what I have to do is provide confidence intervals for the
24 coefficient and then retranslate it, okay.
25 So to get standard errors, I actually -- the
43
1 standard deviation is the coefficient divided by the
2 standard error. It turns out if we take the coefficient
3 and divide it by the standard deviation, we have to get the
4 standard error. So the way the standard can be calculated
5 is we will take this coefficient value and divide by the
6 standard deviation, so 5.55 divided by 14.40, and that will
7 actually -- this is, we're doing this backwards in some
8 sense, because this is actually in the computer output, but
9 just to show where it comes from, this is the standard error
10 of the coefficient, 0.39. That's this value divided by that
11 (indicating). And we can do the same for all of these. So
12 we get 0.44 for 1996, 0.29 for 1997, 0.36 for 1998,
13 0.42 for 1999, and 0.49 for the year 2000.
14 And from this, from those standard errors, we
15 can calculate confidence intervals. The usual way of
16 calculating confidence intervals, I'm going to do it
17 approximately, because it won't make any difference, is to
18 get 95 percent confidence bounds, we take the confidence
19 coefficient, the regression coefficient, and take a plus or
20 minus two standard errors and that gives us 95 percent
21 confidence amounts.
22 So I could do that for this, and so, for instance,
23 in 95 percent confidence, 95 percent confidence interval,
24 then, for the regression coefficient, take 5.55 and subtract
25 two times this, and the lower bound, then, is 4.77 and the
44
1 upper bound is 6.33. So that's a confidence interval for
2 the true value of the coefficient.
3 And I can do that for each of these, and I'm not
4 going to write them all down right now unless you would like
5 me to, but what I can do now is that I can translate these
6 into confidence intervals for the relative odds. And
7 what I do is I use the, let me see, inverse logarithm,
8 exponentiation, so I undo, to get into terms of relative
9 odds, and in terms of relative odds, the confidence interval
10 runs from 118, that's the lower bound, to 561.
11 I think I had indicated earlier that these numbers,
12 I don't believe these numbers exactly, and maybe, you know,
13 in some sense I should never put two decimal points down on
14 something I don't believe too accurately, but we can see
15 that this is a confidence interval for the relative odds.
16 These are all high numbers. You know, I think they are all
17 high values, but these are confidence intervals for the
18 relative odds.
19 So if I might, I think I'll just -- could I write
20 down -- I'll write down the confidence interval for relative
21 odds for each of these, without going through all the other
22 calculations.
23 So the confidence interval for 1996 then becomes
24 130 to 757. Notice they are wide, and they are actually,
25 because of the way it works, they have to be wider on the
45
1 high end, because the relative odds, as they go up, there
2 isn't that much -- as much difference as they go higher and
3 higher.
4 For the 53.49, lower bound is 30, again, and the
5 upper bound is 96, so there is a spread, but these lower
6 bounds are all what I consider big relative odds.
7 THE COURT: Even 30?
8 THE WITNESS: Oh, sure. Oh, sure. Ten is big,
9 okay. I think I said that before.
10 THE COURT: You did.
11 THE WITNESS: Okay.
12 And 64 to 270, 89 to 478, and the last one is
13 166 to 1,176, okay.
14 So the lower bounds are all, you know, in what I
15 consider large relative odds. The upper bounds, of course,
16 are -- well, you know, they just become harder to estimate
17 out at higher relative odds.
18 So they are all -- so this gives you an idea of some
19 variation. So in my estimation, these are all indicating
20 the same thing, that in fact, there is a substantial, a very
21 large allowance for African American applicants for each of
22 the years, and they are consistent.
23 They are consistent, not technically statistically
24 consistent, but substantively consistent, but substantively
25 consistent from year to year. There may be a technical
46
1 difference between them.
2 So that's something I wanted to make sure we
3 demonstrate, the stability in the appropriate scale, which
4 is the regression coefficient scale, and the corresponding
5 confidence intervals that we could generate for relative
6 odds.
7 Q So in short, in a short sentence here, what you have
8 just explained is the basis for your opinion that you
9 disagree with Dr. Raudenbush's conclusion that the relative
10 odds ratios are unstable across years?
11 A Well, that's part of it, yes. That's certainly the
12 basis. This is my thinking of how I would look at this
13 and certainly I believe they are certainly substantively
14 stable, if not technically statistically stable across
15 years.
16 Q Now, I think Dr. Raudenbush criticized you for not
17 reporting confidence intervals as you have just illustrated
18 here for these years for African Americans. Could
19 Dr. Raudenbush have calculated confidence intervals for your
20 relative odds based on the data that he had available to
21 him?
22 A Oh, sure. This is not -- this is straightforward
23 calculations and the computer output was all provided.
24 Everything was there to do this, yes.
25 Q Now, you --
47
1 A But you can do it from the reports, actually, as we
2 just did from that information.
3 Q And you have reported that all of these, as far as
4 you're concerned for these years for African Americans, all
5 of these show very large preferences?
6 A Oh, sure, yes.
7 Q Are you able to tell whether that's true or not for
8 the other races that you did odds ratios analysis on for all
9 these years?
10 A Well, if you computed confidence intervals for
11 relative odds, they would all be, you know, relatively -- or
12 distributed around the estimated values, but they would all
13 show large preferences, yes.
14 Q You know, Dr. Raudenbush testified, I believe, that he
15 believed that the standard deviation -- I probably won't say
16 this correctly so you will have to correct my question,
17 perhaps, and then answer it. I believe he testified that
18 there were 11 standard deviations in different separated
19 odds ratios between 1997 and 2000. Did you understand that
20 he testified along those lines?
21 A Yes, I remember being asked that in cross and I read
22 his testimony to that effect that the difference was about
23 11, I think he said, almost 11 standard deviations.
24 Q Do you disagree with that opinion?
25 A Oh, yes. That's wrong. Excuse me.
48
1 Q Do you have any understanding based on what you have
2 seen from Dr. Raudenbush's testimony and the data that you
3 have as to how he could have come to that conclusion?
4 A Well, if I might, can I do the calculation for the
5 Court, do the calculation in the regression coefficient in
6 the appropriate scale and then show you what the difference
7 is? And then I obviously don't know exactly how he did
8 the calculations, but I have an idea of how he did the
9 calculations, okay?
10 THE COURT: Certainly. Why don't we -- just for the
11 record, why don't we just put some identification on that.
12 MR. KOLBO: I was going to ask if we could mark it,
13 Your Honor, and offer it, as well.
14 THE COURT: Why don't you at least mark it and then
15 we can talk about offering it.
16 MR. KOLBO: 226.
17 THE COURT: What's the next number?
18 MR. KOLBO: Do you want to write 226?
19 THE COURT: Ex. 226, just so we can talk about it,
20 then you can move it later. Just so we have something for
21 the record that we know where it is, okay.
22 THE WITNESS: I need to do the comparison. I have
23 to do the -- compare standard errors.
24 MR. DELERY: We have another easel with another pen.
25 THE WITNESS: That's okay, let me record them, then
49
1 we will do it.
2 1997, the coefficient was what, 3.98, and the
3 standard error was .29. That's okay. We're fine. And it
4 was 6.09, is that right, and the standard error was 0.4,
5 okay. That's all we need.
6 THE COURT: And then if you'd just put on the
7 right-hand bottom corner again, 227, Ex. 227, and the record
8 will reflect that he is now working on 227, proposed. Okay.
9 THE WITNESS: Well, from these values we can see how
10 different they are. And I am going to bring this back just
11 to show you, then I'll move it back over.
12 You can see this value here is what, 12 standard
13 deviations from zero, right, 12 standard deviations from
14 zero, and this one, this one here is what, 14 standards
15 deviations from zero, and what Dr. Raudenbush has testified
16 to is that if we compare these two, we get something close
17 to 11.
18 Well, I mean, statistically, and I think I hope
19 I reacted appropriately when someone asked me, would it
20 surprise you that the difference between these two is 11,
21 given this one all the way down here is twelve and this one
22 all the way down here is 14, would it surprise you that
23 this difference is 11, and the answer is, it would totally
24 surprise me, and the reason is that we're carrying this to a
25 fixed value of zero, and these are compared to each other,
50
1 and each of them has a standard error attached to it, so it
2 doesn't -- it's not reasonable that this difference should
3 be 11.
4 So I went back and did the calculation. And I'm
5 going to show you how to do the calculation now and show
6 you what the number is, okay. What I need is I'm going to
7 compare these two coefficients, so I'm going to compare
8 6.09, the difference between 6.09 and 3.98. That's what I
9 need to compare.
10 In order to compare this difference, I have to get
11 what's called the standard error for the difference, and
12 this is something that we teach in our first course in
13 statistics. And if we have got two standard errors and we
14 combine them to the standard error for the difference,
15 assuming they are computed from different data sets -- which
16 they are, I didn't put anything together, '97 was done
17 completely separate from 2000 -- so the standard error of
18 the difference, actually the way we do it in statistics, we
19 always have to do things in the variance scale, so we have
20 to square these things. So we put them together, 0.29, we
21 square it, and 0.49, we square it, and then to get the
22 standard error we have to go back and we have to take the
23 square root of that. So that's the math. We square these
24 two numbers, add them up, and take the difference.
25 Now, I should have a calculator.
51
1 MR. DELERY: It's .31.
2 THE WITNESS: You got .31? Let's see.
3 THE COURT: He did it in his head, see.
4 THE WITNESS: But he is not testifying.
5 THE COURT: That's okay. For your profession, the
6 calculator is the most important tool. For lawyers, it's
7 their business card.
8 THE WITNESS: Okay. Let's just make sure I do it
9 right. So I have to make sure I do it right, and the value
10 I get, this one is .49, it can't be any smaller than that,
11 it's got to be bigger than that, bigger than the total, each
12 one separately, and it turns out to be 0.57, 0.57, okay.
13 And so we divide this by 0.57. That's the standard
14 error of the difference. And this tells us how many
15 standard deviations these two numbers are apart, and that
16 number turns out to be 3.7, 3.7.
17 Now, that's in the range of big. Two or three is
18 big. This is a big number, but it's not 11. Am I concerned
19 that this number is this big? Well, in fact, I think there
20 probably are some year to year differences, and I think I
21 testified to that before. There probably are. We don't
22 expect these coefficients to be the same year to year.
23 There is variation. There might be differences in the way
24 things are done.
25 Is this a really large number? In the context of
52
1 looking, what, at the most extreme case, right? 1997 versus
2 2000, those are the most extreme cases. Is this a large
3 number? The answer is, it's not large when you consider
4 that it's the most extreme and the fact that we don't expect
5 it to be exactly the same year to year. So this is a number
6 that I think is consistent with the kind of variation we
7 saw on the previous plot, that these are not 11 apart or
8 almost 11.
9 THE COURT: And 11 would be a large number, you
10 would agree with that?
11 THE WITNESS: I think I agreed in my cross
12 examination that I thought 11 was -- I was surprised.
13 I couldn't believe it was 11, okay, but I didn't dare do the
14 calculations right there on the spot. So 3.7, that's given
15 where we are, that's the value. Well, it is the value, so
16 it's reflected in the plot.
17 Q Dr. Larntz, do you have any -- based on what you
18 understand from the data, is there any way you could
19 understand how someone could conclude that the standard
20 deviation was 11?
21 A Well, I don't know exactly what Dr. Raudenbush did,
22 but if, if you didn't go into the regression coefficient
23 scale, if you tried to work in the relative odds scale, if
24 you didn't go to the basic logistic regression coefficient
25 scale and tried to do these directly and used this standard
53
1 deviation, but didn't take this, that we had to work in
2 coefficient scale, if you did the math, the same math that
3 we did over here, but you just took these numbers and
4 assumed that you could take the standard error of this one
5 by dividing by that, if you did that, if you did that,
6 which is --
7 THE COURT: That's not an acceptable statistical
8 procedure?
9 THE WITNESS: Well, it's not right. It's not right.
10 It's not right, because these have more variations. We saw
11 that from the confidences. These have more variations than
12 that. I mean, the standard deviation associated with this
13 isn't about ten, it's much more variation than that, or 11
14 or 15 or whatever.
15 So, but if you did that, if you did the math that
16 way, and did the same, to my understanding, did the same
17 thing, you would get 10.94 if you did it that way. So I
18 don't know what was done, but I know if you did that, you
19 would get 10.94.
20 THE COURT: And you know if that was the way it was
21 done, it's not done directly, in your opinion?
22 THE WITNESS: Oh, absolutely, that's right.
23 BY MR. KOLBO:
24 Q Is that something statisticians could reasonably
25 disagree with, whether it's appropriate?
54
1 A There is no disagreement about that at all. I mean,
2 the logistic regression coefficient scale is the scale for
3 comparison, not the -- and the relative odds are derived
4 from those directly for purposes of understanding the model.
5 Q I just have a few more questions, Dr. Larntz.
6 I think Dr. Raudenbush testified -- I understood him
7 to testify that your analysis really demonstrates only that
8 a race factor was used as a factor in the admissions
9 process. It doesn't demonstrate anything about the extent
10 to which race was used in the admissions process. Do you
11 agree or disagree with that?
12 A Well, I would disagree, and shortly -- can I just
13 briefly describe?
14 I disagree in the sense that these are big
15 representative odds. There is a tremendously large
16 allowance given to race, and it's hard to believe that any
17 other factor could explain that away, unless it were just a
18 surrogate for race.
19 MR. KOLBO: I have nothing further, Your Honor.
20 I would like to offer the exhibits, Your Honor.
21 THE COURT: Any objection?
22 MR. DELERY: No objection, Your Honor.
23 THE COURT: Fair enough.
24 MR. KOLBO: It's 225, 226 and 227. 225 is the first
25 handout.
55
1 MR. DELERY: Oh, the charts, okay. All right.
2 THE COURT: You know, if you remind me Monday, we
3 may be able to see if Court Services -- actually, when we
4 have those and admit them, they have a Polaroid, I think
5 they still have it, and we can take some Polaroids of that
6 so that everybody will -- I don't know how else to get it to
7 a form so everybody will be able to have it, so if somebody
8 reminds me Monday, I'll try to see if they have a Polaroid
9 still up there and take some pictures. If not, I'm not sure
10 how we will do it.
11 MR. PURDY: Just a suggestion, if the Court would
12 prefer, we're happy to have these typed up. We would be
13 happy to do that and then just --
14 THE COURT: That's even better. Nobody objects? We
15 will keep these. You can all sign off on them.
16 MR. PURDY: We won't even take them out of the
17 courtroom. We will just copy them and present them.
18 MR. DELERY: Or mark Dr. Larntz' notes if they are
19 the same thing.
20 THE COURT: I think we should use what he used and
21 preserve those for the official record and type them up and
22 if for some reason you can't do that, or if you can't do it,
23 we will take the numbers, we will get them down here to take
24 pictures.
25 In Judge Taylor's courtroom, they have the white
56
1 board now that you write on and then you push a button and
2 the thing comes out.
3 CROSS EXAMINATION
4 BY MR. DELERY:
5 Q Good morning, Dr. Larntz.
6 A Good morning.
7 Q Welcome back.
8 Like Mr. Kolbo, I think I'm going to -- certainly
9 hope that I stick to what you have covered here this
10 morning. We're not going to go back over the ground that we
11 covered last month when you were here the first time, and I
12 think I'll start where we left off and then work back from
13 there.
14 MR. DELERY: If I may approach, Your Honor?
15 THE COURT: Absolutely.
16 BY MR. DELERY:
17 Q The first two columns here in Exhibit 226 were taken
18 from your expert reports; is that correct?
19 A That's correct.
20 Q The first relative odds numbers came from model one
21 for each year; is that right?
22 A Yes, I think that's right. That's the first set of
23 analyses, that's true.
24 Q You reported three composite odds ratios for each
25 year; correct?
57
1 A That's true. That's true. I was responding to the
2 criticism of this particular comparison.
3 Q Right. I just want to get it clear what these numbers
4 are. These were the model one numbers from each year?
5 A Sure.
6 Q Okay. And then the second column here that you have
7 headed SD are what your report called the standard
8 deviations on the same tables from model one?
9 A That's right.
10 Q And the standard deviations were reported in your
11 report on the log scale; correct? I mean, these are
12 standard deviations in the log scale?
13 A They are standard deviations for the regression
14 coefficient, that's right.
15 Q And the regression coefficients are in the log scale?
16 A Well, the logistic, l-o-g, regression, does analysis
17 on the log odds, so that's correct.
18 Q And the odds ratios, the estimated relative odds in
19 the first column, are not on the log scale?
20 A Oh, that's true, yes.
21 Q And is there a name for not on the log scale, is there
22 a term for that?
23 A You mean on just the normal scale?
24 Q Is that what it's called?
25 A The real scale? I mean, relative odds scale, the
58
1 relative odds scale. It's the relative odds scale.
2 Q It's the normal scale; is that --
3 A I would call it the relative odds scale.
4 Q Okay.
5 A As opposed to the log odds scale.
6 Q And -- okay. The scales are not -- if you line them
7 up, they work differently; correct?
8 A I mean, there is a direct translation one to another,
9 but obviously the relative odds are it's a trans -- they are
10 a direct translation of each other.
11 Q So just so I'm clear, the two numbers reported in your
12 reports for each of these model results were on different
13 scales?
14 A Well, I guess I wouldn't consider it that way. I
15 would consider that, in fact, I gave the relative odds,
16 that's an appropriate way to summarize the value, and this
17 is the significance to understand the relative odds. So
18 I think the standard deviations are directly related to
19 the significance of relative odds.
20 Q The standard deviations are not the standard
21 deviations of the relative odds; is that right?
22 A Oh, no, no. They are an indication of the statistical
23 significance of the relative odds.
24 Q But they are actually the standard deviations for the
25 regression coefficient?
59
1 A For the regression coefficient, that's exactly how you
2 have to do that, yes.
3 Q Okay. And that's the way you reported it in your
4 report?
5 A The report, I reported with those two columns, that's
6 exactly right.
7 Q And then it was the calculations that you have done
8 here that led to the third column for the regression
9 coefficient; is that right? I guess that's actually the
10 fourth column on this page.
11 A The numbers there, I re -- I back calculated, but they
12 are the numbers from the -- if you looked at the computer
13 output for logistic regression, that those are the numbers
14 that you would see in the computer output, yes.
15 Q And so the relative odds is just the -- well, I'll go
16 the other way. To get the regression coefficient, you just
17 take the log of the relative odds?
18 A Of course, the way you get the relative odds is you
19 exponentiate the regression coefficient, that's how, but the
20 regression coefficient is the basic number that you work
21 from and then you exponentiate that to get it in relative
22 odds. That's a standard output in logistic regression, is
23 to report both of those.
24 Q So you can move back -- what you are saying is you can
25 move back and forth between the log scale and the relative
60
1 odds scale?
2 A Well, they represent the same thing.
3 Q Turning now to Exhibit 227, the standard error of
4 the difference here between the odds ratio -- or the
5 coefficient, I'm sorry -- for 1997 and 2000, you calculate
6 as 3.7?
7 A That's right.
8 Q That's a fair statement of what's on this page?
9 A I mean, someone can check the math, but --
10 Q Okay. Am I right that anything over two is considered
11 statistically significant?
12 A Without a selection bias in the sense of looking at
13 extremes, anything over two would have a five percent
14 statistical significance, but we're not doing just any two
15 here, we're doing the most extreme two, so we have selected
16 the two we're looking at out of a group, and I would say
17 that in that case you need a value, I don't know what the
18 value is, but it certainly would range, when you're
19 selecting out of, you know, the most extreme cases, that two
20 doesn't go anymore. You have got to take account of the
21 fact that you're selecting from the most extreme cases.
22 And so typically values of -- you know, you wouldn't be
23 surprised if you got values of three or four when they are
24 the most extreme.
25 Q Okay. Putting aside the most extreme context, two
61
1 ordinarily would be statistically significant?
2 A I mean, you want me to ignore the fact that -- where
3 the data came from?
4 Q For this purpose, I'm just asking you, if --
5 A If I ignore where the data came from, then you have
6 about a five percent chance of getting a value outside the
7 ranges of two if the data comes from a normal distribution.
8 Q So greater than two ordinarily would be considered
9 statistically significant?
10 A In the context of something that wasn't generated as a
11 selection --
12 Q All right.
13 A -- that would be true, in something that wasn't
14 generated as a selected.
15 Q And greater than three would ordinarily be considered
16 highly statistically significant, is that fair to say?
17 A In the same context, where you're not -- where you're
18 just looking straight away, not where you're looking at
19 extremes.
20 Q Okay. Let's go back, if we could, to Exhibit 226.
21 Your view is that all of the odds ratios that you have
22 reported here, both in the first column from your report
23 under the RO column, and then in the confidence interval
24 parentheses, are very large, is that your testimony?
25 A By the standard of statistical practice, they are very
62
1 large, absolutely.
2 Q So in the year 2000, for example, what your results
3 show is that the relative odds for African Americans as
4 opposed to whites could be anywhere from 166 to 1,176?
5 A The data are consistent with those values, yes, at
6 the 95 percent, at the 95 percent confidence level.
7 Q And so we can't really be sure where in that range it
8 falls, but you're fairly confident that it's somewhere in
9 that range?
10 A I mean, the interpretation of a confidence interval
11 now, you want to do that? Okay.
12 Q I mean, have I fairly --
13 A Those are the values that are consistent with the
14 data at the 95 percent confidence interval, confidence
15 coefficient level, that's right.
16 Q Okay. And so we can do the same thing for the other
17 years, 1999, it's somewhere between 89 and 478?
18 A Well, knowing that those statements are made with
19 error rates that are five percent, that's right.
20 Q But the conclusion you draw is that all of these are
21 large, so these differences don't trouble you?
22 A Oh, that's right. They are all substantively
23 consistent, that's what I said, and that's exactly right.
24 Q And then when you plotted it here, you said that the
25 reason that these differences, the differences in the first
63
1 odds ratios that you reported don't trouble you is because
2 they are all quite far away from zero?
3 A They are all quite far away from zero and they cluster
4 together.
5 Q So again, here zero means a relative odds of one, am I
6 right?
7 A That's right.
8 Q Which means that the members of both groups have an
9 equal likelihood of being admitted?
10 A That's true.
11 Q So from these odds ratios you can be quite sure that
12 you have disproved a contention that both groups have the
13 same likelihood of being admitted as a percentage basis?
14 A You certainly can disprove that and you can disprove a
15 lot of other odds ratios, because the lower bounds are far
16 away from one, that's right.
17 Q So you think that you can be confident of more than
18 just that the so-called null hypothesis is disproved from
19 these?
20 A Oh, certainly, certainly, certainly, and from the
21 size of the standard deviations, the number of standard
22 deviations and the corresponding confidence, yes, certainly.
23 Q You believe, in fact, that you have quantified,
24 I believe that you said, the role that race plays in
25 admissions based on these numbers; is that right?
64
1 A Well, I am just a statistician, so I'll be careful of
2 what I did, okay? What I have done is I have described what
3 the admissions office did, okay? And so this quantifies the
4 admissions decisions, okay? And so what I have done is
5 described, quantified the admissions decisions, and this is
6 from their data and this is what their data tells me about
7 that.
8 Q And I want to be clear about that, because we
9 obviously had some fairly lengthy discussions of this issue
10 when you were here last month.
11 Today, Mr. Kolbo asked you some questions about
12 whether anything that Dr. Raudenbush said in his testimony
13 changed your opinions concerning the effect that race plays
14 in the admissions process. Do you remember those questions
15 from Mr. Kolbo?
16 A I'm not sure of the exact wording, but I took that to
17 mean if was there any change in my opinion from previously,
18 and I answered that there was no change in my opinion
19 previously.
20 Q Okay. So putting aside that, now I want to ask the
21 follow-up question. Do you believe that you have expressed
22 an opinion concerning the extent -- and I'm sorry, I guess I
23 may have misspoken a moment ago. I think Mr. Kolbo used the
24 word extent, not effect, if I misspoke, but do you believe
25 that you have expressed an opinion concerning the extent to
65
1 which race is taken into account in the admissions process?
2 A What I have done is expressed an opinion of the size
3 of the allowance that is shown in the data that's from the
4 admissions office for individuals that have the similar
5 credentials, the advantage that's given based on ethnic
6 groups in those for individuals with similar credentials,
7 and that's what I testified to, I hope, and that's really
8 what the conclusion is, is that I have quantified
9 statistically the size of the allowance that's given
10 for individuals with similar credentials.
11 Q And just so we're clear, by size of the allowance, do
12 you mean by an admissions officer sitting down to read a
13 file in making the decision?
14 A I mean describing the admissions decisions, what show
15 up in their -- from their data.
16 Q What shows up as a result of the decisions?
17 A As a result of the decisions, absolutely. It's the
18 results of the decision. I don't think I have ever said
19 or didn't mean to say that I did anything other than to
20 describe. It's their decision. I'm describing what the
21 results are.
22 Q Right. But again, just so we're clear, that's
23 different from describing how the decisions were made;
24 correct?
25 A These are the results of the decisions.
66
1 Q So is the answer to my question yes?
2 A Well, I don't know exactly how decisions are made.
3 Q Okay. And these data don't say anything about how
4 decisions are made?
5 A They say a good deal about the results of those
6 discussions and that's what I would say.
7 Q But not how the decisions were made?
8 A The mechanism for making decisions, that's not
9 statistics.
10 Q The process?
11 A That's not statistics.
12 Q I believe that you said earlier in asking -- answering
13 the questions for Mr. Kolbo that you're confident in the
14 substantive stability of your results, but that the odds
15 ratios were not technically statistically stable across the
16 years, did I get that right?
17 A I think that what I would say is that the substantive
18 results are clearly consistent, okay, clearly consistent.
19 There may be, and I think there probably is, I would expect
20 there to be some technical, in the sense of statistical,
21 variation from year to year. I wouldn't expect the
22 variation from year to year to year, so what I was
23 responding to is I think that the variation from year to
24 year is probably not zero. There is probably some -- in
25 the underlying true scale, that there probably is some year
67
1 to year variation, but it's not large enough to change the
2 substantive conclusions, that's right.
3 Q So the differences among the odds ratios across years
4 don't lead you to alter the basic conclusions that you have
5 reached, that's the bottom line of what you're saying?
6 A Substantively they are the same from year to year,
7 that's what I said.
8 Q I think maybe now I would like to go, if we could,
9 back to the last page of the packet that you have. It's
10 Exhibit 225, I believe, if you could look at that.
11 A I have it.
12 Q The last page was the chart of the relative odds
13 effect on baseline probabilities.
14 A That's correct.
15 Q Am I right to understand that this is basically a way
16 to translate odds to probabilities, is that what this is
17 doing?
18 A It's showing the effect of a relative odds value on
19 certain -- on a range. I tried to do the whole range of
20 probabilities. I think both sides offered their own
21 baseline probabilities. I mean, we both did. And so I
22 thought we should probably, just for clarification, give the
23 whole range of probabilities.
24 Q Okay. So if we look across the top, you have various
25 relative odds, 5, 10, 20, 50, 100, 200?
68
1 A Sure.
2 Q Do you see that?
3 If all we know is an odds ratio number like 100,
4 if that's all we know, we don't know anything about the
5 relative chances of admission of two groups, is that fair
6 to say?
7 A We don't know the probabilities, that's right.
8 Q Okay, because on your chart here, an odds ratio of
9 100 could translate to any of the probabilities that you
10 have listed under that heading as compared to the baselines;
11 right?
12 A Well, it depends on the baseline, yeah, sure.
13 Q But the point is that unless you know, unless you
14 know the underlying probabilities, you can't evaluate the
15 significance of an odds ratio like 100?
16 A Well, it would mean different things for different
17 baseline probabilities, that's what I'm trying to
18 illustrate, exactly.
19 Q Now, I believe you testified when you were here before
20 that an odds ratio of two or three, you would consider
21 large; is that right?
22 A In my work, two or three is a big number in odds ratio
23 terms, yes, absolutely. And I saw one of eight the other
24 day, so everyone was astonished in the room.
25 Q So a select group, I guess.
69
1 A Well, they were clinical medical researchers and they
2 were astonished that it was as big as that. Eight was big.
3 Q And that's -- if you think about it in terms of a
4 medical study for a second, you don't have two or three here
5 on the last page of Exhibit 225, but you do have five.
6 A Oh, yeah, sure, sure.
7 Q Which, if two or three is large, then five is clearly
8 large?
9 A Five is a considerable effect.
10 Q You have, if we look here, a baseline probabilities
11 of .1 or 10 percent for a relative odds of five, the other
12 group's probability would be .35, .36, if you round, is that
13 what this table shows?
14 A I'm not with you. I'm sorry.
15 Q Okay. If we look under relative odds of five, so the
16 first column.
17 A I have that.
18 Q Starting with the baseline prob