In its continuing efforts to keep the public informed about the ongoing admissions litigation, the University of Michigan makes these transcripts of the trial proceedings in Grutter v Bollinger, et al., Civil Action No. 97-75928 (E.D. Mich.), available to the University community and general public. As is often the case with transcription, some words or phrases may be misspelled or simply incorrect. The University makes no representation as to the accuracy of the transcripts.
1
1 UNITED STATES OF AMERICA
2 IN THE UNITED STATES DISTRICT COURT
3 FOR THE EASTERN DISTRICT OF MICHIGAN
4 SOUTHERN DIVISION
5 BARBARA GRUTTER,
6 for herself and all others
7 similarly situated,
8 Plaintiff,
9 -vs- Case Number:
10 97-CV-75928
11 LEE BOLLINGER, JEFFREY LEHMAN,
12 DENNIS SHIELDS, and REGENTS OF
13 THE UNIVERSITY OF MICHIGAN,
14 Defendants.
15 -and-
16 KIMBERLY JAMES, et. al.,
17 Intervening Defendants.
18 ______________________________________/ VOLUME IV
19 BENCH TRIAL
BEFORE THE HONORABLE BERNARD A. FRIEDMAN
20 United States District Judge
238 U.S. Courthouse & Federal Building
21 231 Lafayette Boulevard West
Detroit, Michigan 48226
22 Friday, January 19, 2001
23 APPEARANCES:
24 FOR PLAINTIFF: Kirk O. Kolbo, Esq.
25 R. Lawrence Purdy, Esq.
2
1 APPEARANCES (CONTINUING)
2 FOR DEFENDANTS: John Payton, Esq.
3 Craig Goldblatt, Esq.
4 Stuart Delery, Esq.
5 On behalf of the Defendants
6 Bollinger, et. al.
7
8 George B. Washington, Esq..
9 Miranda K.S. Massie, Esq.
10 On behalf of Intervening Defendants.
11
12 COURT REPORTER: MARY F. WISNESKI, CSR-0231
13 Official Court Reporter
14
15
16 Proceedings recorded by mechanical stenography.
17 Transcript produced by computer-assisted
18 transcription
19
20
21
22
23
24
25
3
1 I N D E X
2 WITNESS PAGE
3 STEPHEN W. RAUDENBUSH
4 Direct Examination by Mr. Delery 5
5 Cross-Examination by Ms. Massie 118
6 Cross-Examination by Mr. Kolbo 121
7 Redirect Examination by Mr. Delery 160
7 DENNIS SHIELDS
8 Direct Examination by Mr. Payton 162
9 Cross-Examination by Mr. Purdy 193
10 Redirect Examination by Mr. Payton 215
11 Recross-Examination by Mr. Purdy 218
12 E X H I B I T S
13
14 NUMBER IDENTIFICATION ADMITTED
15 145 Expert Witness Report of S. Raudenbush 12
16 146-150 Supp. Expert Witness Rep. of S. Raudenbush 12
17 151 Raudenbush Curriculum Vitae 9
18 184-194 Charts of S. Raudenbush 108
19 5 Gospel According to Dennis 188
20
21
22
23
24
25
4
1/19/01 - BENCH TRIAL - VOLUME IV
1 Detroit, Michigan
2 January 19, 2001
3 * * *
4 THE COURT: Good morning, everyone. On the
5 motions, I have nothing else on the agenda this case, why
6 don't we start the case and when we take it a break
7 sometime we'll argue those motions.
8 MS. MASSIE: That sounds great.
9 THE COURT: Is that good for everybody? I'm all
10 prepared, but I just don't want to waste your time this
11 morning. I know you have a witness. This is yours?
12 MR. DELERY: Yes. Good morning, Your Honor,
13 Stewart Delery, Your Honor, again for the university and
14 the individual defendants.
15 THE COURT: How are you.
16 MR. DELERY: If you're ready to proceed.
17 THE COURT: I'm ready. If you're ready, I'm
18 ready. We call.
19 MR. DELERY: We call as our next witness, Stephen
20 Raudenbush.
21 THE COURT: For evidence?
22 MR. DELERY: Thank you, Your Honor.
23 S T E P H E N W. R A U D E N B U S H
24 was called as a witness and after having been
25 sworn was examined and testified as follows:
5
1/19/01 - BENCH TRIAL - VOLUME IV
1 DIRECT EXAMINATION
2 BY MR. DELERY:
3 Q. Could you please state your name and address for the
4 record.
5 A. Stephen W. Raudenbush, 7 Harvard Place, Ann Arbor,
6 Michigan.
7 Q. And where do you work?
8 A. I work at the University of Michigan.
9 Q. What's your job there?
10 A. I'm a professor in the School of Education and the
11 Department of Statistics, and I also have a joint
12 appointment as a Senior Research Scientific at the Survey
13 Research Center.
14 Q. How long have you been at the University of Michigan?
15 A. I've been at Michigan since January 1 of 1998.
16 Q. And where were you before that?
17 A. For fourteen years before that I was at Michigan State
18 University.
19 Q. Well, Professor Raudenbush, could you please, please
20 briefly describe your education, or educational background
21 for the Court.
22 A. Sure. I received my bachelor's degree from Harvard
23 College in 1968 and my doctoral degree from Harvard
24 University in 1984.
25 Q. Has your work at the University of Michigan and before
6
1/19/01 - BENCH TRIAL - VOLUME IV
1 that at Michigan State focused on any particular areas?
2 A. Yes, it has. It's, primarily my work is involved
3 applications of statistics in education, studying student
4 learning, studying student transitions into college,
5 studying how schools and classrooms effect academic
6 achievement. And also looking at other aspects of human
7 development.
8 Q. Okay. And have you published in these fields?
9 A. Yes, I have.
10 Q. About how many publications have you had?
11 A. Well, I guess if you count the second edition of our
12 book on Hierharchical Linear Models, if you count the
13 second edition of our book on Hierarchical Linear Models.
14 THE COURT: Do you want it spelled?
15 (Whereupon an off-the-record
16 discussion was had.)
17 A. H-i-e-r-h-a-r-c-h-i-c-a-l. Okay. There would be, if
18 you count that one, there would be four books and quite a
19 large number of referee journal articles and book chapters
20 that I've published over the years. I'm not sure exactly
21 how many but I publish about four to six articles and
22 chapters a year.
23 Q. Okay. This may be a relative question, but are any of
24 those publications particularly widely known?
25 A. Well, the book I mentioned, I won't mention the title
7
1/19/01 - BENCH TRIAL - VOLUME IV
1 again, has become very, very widely used in education
2 because it deals with the problem of students being nested
3 within classrooms, classrooms within schools. Those kinds
4 of problems become very widely used. And other aspects of
5 social science where we have people in neighborhoods, or we
6 have small groupings of people, basically, which has some
7 relevance to this case.
8 Q. Are you a member of any professional organizations?
9 A. I am. I'm a member of the American Statistical
10 Association, the American Educational Research Association.
11 I'm a member of the National Academy of Education.
12 Q. What's the National Academy of Education?
13 A. Well, the National Academy of Education is an honorary
14 association limited to 125 people in the United States who
15 are involved in education and educational research.
16 Q. Have you held any editorial positions for journals or
17 other publications in your field?
18 A. I have. I've been an Associate Editor of the Journal
19 of Educational and Behavioral Statistics for quite a large
20 number of years. I was the Chair of the Management
21 Committee of that journal for six years. I have served on
22 the Publications Management Committee of the American
23 Statistical Association. I'm also the Associate Editor for
24 the American Journal of Sociology, Educational Evaluation
25 and Policy Analysis and actually several other journals. I
8
1/19/01 - BENCH TRIAL - VOLUME IV
1 won't list them all.
2 Q. Okay. Have you received any teaching or other honors
3 in your field?
4 A. I have. I received, while I was at Michigan State, I
5 received three teaching awards. I've also received several
6 awards for outstanding publications in education and
7 sociology.
8 Q. Okay. Are there any awards or honors that you think
9 are particularly significant?
10 A. I think perhaps the one that I'm, maybe most proud of
11 is that in 1993 I received the Early Career Award for the
12 American, from the American Educational Research
13 Association, which is a very large group of educators and
14 educational researchers around the country.
15 Q. What about national panels or symposia? Have you
16 participated in any of those?
17 A. Yes. In the last, within the last three years, I
18 served on the National Academy of Sciences' panel on the
19 assessment of children in conjunction, basically testing,
20 in conjunction with the Title I Program, which is a
21 compensatory education program. I also served on the
22 National Academy of Science panel on early childhood
23 science, which has just distributed a new book on childhood
24 science with implications for policy and practice.
25 Q. Okay. Professor Raudenbush, I'd like to ask you to
9
1/19/01 - BENCH TRIAL - VOLUME IV
1 look at Exhibit 151, which is, I think in binder six, Your
2 Honor.
3 A. Okay. I see it.
4 Q. Okay. Is that a current copy of your CV?
5 A. It does indeed appear to be that, yes, a current copy.
6 Q. And does it include a current list of your
7 publications and honors and professional experiences?
8 A. Yes, it does.
9 MR. DELERY: Your Honor, at this time, we'd offer
10 Exhibit 151 into evidence?
11 THE COURT: Received.
12 Q. Now, Professor Raudenbush, how would you come to be
13 involved in this case?
14 A. Actually, you asked me if I'd be willing to serve as
15 an expert in this case. Can you hear me?
16 Q. Yes, I can.
17 THE COURT: If anybody can't, let us know.
18 A. Yeah. I had to move this because I can't turn the
19 page.
20 THE COURT: Yeah, that's correct.
21 Q. And what was the purpose of your involvement in the
22 case?
23 A. Well, I started by looking at some of the expert
24 reports written by Professor Kinley Larntz and I then got
25 involved in looking at the database myself, in trying to
10
1/19/01 - BENCH TRIAL - VOLUME IV
1 understand some of the issues involved in this controversy.
2 Q. Okay. Were you present here in court for Dr. Larntz'
3 testimony on Wednesday?
4 A. Yes, I was.
5 Q. And you were here for the entire day for all the
6 testimony?
7 A. I was.
8 Q. And what about on Thursday morning, yesterday morning
9 when he returned?
10 A. I was here then too, yes, correct.
11 Q. Dr. Larntz at one or two points said that he was
12 responding to some criticism of his work. Do you recall
13 that?
14 A. I do.
15 Q. Were you the author of that criticism?
16 A. I'm quite sure that I was.
17 Q. And before this week, had you ever met Dr. Larntz?
18 A. No.
19 Q. Are you being compensated for your work in this case?
20 A. No, I'm not.
21 Q. And have you ever served as an expert witness before?
22 A. No, I have not.
23 Q. Have you prepared expert reports, based on your work
24 in this case?
25 A. Yes, I have.
11
1/19/01 - BENCH TRIAL - VOLUME IV
1 Q. Okay. If you could look in the same binder there,
2 binder six, I'd like for you to look at Exhibit 145 to 150
3 and tell the court whether those are the expert reports
4 that you submitted in this case?
5 A. Yes, these are, these are the expert reports.
6 Q. What information did you consider in preparing your
7 expert reports?
8 A. Well, I read the law school admission policy, which
9 was dated 1992. And I examined data from the database made
10 available by the law school.
11 Q. Did you review the expert reports of Dr. Larntz?
12 A. Yes, I read also each, each expert report that Dr.
13 Larntz wrote.
14 Q. Okay. And what about any deposition testimony in the
15 case, did you review any of that?
16 A. Yes. I read Dr. Larntz' deposition. Of course I read
17 my own.
18 Q. Did anybody help you with your work in this matter?
19 A. Yes. Julia Smith, who was at that time a
20 post-doctoral fellow at Michigan, helped me. She's now an
21 assistant professor. And in certain aspects of the work
22 the, basically the diversity of context for learning part,
23 I received some help from two graduate students at the
24 University of Michigan.
25 MR. DELERY: Your Honor, at this point we'd offer
12
1/19/01 - BENCH TRIAL - VOLUME IV
1 Exhibit 145 through 150 into evidence.
2 THE COURT: Any objection? Received.
3 MR. DELERY: We'd also at this point offer
4 Professor Raudenbush as an expert in the application of
5 statistical methods to education.
6 THE COURT: Any objection?
7 MR. PAYTON: No.
8 THE COURT: Okay.
9 Q. All right, Professor Raudenbush. I believe you
10 mentioned that you reviewed Dr. Larntz' work in this
11 matter.
12 A. That's correct.
13 Q. Do you have an opinion concerning, now just as a
14 summary matter, we'll get into it in more detail. But do
15 you have an opinion concerning the reasonableness of the
16 approach that Dr. Larntz took and his results?
17 A. I do.
18 Q. And what is that opinion?
19 A. I'm actually quite skeptical for two reasons. Dr.
20 Larntz attempted to construct a statistical model that
21 could tell us the extent to which race is taken into
22 account in admissions. And I'm convinced that it's not
23 logically possible to answer that question with such a
24 statistical model.
25 Moreover, certain methodological decisions made by
13
1/19/01 - BENCH TRIAL - VOLUME IV
1 Dr. Larntz, I believe, have led to a misleading impression
2 about the strength of association between minority status
3 and admissions at the law school.
4 Q. Okay. Now, you indicated that in addition to
5 reviewing Dr. Larntz' work, you did some things of your
6 own. What did you do in your analysis?
7 A. Well, as I implied, I think it's, it's not possible,
8 given the data at hand, to organize a statistical analysis
9 that's going to tell us the extent to which race is taken
10 into account in admissions. What we can do, however, and
11 what I think is very useful, is to do a causal analysis of
12 the impact of using race in admissions on those who apply
13 to the university or to the law school.
14 Q. Okay. And what are the basic conclusions again, as a
15 summary matter that you draw from your work in that
16 context?
17 A. What we did, and we'll go into some detail on this, is
18 we compared the current policy, which does use race as a
19 factor in admissions to an alternative policy that would
20 not use race as a factor. And we estimated how that
21 difference in policies would effect the average probability
22 of admission of various people who apply, various
23 sub-groups of people who apply at the University of
24 Michigan.
25 And essentially what we found, first, of course,
14
1/19/01 - BENCH TRIAL - VOLUME IV
1 is that a change in the policy would effect people
2 differently, depending on grades and test scores. It would
3 also effect people differently, depending on ethnic
4 minority status. A switch from the current policy to a
5 so-called race-blind policy would have a fairly substantial
6 effect, negative effect, on the probability of admission on
7 minority candidates.
8 On the other hand, such a change from, again the
9 current policy to a race-blind policy, would have a
10 comparatively modest effect on the positive effect, that
11 is, on the average probability of admission of majority
12 students.
13 Q. And from your work, do you draw any conclusions about
14 the likely effect on the diversity of the law school class
15 of moving to a race-blind admissions policy?
16 A. Yes. We can then take the admissions probabilities
17 under the current policy, as compared to an alternative
18 policy. And from that data, we're able to project the
19 number of applicants, not only who will be admitted, but
20 then using yield statistics, how many would then, in fact,
21 attend the law school. And then we can have an estimate of
22 how the class composition would look of the first-year
23 students at the law school. And so we're then able to make
24 some statements about the likely diversity with that class.
25 Q. And what do you conclude?
15
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. And what we conclude is that switch from the current
2 policy to a so-called race-blind policy would, would quite
3 dramatically reduce the fraction of students who are from
4 underrepresented minority backgrounds, and we'll define
5 that as we go, and to try to understand the practical
6 implications of that, we then took a look at how that would
7 translate into the composition of various contexts for
8 learning that occur in the law school. And, again, the,
9 how different classrooms and other context for learning
10 would look under the current policy versus an alternative
11 policy is really quite different.
12 Q. Well, with that sort of basic overview in mind, let's
13 go back and talk in more detail about how you arrived at
14 these conclusions.
15 A. Okay.
16 Q. What was, basically, the first thing that you did when
17 you approached these data?
18 A. Well, the first thing we did, and we did this for each
19 year between 1995 and 2000, was just to take a look at the
20 basic data; who applied at the law school, who was
21 admitted, who, how many people who were admitted decided to
22 come to the law school, and then what was the composition
23 of the first year class for each of those years.
24 Q. Okay. And I think we've prepared a chart of an
25 illustration of that, is that right?
16
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. Yes.
2 Q. Is that right? And I'd like to put up, if I could,
3 Your Honor, Exhibit 184, the series of exhibits, I think,
4 is in the supplemental exhibit file. And the lights on
5 would be fine, because they're just words today, no screen.
6 A. Your Honor, may I stand up and explain what's on the
7 screen?
8 THE COURT: You may absolutely stand up and
9 explain, yes, or we can move it closer to you so you can
10 sit.
11 A. Yeah.
12 THE COURT: You're a professor, you're used to
13 standing and talking.
14 A. That's right. Either that or I'll have to get new
15 bifocals.
16 THE COURT: Yeah, whatever.
17 A. That's fine, thank you.
18 THE COURT: I've got a pointer here if you'd like
19 one, too, however you got to promise to give it back.
20 A. Right.
21 THE COURT: Because the government, again, we can
22 get almost anything we want, but pointers. They're hard to
23 come by these days.
24 A. It will be hard to walk away with this.
25 THE COURT: Yeah.
17
1/19/01 - BENCH TRIAL - VOLUME IV
1 Q. All right. So this chart is of the 2000 admissions
2 data, is that right?
3 A. That's right.
4 Q. Why don't you explain what's here and what you find
5 significant about these numbers?
6 A. Well, the basic idea behind this chart is that it
7 shows quantitatively how a pool of applicants gets
8 translated into people who actually attend the law school.
9 And the thing to illustrate that, I'll just use
10 the top row of the chart in 2000. And we break this down
11 by ethnic groups. So just to take the first group here in
12 2000, there were 262 African-American applicants and that
13 constituted about 7.4 percent of the applicant pool.
14 And of those 262 people, 36.3 percent were
15 admitted. And that led to 95 offers of admission for that
16 group. Now, of those people who were offered admission,
17 only a minority, 40 percent, decided to come to the law
18 school. So if you multiple 40 percent times the 95 who
19 were admitted, then you get the number of African-Americans
20 who actually were attending the law school in 19, in 2000,
21 and that turns out to be 38.
22 So what you, basically, see is that this number on
23 the left which is 262, ultimately becomes 38, through whose
24 admitted and whether they decide to attend. That's the
25 basic idea on the chart.
18
1/19/01 - BENCH TRIAL - VOLUME IV
1 Now, what we've done is, is to, to make this
2 clear, and I think in conformity with the law school policy
3 of admissions, is we've taken three groups;
4 African-Americans, Hispanics and Native Americans and
5 combined their data in the lower panel here to the data, to
6 a group that we label those of underrepresented minority
7 status. So that --
8 Q. That's UMS?
9 A. And that's called UMS in this table. And then we have
10 taken data from the Caucasian group, Caucasian American,
11 and those, those whose ethnicity is unknown, and again,
12 that's in accord with our understanding of the how the
13 policy works. And we've taken their data and combined them
14 into another group that we call them non-UMS. They are the
15 ones who are not in the underrepresented minority status.
16 Now, that leads one group that I haven't
17 mentioned, and that's the other group, the non-citizen
18 group, and that group, we do not include in this table. We
19 could have looked at underrepresented minority status, yes
20 or no, and foreign or foreign students. But the numbers,
21 in fact, there were only three foreign students attending
22 in 2000, are really too small to do much with. And it
23 seemed that whatever was happening with minority status and
24 non-represented minority status was somewhat different
25 because this group is ethically very diverse, the foreign
19
1/19/01 - BENCH TRIAL - VOLUME IV
1 group, and yet they're not in these categories so we didn't
2 include that small number of applicants.
3 So then down at the bottom what we, basically,
4 have are underrepresented minority students and
5 non-underrepresented minority students and then a total.
6 Q. Do you find anything significant about the pattern of
7 the numbers here on the bottom half of the chart?
8 A. Yeah. There's several significant features of this
9 table. One is we just start by just looking at the
10 applicant pool. So we see that there are 484 applicants
11 who are minority. I'm just going to use the word
12 "minority" and "non", I think, because it gets hard in
13 saying.
14 THE COURT: That would be great. We all
15 understand.
16 A. And I'll try, and I often may use the word "race", and
17 I don't necessarily mean "race". We know there's ethnicity
18 and it's complex, but I'll use it because it gets hard to
19 use so many words.
20 But so 484 minority applicants, and in contrast to
21 2,871, majority applicants, or non-majority applicants.
22 And so the pool sizes are very different. There's a much
23 smaller number of minority applicants than non. So that's
24 one factor that we -- it's very important in
25 understanding -- the dynamics of this whole system is just
20
1/19/01 - BENCH TRIAL - VOLUME IV
1 the different sizes of that applicant pool.
2 The next feature that's very important to look at
3 is just the percentage admitted, because that's a crucial
4 factor in, who ends up being in law school. And what we
5 see is that 35.1 percent of the minority applicants and 40
6 percent of the non-minority applicants are admitted.
7 And these numbers are quite reflective of what
8 happens year to year. Only a minority of people, of the
9 overall applicant pool is admitted. The numbers are pretty
10 similar. In general, the fraction admitted is smaller for
11 the minority group than for the non-minority group.
12 Q. And is that true in each of the years from 1995 to
13 2000?
14 A. That is true. The general pattern is true each year.
15 These numbers will fluctuate but the general pattern is
16 true. And the, from the point of view of promoting
17 diversity, ethnic diversity, which is one of the goals
18 stated in the admissions policy, these two facts; there is
19 the small pool size and the comparatively small fraction of
20 those admitted has important implications for the diversity
21 of the class. Because if this number is lower much, the
22 number of people who actually attend can get very small.
23 Specifically, in this case, with 35.1 percent of
24 the minority applicants admitted, and then with the yield
25 of 34.1 percent out of the 484 applicants who are minority,
21
1/19/01 - BENCH TRIAL - VOLUME IV
1 what we see as actually attending, 58. So 484 goes down to
2 58.
3 And if you're, you know, if you're interested in
4 diversity, the size of this applicant pool, the fraction
5 admitted and the yield are going to strongly effect this
6 number, and I guess this percentage admitted is under --
7 obviously under the direct control of the law school. And
8 if we shadow where we're going with our analysis, if this
9 number were reduced significantly, this number 58 would
10 begin to go down.
11 I mean, if this number were cut in half, then we'd
12 have only 29 minority students, so, and that would assume
13 that the number of applicants and the yield would remain
14 constant. Do you see my point? That if we cut this number
15 in half, hold everything else constant, we're down to 29.
16 THE COURT: Or double it and it may go up?
17 A. Or double it and it will be go up to 116 if we double
18 it. So whatever we do here has big effects on this number,
19 but we also need to take into account the possibility that
20 changing this number could change this number. It could
21 change the number of people who apply. It could also
22 change this number, the number of people once admitted who
23 might then decide to attend.
24 So in particular, if this number were lower, this
25 number could, would likely -- it probably wouldn't stay the
22
1/19/01 - BENCH TRIAL - VOLUME IV
1 same. A more likely outcome, if you lowered the
2 probability of admission of a group, it might encourage
3 fewer people to apply. That's, we don't know. And our
4 analysis won't assume that, but the law school would have
5 to take that into account as a possibility. And lowering
6 this number might also end up lowering the yield because if
7 you, if you reduce this number substantially you would be
8 left with, under a race-blind policy, the people who would
9 be here would be extreme.
10 Q. "Here" being the number admitted?
11 A. The number admitted would be an extremely highly
12 qualified group, in terms of grades, test scores and so
13 forth. And the yield for such a group may be, may be lower
14 because there may be significant competition, among law
15 schools for those people. So changing this number could
16 impact these numbers. And with, with large effects on
17 this, this relatively small number, 58, so that's, that's
18 the key thing that's happening.
19 Q. Right. So this chart is of the 2000 data. Does your
20 report include similar information for the other years?
21 A. It does.
22 Q. For the various reports?
23 A. We have a similar flow chart for each year from 1995
24 to 2000.
25 Q. And is the 2000 data unusual, compared to the other
23
1/19/01 - BENCH TRIAL - VOLUME IV
1 years?
2 A. The 2000 data are pretty similar in virtually all
3 regards. There's one slight difference here. The yield
4 for African-Americans candidates in 2000 was 40 percent,
5 which is, which is higher than it had generally been in the
6 other years. So that number is a little higher than
7 average, but other than that it looks.
8 Q. If we compared the, this data, including the number of
9 applicants to some similar charts in Dr. Larntz' reports, I
10 think there may be some slight differences, is that right?
11 A. I looked at those numbers. The, the exact numbers are
12 not identical and I don't really know why.
13 Q. You worked from the same database?
14 A. We worked from the same database.
15 THE COURT: Are they significantly different?
16 A. They're not significantly different.
17 Q. Okay.
18 A. The patterns that I'm describing are very similar, I
19 mean, they're virtually identical in the two sets of
20 figures.
21 Q. Now, in addition to your point about how, how the
22 various percentages, in particular, can effect the number
23 attending in each year, do you take any other basic
24 conclusions away from looking at this basic descriptive
25 data?
24
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. There are a couple of other conclusions. While,
2 remember, I mentioned that a change in this percentage
3 would lead to, perhaps, fairly large changes in this
4 number; that is, the number admitted and also this number,
5 the number attending, and that's for minority applicants.
6 If changes in this number that are small would
7 have comparatively modest effects, if, let's say, half of
8 these people were rejected instead of, let's say, that
9 would be, that would be, we have 170. That would be 85
10 people. If those 85 places became available to the
11 majority students and then these 2,871 would compete for
12 those 85 places, and so that change, which is big here,
13 that is in the minority row, would have a comparatively
14 modest chain effect on the majority role, so that's one
15 additional piece of evidence from this.
16 Q. And the comparison or the percentage admitted of the
17 two groups, I think, what also might be called the average
18 probability of admission, is that right?
19 A. Yes.
20 Q. Does that comparison tell you anything about the
21 impact of considering race in admissions?
22 A. Well, this, we call it a, yeah, we call this bivariate
23 association. There are two variables. There's the race of
24 the candidate, and then there's the admission decision, and
25 when we look at these two proportions, that gives us
25
1/19/01 - BENCH TRIAL - VOLUME IV
1 evidence about that bivariate association. Is there an
2 association between race and admissions? And we see a very
3 small bivariate association, actually which favors the
4 majority applicants.
5 Now, we use, in statistics, we tend to look at
6 these bivariate associations as a first take on what's
7 going on, just simple data, there's no model, just look at
8 the data. And so we see this relationship. And in
9 conjunction with other bivariate relationships, my
10 conclusion from this was that it, it leads one to be
11 skeptical of a claim that race is a powerful predictor of
12 the admissions decision.
13 Q. Not the end of the analysis but a starting point?
14 A. It's not the end of the analysis, but, let me expand a
15 little bit. If we look at, let's say, just the association
16 between grades and admissions, there's a very strong
17 relationship, even with higher grades are more likely to be
18 admitted. If we look, and we don't have to control for
19 race to see that. We just see that relationship. If we
20 look at the relationship between test scores, LSAT and the
21 probability of being admitted, we see a very strong
22 relationship, we don't have to control for anything else to
23 see that. We look at the relationship between race and the
24 probability of being admitted, we see very little
25 relationship.
26
1/19/01 - BENCH TRIAL - VOLUME IV
1 So that, that tells us that race is unlikely to be
2 a powerful predictor of the outcome. It doesn't mean that
3 race and admissions are not related controlling for other
4 factors, but it does suggest that race will not be a
5 powerful predictor for the admissions decision.
6 Q. Okay. I think at this point you can probably take
7 your seat again.
8 A. Thank you. Your Honor.
9 THE COURT: No, just hold on to it.
10 A. Yeah, I need it again.
11 MR. DELERY: I think we may need it again.
12 THE COURT: Maybe you can move the chart just so
13 the folks in the audience can see.
14 MR. DELERY: Sure.
15 THE COURT: Great. Thank you.
16 MR. DELERY: I apologize.
17 Q. Now, in addition to the examination of the basic
18 descriptive data, what did you do as part of your analysis
19 in the case?
20 A. Well, as I mentioned, I'm convinced, and I think will
21 explain why a little later. But I'm convinced that we
22 can't develop a statistical model that's going to tell us
23 the extent to which race is taken into account in
24 admissions. What we can do and what I think is useful to
25 do is to do a causal analysis. What's the impact of the
27
1/19/01 - BENCH TRIAL - VOLUME IV
1 policy that the university has of using race in admissions
2 on the people who apply. And that causal analysis is
3 something that we can do with a minimum of assumptions.
4 And so that's what I decided to do, and I thought that that
5 would be informative.
6 Q. Okay. Have you prepared a chart to sort of explain
7 that causal connection?
8 A. Yes, I have.
9 Q. Okay. I think for this one, you can probably just
10 stay where you are with the easel where it is.
11 A. Especially with this.
12 Q. This is Exhibit 185, right, exactly with the long
13 stick?
14 A. Right, with the long stick. I don't have to get up.
15 Q. So this chart is called conception for causal link
16 between race and admissions?
17 A. Right.
18 Q. What do you mean by that?
19 A. Well, in causal analysis and statistics, the way we
20 think is that we've got, let's say, two alternative
21 treatments. We've got treatment A and treatment B.
22 Now, for each person that we're interested in, we
23 imagine the following, that that person has an outcome
24 under treatment A and an outcome under treatment B, and the
25 difference between the two outcomes is defined,
28
1/19/01 - BENCH TRIAL - VOLUME IV
1 statistically, as the causal effect of the treatment.
2 So if I, if one person has, let's say, I could
3 randomly assign a person to have surgery for heart problem
4 or I could randomly assign to have medicine, and the, and
5 the person would have one outcome under the first
6 treatment, another outcome under the second treatment.
7 Causal effect is the difference between the two outcomes.
8 So we applied that basic idea to the, to the scenario here.
9 What we have on the left, what we have up here is, is a
10 person, an applicant which and this person.
11 Q. You can tell we're not artistic.
12 A. Right. I wouldn't want to be that person, but we have
13 that person. And this person is going to apply to the law
14 school and that person might apply under policy A. Policy
15 A is the current policy, according to the admissions
16 policy.
17 And in that policy, it states a number of factors
18 that are going to be taken into account, and I guess, I'll
19 read them. I don't know if you can see them all;
20 undergraduate grades, the law school aptitude test,
21 Michigan residency, minority status, gender is, could be
22 considered, I assume as a force, a form of diversity. The
23 quality of the undergraduate school, the curriculum; that
24 is the courses that the applicant took, trend in grades,
25 not just are they how or were they going up, relationship
29
1/19/01 - BENCH TRIAL - VOLUME IV
1 with family members who are alumni. There are essays that
2 are required, letters of recommendation and leadership
3 experience. A person may have displayed other unique
4 experiences and talents and then unusual circumstances. So
5 this -- there's this list of factors that could be taken
6 into account.
7 Q. And these are all things, if I could interrupt you for
8 a second?
9 A. Yes.
10 Q. That are reflected in the policy, as you read it?
11 A. That's right. I, I got these right out of the policy
12 document itself. And so our applicant comes and applies
13 under policy A. All of these characteristics are taken
14 into account and the results is this person has a certain
15 probability of admission. We call it a probability because
16 there's some uncertainty in what's actually going to happen
17 here. There's subjective judgments being made and there's
18 some probability of admissions. So we call that
19 probability A. So that's policy A.
20 Now, if our same applicant were to apply under a
21 different policy, and we're going to call that policy B,
22 the result might different. Policy B is, we label a
23 race-blind admissions policy. And the way we're, the way
24 we're defining that is that all of the same factors that
25 were taken into account under policy A would be also taken
30
1/19/01 - BENCH TRIAL - VOLUME IV
1 into account under policy B with one exception, and that is
2 underrepresented minority status. That would not be
3 considered. So we call that a race-blind policy.
4 So our applicant comes along now, low and behold,
5 policy B is in effect. These are taken into account, these
6 factors, and the result is that our applicant has a
7 probability of admission, a piece of B.
8 And so with that scenario in mind, we can define
9 the causal effect of policy A versus policy B as being the
10 difference in the two probabilities of admission. So if,
11 let's say our applicant applied under policy A and got a
12 piece of A, probability under B, a piece of B.
13 Suppose those two probabilities were the same,
14 identical, there would be no causal effect of a change in
15 policy on that person. Suppose, on the other hand, that
16 these probabilities were very different. A person was,
17 let's say, you know, very unlikely to get in under policy A
18 and very likely to get in under policy B, big causal effect
19 of the policy. So that's, basically, how we defined the
20 causal effect. And that was what set up our analysis.
21 Q. Now, why do you think it's important to look at this
22 contrast between two policies in this case?
23 A. There are two reasons. One is that a change from
24 policy A to policy B could effect the diversity of the
25 incoming class and that's one of the goals stated in the
31
1/19/01 - BENCH TRIAL - VOLUME IV
1 admissions policy is to have an ethically diverse class,
2 and so we can use this framework to assess the effect,
3 causal effect on the change of policy on the diversity of
4 the class.
5 The other reason that it's important is that it,
6 it's a way of gauging the causal effect of, on those who
7 apply, I mean, I think that a person who applied to the
8 university, or to the law school, would be very concerned
9 about, are my probabilities going to be very different
10 under these two, under these two policies. If they were,
11 that would have important effect on behavior of people who
12 apply and it's just an important issue and it gauges the
13 extent to which the current policy is strongly effecting
14 the outcomes of people who apply.
15 Q. Okay. And is this kind of comparison between
16 alternative policies the standard way in your field to get
17 at causal questions?
18 A. This has become the, essentially, the consensus in how
19 we think about causation in statistics, two alternative
20 policies, an outcome under each for each person and the
21 causal effect being defined, as I mentioned.
22 Q. Now, how, if at all, does this conception, this
23 approach, differ from what Dr. Larntz did?
24 A. Okay. In Dr. Larntz' analysis, he's analyzing the
25 data that were generated under policy A and computing
32
1/19/01 - BENCH TRIAL - VOLUME IV
1 correlations or associations and trying to use those to
2 make strong causal inferences. And, as I mentioned, I'm
3 convinced that that's not logically possible to do in this
4 case. This kind of analysis --
5 THE COURT: You say in this case, in any case?
6 A. With, well, I think part of the problem is the amount
7 of available information. With, if, with a great deal of
8 information, one might be able to make a better, I think
9 that's an important constraining piece, if there were
10 enough information, but we really had very limited
11 information about the people who apply, numerical
12 information, so I think that's a key constraint on the, on
13 a correlational approach. Generally.
14 THE COURT: Well, you say.
15 A. Sure.
16 THE COURT: Limited numerical information. What
17 other, on your list, there's only certain things that can
18 be equated to numbers.
19 A. Right. And that's one of the difficulties in drawing
20 a causal inference from numerical data is the --
21 THE COURT: Oh, I see.
22 A. If the important, if many of the important factors are
23 not co-indentifiable.
24 THE COURT: I see. Thank.
25 A. That would be a good reason why we didn't have that
33
1/19/01 - BENCH TRIAL - VOLUME IV
1 information.
2 Q. Now, with this conception for the causal analysis in
3 mind, what did you do next in your analysis?
4 A. What we tried to do then was to compare policy A and
5 B, and I think we have an exhibit that displays how we
6 approach that.
7 Q. Okay. Let's put up Exhibit 186 now, the next chart.
8 Does this chart illustrate how you approached your
9 analysis?
10 A. It does. Simulating would happen under policy A was
11 very easy because we actually didn't have to simulate it.
12 We have the data from the years '95 to 2000. So we just
13 actually used, we used the actual reported admissions
14 results to compute probabilities of admission, average
15 probabilities, of admission for various sub-groups who
16 applied, and those were just based strictly on the data.
17 Policy B posed us with a more challenging problem.
18 We don't know what the effect will be on the probability of
19 admission under policy B, because it's never been
20 implemented. So we have to make some assumptions.
21 Essentially what we did was we had data on grades,
22 on test scores, Michigan residency and gender. And we can
23 develop, based on past data a prediction equation that
24 would predict the probability of admission, based on past
25 data. And then from that we can simulate what's happening
34
1/19/01 - BENCH TRIAL - VOLUME IV
1 under policy B. The problem we face is the same problem
2 that Professor Larntz faced. There's a lot of information
3 that we don't have. We don't know anything about the
4 undergrad school curriculum, etc., essays, recommendations,
5 all these other things, these long list of factors. We
6 don't have any numerical data.
7 Q. When you say "we don't know about those things", you
8 mean that, as a statistician looking at the data you don't
9 know?
10 A. Exactly. As a statistician analyzing the numerical
11 database, I only have access to a small fraction of the
12 relevant information used in make admissions decisions, so.
13 Q. The admissions officers have more information than you
14 have?
15 A. Exactly. And that's why, that's one of the reasons
16 why it's difficult to model those decisions. They know a
17 lot more than we do. And we have to make assumptions about
18 what we don't know. In order to do this simulation, we
19 have to assume, essentially, that all of these factors that
20 we don't know anything about are not associated with the
21 factors that are in our model.
22 THE COURT: So you have quite a few there?
23 A. That's right.
24 THE COURT: And Dr. Larntz testified that the
25 fewer assumptions you make, and I'm not saying you have to
35
1/19/01 - BENCH TRIAL - VOLUME IV
1 agree or not agree, but I'd like your opinion on it. He
2 testified that the fewer assumptions you make, the better
3 your results are. That when you start making assumptions,
4 that it may skew it to subject -- I don't think you used
5 the word, subjective, but at least it's more extensive. In
6 your model you're making assumptions, at least, as to one,
7 two, three, four, five, six, seven, eight, nine, ten areas?
8 A. That's right, exactly.
9 THE COURT: So do you disagree with him?
10 A. Oh, I agree with him on that, absolutely, yes. We're
11 very concerned about the impact of the possible falsehood
12 of these assumptions. And there are almost certain to be
13 some falsehoods here. The question is the falseness of
14 these assumptions, the question is to what extent does that
15 effect the result.
16 We know we're not going to really have the model
17 right, but to the extent we have it wrong, to what extent
18 does that have some effect on our results. And that's what
19 we then had to do in this was to, what we actually did was
20 we did this simulation.
21 We looked at the results. We repeated the
22 simulation a couple of other ways, but actually, this is,
23 in some ways that I believe the great strength of the
24 causal analysis. We can put bounds on the errors of your
25 our estimates that require virtually no assumptions, so we
36
1/19/01 - BENCH TRIAL - VOLUME IV
1 can actually assess the extent to which errors in our
2 assumptions effect our results in a very sure-minded way,
3 and I'll try to explain how we did that as we go.
4 So the way, the way it works is, is you do an
5 analysis, based on assumptions, you look at the results,
6 you try another analysis, generally, that's based on maybe
7 some different, slightly different assumptions. But then
8 you try to bound the error in your results as a function of
9 your assumptions, and we we'll show how we do that.
10 MR. DELERY: I think it will be easier to see
11 that, Your Honor.
12 THE COURT: That's fine.
13 MR. DELERY: After we see the results.
14 Q. But before we leave this point, while we're on
15 assumptions and just so we're clear, what, what is, or what
16 are the assumptions about the factors below the line on the
17 chart, as related to the factors above the line, just so we
18 have that in mind?
19 A. Right. Basically the assumption is that if any of the
20 factors below the line are correlated with, with the
21 factors above the line, then our estimate of the effects of
22 the factors above the line will be biased.
23 Q. So --
24 A. And if they're biased, the predictions, the predicted
25 probabilities will be potentially biased as well.
37
1/19/01 - BENCH TRIAL - VOLUME IV
1 Q. I think we'll come back to that as to how you dealt
2 with that, is that right?
3 A. Yes.
4 Q. All right. But before we go to look at the results,
5 let me just ask you a couple questions about exactly what
6 you did. Did you, just as a general matter, did you use
7 any particular kind of, of analysis to undertake the
8 simulation?
9 A. We did. We used -- the first method we used was
10 called, logistic regression. And I think we've had a
11 discussion of that. You have a binary outcome which is
12 admitted, yes or no, and then you have a number of what we
13 call explanatory variables, which are the ones here above
14 the line. And you are able to estimate an equation that,
15 that estimates the relative weights of these factors on the
16 probability, the log odds of admission, and ultimately we
17 can translate like that into the probability of admission.
18 Q. So -
19 A. We've talked about that in court. And I assume we
20 don't need to necessarily say much more about it. I think
21 Professor Larntz explained what that was.
22 Q. And so Dr. Larntz also used logistic regression, of
23 course, as part of his analysis?
24 A. Yes.
25 Q. And we'll get back to Dr. Larntz' regression models.
38
1/19/01 - BENCH TRIAL - VOLUME IV
1 But are there general things that you can say about how
2 your regression analysis differed from, in addition to the
3 conception from what Dr. Larntz did?
4 A. We, yes. We actually estimated our models separately
5 for minority and majority applicants. And the reason we
6 did that was that we found that the association between
7 minority status and admissions was strongly dependent on
8 grades and test scores; that is, we found that, for
9 example, applicants who had very high grades and test
10 scores, for those applicants minority status has a very
11 small effect, or very small association. And for
12 applicants in other cells the association is considerably
13 stronger. So because the association between minority
14 status and these factors varied, what statisticians then do
15 is, they say we can't estimate one model for everybody, we
16 then do the models separately.
17 Q. Did you exclude any of the applicants for which you
18 had data from your analysis?
19 A. No. We used -- oh, I should say, we did exclude
20 people, a very small number of people have have no grades.
21 There's just, they don't have grades in the database. It's
22 a tiny fraction, or they don't have LSATs, so those people
23 we excluded. But we excluded no cases based on their
24 outcomes.
25 And this is a very important point. When you
39
1/19/01 - BENCH TRIAL - VOLUME IV
1 start excluding cases from an analysis based on the outcome
2 of the admissions decision, you get into some significant
3 biases and we did not do that.
4 (Whereupon an off-the-record
5 discussion was had.)
6 Q. All right. So with the simulation model or regression
7 model, how did you conduct your simulation?
8 A. So what we did was we actually, for each year, we did
9 the analysis I mentioned, we did it separately for majority
10 and minority applicants. We actually used the majority
11 equation in predicting the probabilities of admission under
12 the race-blind policy. We assumed that under the so-called
13 race-blind policy that the majority equation, which has
14 more cases involved in the estimation would be more like, I
15 mean, the average equation would be more like that. So we
16 used that equation.
17 Q. Okay. And with that equation, what did you do?
18 A. Well, based on that equasion we could compute the
19 predicted probability of admission under policy B for any
20 applicant, and then we could combine those within ethnic
21 groups to predict the average probability of admission for
22 any sub-group of applicants in this case, as a function of
23 ethnicity.
24 Q. And so from that you can estimate how, what the
25 percentages admitted would look like?
40
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. Exactly. From that we're able to compute the average
2 probability of admissions for ethnic, for minority and
3 majority applicants, and compare it to the observed
4 probability of admission under the current policy.
5 Q. All right. I'm going to ask just one other thing
6 about the simulations. Are you able to, are you able to
7 say, based on the simulation, what would happen to any
8 particular applicant under the alternative policy?
9 A. No, we're not. And this is one of the ironies of
10 causal inference and causal modeling. For any person,
11 we'll never know the two probabilities. In order to do
12 that -- we can't even imagine how to do it. We'd have to
13 have both policies in operation and we'd have to have them
14 implied under both policies and see all the results. But
15 we can't do that. And that's generally true in causal
16 inference. We can't compute the causal effect for any
17 specific case. What we can compute is called the average
18 causal effect. In this case, it would be the average
19 probability of admission under policy A, minus the average
20 under policy B for sub-groups of applicants.
21 Q. Now, let's look at, if we could what happened in your
22 simulations. I think the next Exhibit is 187 in the same
23 category.
24 A. Now, mind you --
25 Q. Yeah, why don't you first tell us what the columns
41
1/19/01 - BENCH TRIAL - VOLUME IV
1 are.
2 A. Right.
3 Q. And then --
4 A. Yeah.
5 Q. Explain what the results are?
6 A. Let me just preface it by saying that the results of
7 policy B are going to be those based on the model I just
8 described, but we also replicated this analysis using
9 another, actually a couple of different regression models
10 we tried. But we also used another method, which we can
11 describe a little bit later. But under the method that I
12 just described --
13 Q. Can I, let me just ask you --
14 A. Yeah.
15 Q. Are the results under the other methods substantially
16 different?
17 A. They're not substantially different. They're somewhat
18 different but in the main, they're very, very similar.
19 Q. All right. So why don't you explain what you have on
20 the chart and then what the results showed.
21 A. Okay. What we have on the chart are two columns,
22 policy A, again, that's the current policy; policy B, this
23 is the so-called race-blind policy that I mentioned.
24 Q. And just so we're clear, the number in policy A is the
25 actual observed data?
42
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. Right. And so we have for minority and non-minority
2 applicants, and for each year, the predicted -- well, in
3 this case under policy A, the actual observed average
4 probability of admissions. And then under policy B, the
5 average probability of admission for that same group.
6 A. So again looking at 2000, we've been looking at 2000.
7 The average probability of admission in 2000 for minority
8 applicants was .35. We project that under policy B the
9 average probability of admissions would be .10, which is,
10 which is quite a large difference. And that type of result
11 occurs in each year. They're pretty similar. There's some
12 exceptions.
13 It turns out that 1995 is a bit extreme in terms
14 of the change in the probabilities for the minority group.
15 But, but it follows the same pattern. It's, and the other
16 years are very similar to, to the year 2000. So we see
17 then in some, a quite sharp reduction in the average
18 probability of admission of the minority applicants under
19 policy A and policy B.
20 A. Now, if we move down to the bottom panel, we have the
21 results for the non-minority applicants under each year.
22 So, again let's just take a look at, for illustration of
23 the year 2000 under policy A the average observed, average
24 probability of admission was .40, 40 percent of those who
25 applied were admitted. We project that under policy B,
43
1/19/01 - BENCH TRIAL - VOLUME IV
1 this is a race-blind policy, that would increase. It would
2 increase from .40 to .44. So it would be rather marked,
3 small or marginal increase in the average probability of
4 admission, .40 to .44.
5 Q. And are the results similar for the other years?
6 A. And the results are very similar for other years. It
7 tends to be, .99 goes 41 to 45, again the difference being
8 .04. In some cases it's .05. I think the actual biggest
9 one we see is in '95, not surprisingly, which is .06, .28
10 up to.34.
11 Q. Now, why is it, Professor Raudenbush, that the change
12 in the average probability of admission is fairly large for
13 the minority students and fairly small for the non-minority
14 students?
15 A. It's a very straight-forward result of the difference
16 in the sizes of the applicant pools. There are relatively
17 few minority applicants, a small -- a large change in the
18 probability, a large reduction in the probability of
19 admission of those candidates translates into a very small
20 increase in the probability of admission of the majority
21 group, because it has so many more applications; basically,
22 any extra, sort of admission seats, if you will, or admits,
23 could become available by reducing this probability, will
24 be competed for by a large number of people.
25 Q. Now, as Judge Friedman alluded to earlier.
44
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. Right.
2 Q. These simulation results are based on regression
3 models which involve assumptions, correct?
4 A. Right.
5 Q. How can you be confident, given those assumptions
6 about these results here?
7 A. Right. Well, the first thing we did, as I mentioned,
8 was we did use an alternative method to do the simulation,
9 and as you asked me were the results similar and the answer
10 was, yes, they were very similar.
11 Q. So the fact that you got similar results says what
12 about these?
13 A. From an approach that did not use logistic regression
14 at all, and I'll explain exactly what we did a little bit
15 later. But the most important way that we can bound our
16 error, if you will, is much more straight forward and
17 requires an absolute minimum of assumptions. And I think I
18 can maybe demonstrate that with a different exhibit.
19 Q. All right. Why don't we go.
20 THE COURT: Let me ask you one question?
21 A. Sure.
22 THE COURT: You can also conclude from that chart
23 that by having a race-blind policy that, looking at 2000,
24 for example, that there's a 25, obviously a 25 percent
25 difference, so that.
45
1/19/01 - BENCH TRIAL - VOLUME IV
1 A. Right.
2 THE COURT: That's right. So you could also say,
3 could you not, that the effect is, the effect having a
4 policy that's not race blind is about 25 percent?
5 A. A difference in probabilities of .25, yes, right. And
6 people do this in different ways. We talk about odds,
7 ratios of probability. Sometimes differences in
8 probabilities are the most straight-forward way of
9 interpreting the results. It depends on the situation.
10 Q. Let me ask a related question.
11 THE COURT: Well, go on.
12 MR. DELERY: Please.
13 THE COURT: You ask, I'll get mine later. He may
14 answer. If he doesn't.
15 Q. Before we look at the bounding point, in your view, do
16 these numbers here, the results of your simulation analyses
17 say anything about the extent to which race is considered
18 by admissions officers in making their decisions?
19 A. They don't.
20 Q. And why is that?
21 A. Let me explain what could generate this difference in
22 probabilities. If you have a large, a much larger
23 applicant pool than can be admitted, so you have many more
24 people apply than you can accept; and if grades and test
25 scores are very important, play an extremely important role
46
1/19/01 - BENCH TRIAL - VOLUME IV
1 in the admissions decision, then a very small difference
2 between two groups can lead to a large difference in the
3 probability of being admitted.
4 And so under this, under our simulation of the
5 race-blind policy, grades and test scores are playing a
6 very important, and extremely important role because we
7 don't have any other data, basically. We know that there
8 are many more applicants than there are seats. And we know
9 that there's a small difference between minority and
10 non-minority applicants. And that explains why this
11 difference turns out to be big.
12 Q. So.
13 A. And it doesn't.
14 THE COURT: Turns out to be big?
15 A. Big, yes, these numbers are quite different. That
16 doesn't depend on how heavily the admissions officer weigh
17 race. It's a function of the fact that you're heavily
18 weighing a factor on which two groups have a different
19 mean.
20 THE COURT: A different what?
21 A. A different mean, a different average.
22 THE COURT: Mean.
23 A. Right.
24 Q. Just so I have a sense of the terminology here, is it
25 your view that there's a difference between measuring the
47
1/19/01 - BENCH TRIAL - VOLUME IV
1 effect or impact of the policy on the one hand?
2 A. Right.
3 Q. And the extent to which a particular factor is
4 considered in an admissions process on the other?
5 A. There's a great deal of difference. And I might add,
6 especially in this case, the causal impact of the policy is
7 much more excessible to statistical investigation than is
8 an attempt to discern how people who are making decisions
9 about admissions are weighing one of many factors, when we
10 don't have any information about most of the factors. It's
11 just a very difficult thing to do, statistically. We
12 basically can't do it.
13 So, but we can assess the impact of what they do.
14 We don't know why it has that impact. You see, there's a
15 big difference between finding a causal effect and
16 explaining the causal effect, knowing why it happens.
17 There are lots of things in social science,
18 medical science, where we know there's an impact on
19 something, but there's so many possible explanations. And
20 we don't have the information to explain the explanation.
21 So this analysis can be conducted with a minimum of
22 assumptions and with a considerable amount of confidence,
23 whereas the more, the much more challenging task of trying
24 to use statistical information to discern how people who
25 have much more information than we do, how they think.
48
1/19/01 - BENCH TRIAL - VOLUME IV
1 This is much more difficult.
2 Q. I think we'll come back to this question of extent a
3 little bit with some additional illustrations, but let's
4 return to the bounding point?
5 A. Right.
6 Q. That you were on, if we could. And I think the next
7 exhibit is 188.
8 MS. MASSIE: Judge Friedman, I don't know if this
9 is, if we could take a quick break, that would be great.
10 THE COURT: Of course, how much do you want?
11 MS. MASSIE: Five minutes.
12 THE COURT: Okay. We'll take a five-minute break.
13 (Whereupon an off-the-record
14 discussion was had.)
15 THE COURT: Okay. You may be seated. Thank you.
16 MR. DELERY: Thank you, Your Honor.
17 Q. Professor Raudenbush, I believe we had been talking
18 about the simulation results for the minority students on
19 the one hand and the non-minority students on the other
20 hand and the bounding issue that you?
21 A. Yes. Just to recreate where we were, the key result
22 here was that the effect of going from policy A to policy B
23 was quite big for the minority students. Like in 19, in
24 2000 it was 25 percentage points, whereas the effect going
25 from policy A to policy B on the non-minority students was
49
1/19/01 - BENCH TRIAL - VOLUME IV
1 quite small.
2 So in 2000, going from forty, .40 to .44, so going
3 up on four percentage points. So that's where we were, and
4 the question is the problems with this model.
5 As we discussed, policy A, policy B is based on a
6 simulation. It's based on a model. The model has to make
7 assumptions. The assumptions, not might be, but probably
8 are wrong, and so how far off might we be, as a result of
9 failure of those assumptions, and that was our next step.
10 Q. Okay. And here, are you talking about the assumptions
11 that the factors not in the model are unrelated to the
12 factors in the model?
13 A. Correct.
14 Q. Did Dr. Larntz' model include the same assumptions?
15 A. Yes.
16 Q. Well, why don't you move to the next chart, actually,
17 and tell us what you did. The next chart will be 188.
18 Tell us what you did to evaluate how reasonable your
19 results were, in light of the assumptions.
20 A. What we did was we used an idea that has a fancy name
21 but it's a real simple idea. The fancy name is, these are
22 non-parametric upper and lower bounds on causal effects.
23 The simple idea is how, how small could the effect be and
24 how big could it logically be. And here's how simple it
25 really is.
50
1/19/01 - BENCH TRIAL - VOLUME IV
1 Again, let's just focus on 2000. And we're
2 looking at majority students here. And we see that in 2000
3 40 percent of them were admitted. How small could the
4 effect be of going to policy B? Well, logically it seems
5 that the smallest the effect could be would be there
6 probability would stay the same.
7 In other words, we go to a race-blind policy and
8 there's no impact. It goes from .40 to .40. It logically,
9 it logically can't really go down. It's hard to imagine
10 how eliminating race as a factor would make things worse
11 for, for majority students. So .40 is the lower bound for
12 the effect. So zero percentage points, .40 to .40. The
13 upper bound is, is constructed, again, very simply; how big
14 could the effect be. The biggest it possibly could be
15 would be if every minority students were rejected under
16 policy B. If you eliminate race as a factor and every
17 single minority students were rejected, then that means
18 that's the biggest effect it could be.
19 And under that scenario, the upper bound is .46,
20 so that means the difference between the lower bound and
21 the upper bound is .06. That's six percentage points. Our
22 estimate, based on our simulation is .04. It's kind of in
23 between the lower bound and the upper bound. So our .44 is
24 undoubtedly wrong, to some degree, but to what degree can
25 it be wrong, the upper and lower bound tell us, it can't
51
1/19/01 - BENCH TRIAL - VOLUME IV
1 be -- the lower bound is a .04 error, the upper bound is a
2 .02 error and those bounds don't require me to make any
3 assumptions about what's in the model, what's not in the
4 model. Those are logical upper and lower bounds.
5 Q. So based on the bounds that you found and, as compared
6 to the simulation results, do the bounds give you
7 confidence in, in your models and in your analysis?
8 A. They give us confidence in the causal effect of the
9 policy change on the majority students.
10 Q. And that's what this chart shows?
11 A. That's what this chart shows. Now, I should add that
12 the bounds on the causal effect for the minority students
13 are wider because like when, I think in 19 -- in 2000 we
14 went from, I think it was something like .34 to ten. The
15 extreme bound would be to zero. So from .34 to zero. So
16 they were a little bit wider.
17 There's a little more uncertainty as to how the
18 switch in policy would effect the minority students. But
19 there's a great deal more -- I should say a great deal less
20 uncertainty about how the change in policy would effect the
21 majority students.
22 Q. Did Dr. Larntz do any kind of similar bounding
23 analysis on the results of his regression model?
24 A. I didn't see any evidence of it in the reports. And I
25 didn't hear him put an upper and lower bounds or a
52
1/19/01 - BENCH TRIAL - VOLUME IV
1 confidence interval on the odds ratios.
2 A. By the way, a confidence interval is a weaker bound,
3 much weaker than a non-parametric up upper and lower bound
4 because this bound has virtually no assumptions. The only
5 real assumption I'm making is that going from policy A to
6 policy B wouldn't hurt the majority students, and that seem
7 indisputable.
8 Q. So these results tell us what the expected
9 probabilities of admission are for, on this chart, the
10 majority students and on the earlier chart also, the
11 minority students?
12 A. Correct.
13 Q. Did you take that analysis any further?
14 A. Yes, I did. Once we have predicted probabilities of
15 admission or average probabilities of admission for
16 sub-groups, we can then develop a picture of what the
17 composition of the first-year class would look like under
18 policy B. Of course we already know the composition of the
19 class under policy A. It's what we observed.
20 And to do this is really very straight forward.
21 We take the probabilities of admission under policy B. We
22 multiple that by the yield which is what fraction of people
23 who were admitted decided to come to Michigan, the one that
24 was actually observed. And that can then give us the
25 expected number of people in each, of each group for each
53
1/19/01 - BENCH TRIAL - VOLUME IV
1 year.
2 Q. Okay. I think we have a chart showing those results.
3 A. Yes.
4 Q. It's Exhibit 129. Just so I'm clear about your last
5 point, Professor Raudenbush, you're assuming in this part
6 of the analysis that the yield rate would not change?
7 A. That's correct.
8 Q. If the university moved to a race-blind admissions
9 policy?
10 A. Exactly. We're, it could arguably go down if this
11 change were made, in which case our results would
12 understate the impact on diversity.
13 We're also assuming, as years go by, that the size
14 of the minority applicant pool would not be effected by a
15 sharp reduction in the probability of admission, which is,
16 which is another conservative assumption. It seems
17 reasonable that if the probability of admission goes down,
18 the number of people who would take the time and effort and
19 pay the price of climb might well go down, but we didn't
20 assume that that would happen.
21 Q. Why don't you look at this chart, Exhibit 189, and
22 tell us what it shows about this next step of your
23 simulation analysis?
24 A. Okay. Again, it's divided. As we go down the, down
25 the rows, we see the years. We have under policy A and
54
1/19/01 - BENCH TRIAL - VOLUME IV
1 under policy B and in each case what's in here is the is
2 the composition of the class. So for policy A it's going
3 to be the actual composition that happened in that year.
4 Under policy B, it's what we would predict, based on the
5 simulation.
6 And again, why don't we just, for illustration,
7 stick with 2000. Under the current policy, 170 minority
8 students were admitted and based on the yield, 58 actually
9 attended. And that was, that turned out to be 14.5 percent
10 of the class.
11 Q. Those numbers were taken from the first chart that we
12 saw today?
13 A. That's right. Those are just the actual observed
14 numbers. Under policy B, we, we would predict that only 46
15 minority students would be admitted. And then applying the
16 yield, that would lead to 16 attending. So only 16
17 minority students, from 58 down to 16, and then that would
18 be four percent of the class, so our, our analysis would,
19 would predict a reduction in the fraction of students who
20 are minority from 14.5 percent to 4.0 owe percent.
21 Q. So what, if anything, do these results, I guess I
22 should back up and ask, is 2000 unusual in this respect,
23 or?
24 A. The basic pattern of 2000 appears each year. We see
25 very similar results. Again, there's a little more extreme
55
1/19/01 - BENCH TRIAL - VOLUME IV
1 result in 1995, but it's basically in the same direction,
2 same pattern, and the other years are very similar.
3 Q. So what did these results tell you, if anything, about
4 the expected diversity of the law school class under a
5 race-blind admissions system?
6 A. Right. So we did see that under this simulation, that
7 the overall composition of the class, which, in 2000 was
8 14.5 percent minority, would be very substantially less
9 diverse with only four percent of the students being from
10 minority background.
11 Q. I think you indicated that there would be somewhat
12 over a hundred fewer minority students admitted, your model
13 predicts, under the alternative race-blind policy?
14 A. Right.
15 Q. What would happen to the spaces in the class that, I
16 guess, those students had accounted for under the current
17 policy.
18 A. Right. Well, --
19 MR. KOLBO: Object to the form, basis, Your Honor.
20 THE COURT: I think it's a pretty obvious answer,
21 but why don't you rephrase it.
22 MR. DELERY: Okay. I'll rephrase it.
23 Q. Can you tell us anything about what the model predicts
24 about where the hundred-plus spaces that had been under the
25 current policy given to admitted minority students? What
56
1/19/01 - BENCH TRIAL - VOLUME IV
1 would happen to those spaces under your alternative
2 simulation?
3 A. Right. Under our alternative simulation, those places
4 which look to be approximately 134 places would be competed
5 for by all of the non-minority students; that is,
6 approximately three, 2,800, whatever the number was, of
7 students that would compete for those places. That's the
8 way we've constructed the simulation.
9 Q. Okay. Now, using these numbers, the predicted
10 composition of the law school class as a whole under your
11 alternative policy, did you do anything to look at how that
12 would translate into the more day-to-day activities of the
13 law school?
14 A. Yes, I did. And I believe we have an exhibit that
15 displays that. Essentially, what we --
16 Q. Why don't we put the exhibit up, if we could.
17 A. What we did, while that's being put up --
18 Q. -- This is 190, by the way.
19 A. People at the law school supplied me with a list of
20 some of the important contexts for learning that arise at a
21 law school. They're listed here and they range in size.
22 The first-year section is the biggest one, 85 students are
23 in the first-year section where students take many of
24 their, several of their required classes. The smallest is
25 a moot court team which is just pairs of people in a moot
57
1/19/01 - BENCH TRIAL - VOLUME IV
1 court, and, and there are other contexts. Each one has a
2 size. And what we did next was to ask questions about the
3 likely composition of these contexts for learning under
4 policy A, which is the current policy again; and policy B.
5 And that's essentially what we did. And I think we have an
6 exhibit that displays the results.
7 Q. Okay. In your view, these, these contexts were
8 representative?
9 A. I was told by the people that supplied these, actually
10 through your office, that these were the representative
11 contexts. And they cover the range of sizes of various
12 contexts. And what's really important from the point of
13 view of statistics here is the size of the context and how
14 does that then look, in terms of its ethnic and
15 composition.
16 Q. Why don't we put up the next chart, if we could.
17 That's 191. What does this chart represent, Professor
18 Raudenbush?
19 A. Okay. So what we've been done is asked questions
20 about the expected composition of each learning context,
21 from the standpoint of a majority student and from the
22 standpoint of, we just picked African-American students We
23 wanted to have a definite type of person, rather than a
24 minority student in mind when we thought about this. And
25 we didn't do it for all of the contexts.
58
1/19/01 - BENCH TRIAL - VOLUME IV
1 We picked three represent -- three that were sort
2 of across the range of sizes. We picked the first-year
3 section, which has 85, then the second row is the half
4 section. And then the residential dormitory entryway.
5 This is an entryway of a dormitory and approximately 25
6 students would be in that entryway.
7 Q. And the results for the other contexts are reflected
8 in your report?
9 A. They're in my report, right. And I think you, this
10 basically captures what's going on here. I don't think
11 it's necessary to go through all these numbers. I might
12 just pick one of them and kind of explain. The first-year
13 section, the biggest context, let's take it from the point
14 of view of the majority student.
15 What's the probability that that would be
16 segregated in the sense that that would be no minority
17 students under policy A and policy B. And the answer is
18 it's a very small like likelihood. Under either policy
19 it's unlikely that there would be no minority students.
20 It's actually .00 versus .03.
21 But then let's ask another question, well, what's
22 the probability that there would be at least, at least
23 three minority students. And it could be nearly certain,
24 which is, approximately, pushing toward 1.0 under policy A,
25 whereas under policy B that would only happen two thirds of
59
1/19/01 - BENCH TRIAL - VOLUME IV
1 the time. There would be a one-third chance of not having
2 as many as three in that section.
3 And then for, what's the probability that there
4 would be, at least three African-American students and at
5 least three Hispanic students in that group of 85. Under
6 policy A it's almost certain to occur. Under policy B,
7 approximately one time out of four. So it's actually not
8 likely to have that agree of diversity. That's the biggest
9 section. The effects of the policy are more pronounced
10 when we go to smaller-size sections.
11 For example, for example, just take, take the, the
12 residential dormitory, what's the probability of having at
13 least three minority students, .75 in that residential
14 dormitory, to picture, 25 people who live in the dormitory,
15 .75 probability that at least three of those people would
16 be minority under policy A. Under policy B, .08, a very
17 unlikely matter. So that kind of demonstrates what's going
18 on from the point of view of the majority student.
19 Things are a little bit different from the point
20 of view of an African-Americans student because, you know,
21 the African-American has to be in the context before we can
22 ask what's happening. So given that there is an
23 African-American, we ask questions, the following
24 questions; what's the probability that you'd be the only
25 African-American student in that context, or, you know
60
1/19/01 - BENCH TRIAL - VOLUME IV
1 what's the probability of three or more of those.
2 So just, we could say, again, take, take the
3 residential dormitory example, under policy A, that's the
4 current policy -- there's a pretty small chance that you'd
5 be the only African-American student, .18, in this
6 residential dormitory. Under policy B, .69, it's very
7 likely that you would be the only African-Americans student
8 in the dormitory. And the probability of at least three,
9 at least two other African-American students would be,
10 would be relatively high under policy A, .56, at least
11 better than half, and very low, .07, under policy B.
12 So I think this gives some flavor of our
13 expectations about what would happen to the diversity of
14 certain contexts for learning under a change in policies.
15 Q. All right. Now, taking all of these simulation
16 analyses together, the overall picture that you've
17 presented here this morning, what conclusions, if any, do
18 you draw about the impact of using race in law school
19 admissions at the university?
20 A. I draw several conclusions. The first is that the
21 impact on the probability of admission of minority
22 candidates would be quite substantial. There would be
23 quite a sharp reduction in the probability of admission.
24 The second conclusion would be that the impact on majority
25 applicants would be modest, by comparison. There would be
61
1/19/01 - BENCH TRIAL - VOLUME IV
1 a small increase in the average probability of admission
2 for majority candidates. And about that conclusion, I feel
3 considerable confidence.
4 Q. And again, why do you think there is that difference?
5 A. And the reason that that's, that difference occurs,
6 that is, you know, why does it effect minority students
7 more than majority students, it's simply a result of the
8 smaller pool of applicants of the underrepresented minority
9 group than of the majority group.
10 Q. Now, so by giving these views and these estimates of
11 the impact of considering race and admissions, are you
12 saying anything about the extent to which the race of an
13 applicant is considered by the admissions people?
14 A. No. We're not making any inferences about how heavily
15 this is being weighed by the people who are making the
16 admissions decisions. We don't have information about that
17 question. But we do have information about the impact.
18 Q. And are these impacts, estimates, telling anything
19 about the relative weights of any of the factors in the
20 admissions process?
21 A. No. They're not quantifying the relative weights of
22 anything in the process.
23 Q. Okay. So as I think you indicated before, this
24 simulation results, simulation analysis, I should say, is a
25 different approach from the approach that Professor Larntz
62
1/19/01 - BENCH TRIAL - VOLUME IV
1 took?
2 A. Correct.
3 Q. Is that your view?
4 A. Correct.
5 Q. Is that your view?
6 A. That's right.
7 Q. How does your simulation analysis bear on an
8 evaluation of Dr. Larntz' work?
9 A. Well, I think that the simulation analysis gives a
10 framework of a policy framework. We've looking at policy
11 options faced by the law school that we can use to
12 understand the reasonableness of some of the results
13 results of Professor Larntz' work.
14 Q. And in your opinion does Dr. Larntz' work provide an
15 accurate or realistic picture of the role that race plays
16 in law school admissions?
17 A. And of course the answer is, no. As I stated at the
18 outset, Professor Larntz attempted to construct a
19 statistical model that could tell us the extent to which
20 race played a role. And I don't believe that we have
21 information that can enable us to do that.
22 Q. And on its own terms, do you believe that Dr. Larntz'
23 approach was appropriately executed?
24 A. Well, I believe that certain key methodological
25 choices that Dr. Larntz made led to a, an exaggerated
63
1/19/01 - BENCH TRIAL - VOLUME IV
1 impression about the association between minority status
2 and admissions.
3 Q. And what were those?
4 A. Well, they're essentially --
5 Q. Just briefly and then we'll get into them a little
6 more?
7 A. I'll give you three types, and I know we'll talk about
8 some of the details.
9 The first was that his analysis selectively
10 attended to the data; that is, it discarded data based on
11 the outcomes of the admission process. And it discarded
12 data that was, in fact, discrepant with the hypothesis that
13 there is a strong correlation between race and admissions.
14 That was the first.
15 The second was that his analysis was based on
16 strong assumptions, as our policy via regression, as I
17 explained the same kinds of assumptions that we had.
18 And that in one important case, I did an analysis
19 that showed that a key assumption that he made and was an
20 important one, was not true. And in the second case, the
21 other, another key assumption is like what I described
22 before. It's probably not true, almost certainly not true,
23 the problem being we don't know the impact. We can't gauge
24 the impact of the falsehood of the assumption on the
25 validity of the results.
64
1/19/01 - BENCH TRIAL - VOLUME IV
1 And thirdly, the results of his analysis were
2 extremely unstable. They were very different from year to
3 year, and the size of the differences from year to year
4 really can't be explained by the process, or by the data at
5 hand. And so my conclusion is that there are aspects of
6 the methodological approach that create the instability,
7 not the admissions policy or the data.
8 Q. Before we talk about those problems that you found
9 with Dr. Larntz' work in more detail, I'm wondering if you
10 could give us a sense of, of how his overall approach, his
11 conceptual framework differed from your's?
12 A. Right. Well, his, his conceptual framework was,
13 again, the idea of constructing a model that would tell us
14 about the role of admissions, the extent to which they're
15 taken into account by the admissions people, which I view
16 as a very challenging thing. You have to have tremendous
17 amount of information to assess peoples' thinking and the
18 extent to which they're weighing factors. My question is
19 actually a more limited one but one that I think we can
20 approach with minimal assumptions through statistical
21 inference and still get some very useful information. It
22 doesn't tell you, it doesn't give us the answer to that
23 question, but it gives us extremely important information
24 about the impact of taking race into account.
25 Q. Now, obviously you were here the other day when Dr.
65
1/19/01 - BENCH TRIAL - VOLUME IV
1 Larntz testified, and there was a lot of discussion about
2 odds ratios, yes.
3 A. Right.
4 Q. Obviously we all remember that.
5 A. Right. I'm just glad I don't have to explain what
6 they are.
7 Q. Is, well, I'm going to ask you to give some examples
8 in a second.
9 A. Okay. I couldn't get out of that one.
10 Q. No such luck. I guess my first question, though,
11 about this is, is computing odds ratios an accepted method
12 of statistical analysis?
13 A. It is. It's widely accepted. It's widely used.
14 Q. And in what context is it appropriately used?
15 A. Well, the thing about odds ratios is that typically an
16 odds ratio by itself doesn't tell us what we need to know.
17 It's a piece of information. But to interpret the meaning
18 of the odds ratios, we, odds ratios, we really need to know
19 something about the probabilities that went into computing
20 the odds ratio because depending on what the probability,
21 you know, an odds ratio controls a function of the
22 probabilities for each group. And depending on what those
23 two probabilities are, the odds ratio could be very, very
24 different things. So my, my general rule of thumb is to
25 always keep in mind the probabilities as well as the odds
66
1/19/01 - BENCH TRIAL - VOLUME IV
1 ratios, for that reason.
2 Q. And you have used odds ratios in your work?
3 A. Oh, yes.
4 Q. Is that right?
5 A. Yes, I have.
6 Q. Okay. In your opinion, do odds ratios provide an
7 accurate or appropriate way to look at the role that ratios
8 make in the law school admissions process?
9 A. There's some problems with using, there's some huge
10 problems with using them alone, again, without, without
11 accompanying them with other information. Generally what
12 happens to the odds ratio is that it becomes very unstable
13 when one group or the other has a probability or, of either
14 nearly one or nearly zero.
15 Q. Do you have some illustrations of that effect?
16 A. Well, I thought we might actually just revisit some of
17 the odds ratios we looked at. Was that, the day before
18 yesterday I think it was, right. The day before yesterday.
19 And maybe we could even just quickly review those. I don't
20 know if we still have those charts or if we need to
21 scribble down those things again.
22 Q. I think we do. I think the page that we have before
23 is gone.
24 A. May I.
25 MR. DELERY: I'll move the easel out a little bit
67
1/19/01 - BENCH TRIAL - VOLUME IV
1 here.
2 A. Thank you. I think what we had the other day was we
3 had a group, some group. Let's call this group one, that
4 had a probability of admission of .99. And then we had
5 group two that had a probability of admission of .90, and
6 the odds ratio turned out to be eleven.
7 So, basically, this was saying group one had
8 eleven times the odds of admission of group two. And then
9 we had another example where group one had a probability of
10 admission of .999. Group two still had a probability of
11 .90. And what happened to the odds ratio was that it
12 became 111. And then just, you can see the pattern here.
13 If group one had a probability of admission of .9999 and
14 group two system had a probability of admission of .91, the
15 odds ratio went to 1,111. Now, those are, those are facts.
16 There's no problem with that.
17 The only problem is, if all we saw, if I hid these
18 probabilities, and all I saw were the odds ratios, I might
19 get the impression that those are three extremely different
20 results. Eleven times the odds, 111 times the odds, 1,111
21 times the odds. These look so different. But when I look
22 at the probabilities of admission from a practical point of
23 view, if I'm a candidate, and my probability is .99 versus
24 .90, that's about ten percentage points.
25 And I'm nearly certain to be admitted. If I go up
68
1/19/01 - BENCH TRIAL - VOLUME IV
1 to .999 versus .91 it's still about ten percentage points.
2 I'm still nearly certain, but yet my odds ratio went up by
3 ten, a factor of ten. And then another factor of ten as we
4 go to .999. So all I'm saying is the odds ratio by itself
5 can create a misleading impression if you don't also see
6 these numbers.
7 Q. Is there something about the mathematical
8 characteristic of the odds ratio that causes this, I mean,
9 is that the reason?
10 A. The basic problem is that an odds ratio requires
11 division. And if one of the probabilities is either near
12 one or near zero, we encounter something called division by
13 zero which is prohibited, mathematically. We can't have a
14 fraction that has zero and nine.
15 Q. And so what's the results of that?
16 A. And so as the denominator goes towards zero, the
17 fraction increases without bound to incredibly large
18 numbers. If we keep adding nines, this thing keeps going
19 up and up and up.
20 Q. And does the same pattern happen when you're talking
21 about small probabilities at the other end?
22 A. Exactly the same pattern happens, so, for example, if,
23 I just switch it around. If group one had a probability of
24 admission of .10, and group two had a probability of .10,
25 the odds ratio would be eleven.
69
1/19/01 - BENCH TRIAL - VOLUME IV
1 If I went from, again, group one, .0 to group two
2 .001, 111, .10 to .0001, 1,000, 111. So again, group one,
3 ten percent chance of getting in, group two, very small
4 .10, very small, very small. Ten percentage point
5 difference leads to very, very different odds ratios.
6 Q. Do you have an example of a situation in which two
7 people might have similar probabilities of something
8 happening, but very different odds or a real world example?
9 A. Yes, actually, I did think of one. It actually
10 involved the lottery. Suppose that, you know -- I get
11 excited about the lottery and I buy a lottery ticket. And
12 you say, well I'm going to outdo you, I'm going to buy
13 fifty lottery tickets.
14 So what would happen is your odds would be roughly
15 fifty times, mine. But yet both of us would have near zero
16 probability of winning the lottery. I mean, it's wise,
17 you'd say, I'm going to be really smart and go buy
18 thousands of tickets to the lottery. Everybody would be
19 buying. Of course they are, but.
20 THE COURT: Actually this week it's fifty-nine
21 million. There's a sign on my way home. Every time I keep
22 looking.
23 A. They're doing it. They're rapidly increasing their
24 odds, but what they don't know is their probability is
25 staying right almost exactly at zero.
70
1/19/01 - BENCH TRIAL - VOLUME IV
1 Q. All right. Okay. If you could take the stand.
2 Professor Raudenbush, in your view does this pattern that
3 you've just described to us examples have any relevance to
4 the data we have in this case?
5 A. They do. There are combinations of grade point
6 average and LSAT where the probability of admission of
7 anyone who applies to the law school is extremely high. I
8 mean, people who have near A averages who are up in the
9 upper 160's or 170's on their LSAT have an extremely high
10 probability of admission.
11 Of course in the data what we see is that the
12 proportions are something like 1.0 for minority applicants,
13 and something in the .9 range, or in a very high range for
14 majority applicants. And so in that sense, the examples
15 that I was presenting were not unusual. And something
16 similar can also, and does appear at the lower end of
17 people who have fewer qualifications where the differences
18 may be small in probability terms, but the odds ratios may
19 be big.
20 Q. With that background in mind, I'd like to ask some
21 questions about the cell-by-cell analysis, that Dr. Larntz
22 conducted.
23 A. Okay.
24 Q. Just, let's start with a general question. What's
25 your opinion about the of the appropriateness or the
71
1/19/01 - BENCH TRIAL - VOLUME IV
1 validity of that approach?
2 A. Well the problem, well, one of the problems with that
3 approach is that it requires that an odds ratio be
4 computable for every single one of the hundred plus cells
5 that appear in any year in Professor Larntz' reports. And
6 since the odds ratio is not computable in a number of
7 cases, what this leads to is a discarding of data in those
8 cases where there can't, where no odds ratios is
9 computable. And this ends up discarding considerable
10 evidence that are relevant to how the university is
11 handling the admissions decisions.
12 Q. And I believe we had some examples?
13 A. Yes.
14 Q. Of those situations?
15 A. We do.
16 Q. I think this is Exhibit 192. If you'd put that up.
17 Maybe, David, if you could put the easel back where it was.
18 Can you read it from there?
19 A. Yeah, I can see the numbers from there.
20 Q. Okay. There's very small, actually.
21 A. I'll --
22 Q. I think I need to come closer.
23 A. Okay. All right.
24 MR. DELERY: If that's all right, Your Honor.
25 THE COURT: Of course.
72
1/19/01 - BENCH TRIAL - VOLUME IV
1 Q. So I guess let me first just ask, I take it this page
2 here on the left is a little image of a page from one of
3 Dr. Larntz' reports?
4 A. Yes. That's page six of six from the March 20, 2000
5 report. And we selected that page. It was just convenient
6 because it had three examples that I wanted to say
7 something about, because it has a bearing on what we're
8 discussing, and they all came from the same page.
9 And the first example, actually, the first two
10 examples involve cases in which the admissions process
11 treated people the same, in terms, they had the same
12 admission decision regardless of minority status. So in
13 the first cases, and we're looking here at, at students who
14 have relatively low grade point average. It's down 2.25 to
15 2.49, but relatively high LSAT's, 161 to 163. There was
16 one minority applicant in that, in, who had those
17 characteristics. And that person was rejected. There were
18 two majority applicants and they were rejected because both
19 people were rejected. Of course, what we know is they both
20 had the same admissions decision. There was no different
21 decision for the minority and majority applicants. But
22 because none of them were admitted, we can't compute the
23 odds ratio. So if you've developed a statistical approach
24 that requires cell-by-cell computation of odds ratios, you
25 can't compute the odds ratio.
73
1/19/01 - BENCH TRIAL - VOLUME IV
1 Basically what happens is you have to discard
2 their cell. But when you discard this cell, you're
3 discarding information that's relevant to the decision of,
4 it's relevant to the decision made by the admissions
5 committee. That is, essentially, you're waiting to see
6 what the admissions committee decides.
7 And if they make it a certain decision, which in
8 this case is treating everybody the same by rejecting them,
9 discard the data. If the admissions decision had been
10 different, if, if someone had been admitted, then the cell,
11 the data would have gone into the analysis. So that means
12 the data goes into the analysis conditional on the decision
13 of the university.
14 If the university makes a decision to treat
15 everybody the same, we throw the data out. If the
16 university decides to treat them differently, the data go
17 in and we -- we don't like that situation in statistical
18 analysis. This, we don't wait to find the outcomes of the
19 data and then decide whether to use the data. We decide
20 what data we're going to use prior to, to, to investigating
21 the outcomes, or without any attention paid to what the
22 outcomes are that we're trying to discuss.
23 Q. Okay. And what about the second cell here?
24 A. Well, the second cell is another example of the same
25 thing but it's at the upper end of the distribution.
74
1/19/01 - BENCH TRIAL - VOLUME IV
1 In this case we have people whose grades are 3.75
2 and above. This is very high. These people are getting
3 basically A's, maybe a few A minuses. Their LSATs are also
4 very high. They're 167 to 169, which I think is very high
5 up in the percentiles of that distribution so these are, in
6 terms of just grades and test scores this is a very able
7 group of applicants.
8 In 1999 there were two minority applicants. They
9 were both admitted. There were 106 majority applicants.
10 They were all admitted. So you look at those data, and I
11 think reasonable people would say, did race play a factor
12 in the decision for those people. And the answer seems to
13 be no. They had very high grades and very high test
14 scores. They all had the same decisions. The decisions
15 weren't different. However, can't compute the odds ratio,
16 throw out the data.
17 Q. Well, let me ask you about that, because Dr. Larntz
18 said that these cells don't have comparative information in
19 them, as I understand it, and so a principle or fair
20 comparison should mean that you would discard them. You
21 would look at only cells where you have different results.
22 Do you disagree with that?
23 A. I strongly disagree with that, and I'll try to explain
24 why. We only know after the fact that these people had the
25 same treatment. To then say, well, because they had the
75
1/19/01 - BENCH TRIAL - VOLUME IV
1 same treatment we're going to throw them out, no, you can't
2 do that. The admissions decision could have gone the other