Posts Tagged ‘statistics’

Is the bar exam pseudoscientific?

July 9, 2013

Ah, July. The sun is shining, the flowers are all in bloom, it is a time of vacation and warm beer, a time where slower clocks strike happier hours… except for the thousands of recent and not-so-recent law school graduates (and congratulations to all of y’all) around the country who locked in libraries studying for the Multistate Bar Exam and, more or less universally, at least one state’s version of that test. The MBE and its state equivalents are, for first-time takers, required in most jurisdictions to get the right to practice law in that jurisdiction (though once you’ve passed the bar in one state, you can often gain admission to the bar of another state by motion instead of by examination), which is to say, passing the bar the first time around is of the utmost importance to law school graduates. The oncoming train of massive student loan payments, measured against the time that is lost and the new debt that is incurred in preparation for the MBE, have even led some states to call for the abolition of the bar exam, or at least, to let law school students take it before graduating. 

The bar prep process is a long and arduous one; in a way, the entirety of law school is a form of bar prep. Institutions both public and private run bar preparation courses that can run students thousands of dollars, and the fees for the exam itself, just for the application to take the bar, put hundreds of dollars on top of that – fees that you don’t get back if you fail. But what is it all for?

Ostensibly, the goal of the bar exam is to ensure that only competent practitioners are able to advertise and provide their services, under the auspices of the National and the fifty state Bar Associations. The goal seems worthy enough to me – I rather enjoy the prospect of not competing with incompetents, and enjoying at bar association meetings only the mighty company of titans of legal acumens. Except, of course, that such is not the case. Cretins abound in the legal profession, as any lawyer, cretinous ones included, will tell you, and the ranks of the MBE’s failure include powerful career success stories  ranging from Kathleen Sullivan and John F. Kennedy to Michelle Obama and Hilary Clinton. Evidence both anecdotal and empirical such as the success stories cited above show it to be plainly false that the MBE screens out bad lawyers and admits only good ones.

There is shockingly little data available on whether or not the MBE or any particular state bar exam actually correlates with career success, though what data does exist shows a sort of self-serving prophecy at work: given that bar exam failure delays the start of a career by many months until the next exam date, and that no law firm has any reason to hire someone who has not passed the bar, bar exam passage really does have some correlation with career success. So it seems unsurprising that advocates for the MBE could say that MBE success accurately predicts a certain level of career success; career success isn’t even possible until MBE success is had!

What should be more troubling for any consideration of the MBE is that failure is statistically a greater affliction of the non-white and the non-male than for the white and the male. I have a difficult time imagining any reasons of any academic or jurisprudential virtue that  would cause a supposed test of legal competence to disproportionately disqualify those who are merely demographically different from the last generation’s lawyers. Perhaps, however, since the MBE reflects systemic problems in American law itself, problems that build institutional barriers to non-whites and non-males.

But this, too, is empirically known to be untrue, since the available evidence suggests that MBE passage rates has remarkably little to do with what is actually taught at law schools. Unless we are really willing to swallow the proposition that more is taught in eight-week bar prep crash courses than in the entirety of the rest of law school, this I think raises the troubling possibility that the MBE, whose success does correlate strongly with LSAT scores though the LSAT has nothing at all to do with the law according to a study cited above, is actually a measure of the kind of intelligence that the LSAT itself measures.

The LSAT does a remarkable job of predicting law school performance. Law school performance strongly predicts bar exam results (along with gender and race). Gender and race track to LSAT scores almost precisely as well as they do for the MBE. Yet while the MBE tracks well to overall law school performance, it does not track well to what one actually studies in law school – whether one studies bar-intensive subjects like contracts and constitutional law or more esoteric bar-unfriendly courses like entertainment law or the laws of war does not have a significant impact on MBE performance. So what is the MBE actually measuring? Is it measuring anything other than general “intelligence” where “intelligence” is defined as “an ability to succeed in law school?”

And do we want lawyers who are good at law school, but whose actual abilities to be lawyers are, as a statistical fact about the MBE, completely unknown?

Stranger still, the MBE is basically a gatekeeper to state and national Bar Association membership – yet those organizations do not, strictly speaking, exist to discipline or screen out bad lawyers, but rather unethical lawyers. Bar associations technically have no jurisdiction whatsoever, but as a practical matter they are the ethicists and comptrollers of the legal profession; bar associations punish for accounting shenanigans and violations of ethical rules, but only legislatures can properly create actual crimes, even where the legal profession is concerned. And genuine incompetence is actionable only by bar associations when it amounts to a matter of ethics, when it would actually be immoral to let a person continue to practice based on a history of misrepresentations or false promises, not on how many cases somebody has won or lost. So the MBE supposedly (but I think, as I’ve shown, falsely) promises to screen incompetent lawyers from entry into an organization that has no power whatsoever to punish lawyers for general incompetence.

Based on the evidence that I’ve prevented here, the multistate bar exam is by many relevant measures pseudoscientific. It is circular and self-fulfilling insofar as the success it measures is governed by success on the test itself. It claims to be a measure of competence at law but has statistically significant gaps in success based on factors that are utterly irrelevant to legal competence, such as gender and race. It tests something that bar associations have no technical ability to even control for post-passage, which is actual success. And yet as we speak, there are thousands of young Americans around the country cramming for it as if (because?) their careers depended on it. Quality controls of are the utmost importance in the legal profession, but unfortunately, we have too many good reasons to suspect that the MBE is not even giving us that.