Rankings Stink Stirred up again: Gotanda and Henderson and the Distortions the Rankings Have Wrought

National Jurist has two articles that raise some new and unpleasant questions about the rankings game. A New Low in the Rankings Arms Race, by Jack Crittenden chews over again Villanova's February admission that it misled the ABA in reporting LSAT and GPA statistics for years. Villanova's new dean, John Gotanda, announced last January that he found that the previous administration had been knowingly reporting false LSAT and GPA data to the ABA for some years up to 2010, in an attempt to raise its ranking with the US News and World Report. This announcement confirmed long-running rumors of smoke and mirror tricks at certain schools where the numbers just seemed too good to be true. What is surprising is that the first school stepped forward voluntarily, rather than slipping up and being caught. Villanova and Dean Gotanda deserve a round of applause for rock-ribbed morals and principled behavior. The new dean launched an immediate internal investigation, hiring Ropes & Gray for an independent audit, which found significant differences between what should have been reported and the figures which were turned in. The admissions reporting process and organizational structure will be reconfigured as part of the housecleaning.

LSAT score has a weight of 12.5% and GPA weights 10% in the US News algorithm. In his Morse Code blog, Bob Morse said in February that US News will not revise the older rankings to take the inaccurate data into account, though he believes that the weights of the LSAT and GPA scores in the US News algorithm are enough that it would have affected Villanova's ranking.

How does the rankings data for the J.D. class entering in 2010 compare to the previous year's? The difference is significant enough between the older and newer data to have a meaningful negative impact on Villanova's upcoming ranking: For the fall 2009 entering class, Villanova reported inaccurately a median LSAT score of 162 and median undergraduate GPA of 3.44. For the fall 2010 entering class, Villanova certifies its median LSAT score was 160 and its median undergraduate GPA was 3.33.

According to the National Jurist article, Villanova's ranking in 2011, with the new, accurate data, dropped to 67, from 61 in 2010, (though it was 68 in 2009, and 60 in 2008 – the mysteries of the rankings!).

The article goes on to quote Bucky Askew, long-time consultant on legal education for the ABA, as saying that it is very rare for inaccurate data to be purposefully submitted to the ABA questionnaires, though there are currently concerns over the accuracy of employment data and that the ABA is reviewing that. Oh, what a very optimistic man! Or perhaps, a diplomatic one.

Paul Caron, is a tax professor at the University of Cincinnati and leading blogger at TaxProf Blog and Editor in Chief of the Law Professor Blogs Network. He also publishes Leiters Law School Rankings, an alternate ranking system that competes with US News and World Report's ranking system. Prof. Caron critiques the collection of data by the ABA, and suggests that using information directly from the Law School Admissions Council (LSAC) , for instance, where law students must register to take the LSAT and have their undergraduate school send their GPA before applying to law school would be a better and more reliable resource for the LSAT and GPA data. At this point, the LSAC is not at all involved in the ABA questionnaire data gathering process, and there are no checks and balances on the law schools' self-reporting.

Probably the reason to pull the Villanova issue back up is two-fold. One is simply to encourage folks not to beat on the school – that is the last line of the article, “If we place too much scorn on Villanova, other schools will be scared from stepping forth and doing what is right,” quoting Professor Bill Henderson of Indiana University. Henderson writes an opinion piece that is the second article on the topic in the National Jurist, “How the Rankings Arms Race Has Undercut Morality.” Henderson has some excellent insights:

For better or worse, the rankings created new metrics of competition. … we never expected to get an education on how competition in a magazine could produce a competitive environment in which bad behavior becomes the norm. Each year the rankings arms race seems to produce a new low in law school conduct. …

The story we can write now is a tragedy. It has many victims and no heroes. And it is not yet over. (snip)

Over time, a terrible dynamic has emerged in which bad behavior among law school administrators is rewarded and principled behavior gets you fired. Virtually everyone hates this system. Unfortunately, it is all too easy to rationalize unethical actions for the greater good of one's school. … Career services and admissions professionals – who are not tenured and so have no strong job security – are under tremendous pressure to increase or hold onto their numbers. They justify their actions (to themselves and others) by saying the practices are the norm throughout the law school hierarchy. As a result, pretty soon they are the norm.

Henderson describes how the rankings initially reflected the general consensus among academics about how the various schools were ranked by public opinion. But as alumni and university administrators began to realize that the rankings really influenced applicants, pressure began to be applied on deans and the professors of the law schools to improve the ranking of the school. By 1995, the US News editors had been seeing “disturbing discrepancies” between median LSAT scores as reported to the magazine and the data submitted to the ABA for accreditation purposes. The editors of the US News and World Report wrote in their Law Schools rankings issue that year a plea, “Will the ABA eventually insist that schools make such data public?” Eventually, the ABA, under public pressure, began to publish an annual ABA-LSAC Official Guide to Law Schools, which included such data, and US News began to rely on this resource for most of its rankings data.

Later in the opinion piece, Henderson explains the miracle of the increasing percentages of law school graduates employed after graduation reported to US News, even during the worst legal hiring slump in memory, and when lower percentages of these same graduates have managed to pass the bar exam. The “employment” need not be legal; it can be a job at a fast food joint, or the law school itself, temporary or part time. And because US News automatically counts 25% of “unknown” graduates as employed, canny career services personnel may avoid contacting graduates whom they suspect might not be employed. Henderson asks, if this is not consumer fraud, are we not at least perilously close? Yet the employed at 9-months statistic can drop a school several places in the rankings if it is not high enough. Students pay close attention to this number, naturally enough. Recent pressure on deans and law schools, and the ABA itself from students and alumni who have been embittered by the fraud their schools have perpetrated in this one area, led by the folks at www.lawschooltransparency.com, has accomplished a great deal.

Prof. Henderson recounts how, as pressure mounted to maintain or increase ranking, law schools changed how they admitted students. They no longer looked at resumes or personal statements or letters of recommendations, focusing more and more on formulae to increase the LSAT median scores and increase undergraduate GPAs. Schools changed how they awarded scholarships, giving less on the basis of need, and more on the basis of merit, in order to lure and retain students with the higher LSAT scores and GPAs in law school.

The effect has been to reshuffle the priorities and the make-up of the law schools. Rather than admitting students who may not have stellar pre-law credentials, but have potential to become great lawyers and add something to the school or bar, or who will increase the socio-economic and ethnic and racial diversity of the school and bar, schools are focusing solely on increasing the LSAT and GPA scores in order to maintain or better their rankings. This focus almost naturally brings a higher percentage of students from affluent backgrounds, and tends to reduce the levels of diversity in all categories. They also tend to need the scholarships less in terms of financial need. Yet, that is where the law schools are spending their scholarship money.

Students who have to work during law school tend to get lower grades because they cannot put in the longer hours of study, and are distracted. So there is a vicious cycle at work for the lower-income students who do not receive need-based scholarships. They have a higher debt load, and thus need to work more. The more they work, the less they can study. And so their GPA drops, and they are less likely to receive a merit-based scholarship. And because their GPA and class rank is lower, they are less likely to get jobs upon graduation, when they leave law school with a higher debt load.

Henderson argues that, rather than creating an objective, measurable ranking system that creates a fair way to distinguish among the law schools for consumers, the rankings systems have created instead, a warping gravity well, that pulls law schools out of shape and into its orbit. He calls for the ABA to create a clear and rigorous standard that requires reporting of data on post-graduation employment, including clear explanation of how the data was compiled and documented. The data should then be verified as accurate and randomly audited by neutral third party professionals. Discrepancies should result in severe penalties for law school administrators. The data should be open to the public in a way that makes it easy to compare school to school. Henderson notes that the information is already reported to National Association of Law Placement, but is never made public by school. He calls for the lawyer members of the ABA to override the academic members who have such obvious self-interest in this matter. And if the ABA cannot act, Henderson goes so far as to call upon the Department of Justice and Congress, and various state supreme courts, which regulate the legal profession to take away the ABA's accreditation authority and act directly.