Deception With Statistics


Burt Likko

Pseudonymous Portlander. Homebrewer. Atheist. Recovering Republican. Recovering Catholic. Recovering divorcé. Editor-in-Chief Emeritus of Ordinary Times. Relapsed Lawyer, admitted to practice law (under his real name) in California and Oregon. On Twitter, to his frequent regret, at @burtlikko. House Likko's Words: Scite Verum. Colite Iusticia. Vivere Con Gaudium.

Related Post Roulette

49 Responses

  1. Avatar Jaybird says:

    I wonder how much is putting the cart before the horse… someone who is fairly smart and fairly driven will, all things considered, make more money than someone who is one but not the other and a lot more money than someone who is neither (all things considered).

    And, as far as I can tell, you have to be fairly smart and fairly driven to go to law school, graduate in something like the top (whatever’s worth mentioning) and pass the bar the first or second or third time.

    If we want to use money as a good-enough marker for “success” (and why not?), I’m pretty sure that the folks who were capable of going to law school and passing the bar would be folks who would have made money had they instead gotten MBAs, jumped into the whole sysadmin thing, or did all they could to swim with the sharks in academia. (Of course: some of these jobs are more suited for some personalities than others and you need to take that into account… but smart/driven is smart/driven.)Report

    • Avatar Burt Likko in reply to Jaybird says:

      “Smart and driven” is probably as big a factor as “has a graduate degree,” that’s certainly true.Report

      • Avatar Jaybird in reply to Burt Likko says:

        I’m just saying that you take that same kid at 18 and dump him in one of the military academies, you’d end up with a full-bird colonel (if not general).

        All things considered.Report

        • Avatar Barry in reply to Jaybird says:

          “I’m just saying that you take that same kid at 18 and dump him in one of the military academies, you’d end up with a full-bird colonel (if not general).”

          I was told that there are generally 6 people making colonel for every thousand LT’s, so that would probably only apply to the grads of top law schools, and the top grads of some other law schools.

          OTOH, most law school grads will have possession of what a Michigan law student told me years ago was the key to getting through law school – the ‘iron butt’ (to study for a loooooong time).Report

    • Avatar NewDealer in reply to Jaybird says:

      The joke we had in law school was that lawyers were people who wanted to make money but were too nerdy for business school.

      In my school, we can from a variety of academic backgrounds but I think most of us tended more towards the arts and humanities than the sciences. There were a few science people among us. One classmate decided to go to law school because she did not want to get a PhD.

      I toyed with the idea of getting a PhD but my fantasy (assistant professor of drama in a small picturesque liberal arts college) did not team up with the current reality of academia (adjunct hell).Report

    • Avatar Barry in reply to Jaybird says:

      Something which Paul Campos pointed out repeatedly (and probably others have) is that the current system takes 22-year olds with good to great grades and lousy immediate job prospects, and makes it incredibly easy to go into $100K – $300K into debt (i.e., buying a house level of debt). Given bleak immediate prospects, it’s hard thing to turn down.Report

  2. Avatar Brandon Berg says:

    If anyone else was wondering, I checked the paper, and those figures are net present values, and they also controlled for ability bias.Report

  3. Avatar Don Zeko says:

    I have nothing of note to add, other than that I think this is a good post and that I have an intense, personal, and entirely selfish interest in the answer to the question considered.Report

  4. Avatar Griff says:

    One man’s opinion — law school is “worth it” if you meet any two of the following three conditions:

    (1) You really, really, REALLY want to be a lawyer;

    (2) You get into a really good law school;

    (3) You get something very close to a full scholarship (provided your school doesn’t habitually revoke these).Report

    • Avatar NewDealer in reply to Griff says:

      1. I decided to become a lawyer later in life after not making it in theatre. Like many lawyers, I am a bit too academically inclined for business but not academically inclined enough for the academy.

      2. My school was a respectable Tier II when I attended but not Tier I

      3. I did not get a full scholarship but was lucky to graduate with no debt.

      I’m still doing better than I ever did in theatre and working project/temp jobs.Report

  5. Avatar NewDealer says:

    The joke I’ve heard a million times and I am sure you heard a million times is this:

    “A students become professors, B students become judges, and C students become really rich lawyers.”

    It has always been my thought/impression that most of these really rich lawyers were in the plaintiff’s bar. That place of low respect (ambulance chaser, shark, enemy of business etc) but if you do it right you can become very successful in financial terms.

    My dad graduated law school in the 1970s. It took him one year to find his first law job. His first law job paid less than his day job (he went to law school at night). Now he is doing much, much better than he ever would have done if he kept at his day job. Did it take him a while? Yes, it took him decades.

    Flash forward forty or so years later. I graduated law school in 2011. I should be railing against the world according to Paul Campos because I was “scammed” But I knew I would never be a big firm lawyer. I don’t like that I am still working on a project to project basis but I am slowly (very slowly) building a career (I hope) and doing much better financially than any other time in my life. Not saving much but I pay my rent and my insurance. So I am somewhere between that brass ring and the guy or gal working at a coffee shop or bagging groceries. What does Paul Campos want me to do? Give up so he can have one more statistic? Honestly the law school scam people piss me off. Though I said this a million times and most people still rush to the defense of St. Paul Campos, Brave Truth Teller (TM).

    Do I like this project to project life? Not super-much. Do I get scarred about it being my permanent condition? Yes! Does that mean I am going to throw in the towel? No.

    I think in American culture we are too focused on instant success. As a society we have come up with the idea that you need to be the person with the big firm associateship right out of the gate or you will be a loser. We don’t focus on people who took a longer time to build their careers. I think it takes decades to build a career and struggling a bit in your 20s and 30s should not mean damnation to an unsuccessful life.Report

    • Avatar Burt Likko in reply to NewDealer says:

      I think your last point is very well-taken and one that I don’t think the study spent a lot of time looking at: it takes a lifetime to build a successful career. And while there are exceptions, the typical story is that all that hard work goes in before the money comes back.Report

      • Avatar NewDealer in reply to Burt Likko says:

        Agreed. I don’t really blame the paper for this. This is endemic to American society. We like the young and successful like Lena Dunham, Athletes, Rock Stars, etc. We don’t really pay attention to the fact that Gene Hackman was working as a doorman well into his 30s before his acting career took off.

        Something like this list seems more common but probably more of a downer to talk about:

        • Avatar Barry in reply to NewDealer says:

          “Agreed. I don’t really blame the paper for this. ”

          This is not celebrity new, this is supposed to be a factual analysis. If they focus only on those who succeed, then they haven’t done an NPV. There’s a joke about people doing a ‘benefit analysis’ when they are supposed to be doing a ‘cost-benefit analysis’.Report

          • Avatar NewDealer in reply to Barry says:

            Okay. It is not the best example but I could not find a real world version.

            The truth of the assertion that it takes a long time to build a career still stands and that graduating into a bad economy is not necessarily going to damn someone or at least hopefully will not.

            Burt pointed out that he went to law school during the Clinton admin, a much better time for law school grads and the economy. He still heard stories that it was the brass ring or packing groceries. There don’t seem to be many stories about the person who worked in government or started out at the small/medium sized firm. It is brass ring or bust while ignoring everything inbetween.

            The same is true for college. We seem to think it is all people will end up at Google/Apple/Wall Street/Consulting or they will be working in a coffeeshop, as an admin assistant, or some other job that might not require a college degree.Report

      • Avatar Barry in reply to Burt Likko says:

        Burt, I’ve heard comments about this but haven’t read it yet. From what other people are saying:

        1) They used a sample of currently practicing lawyers who graduated in 1985 and later. IS that true? Because if so, there’s massive survivorship bias there – the ones who were not successful, and who left the practice of law are not included. Now, some of them may have left law to become millionaires in another field, but my money would be on them having spent several years to over a decade miserably struggling before admitting failure.

        2 If they don’t account for the cost of law school, then they don’t have a NPV calculation. You said in your post ‘Nothing really has changed since these long-ago days. Law school is still ridiculously expensive.’ That’s not really true, the cost of law school has changed vastly since those days, *becoming* far more expensive. A lawyer graduating in 1990 might have thought that they paid a lot for their JD; a lawyer graduating in 2010 paid far more, even if the the earlier lawyer had a degree from an elite school and the 2010 law grad had a degrees from the UrFookdNow School of Law.

        3) Adding on to the survivorship bias, present grads have an overall chance of getting law jobs of 50% (at nine months after graduation, but at that point a fresh class is coming into the job market). Multiplying the law career salary by 50% up front, even before attrition, knocks the NPV down quite a bit. If they didn’t account for this, they are bullsh*tting. Law professors don’t have an excuse for being ignorant about the job prospects of their students, and law professors alleging that they studied the worth of a law career have even less than that.

        4) Bayesian prior. The bleak prospects and astronomical tuition for law schools shocked me, as I read ‘Inside the Law School Scam’ and other blogs, but not as much as the raw stinking fraud of the law professors and especially deans of law schools. I’m not sure what the limit is on the depth of this, but it’s not been encountered yet. And we’ve seen things like schools which counted all jobs as ’employed’, but only some jobs for salary statistics (they’re admitting it now, but the stats haven’t changed from when they weren’t admitting it, which is flat-out fraud in my book). I’ve seen a school baldly stating that 24% of their new grads were unemployed and not seeking employment, which had to have been a lie, and then change that the very next year when the rules changed – which means that either their demographics changed radically within one year, or that they were lying. The reason I say this is that this analysis can’t and shouldn’t be treated with any sort of up-front respect; it should be treated as if it were produced by Goldman Sachs – no figure can be trusted, all definitions and assumption are to be considered as having been made up to prove the desired conclusion, no matter how far they are from what a ‘reasonable person’ would think.Report

    • Avatar Barry in reply to NewDealer says:

      NewDealer: “What does Paul Campos want me to do? Give up so he can have one more statistic?”

      Please stop lying. Paul Campos has *never* said that, and never *implied* that. What he has pointed out is that outcomes like yours are far more common than the law schools state. He points out that you are far from a failure; you’re probably modal now, and that people should take into account the *fact* that a goodly proportion of grads will end up envying your position.Report

      • Avatar trumwill mobile in reply to Barry says:

        Crikey, Barry. There is a difference between ‘says something that needs to be corrected’ and ‘is lying’.Report

        • …yeah. I like Barry’s spirit and energy A LOT already (do we know him under another name? …he’s comfortable like he’s an old-timer, which if he isn’t is awesome), but I was inclined to ask him to put a safety on the L-word trigger, as it really shouldn’t have been pulled there.Report

          • Avatar Barry in reply to Michael Drew says:

            Will do. Please change my comment to say:

            ‘NewDealer, what you’re saying is factually wrong, and I’d appreciate it if you’d post any facual basis for thinking that. Where ‘factual basis’ is on the level that a professional in evidence and facts is supposed to operate.’Report

            • Avatar Jaybird in reply to Barry says:

              Yeah. Something like that is less googleable than doing a site search for “stop lying”.Report

            • Avatar trumwill mobile in reply to Barry says:

              I’d prefer something like ‘You misunderstand Campos’s argument, which is advice for people considering law school. He doesn’t have suggestions for those who already went to law school. He just wants to prevent more people from going that route with false expectations, which is important in its own right even if not helpful to people who have already gone that route. ‘Report

              • Avatar Barry in reply to trumwill mobile says:

                Accepted, with the waiver – it’s not like Paul has ever presented himself as not caring about people who graduated. Analogy – a guy who advocates for wearing helmets while motorcycling isn’t responsible for doing brain surgery on people who’ve cracked up a bike without one.Report

              • Avatar NewDealer in reply to trumwill mobile says:

                Perhaps I do but I also find is tone to be very grating. I don’t like it when people style themselves as moral crusaders for truth and justice and think they are heroic warriors speaking truth to power. There is a certain lack of humility in his prose. There is something about how he views himself (or I perceive he views himself) that is off-putting.

                Then again, my wish for humanity is that more people would express doubt in their beliefs openly. Doubt and self-reflection are good for the soul and humility.

                There is a lot I see about “If you can see yourself being happier doing anything but X….do it” I don’t know why it is so radical to think that people who make such statements should have some constructive/useful advice on helping people find X”

                A lot of people use grad school (including and maybe usually law school) as a way of getting out of dead-end and lack of promotion employment. So even if you are telling people don’t go to grad school or law school, it is nice if you have some idea as to how people can improve their lots without those tools.

                Otherwise you are not doing anyone a service in my opinion.Report

              • Avatar Michael Drew in reply to NewDealer says:

                As a general matter I share your response to self-styled righteous truth-tellers. It’s usually good that they’re telling the truth as they see it, but the self-righteousness can be grating.

                I guess I’ve always responded differently to the particular case of Paul Campos, however, for a few reaosns. First, I actually just don’t find his writing be that bad a case of the tone you’re talking about. It’s there a bit, but it doesn’t just bowl me over the way a Glenn Greenwald does. (And again, this kind of tone can be bothersome regardless of the actual correctness/righteousness of what’s being written.)

                Second, I take him to be an example of sort of a special case among truth-tellers: the acknowledgedly guilty co-party to the wrong being described. I’ve read his blogging at LGM since before he started the ‘Scam’ blog, was outed, and then continued his blogging on the topic back at LGM. I read the entire ‘Scam’ blog concurrently with my girlfriend deciding she wanted to switch her grad school plans from social work to law, applying, choosing Hamline (ahem, review Campos’ recent posts at LGM), and completing her 1L. From the beginning, he presented himself as someone benefitting from the very scam he was describing, who had no intention of removing himself from that position. Perhaps that colored my reading of the writing itself and my judgement of its tone, but I took that as a surrender of really any claim to personal righteousness.

                Which leads to the third factor: where he does get righteous is not in response to the simple fact of the economics behind law schools today, but to law schools’ reprenetaiton of that reality, their value, and their trading on exactly the undifferentiated ambition that is common to so many college graduates (esp. in the humanities and social sciences) trying to figure out how to proceed career-wise). It’s one thing to be self-righteous in simply describing the ways things are as if you have some special right to be particularly outraged about it just because you’re the one who’s saying it. It’s another to let your outrage show at sustained misdirection and dishonesty that comes back from those about whom you’re telling the truth. (I try to keep this in mind myself when reading Greenwald.) Campos lets his outrage show not because he thinks law school is a bad investment for a large number of college graduates, but because of representations to the contrary are made by so many law schools to so many students in an endeavor to get them to take steps that amount to putting their life in hoc to those institutions. If the underlying reality is the way Campos understands it to be (about which there is certainly legitimate debate), it seems to me he is quite justified in trying to draw attention to that by striking a tone that get people to notice what he’s saying. (Again, I try to keep that in mind in thinking about my assessment of Greenwald’s decision to use the tones he does: ie. to me he’s entirely justified in doing so, even though if truth be known it’s caused me to stop reading his actual writings entirely, though that also relates to being able to divine exactly what he’s going to say just by knowing what he’s writing about after reading about ten of his posts.)Report

        • Avatar Barry in reply to trumwill mobile says:

          Trumwill, the reason I’m harsh is that it’s a common whine about Campos. He’s trying to keep more people from being f-ed; a bunch of other people are ragging on him to somehow magically save those who already are.Report

    • Avatar Mad Rocket Scientist in reply to NewDealer says:

      I get that. Even in the sciences, it takes a long time to make a name for yourself. You need a significant body of work behind you before people take you seriously. Even the guys who break big early with a cool invention/discovery are often looked upon as lucky, one-hit-wonders until they show that they can keep the work going.

      I wonder how often those stories of early success fail to mention the amount of soul-crushing work that went into it, or how many later faded out because they rested too much on their laurels?Report

      • Avatar Barry in reply to Mad Rocket Scientist says:

        Mad Rocket Scientist, the thing that most people don’t realize is that it’s closer to a Ph.D. in science (namely most don’t get the spotlighted jobs, and most of those who don’t really struggle to even make a middle-class living), with the addition of hundred(s) of thousands of dollars of debt. It’d be like you had to take out student loans for your Ph.D. program.Report

        • Avatar Kim in reply to Barry says:

          Many people do take out loans for PhD programs. Not physicists, mind.Report

          • Avatar Barry in reply to Kim says:

            The rule of thumb I was told was that if you were not supported in a Ph.D. program, do not go. The program didn’t care if you went or not; you were a marginal student from their viewpoint, and they weren’t going to help you.

            And given that overall Ph.D. completion rates are very low (I’ve seen multiple casual estimates of 30%), this also meant that your chance of getting a Ph.D. was about zero.Report

            • Avatar Chris in reply to Barry says:

              Depends on the field. In some humanities departments, few students are supported, particularly in their first few years, while in most sciences, everyone is supported if they are admitted.

              Not being supported does make completing a PhD program without loans or being independently wealthy difficult, but not impossible.Report

        • Avatar Mad Rocket Scientist in reply to Barry says:

          Loans? For a PhD? That’s what TA/PA/RA positions are for!Report

  6. Avatar Michael Drew says:

    What bugs me about the Simkovic-McIntyre study is that it suggests that the typical law school graduate can look forward to making it out of the long tail and breaking in to the tall head — to topping out the curve. I don’t think that’s realistic. Yes, it can happen. But just like a kid who is pretty good at baseball in high school can dream about making it to the big leagues, but will probably play A and AA ball for most of his career, so too will most students who mortgage their futures on a law degree never really get into the income stratosphere.

    Yes, this. But it’s not just the idle question facing basball players, “Do they get rich or are they just left having to work for a living like the rest of us shmoes?” With current tuition and thus debt levels, the question is much more daunting: “Do they get rich; more or less break even financially but attain a level of social status that may make the lack of riches and opportunity cost pretty well worth it on the whole; or do they end up essentially failed in an career track in which they invested up to three years and at least mid-five-figures to often much more money, financed with non-dischargeable debt, and with potentially net-diminished employment prospects (i.e. if you have a JD there are lots of places where you can’t be a paralegal or are unlikely to be hired as one)?”

    With those as the disparate outcomes, and with the right-side tail of the income-outcome curve extended ever further rightward, calculating the mean of those outcomes seems – you nailed it – like the definition of misleading with statistics.

    It’s also necessarily the case that this study examines, like, the past, since that’s where real data comes from. And it seems to me that part of the thesis of the Campos view of the profession(al school) is that we are in the midst of a rapid shift in expected net benefit from the attainment of JDs *from the total corpus of law schools that exist today*, that a showing that over some period in the past, “the (average) expected added benefit is X” might even on its explicit terms be, well, maybe the term would be, “not (necessarily) responsive.”Report

    • Avatar Barry in reply to Michael Drew says:

      “… and with potentially net-diminished employment prospects (i.e. if you have a JD there are lots of places where you can’t be a paralegal or are unlikely to be hired as one)?””

      And from what people say, having a JD make one’s non-law job prospects much worse.Report

    • Avatar Barry in reply to Michael Drew says:

      Another way to phrase it is that current law students are basically buying a rental property for income. If you find that you can’t rent it out, or that the rent is low, there’s a good chance that you just wouldn’t do it.Report

      • Avatar Burt Likko in reply to Barry says:

        I like that analogy. The price tags are about the same, too, as is the prevalence of true market conditions being less lucrative than what was hoped for.Report

        • Avatar Barry in reply to Burt Likko says:

          Thank you, Burt. I have two reasons for using it – first, that this is not a liberal arts degree; it’s not going to expand your horizons, except in a very narrow, vocational direction. Second, it’s a huge honking amount of money. A friend once used the term ‘life-changing amount of money’, and even a cheap JD ($100K) is enough to financially ruin a person.Report

    • It’s fair to correct myself that the study authors do break out law grads into percentile groups (I’m actually not clear yet as to percentiles on what scale, but I’ll figure it out), so they’re not truly just averaging everyone together. But they’re still using the great success of some portion of each quintile to smooth out the reality of the desperation of a nontrivial number of grads’ situations.Report

      • Avatar Barry in reply to Michael Drew says:

        “It’s fair to correct myself that the study authors do break out law grads into percentile groups (I’m actually not clear yet as to percentiles on what scale, but I’ll figure it out), so they’re not truly just averaging everyone together. But they’re still using the great success of some portion of each quintile to smooth out the reality of the desperation of a nontrivial number of grads’ situations.”

        It’s been pointed out that overall, 50-60% of grads have real legal jobs after nine months (i.e., the point at which recruiting for new grads has long since moved to the next graduating class). This suggests that the 25th percentile would be grads who didn’t even get the odd document review temp job.

        However, this analysis states that they are successful.

        In the end, the analysis (a) looks at a historical period which probably won’t repeat, and (b) has extreme survivorship biars – ’25th percentile’ means ’25th percentile of people who are still full-time lawyers after up to 23 years’.Report

  7. Avatar Barry says:

    Oh, another thing – the guys over on Balkinization also made a nasty comment about Paul Campos (I can’t find it now; I suspect that it’s been removed). They accuse Paul of not caring about what students do, if they don’t go to law school. Paul’s whole point is that for most students, they may be in a bad situation, but going to law is going to make their previous situation look pretty good.Report

    • Avatar LeeEsq in reply to Barry says:

      I agree with this. I think a lot of people go to law school because they see it as a way to earn big bucks. That’s partly because only the most glamorous and profitable aspects of law are depicted in media usually. One reason why Win-Win was that depicted a struggling solo practitioner. Campos just wants people to know this.Report

  8. Avatar Barry says:

    One of the things mentioned (I believe on Inside the Law School Scam) was that when looking at BLS figures vs. school graduation rates, for doctors and dentists the number practicing was very well predicted by a model assuming that all grads practiced, for 30-35 years, and then retired. In other words, it looked like an extremely high percentage of those grads had a full career in their fields.

    For lawyers, on the other hand, the number practicing was something like half of what the above model predicted. Half or so of law grads did not have a full career in law.

    Now, if you think that those people used their ‘versatile’ JD degrees to pursue promising careers in whatever, careers which would not have been open (or as much) to them without a JD, then the analysis in question makes sense.

    If you don’t think that……………………….Report

  9. Avatar Kazzy says:

    Does the study account for lost earning power during law school? Most people I know didn’t work durig law school and some went into debt just living.Report

    • Avatar Burt Likko in reply to Kazzy says:

      No, I believe it only looks at post-degree earnings. Some people don’t work, go into debt, and are in theory supposed to work themselves out of debt afterwards. Others have good jobs, go to law school at night, and get even better jobs. A good friend (my wife’s attendant at our wedding) did just that. always earning pretty good money, but doing better after she got the J.D.Report

      • Avatar Kazzy in reply to Burt Likko says:

        That seems like a pretty big piece. If you earn an extra $500K with your J.D. but forgo $150K of earnings while in law school, that’s a pretty big hit to the premium, no? Or am I thinking about this incorrectly?Report

      • Avatar Brandon Berg in reply to Burt Likko says:

        On page 40: “Our estimates account for the opportunity cost of lower earnings during
        law school compared to the earnings of a bachelor degree holder who is not
        attending school.”Report