Ajani Jain, Deputy Dean of Yale School of Management, sparked a controversy with his recent analysis demonstrating how inaccurate and distorted the Bloomberg Businessweek is, in terms of their MBA ranking and methodology, when he attempted to recreate their approach.
In this episode of Business Casual, Maria, John and Caroline discuss the Businessweek MBA rankings and their overall impact. Listen to hear discussion on:
- Why all three of them were already skeptical about the “rankings” (And what exactly John means by that)
- Why Maria thinks that of all the rankings, Businessweek is always the one with the ‘wild swings’
- Why Maria praises Jain’s writing on how he approached the matter with Bloomberg in this episode. (It is certainly without a doubt intellectually humorous!)
- The team debate the question “Do MBA rankings really matter?” Is it more important than a genuine and authentic reputation that data analysis and statistical ranking systems do not obtain?
- Is ranking manipulation a criminal offense punishable by law?
- One fact that will enrage the faculty: applicants value rankings three times more than faculty quality when choosing a program. (The statistics gathered from applicants will astound you!)
Episode Transcript
[00:00:07.330] – John
Well, hello, everyone. This is John Byrne with Poets and Quants. Welcome to Business Casual, our weekly podcast. We’re going to talk about a big controversy that’s in the news right now. We broke the story upon Poets and Quants. It’s an analysis by Anjani Jain, the deputy Dean of Yale School of Management, on the latest Businessweek ranking of MBA programs. He pulled apart all the numbers and put them into an optimization model, and Lo and behold, found that he could not replicate their ranking if he used the exact weights on five indexes that they used to measure the quality of an MBA experience. In fact, if he runs the analysis on the weights that they provide and state in their methodology, 79 of the 84 schools that would be ranked would have a different rank, and some of the schools would have dramatically changed rankings. One glaring example, Wharton, which already was ranked 9th in the Businessweek ranking, would actually fall to 28th if the proper weights were accorded based on this analysis by Anjani Jain. Now I should point out that Businessweek is saying that Anjani’s analysis is incorrect. It’s inaccurate. You can’t possibly attempt to replicate the data without having the raw scores as opposed to the scores that they actually published online and in the magazine.
[00:01:50.930] – John
Anjani argues that it doesn’t make any difference if the actual scores published in the magazine and online are truly reflective of the raw scores. It doesn’t matter whether you have the raw scores or not. Caroline, what do you make of this?
[00:02:06.100] – Caroline
Yeah, I agree that it looks very dodgy indeed, and that Businessweek has not responded very clearly to the challenges, so definitely looks very suspicious. I think they’re trying to make some excuses now and close the discussion, but it definitely looks like something is a mischief, which is very concerning.
[00:02:29.450] – John
Yeah, exactly. And now, Maria, I know. Well, all three of us are skeptical of these rankings to begin with. If you’ve listened to our podcast in the past, you know that. And we had a whole podcast not long ago devoted to the Bloomberg Businessweek ranking, where we kind of tore it apart. But, Maria, what’s your take on all this?
[00:02:50.420] – Maria
Yeah, I think of all of the rankings, Bloomberg Businessweek has always been the one that kind of has these wild swings back and forth. And I think it’s because my impression is that it’s a very subjective ranking. And so if you organize a bunch of people together and you say, okay, guys, Businessweek is going to be reaching out to you, and it’s really important that you say that our academic experience was the best thing in the world because that’s how they rank, whether or not it’s a good academic program. I don’t think that you should rank a business school based on a popularity contest solely on people’s opinions. Or what if a bunch of graduates are grumpy in a particular year? And I don’t know, I do think that there should be some emotional component. Right. Are you happy you went to the school? Something like that. But I don’t think that so much of it should be so qualitative. So, yeah, I think there’s just something fundamentally wrong with Businessweek, how they’ve been doing this anyway. And so even if they had like, let’s say, let’s say if what Deputy D and Jain is saying is correct, and they got these weird results, like super weird results.
[00:04:00.710] – Maria
And they were like, oh, no, this doesn’t look good. So we need to kind of mix things up a little and change it around. Even if that’s true, the fact that they are getting such weird results, that should be a signal to them as an organization. Like, wait a minute, this is weird. One school in Texas, does it really have the best academic experience? I’m sure it’s a great academic school, but isn’t really the best. Maybe we should look inward and with a bit of maturity and say maybe our own ranking isn’t very our own methodology is not that great. I will say, as a side note.
[00:04:35.510] – John
In context of what you just said, Marie, if I may, Marie is referring to the fact that under the Businessweek surveys of students, UT Dallas is number one in terms of their learning category, which seems pretty surprising, right?
[00:04:54.130] – Maria
I’m sure, I’m sure it’s an amazing school. But come on, the number one learning. Come on. Even when they first get that data back, even before they move to analyze it, and before they move to do all these machinations and this sort of Cirque du Soleil twisting themselves into a pretzel to try to get the to try to claim that their methodology works. Like, maybe internally they should be like, maybe our methodology is kind of lousy because this doesn’t really pass the test. Although I will say, just as a side note for all of this, if I were Businessweek, I would sort of snarkily point out that, hey, Jain, when you redo it, it just so happens at your own school, jumps up four points of four places. And it just so happens that Yale is now again in the top ten, especially because we know, I think we might have mentioned at some point before in one of our conversations that I think Yale in particular has been really trying to game the rankings or to increase the ranking these past several years, anecdotally I’ve seen a much bigger emphasis on GMAT scores at Yale, for example, than I did 15 years ago.
[00:06:09.550] – Maria
If I were Businessweek, I would sort of reply with that. But regardless of where you stand on this, this is like, get out the popcorn, read this article, go to Poets and Quants. And right now, because it is such great, it’s such compelling reading, although I don’t remember what his E score is to save my life. But the other thing that’s great is that really just like James writing is so awesome. I love my favorite phrase that he uses it’s a deceitful canard. As a society, we don’t use the term canard enough. And I, for one, applaud Deputy Dean Jain for his wonderful writing, which has just been a delight to read, even if I don’t understand the statistics behind it.
[00:06:53.890] – John
Yeah. His response to Businessweek is a really wonderful reading. He calls Businessweek statement disingenuous and nonsensical and yes, a canard, among other things. And then there’s all the analysis, which some of it is above my pay grade in my head. And I never had the benefit, like you two, of taking statistical courses. So while you 2 may know everything about this, including the difference between a normalized score and a Z score, I don’t have a clue.
[00:07:33.950] – Maria
Don’t look here.
[00:07:36.110] – John
Now, again, I need to point out that I’m the culprit who originally created the MBA ranking back in 1988. But I want to add some context here. The ranking has changed dramatically. Back when I did it, it was a simple customer satisfaction survey that measured the satisfaction of the latest graduating class and the employers who hired them. That’s it simple. And it was cleaned in that way and pure because it was so simple. And I had hired a PhD in statistics who ended up becoming the President of Cooney, the College system in New York, to be my consultant to make sure that we were doing the right things. And I think a lot of these ranking organizations put the fear of God in business schools to report correct data and not to interfere with any surveys of student opinions that they may do. It’s also incumbent on them to have transparency, to operate with full integrity, and to provide the data that would actually allow for more rigorous analysis of the results. And, of course, that’s not happening here. The big question is just an interesting, entertaining fight that we can sit by the sidelines with a bag full of popcorn and enjoy.
[00:09:03.350] – John
Or does this fight really matter?
[00:09:05.430] – Caroline
Caroline, the rankings do carry rates, right? People pay a lot of attention to them, for better or for worse, from prospective students who are thinking about ways to apply to recruiters, to faculty who are looking at which schools they want to work at. It does have a broad impact. And so they have some responsibility here to respond more clearly to the claims, I think.
[00:09:35.190] – John
Yeah, that’s really true. It’s worth pointing out that there’s only one of the five major rankings that actually does audits of school provided information, and that’s the Financial Times. And I believe that they audit programs every three years. And I got to tip my hat off to them for actually going to that extra step to ensure more integrity by the schools, because obviously the schools are under significant pressure to push numbers in their favor, no matter what they are. But to your point, rankings are consequential. They do matter. Two years ago, the association of International and Graduate Admissions Consulting, we call that AJAC. And we’re here as a member. They surveyed MBA applicants, and out of nearly 2000 responses, here’s what they found. They found that nothing mattered more in school selection than rankings and reputation, two factors that, let’s face it, that largely become inseparable. So when you ask the applicants what were the top factors influencing specific choice of schools, 63% of them sided rankings with an equal number of sighting reputation. Now, I’m going to state a fact that it’s going to enraise a lot of the faculty who listen to our podcast.
[00:11:02.270] – John
That means, according to this survey of applicants, that applicants value rankings three times more in selecting the program than faculty quality. Faculty quality got 21%. The cost of the program got 23%. GMAT or GRE scores attained, meaning it affects where you apply, 27%. Think about that. 63% said ranking, 63% said reputation. They both go hand in hand. It’s kind of remarkable, frankly. And then when GMAT surveys its folks and they did so in 2021, earlier this year, they found that rankings was a key information resource for researching programs. 45% of domestic MBA applicants said it was 49% of international MBA applicants said it was. So it’s clear that these rankings sway people one way or the other because they do affect your impression of brand. And this is particularly true when you get past Harvest Erin Wharton, and then you start getting into the other schools, which, after all, enroll and get applications from far greater numbers of people than the combined numbers at Harvard, Stanford and Wharton, where I think no matter what their rankings are, it really doesn’t matter because people won’t believe that they’re ranked poorly. No one’s going to believe Wharton’s current rank of nine in Bloomberg Businessweek, and no one would believe the reconstituted ranking, which shows Wharton is number 28.
[00:12:45.070] – John
So it doesn’t matter for those schools, but it matters for everybody else pretty much. And then you have this sense that, look, you know, it matters because people cheat. Next month, the former Dean of Temple University’s business school is actually going on trial in federal court for allegedly cooking the books and providing fraudulent data US news that allowed his online MBA program to be ranked number one for four consecutive years. And this is a criminal indictment. It’s the first time the government has actually gone after a school. And it’s been for alleged fraud, which is pretty darn remarkable. And that trial is going to be getting a Philadelphia courtroom, I believe, in the second week of November. Sorry. But Marie, when you chose the school, you weren’t thinking about rankings, were you, because you went to Harvard?
[00:13:50.850] – Maria
No. You know, it’s funny. I basically looked at, like, people I admired and where they had gone to school. And then also I looked at people that I didn’t like so much. And a lot of them all randomly, perhaps coincidentally, had all ended up going to another well known school. And so I was like, well, that helps me narrow it down pretty considerably. And yeah, actually, I believe at the time Wharton I mean, Harvard has always been obviously, because it’s Harvard. But Wharton at the time I was living in Asia, and my boss was like, well, Wharton is a better school. Right? And I was like, I don’t know. Wharton is sort of famous worldwide for being like the business school. And so I don’t even know if I looked at the ranking. We’ve talked in the past about why we chose our respective schools, but the people around me were at least my boss at the time was like, oh, I thought Wharton was the better. I’m surprised that you’re taking Harvard INSEAD, but they were just such different programs that I chose the one that I thought was best for me. So I don’t know.
[00:14:53.650] – Maria
It’s a shame because I do think that it’s interesting that the three of us have spent so many years in the space, and we’ve seen not just the fluctuations in the rankings, but we’ve seen clients or people that we’ve interviewed or anyone like that who goes to different programs. And many of them have wonderful career outcomes, really, regardless of whether they go to a school that one year is ranked second and the next year is ranked 7th. And so we can see that the rankings themselves, in terms of outcome for your life, don’t really matter a ton. But every year, people coming fresh into this process, it’s a fresh batch of people every single year. And so it’s brand new to them. And so it’s understandable that the rankings is the first place they would look. And I guess it’s understandable that they would place a lot of weight on it. It’s just sort of a shame that we can’t sort of transmit our collective wisdom into their brains and feel like it’s not nice, but it’s not the be all and end all of the experience.
[00:15:53.530] – John
That’s really true. And Caroline, obviously you picked INSEAD because you wanted a far more international global experience, and that’s what you got there.
[00:16:02.450] – Caroline
Yeah.
[00:16:03.550] – John
INSEAD great school was a terrific MBA program. It has been for a very long time. I’m sure rankings didn’t come into the picture for you either.
[00:16:14.240] – Caroline
No, they didn’t. I don’t remember consulting the rankings, but I think people it’s something that’s sort of in the background.
[00:16:20.490]
Right.
[00:16:20.750] – Caroline
People are aware of them, and I know that it does create a reaction. So I’ve seen that when I was working at it yet that when the score went up in the rankings, stakeholders get very excited. And when the score went down in the rankings, they’d been knocking on the Dean’s door and wanting explanations. So people do take notice. Right. Even people who know these schools incredibly well. Alumni, donors, et cetera. They do keep an eye on these things and they want to understand what’s going on. So even though you may know it’s all very well and appreciate its value, regardless of what particular publication says, people are aware of the impact of the rankings have on other people. So the influence that that has and they care about the sort of public reputation of the schools. And so I do think the rankings have disproportionate influence, unfortunately.
[00:17:29.450] – John
Absolutely. Well, no matter how this plays out, meaning, will Businessweek ultimately concede that it made an error or will it never do that? And the argument will just be out there? I think the one good outcome of all this is that it makes us do exactly what we should be doing treating these lists more skeptically, looking at them probably more for entertainment value than for the reality that this school is number three and that school is number five and the other school is number eight. Again, I’ll always say the benefit of a ranking is generally the underlying data that comes with it. And without rankings, I don’t think we would have had the kind of disclosures that we’ve seen in the last quarter of a century where you know what the average GM out of entering classes, you know the average GPA, the number of years of work experience, you know what the outcomes are going to be in terms of average salary, signing bonus, percentage of people who have a job offer at graduation and three months later. These are all stats that often go into rankings and they are public because the rankings exist.
[00:18:48.140] – John
And I think that’s a positive thing for people to help make informed decisions about where to go. Meantime, pop your popcorn set in your sofa and read all the stories and Poets and Quants in the back and forth between Bloomberg this week and Anjani Jain and make up your own mind. That’s all makes sense. All right, Maria Caroline, thank you so much. Watch again for your wisdom and your smarts. This is John Byrne with Poets and Quants you’ve been listening to Business Casual.
