In this episode of Business Casual, our hosts critically assess the Bloomberg Business Week MBA Ranking, expressing profound reservations about its credibility and methodology. They highlight the ranking’s complexity, which obscures the results derivation process, and call attention to potential bias in responses, particularly from alumni inclined to portray their alma mater favorably.
Additionally, they mention the ranking’s lack of transparency and reliability, cautioning MBA applicants against relying solely on it for decision-making. Instead, they advise listeners to approach it with skepticism and explore alternative resources when evaluating business schools.
The hosts suggest that the Bloomberg Business Week MBA Ranking may serve more as entertainment than a credible source for prospective MBA students, as it falls short of being a dependable reference.
Episode Transcript
[00:00:07.450] – John
Well, hello, everyone. We have another ranking. Yes. It’s the Bloomberg Business Week MBA Ranking just came out. I’m John Byrne with Poets and Quants. You are listening to Business Casual, our weekly podcast with my two partners who are ranking haters, rightfully so, and Caroline Diarte Edwards. We’ve been in our little pre session here discussing this ranking and scratching our heads a little bit, trying to even understand it. Even if you have the willingness to read through the dense methodology, it’s still somewhat mystifying and hard to not only comprehend, frankly. But it’s weird that Business Week feels the need to make it so complicated and not to be very transparent. And we’ll get into that in a ways. But let me just tell you what the results are first. So Stanford is number one, and that’s perfectly no big surprise. This is the fifth year in a row that Stanford is number one. Number two, Chicago booth number three, Dartmouth. Tuck, tied for number three, UVA Darden number five, Columbia. Are you wondering where Harvard and Wharton are yet? Well, Harvard is 6th, Kellogg is 7th. Wharton is number eight in this ranking, Michigan Ross, nine, and MIT Sloan, ten, tied with Berkeley also at ten.
[00:01:42.050] – John
That kind of tells you a lot. I mean, no one would think that Harvard should be languishing in 6th place below the likes of Chicago, Booth, Dartmouth, and Virginia and Columbia for that matter. And no one would imagine that Wharton would be in 8th place. But then again, what rankings often do is they challenge us to rethink what we think in general about reputation that has been built over the years and is really a snapshot of a given school. Now, how credible it is, that’s a whole other issue. How flawed is this ranking and how intellectually dishonest is it? Well, we’ll get into that too. What’s your basic thinking about this, Maria?
[00:02:31.170] – Maria
I think when you go to their methodology page, I think it’s intentionally meant to give you a bit of a headache. It is a suspiciously opaque methodology. So, for example, a large portion of the ranking is based upon qualitative surveys that are asked of recent graduates and also alumni. Now, first of all, the alumni, it’s not a broad sampling of alumni across decades. It’s only alumni who graduated five, six, and seven years ago, which is kind of an odd. I can understand maybe 510 15 or 510 25. But five, six, and seven years ago is an odd timing, especially because one of the or some of the things they ask about are things like networking and entrepreneurship. And in many cases, for example, for entrepreneurship, people who do start businesses don’t start it until maybe they’re 10, 15, 20 years out of business school. So it’s a bit strange that they ask that. But anyway, in the subjective ranking, it will be things like I agree or disagree. I strongly agree or I strongly disagree with certain statements like this school taught me how to build my professional network. But then it’s like, okay, if you completely agree, it gets seven points all the way down to one point if you completely disagree.
[00:03:46.700] – Maria
And then after that, they say, well, there are five points awarded to the top ranked factor. Wait, sorry. So it’s on a scale of one to seven, but then different point levels are awarded based on where they I mean, it’s just like you’re going in circles, and it’s super, super suspicious. And as I said during our little pre conversation, there are parts of this that remind me a lot of the movie The Big Short where there were just, like, these mortgage backed CDO securities, and they were just so complicated that nobody understood them, so people just shrugged and were like, well, I guess it’s a good idea. And also in that movie, as I mentioned, the late, great Anthony Bourdain had a cameo where he basically points out that a fish stew at a restaurant is where chefs will put the grossest parts of the fish that they can’t otherwise sell into the stew and sell it as a new thing. And it kind of feels like this is kind of like the fish stew of methodologies and opacity.
[00:04:41.930] – John
Yeah. And fish often stink.
[00:04:47.050] – Maria
Yes, they do. God, yes, they do.
[00:04:49.200] – John
I know. You’re holding your nose. I am, too. Even just compensation. Okay. In their compensation metric, there are six different measures. One of them includes something called compensation related questions to either students or alumni. We have no idea. We have no idea what the questions are. We have no idea how they were scored. It says percentage of class with a job. We have no idea if that means is that the percentage of the class with a job at graduation or three months later? There’s no explanation. There’s median salary after graduation. That’s fine. Then there’s median alumni salary. But they are surveying three different alumni classes. I mean, it’s just all over the place. It really then, you know, they do break the results out by different regions of the world Asia, Pacific, Canada, and Europe. And I want to bring in Caroline here because their number one school in Europe is Bocconi in Italy, and they have Bocconi above INSEAD, London Business School, IMD, Cambridge, Oxford, and other really, institutions that have MBA programs that tend to rank on other lists way above Bocconi. Caroline, what do you make of that?
[00:06:16.100] – Caroline
Yes, it’s a very bizarre result, and I’m glad that Maria said that. She was scratching her head trying to read through the methodology, because I was just looking through it this morning, I thought, my God, my caffeine has really not kicked in. I have no idea what’s going on here. It’s bizarre. And in the real world, I don’t see many candidates choosing Bocconi over schools like INSEAD or London Business School. They have said that Bocconi rates more highly than INSEAD, for example, on networking and compensation. And, I mean, that’s just not the case, right? So I do not know what is going on behind the scenes or how they’re cooking this up, but it’s just not credible, right? So I don’t think that candidates should really spend much time on this. And I think it’s a shame because one of the valuable parts of these ranking publications is that often they collect a lot of data. They do surveys, they publish data that might not otherwise be easy for candidates to find or might not otherwise be publicly available on school websites and so on. So they can be useful for candidates to dig. You know, I think it’s very important for candidates to understand a methodology behind a ranking and delve into the data, because what Bloomberg Business Week may decide is really important and how a school should be ranked may not be how an individual candidate may want to rank their individual choice of schools.
[00:07:52.720] – Caroline
And so being able to dig into the methodology and being able to dig into the data can be very useful and by making it so opaque, right? So a very bizarre methodology that is not clearly explained and then not actually publishing the data behind it, to me, it’s just futile as far as the candidate is concerned. Right. It adds zero value to candidates who are doing research and trying to figure out which are the best schools and which schools might actually be the best fit for them given their so, you know, I think it’s a bit of a waste of time, waste of money by Bloomberg Business Week.
[00:08:34.710] – John
And, you know, the more you dig, the more discomforting the results are and give you one really good example. One of the major metrics that they attempt to measure one area is learning. Now, they have a really nice definition for learning. You can look it up. But here’s what’s amazing. Harvard in learning is number 26, just above Holt at number 27, the number one school with learning, William and Mary. Number two university of Georgia. Number three, University of Maryland. Number four, Georgia Tech. Now, I have nothing against these schools, but I will just tell you that it’s very hard for me to believe that the learning isn’t better at a place like Berkeley Haas or Stanford or Harvard or Kellogg. And yet the learning index tells you that, no, William and Mary is the best place. If you want a great learning experience in an MBA, it just doesn’t compute. There are other things that don’t quite know. The number one school for networking is University of Florida. Now, I have nothing against Florida. They have a great MBA program. It’s really their flexible options for you. But you can’t tell me that the school or the MBA program that has the best networking in America, it’s going to be Florida.
[00:10:08.250] – John
It’s not going to be Dartmouth Tuck, where people live together for two years. It’s a residential, unique residential MBA experience at Dartmouth, a higher percentage of Tuck MBA alums contribute back to the school every year than any other school in the world, meaning business school. And in fact, the participation rate at Tuck rivals the participation rate at almost every school in the world. That’s how loyal the alumni are and how networked that school’s alumni are. So Florida, Georgia Tech, and Michigan are number one, number two, number three in just the more you tear into this, the more you’re going to say, hey, how can this possibly be? And one explanation for it is that it is, as Maria pointed out, highly dependent on surveys and the views of alums, whether they be recently graduated or a few years out. And truth be told, if you’re an alum from a school and you get one of these surveys and you know that based on your answer or answers plural, your school is going to receive a certain rank, I think you have every justification in the world to be very positive and very favorable as opposed to very true.
[00:11:34.880] – John
And so what happens is if you are in fact from a school that doesn’t have the networks, the reputation, the resources of a Harvard or a Wharton or a Stanford or a Kellogg or what have you, or INSEAD or London Business School, you actually may feel, you know, my school did fine and I’m going to reward at the top scores. And after all, I don’t know what an education at Harvard or Stanford would have been like, or in Seattle or London Business School. So you fill out those surveys with certain kind of expectations that may favor a school. Now, there are some cases, of course, at the bottom of these rankings where schools don’t do so well. And the other problem with all of this is how closely clustered together the scores are on any of these dimensions, because they’re so closely clustered together that it’s not statistically meaningful to say that one school is five, one school is seven, one school is ten or twelve or 16. And that’s another big problem with this ranking and most rankings. Maria, any other insights?
[00:12:48.130] – Maria
Yeah, I mean, to that last point that you were making, like, the total points given to Stanford were 86.8 versus the total points given to Harvard 85.2, it’s a 1.8% difference. So not super statistically relevant. But I think one of the things that should make people the most suspicious right, regarding this learning rank, for example, Virginia Darden is a case based primarily a case based pedagogy, and as is Harvard Business School. So if both schools are heavily focused on using the case methodology, whether you like the case methodology or don’t like it, that’s a debate for another day. But why is it that Virginia ranks fourth for learning and Harvard Business School ranks 26th? If they both have obviously they don’t have identical curricula, but they use the same stuff. So if that doesn’t immediately kind of raise some alarm bells and some little red flags in your mind regarding the soundness of this methodology, I find it hard to believe that two schools with nearly identical pedagogies would have such a vast difference in their learning rank. So right off the bat, that’s one of the biggest grains of salt I will lodge at this ranking.
[00:14:01.240] – John
Yeah, actually that’s a really good thing to point out because obviously I’m just going to say this in case study schools, they attract master teachers who have to orchestrate a very fast moving, dynamic discussion involving lots of different people, making lots of different points, some relevant, some not. And it takes an incredible skill to do that. And you would think that the teachers at Darden and Harvard would be much closer together and it wouldn’t be that great a difference at all. And here’s how Business Week defines learning. We explore the quality, depth and range of instruction, focusing on the curriculum relative to real world business situations that would seem to favor every case study school emphasis on innovation, problem solving and strategic thinking, mentoring and support from instructors, class size and collaboration. I don’t think anyone would disagree with the definition, but what they’re actually asking and what the answers are, we have no idea because there’s no transparency here. And that just goes from one thing to the next. I mean, networking also measures the so called effectiveness of the Career Services office. Now, sorry, but I don’t know what that has to do with networking and the school’s brand power.
[00:15:23.000] – John
According to recruiters, what does that have to do with networking? So it’s almost like even within these definitions, they’re shoehorning in all kinds of different things that aren’t entirely related to learning, networking, entrepreneurship, or compensation. Wow. Caroline, would you rely on such a list?
[00:15:50.030] – Caroline
No, definitely not. And I think it’s a very good point you made about the survey and the fact that people just don’t have a point of comparison, right? No one goes to INSEAD and Bocconi to do their MBA, so no one can compare those two schools. The survey just, I think from that perspective doesn’t work. And even with recruiters, right, a lot of recruiters will not have recruited at all of the schools, right? They may have recruited at a few of the schools. They may have experience of hiring candidates from a handful, but they won’t have necessarily a really broad perspective across all of the schools in that market or in the region. So I think that a survey is too subjective and it’s incredibly frustrating for the schools, right, because they put a huge amount of effort into working with rankings publishers. It’s a lot of work. I remember having to do this when I was in INSEAD you know. It takes a huge amount of time and effort to provide the data that rankings publishers need and make sure that we’ve done everything properly. I’m giving them everything they need. There can be quite a bit of preparation that goes on behind the scenes to pull everything together, and then this is what they come up with, right?
[00:17:18.880] – Caroline
Why bother? You mentioned earlier that, for example, Rotman has decided not to are people at these schools are busy people. They have other things that they could be doing beyond working with these publishers and spending a great deal of time on it, and then this is what gets thrown back at them. So I wouldn’t be surprised if other schools decide in the future not to collaborate with this particular ranking. And actually, as your article from Jain at Yale commented, actually, you can’t replicate those results with the data that they have given. And actually, it looks like they are massaging things behind the scenes to make the ranking less bizarre than it would otherwise have been. So that’s very strange, right? I think they’re sort of cooking it up behind the scenes to make the methodology or make the ranking look halfway reasonable. But actually, if they followed through on their methodology as they state it, it would give an even more bizarre and result that lacks even more credibility.
[00:18:35.190] – John
Here’s the other thing. They say they surveyed 713 employers. They never tell you how many people actually responded to the survey. Same is true of the surveys. They went to students or alumni. They give you no response rate, so you have no idea what’s the strength of the response rate and therefore the sample. And on 713 employers, I hate to say this, but the truth is, if you were to look at the mainstream companies that literally make up the MBA market, they’re probably less than 100. Let’s face it. If I want to really get particular and I would say how many companies in the world recruit 20 or more MBAs a year? I can tell you that it’s fewer than 50. All right, so how do they have 713 to even survey? I’ll tell you how. Because they go to each school, they ask them to provide a list of all the recruiters who visited the school, and then they survey all of them. And they do that every school. So in other words, the McKinsey partners that go to Harvard get a survey. The McKinsey partners that go to Stanford get a survey. The McKinsey partners that go to INSEAD get a survey.
[00:19:53.720] – John
Probably none of them fill it out, because that’s the nature of McKinsey. And if they did, guess why the McKinsey partner goes to Harvard? Because they’re a Harvard Alum. And guess why the INSEAD McKinsey partner goes to INSEAD? Because they’re from INSEAD. So now you have the built in bias of these employers who actually do fill out the survey, because inevitably, among the better recruiting organizations of MBAs, they always send the alma mater, the people back to the alma mater. And there’s no mention of this in the past. Business Week actually admitted that among the recruiting sample, those who in fact went back to their old schools, put in inflated values on the value of those schools, and they had to actually discount it. Now, we don’t know what’s going on because there’s no mention of this at all. But these are some of the problems that exist in this survey. And Caroline was mentioning a critique that we’ve published from Anjani Jain who is the deputy dean at Yale School of Management. For the last three years, he’s been criticizing this particular ranking and its methodology, not only because he can’t replicate it, but because he’s discovered that there’s a step in statistical analysis that’s required to get a solid result and they’re not using that step.
[00:21:31.140] – John
It’s to normalize the data, essentially, and I’m not going to get into the technical definition of what it is to normalize this data. But nonetheless, he says, and you could read this on her site, that if in fact, they applied the weights to the data that they do provide here, and the data corresponded to the weights that they tend to advertise in this ranking, that Wharton, instead of being number eight, would actually be ranked 20th. And his conclusion is that obviously business week doesn’t want to come out with a ranking where Wharton is ranked 20th in the US. So they’re putting their hands into the methodology, into the black box and moving things around so that it’s not as embarrassing as it actually would be. That’s his conclusion. So it’s kind of wacky. Now, I will say this. They clearly devote a lot of time and a lot of resources to this. There’s a lot of interesting data buried in the ranking and the profiles in Business Week. The entertainment value of it is, I’d say it’s pretty good. It might be a cheap reality TV show, but sometimes that can be a guilty pleasure for people.
[00:22:54.310] – John
But in terms of a serious undertaking created and nurtured by people who really want to help people make an important decision in their lives, I would say this fails very notably. Maria, I’m assuming you would never recommend this ranking to your clients.
[00:23:18.110] – Maria
I would not because in the past when we have also, I mean, we don’t really like any of the rankings except for of course, the Poets and Quants ranking. But when we have always tried to find the silver lining of rankings, we say, well, at least they provide a consolidated starting point for research. They provide some good data, things like the average GMAT scores or sort of at least a place for people to start to collect information. But since this information is not scientifically sound like as you pointed out, there’s nothing stopping someone from just saying, I’m just going to put all sevens strongly agree with everything about my school because I want to give my school this artificial bump up. I mean, that’s really not defensible so because they don’t provide really any actual data. It doesn’t even really help from that perspective, from a data collection perspective. So I think that this is probably one of the least useful ones. Sorry, guys, but sorry, Bloomberg Business Week, but I don’t really see the point of spending time on this.
[00:24:24.140] – John
Yeah, it’s a shame because clearly they’re devoting a lot of effort to this to survey all those folks, crunch all these numbers to create an overly complicated methodology that no one can understand, and that gives Maria a headache. Clearly they could have done a much better job.
[00:24:47.530] – Maria
I was just going to say or provide everyone who reads it with a free bottle of headache medicine and a free cup of coffee, like a free espresso and a free aspirate. Maybe that’s the solution. They can partner with companies to provide that with Johnson.
[00:25:04.420] – John
Johnson, exactly. Tylenol here.
[00:25:07.560] – Maria
Yeah, exactly.
[00:25:11.010] – John
Caroline, last words on this.
[00:25:14.050] – Caroline
Well, another thing that struck me about. It is you look at the Asia Ranking and INSEAD is nowhere to be seen there. Right. If you want to do an MBA in Asia, then INSEAD is actually a fantastic option.
[00:25:30.010] – John
MBAs that are admitted to INSEAD, have a choice of starting the program or completing it in Singapore or France.
[00:25:38.860] – Caroline
Yeah. And you get an incredible network in Asia if you go to INSEAD, so. Really, you know, too many anomalies in this ranking for it to be a useful reference point, I think.
[00:25:50.220] – John
All right, if you want to read our story on this, it’s Harvard and Wharton Slump and New 2023 Business Week MBA ranking. And if you want to read Anjani Jain’s critique of the ranking, it’s called Flawed Again: Bloomberg Business Week’s New MBA Ranking. Check it out. Meantime, remember, it’s just entertainment. It’s just fun. It’s just like a cheap reality TV show. A guilty pleasure. This is John Byrne with Poets and Quants. Thanks for listening.
