The College Rankings and Methodology Thread
Posted: Tue Mar 24, 2020 1:47 pmOn March 23, 2020 MUBoxer wrote:
Anyone using Niche to rank universities instantly loses credibility.
1. They factor in neighborhood safety and quality which instantly puts schools in large urban settings, particularly Jesuit schools which are usually in rougher neighborhoods, at a major disadvantage.
2. They factor in a ton of other useless metrics. Diversity? Loan amount? I went to a CPS HS then a suburban HS that was even more diverse than my CPS school, it's great for diverse thought but not correlated to great school or else cheaper directional schools would own the more expensive private schools. Loan amount? This is idiotic, it has zero to do with college quality.
Bottom line is Niche is a joke, in an attempt to go wayyyy overboard with analysis Fieldhouse Flyer has a tendency to not review data.
MUBoxer – I can assure you that I review the data I post very carefully before I post it.
It appears that you are the one who has a tendency not to review data. Loan amount is a Forbes criterium which is not used by Niche, so you have just trashed the Niche Rankings because of Forbes’ methodology. See Part 19 below.
Also, Niche assigns a 0.025 weighting factor to Safety Rating and a 0.05 weighting factor to Diversity. The other rating services do far worse.
Since you decided to make it personal, this post is for you, MUBoxer.
I have followed College Rankings very closely since their inception. Below is a post that I prepared last spring concerning the 2019 college rankings. I have deleted most of it for this HLOH post (for the sake of brevity), but I would be happy to post any of the deleted sections if requested. Those of you who read my posts already know of my considerable interest in the history of things, so I’ll start with that before proceeding.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
History of College Rankings Services
1917 • Weekly news magazine Forbes founded in New York.
1933 • Weekly news magazine U.S. News[/url] founded, headquartered in in New York.
1946 • Weekly news magazine World Report founded, headquartered in in New York.
1948 • Weekly news magazine U.S. News & World Report founded by merger.
1954 • Ivy League founded, headquartered in Princeton, New Jersey.
1979 • Big East Conference founded, headquartered in in New York.
1981 • A company named ’The Princeton Review’ is founded as a college admission services company offering test preparation services, tutoring and admissions resources, and online courses. The company is headquartered in New York City, and is privately held. Despite the title, it is not associated with Princeton University.
1982 • Atlantic 10 Conference founded, headquartered in in New York.
1983 • U.S. News & World Report publishes its first "America's Best Colleges" report in its news magazine. The rankings have been compiled and published annually since 1987.
1992 • The Princeton Review publishes its first "Best Colleges" rankings, some 11 years after the company was founded, and nine years after U.S. News & World Report published its first "America's Best Colleges" report .
1994 • The Wall Street Journal published disclosures about institutions flagrantly manipulating data in order to move up in the rankings in U.S. News and other popular college guides.
1995 • , Reed College refused to participate in U.S. News & World Report annual survey. According to Reed's Office of Admissions, "Reed College has actively questioned the methodology and usefulness of college rankings ever since the magazine's best-colleges list first appeared in 1983, despite the fact that the issue ranked Reed among the top ten national liberal arts colleges". Reed's rankings were then artificially decreased by U.S. News after they stopped sending data to U.S. News & World Report.
1996 • On September 23rd, Stanford President Gerhard Casper sent a letter to James Fallows, editor of U.S. News & World Report, stating: "As the president of a university that is among the top-ranked universities, I hope I have the standing to persuade you that much about these rankings - particularly their specious formulas and spurious precision - is utterly misleading."
1997 • In January, president of Alma College, Alan Stone asked 480 colleges to boycott the U.S. News & World Report Rankings due to the peer assessment survey which counts for 22.5% of a college's ranking. According to the Chronicle of Higher Education, in 1996, Alma College surveyed 158 colleges about the rankings. The result of the survey indicated that "84 per cent of the respondents admitted that they were unfamiliar with some of the institutions they had been asked to rank. Almost 44 per cent indicated that they 'tended to leave responses for unfamiliar schools blank.' " Stone stated, "this makes me wonder just how many votes are being considered for each school's academic-reputation ranking."
1998 • Stanford posted an alternative database on its website stating: "This page is offered in contrast to commercial guides that purport to "rank" colleges; such rankings are inherently misleading and inaccurate. Stanford believes the following information, presented without arbitrary formulas, provides a better foundation for prospective students and their families to begin comparing and contrasting schools." It has since been posted annually as the "Stanford University Common Data Set."
1998 • In the autumn, Luke Skurman, a San Francisco native, enrols as a freshman at Carnegie Mellon University.
1998 • Princeton University is ranked No. 1 in the U.S. News & World Report rankings.
1999 • Amy Graham is appointed as the new lead statistician for the U.S. News & World Report’s College Rankings. Ms. Graham changed the rankings formula to one she considered to be more statistically valid, and the California Institute of Technology jumped to first place.
2000 • In the summer, Amy Graham leaves U.S. News, and the rankings system is modified again, pushing Princeton back to No. 1 in the rankings.
2000 • In the autumn, Carnegie Mellon University junior Luke Skurman enrols in an undergraduate entrepreneurship class and begins work on a new college rankings methodology for his class project. It can be taken as fact that Luke Skurman would have been aware of the college rankings published by the U.S. News & World Report and by The Princeton Review, and their respective methodologies. Being a California native, it is also highly likely that Luke would have been aware of the controversial rise and fall of the California Institute of Technology in the U.S. News & World Report’s college rankings as well as Stanford’s war against the U.S. News college rankings. Therefore, it can safely be presumed that Luke Skurman consciously set out to ‘build a better mousetrap’ for his class project.
2002 • Luke Skurman complete his entrepreneurship class and new college rankings, published them under the name “CollegeProwler”, receives his Bachelor of Science degree from Carnegie Mellon University, and enrols as a graduate student at Carnegie Mellon. In addition to its burgeoning popularity among its target audience, CollegeProwler has won awards at Carnegie Mellon and in Pittsburgh. CollegeProwler won Carnegie Mellon's 2002 Undergraduate Entrepreneurship Association's Business Plan Competition and received $500 from the School of Industrial Administration. Accountant Ernst & Young's Enterprise Creation Competition named CollegeProwler as one of the top nine national finalists.
2002 • In September 2002, CEO Luke Skurman and the CollegeProwler staff travelled to Salt Lake City to officially launch their company at the National Association for College Admission Counselling (NACAC) Conference. They left the conference with more than 700 business leads, including those from college admission staffs, high school guidance counsellors, and major publishers.
2004 • Luke Skurman receives his Master of Science degree from Carnegie Mellon University.
2005 • College Prowler was recognized by Fast Company for being one of the 50 fastest-growing companies in the nation.
2007 • College Prowler (which produced only printed guidebooks from 2002-06) makes their content available online for a subscription fee.
2007 • On June 19th, during the annual meeting of the Annapolis Group, members discussed the letter to college presidents asking them not to participate in the "reputation survey" section of the U.S. News & World Report survey (this section comprises 25% of the ranking). As a result, "a majority of the approximately 80 presidents at the meeting said that they did not intend to participate in the U.S. News reputational rankings in the future."
2008 • CollegeProwler became embroiled in a scandal known as "Facebookgate" when hundreds of spurious "Class of 2013" groups were created on Facebook for the purpose of promoting CollegeProwler. Such groups would normally be created by actual students or colleges themselves. According to CEO Luke Skurman, "The original purpose was to use these groups as a way to inform students that they can access a free guide about their new college on our site." He also added, "No employee or anyone else associated with College Prowler has used these groups to send out messages or wall posts." College Prowler later removed all administrative access from the 125 groups, admitting "It was clearly over the line".
2008 • In July, CollegeProwler CEO Luke Skurman is appointed as a Trustee of Carnegie Mellon University – a position he still holds.
2008 • Forbes publishes its first "Top Colleges" rankings.
2009 • CollegeProwler removes their subscription fee, making all web content free.
2009 • The New York Times reported that: "40 percent of the Forbes Media LLC enterprise was sold ... for a reported $300 million, setting the value of the enterprise at $750 million". Three years later, Mark M. Edmiston of AdMedia Partners observed, "It's probably not worth half of that now". It was later revealed that the price had been $264 million.
2009 • In January, U.S. News & World Report becomes a biweekly publication, citing the decline of the magazine’s circulation and advertising.
2009 • In June, U.S. News & World Report becomes a monthly publication, due to the continuing decline of the magazine’s circulation and advertising.
2010 • On November 5th, a U.S. News & World Report internal memo was sent on to the staff of the magazine informing them that the "December issue will be our last print monthly sent to subscribers, whose remaining print and digital replica subscriptions will be filled by other publishers."
2013 • CEO Luke Skurman changes the name of his company from “CollegeProwler” to “Niche”.
2013 • The Big East Conference becomes a ten-member, private-school-only conference and begins play in the 2014-15 season.
2014 • The popularity of U.S. News & World Report’s Best Colleges rankings is reflected in its 2014 release, which brought 2.6 million unique visitors and 18.9 million page views to usnews.com in one day. Traffic came from over 3,000 sites, including Facebook and Google.
2016 • Niche CEO Luke Skurman receives Ernst & Young’s Entrepreneur of the Year - Western Pennsylvania & West Virginia award in June.
2018 • Niche raises $6.6 million and aims to be one of Pittsburgh’s ‘game-changing’ startup companies. The Pittsburgh-based startup, a ranking and review site for K-12 schools, colleges, and neighborhoods around the country, today announced a $6.6 million Series B funding round led by Allen & Company LLC and Grit Capital Partners. Its platform uses more than 120 million ratings and reviews from real people along with hoards of data that rank schools and neighborhoods, helping inform students and parents about important decisions. Traffic is up, too, with 8.8 million visits in January and 51 million uniques last year. Niche CEO Luke Skurman said: ”Half of college-bound students have a Niche account. Google continues to recognize us as a best in class site.”
2019 • Niche growth accelerates and subscription revenue grows by 200% in 2018. Today [February 7, 2019] Niche announced that it has added a $2 million investment to its Series B financing, bringing the total Series B proceeds to $8.6 million. Every year, over 60 million people use Niche to choose the schools, jobs, and neighborhoods.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2019 COLLEGE RANKINGS AND METHODOLOGY THREAD
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. Introduction
2. Methodology – U.S. News College Rankings
3. Methodology – Niche College Rankings
4. Methodology – Forbes College Rankings
5. Methodology – The Princeton Review Rankings
6. Schools of Interest
7. The U.S. News ‘Best Colleges 2019’ Rankings
8. The Niche ‘Best Colleges in America 2019’ Rankings
9. The Niche ‘Best Catholic Colleges in America 2019’ Rankings
10. The Forbes ‘America’s Top Colleges 2018’ Rankings
11. The Princeton Review ‘Best 384 Colleges 2019 Edition’ Rankings
12. Analysis of Data
13. Analysis: Forbes’ Room and Board, and ‘Other’ Living Expenses
14. Analysis: Calculation of Forbes 2018 'Implied Net Price'
15. Analysis: The Effect of U.S. News Dividing the Data Population Into 10 Different Categories
16. Analysis: The Effect of U.S. News’ Six Subdivisions on the Rankings of Big East Schools
17. Analysis: Which College Ratings Service Gives the ‘Best’ Rankings to Big East Schools?
18. The Backlash Against the U.S. News College Rankings
19. The Problems with the Forbes’ College Rankings
20. The Princeton Review Rankings
21. The Niche Rankings
22. Ranking the College Ranking Services
23. The Best Colleges in America 2019
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1. Introduction
The OT: Creighton moves from Midwest to National University thread caused me:
(1) to wonder about the reasons for the differences in the school rankings by various college ranking services, and
(2) to wonder if it can be demonstrated that some college ranking services are better than others?
The purpose of this comprehensive study is to answer those two questions. Please note that this thread is intended only for those of you who share a similar interest in this fascinating and controversial topic.
For an illustrated overview of the topic, please see the following well-diagrammed article from 2016, followed by other key articles.
Comparing College Rankings: U.S. News, Niche, The Princeton Review, and Forbes – BestCollegeReviews.org – 2016
All the College Ranking Lists You Should Read: U.S. News, Forbes, and Niche – PrepScholar.com – August 16, 2018
How To Rank College Ranking Sites – Forbes - August 29, 2018
1. U. S. News' 2018 Best Colleges, 3. Forbes, 7. The Princeton Review's Best 384 Colleges 2019, 8. Niche
Rankings of Universities in the United States - Wikipedia
Table 1(a): The ‘Big Four’ College Ratings Services
• The U.S. News Best Colleges 2019 Rankings were first published in 1983 and are the most widely quoted of their kind in the United States. The U.S. News Best Colleges database contains more than 1,800 schools and the 2019 Edition lists 312 National Universities, with ranking numbers assigned to only the Top 226 schools. The US News also ranks what they call ‘Regional Universities’ in four geographic designations: North (196 schools listed), South (148 schools listed), Midwest (172 schools listed), and West (140 schools listed), for a total of 968 schools listed. I have not found any other College Ranking Services subdivides the population of universities into multiple categories before ranking the schools. Rather than assigning a unique sequential ranking number (as the other college ratings services do), the vast majority of the schools ranked by U.S. News are ‘tied’ with a few or many other schools in a designated ‘tier’. The U.S. News’ [unpublished] internally-computed raw scores are then rounded, and schools with ‘similar’ raw scores are lumped together as a ‘tie’. [Presumably in this instance, in order to prevent sixth-listed Yale being ranked below fourth-listed M.I.T and fifth-listed University of Chicago. These three schools along with third-listed Columbia are all lumped together and assigned a ranking of ‘#3(Tied)’ by U.S. News, followed by #7 Stanford.]
• The Princeton Review Best 384 Colleges 2019 Rankings were first published in 1992. For the 2019 Edition, the Princeton Review surveyed 138,000 students from across the country. The Princeton Review does not rank schools overall 1-384, but lists the Top 384 schools alphabetically and provides 62 lists of Top 20 schools in 62 different categories. The Princeton Review added five colleges to their 2019 edition: Butler University, Drury University, Marlboro College, Michigan State University, and Oregon State University. Unlike other college ratings services, The Princeton Review requires a free registration in order to access all of their information, or to access a given webpage more than one time. Creating a free Princeton Review account took me less than a minute, and is well worth the effort, as The Princeton Review has vast amounts of information and numerous interesting rankings.
• The Niche Best Colleges in America 2019 Rankings were first published in 2002 under the name “College Prowler”. In 2013 the company was renamed ‘Niche’. The Niche Best Colleges in America ranking is based on rigorous analysis of academic, admissions, financial, and student life data from the U.S. Department of Education along with millions of reviews from students and alumni. The ranking compares more than 1,000 top colleges and universities in the U.S., with ranking numbers assigned to the Top 880 schools. The Niche Rankings uses the term “Net Price”, which they define as: “The average cost after financial aid for students receiving grant or scholarship aid, as reported by the college”. Unlike other college ratings services, Niche assigns an Overall Niche Grade (A+, A, A-, B+, etc.) to each reviewed university.
• The Forbes ‘America’s Top Colleges’ 2018 Rankings were first published in 2008, and ranks the Top 650 schools 1-650. The Forbes Rankings uses the term “Cost” and defines it as “the estimated total cost of attendance for out-of-state students living on campus in 2017-2018. This estimate includes the reported average cost of tuition and fees, books and supplies, room and board, and ‘other’ living expenses. This figure is an estimate and not a guarantee of the actual amount the student may be charged.” The 2019 Edition of Forbes’ Rankings is scheduled to be published in August, 2019.
Table 1(b): Examples of Information Provided in Scrollable Rankings Tables, and the Formats of ‘School Pages’
Click on the Georgetown links below to see the format of each ratings service’s ‘School Page for Georgetown. The information displayed in the Table below is the information which is displayed along with the school’s name in the scrollable Rankings Tables for each of the four rankings services.
Forbes 2018: #12 Georgetown University • Cost: $71,580 • Average Financial Aid: $40,346
U.S. News 2019: 22(T). Georgetown University • Tuition and Fees: $54,104 • Enrolment: 7,463
Niche 2019: 22. Georgetown University • Overall Niche Grade: A+ • Acceptance Rate: 16% • Net Price: $27,420 • SAT Range: 1350-1520
The Princeton Review 2019: Georgetown University • 7,459 Enrolled
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
12. Analysis of Data
Financial Data Mined in PART 1
Institution ………………………………………… Endowment [1] … Cost [2] … Tuition and Fees [3] … Net Price [4] … Average Financial Aid [6]
Harvard University ………………………. $37,100,000,000 ………. $69,600 ………. $50,420 ………. $17,030 ………. $48,195
Yale University …………………………….. $27,800,000,000 ………. $71,290 ………. $53,430 ………. $18,053 ………. $48,126
Princeton University ……………………. $23,800,000,000 ………. $66,150 ………. $47,140 ………. $16,302 ………. $44,128
University of Pennsylvania ………….. $12,200,000,000 ………. $71,715 ………. $55,584 ………. $24,539 ………. $43,856
Columbia University …………………….. $10,000,000,000 ………. $74,199 ………. $59,430 ………. $22,824 ………. $46,127
Cornell University ………………………….. $7,200,000,000 ………. $70,321 ………. $55,188 ………. $31,449 ………. $35,445
Dartmouth College …………………………. $5,000,000,000 ………. $71,827 ………. $55,584 ………. $22,303 ………. $45,867
Brown University ……………………………. $3,500,000,000 ………. $71,050 ………. $55,656 ………. $25,651 ………. $40,116
Stanford University ………………………. $24,800,000,000 ………. $69,109 ………. $51,354 ………. $16,562 ………. $47,782
Mass. Institute of Technology ………. $16,400,000,000 ………. $67,430 ………. $51,832 ………. $22,230 ………. $41,674
University of Chicago …………………….. $8,200,000,000 ………. $75,735 ………. $57,006 ………. $34,834 ………. $39,032
University of Notre Dame ……………. $13,100,000,000 ………. $69,395 ………. $53,391 ………. $27,453 ………. $38,080
Georgetown University …………………. $1,770,000,000 ………. $71,580 ………. $54,105 ………. $27,420 ………. $40,346
Villanova University ………………………….. $715,000,000 ………. $68,231 ………. $53,458 ………. $35,491 ………. $29,321
St. John's University …………………………. $648,000,000 ………. $59,865 ………. $41,760 ………. $25,981 ………. $24,963
DePaul University …………………………….. $596,000,000 ………. $56,275 ………. $39,975 ………. $32,927 ………. $19,085
Marquette University ……………………….. $550,000,000 ………. $55,273 ………. $41,870 ………. $32,377 ………. $17,935
Creighton University …………………………. $448,000,000 ………. $53,452 ………. $39,916 ………. $31,481 ………. $19,913
Seton Hall University ………………………… $243,000,000 ………. $57,090 ………. $42,170 ………. $28,700 ………. $23,572
Providence College …………………………… $213,000,000 ………. $64,984 ………. $50,390 ………. $36,099 ………. $24,941
Butler University ………………………………. $174,000,000 ………. $56,740 ………. $41,120 ………. $38,051 ………. $18,926
Xavier University ………………………………. $151,000,000 ………. $52,380 ………. $38,530 ………. $30,190 ………. $21,463
Saint Louis University …………………… $1,230,000,000 ………. $57,214 ………. $43,996 ………. $31,718 ………. $21,675
Fordham University …………………………. $739,000,000 ………. $72,239 ………. $52,248 ………. $38,503 ………. $25,892
University of Dayton ………………………… $524,000,000 ………. $57,680 ………. $42,900 ………. $35,077 ………. $19,511
Saint Joseph's University ………………… $279,000,000 ………. $60,854 ………. $44,974 ………. $35,290 ………. $22,407
Duquesne University ………………………… $261,000,000 ………. $52,108 ………. $38,178 ………. $30,540 ………. $19,310
La Salle University …………………………….. $88,000,000 ………. $45,476 ………. $30,710 ………. $26,276 ………. $26,430
St. Bonaventure University ………………. $60,000,000 ………. …. [6] … ………. $34,301 ………. $22,409 ………. …. [7] …
Virginia Commonwealth U. …………. $1,840,000,000 ………. $51,204 ………. $35,138 ………. $20,741 …………. $9,207
University of Connecticut ………………. $447,000,000 ………. $53,112 ………. $38,098 ………. $18,699 ………. $12,877
Gonzaga University …………………………. $276,000,000 ………. $57,350 ………. $43,210 ………. $34,085 ………. $19,529
[1] – The Endowment amounts were mined from Wikipedia (from the appropriate Conference webpages were available). The 32 Schools of Interest are sorted by Endowment in Table 6(d) above.
[2] – The Cost amounts were mined from the Forbes ‘America’s Top Colleges’ 2018 Rankings. Forbes defines “Cost” as ”the estimated total cost of attendance for out-of-state students living on campus in 2017-2018. This estimate includes the reported average cost of tuition and fees, books and supplies, room and board, and ‘other’ living expenses. This figure is an estimate and not a guarantee of the actual amount the student may be charged.”
[3] – The Tuition and Fees amounts were mined from the U.S. News Best Colleges 2019 Rankings.
[4] – The Net Price amounts were mined from the Niche Best Colleges in America 2019 Rankings. Niche defines “Net Price” as “The average cost after financial aid for students receiving grant or scholarship aid, as reported by the college”.
[5] – Basic Relationships Between Variables[/b]
…… Forbes’ Cost = (tuition and fees) + (books and supplies) + (room and board, and ‘other’ living expenses)
…… Niche’s Net Price = (tuition and fees) + (books and supplies) + (room and board, and ‘other’ living expenses) – (average financial aid)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
17 Analysis: Which College Ratings Service Gives the ‘Best’ Rankings to Big East Schools?
17.1 Issue: College Ratings Services - ‘Best’ Rankings for Big East Schools.
17.2 Question: Which College Ratings Service Gives the ‘Best’ Rankings to Big East Schools?
17.3 Answer: There is no objective answer to this important question because the population sizes (i.e., number of schools) used by Niche, Forbes, and U.S. News are significantly different. In order for direct comparisons to be valid and meaningful, all three ratings services must use the same population size with the exact same schools comprising the population – something which has clearly not occurred.
The other problem with answering the question hinges on the definition of ‘best’. Best published rankings (‘raw scores’) of Big East schools ? Or best percentile ? (i.e., best average placement of Big East schools relative to the population size of the dataset). Table 20(a) below shows both (noting that the 95th percentile always means 95% of the other values lie below it, and 5% lie above it.)
Table 17(a): Analysis: Average Ranking and Average Percentile of Big East Schools
236 • AVERAGE RANKING OF 10 BIG EAST SCHOOLS – Niche (population size = 880 ranked schools)
208 • AVERAGE RANKING OF 10 BIG EAST SCHOOLS – Forbes (population size = 650 ranked schools)
92 • AVERAGE RANKING OF 6 BIG EAST SCHOOLS - US News (population size = 226 ranked National Universities)
Percentile Calculations: Average Percentile of Big East School Rankings:
…… Number of schools ranked worse than the Average Ranking of Big East schools = (population size) - (Average Ranking of Big East schools)
…… Average Percentile of Big East School Rankings = (100)x[(No. of schools ranked worse)/(population size)]
73 = (100)x(880-236)/880 • Average Percentile of 10 Big East Schools - Niche
68 = (100)x(650-208)/650 • Average Percentile of 10 Big East Schools - Forbes
59 = (100)x(226-92)/226 • Average Percentile of 6 Big East Schools - U.S. News
Niche ranks 73% of the 880 ranked schools WORSE than the Average Ranking of Big East schools.
Forbes ranks 68% of the 650 ranked schools WORSE than the Average Ranking of Big East schools.
U.S. News ranks 59% of the 226 ranked National Universities WORSE than the Average Ranking of Big East schools.
It can be seen from the calculations above that as more schools are added to the size of the population, the Average Big East Ranking becomes worse, but the Average Percentile of Big East School Rankings becomes better. To further illustrate this point, I’ll rerun the calculations above, substituting St. John’s rankings for the Average Big East Ranking:
17.7 Confessions from the Data:
US News gives the Big East schools the best raw score (average ranking), but ranks only 59% of the ranked National Universities WORSE than the Average Ranking of Big East schools – which is directly attributable to US News having the smallest dataset, consisting of only 226 schools.
Niche ranks 73% of the ranked National Universities WORSE than the Average Ranking of Big East schools, but gives the Big East schools the worst raw score (average ranking) – which is directly attributable to Niche having the largest dataset, consisting of 880 schools.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
18. The Backlash Against the U.S. News College Rankings
For an overview, here are some excerpts from the two articles listed in 1. Introduction:
18.1 Rankings of Universities in the United States - Wikipedia
… 1.20 U.S. News Rankings
U.S. News determined the relative weights of their weighting factors and changes them over time. The National Opinion Research Center reviewed the methodology and stated that the weights "lack any defensible empirical or theoretical basis". The first four of the listed factors account for the great majority of the U.S. News ranking and the "reputational measure" (which surveys high-level administrators at similar institutions about their perceived quality ranking of each college and university) is especially important to the final ranking.
A New York Times article reported that, given the U.S. News weighting methodology, "it's easy to guess who's going to end up on top: the Big Three, Harvard, Yale and Princeton round out the first three essentially every year. When asked how he knew his system was sound, Mel Elfin (M.A. Harvard University, 1952) the rankings' founder, often answered that ”he knew it because those three schools always landed on top.“
A 2010 study by the University of Michigan found that university rankings in the United States significantly affect institutions' applications and admissions. The research analyzed the effects of the U.S. News & World Report rankings, showing a lasting effect on college applications and admissions by students in the top 10% of their class. In addition, they found that rankings influence survey assessments of reputation by college presidents at peer institutions, such that ”rankings and reputation are becoming much more similar over time.”
18.2 Rankings of Universities in the United States – Wikipedia
… 2. Criticisms of the U.S. News & World Report’s College Rankings
Critics charged that U.S. News intentionally changed its methodology every year so that the rankings change and they can sell more magazines. A San Francisco Chronicle article argues that [/i]"almost all of US News factors are redundant and can be boiled down to one characteristic: the size of the college or university's endowment."[/i]
Some higher education experts, like Kevin Carey of Education Sector, have argued that U.S. News & World Report's college rankings system is merely a list of criteria that mirrors the superficial characteristics of elite colleges and universities. According to Carey, "The U.S. News ranking system is deeply flawed. Instead of focusing on the fundamental issues of how well colleges and universities educate their students and how well they prepare them to be successful after college, the magazine's rankings are almost entirely a function of three factors: fame, wealth, and exclusivity."
18.3 How To Rank College Ranking Sites – Forbes - August 29, 2018
The U. S. News's Best Colleges is the grandaddy of all the lists. Over the years its methodologies have changed and a few schools have been caught fudging their numbers or gaming the rankings. Unfortunately, some trustee boards put stock in the rankings and heads can roll if a school slips. This list and its creators have assumed an arrogance over the years similar to the College Board's. Their attitude is "Don't like our rankings? What're you gonna do about it?"
It has now been 25 years since the 1994 The Wall Street Journal published disclosures about institutions flagrantly manipulating data in order to move up in the rankings in U.S. News. Others soon followed, and for decades such articles have become an annual event. Here is the latest round:
18.4 15 Things to Know About U.S. News’ College Rankings - TheCollegeSolution.com - May 8, 2018
While U.S. News and World Report’s college rankings are wildly popular, few families know much about how they are created. Before you place too much faith in U.S. News’ college rankings, here are 15 things that you should know about them:
1. U.S. News relies on rankings to stay alive.
2. U.S. News’ college rankings have encouraged institutional bad behavior.
3. The college rankings formula can be gamed.
4. Popularity is a big ratings factor.
5. U.S. News measures six-year graduation rates.
6. Rankings encourage colleges to favor affluent students.
7. Rankings encourage the use of merit scholarships.
8. Elite schools are the exception to merit awards.
9. Rankings encourage admission tricks
10. Rankings don’t measure what’s important.
11. Rankings encourage cheating.
12. Rankings encourage debt.
13. Don’t believe the numbers.
14. Use U.S. News as a tip sheet.
15. U.S. News is here to stay. A few years ago, Brian Kelly, the U.S. News editor made this promise during an press interview: “You can love us or hate us, but we’re not going away.”
18.5 14 Reasons Why US News College Rankings are Meaningless – LiberalArtsColleges.com - August, 2018
US News and World Report released its 2018 US News College Rankings. Parents, high school guidance counselors, students, and even the colleges themselves eagerly await the rankings so they can brag and attempt to impress others with the numbers and hierarchies.
We’re here to tell you that the 2018 US News College Rankings are useless. Unproductive. Meaningless. Even harmful. It is time to expose the shortcomings. Here, we outline the main problems with the US News College Rankings, along with examples and consequences of the nonsense we’re talking about.
The Major Problems with the US News College Rankings Are:
• They are inconsistent
• They are missing critical pieces of information
• They use a bogus “Reputation” measure
• They encourage cheating and schools game the system
• They lead to overall negative outcomes
Let’s look at each of these areas in detail and explore the 14 reasons why these rankings are meaningless.
Inconsistency in the US News College Rankings:
The metrics used by US News College Rankings are changing constantly, so it’s impossible to make meaningful comparisons between schools from year to year. Is this a tactic to get rankings to shift slightly to keep people interested, or is it really necessary to change the ranking system from year to year?
Reason #1 – Unstable rankings and irregular shifts continue to be a problem for US News.
Reason #2 – Inconsistencies cause problems for students.
Reason #3 – Flawed metrics have persisted for over a decade.
Reason #4 – Moving up the rankings has nothing to do with improving the student’s experience.
Reason #5 – Failure to adapt.
Reason #6 – Failure to consider student debt load.
Reason #7 – Doesn’t take into account the student experience.The Bogus “Reputation” Measure
US News asks for “peer assessments” from various college presidents, high school counselors, admissions directors, etc. They ask about “intangibles” such as “faculty dedication to teaching.” This can turn the rankings into a popularity contest, fueled by existing prejudices, and allowing prestigious reputations to work in a positive feedback loop to reinforce themselves. This reputation measure comprises more than 20 percent of the overall ranking.
Reason #8 – The popularity contest.
Reason #9 – Skewed rankings by insiders.
Reason #10 – When colleges refuse to participate, US News penalizes them.
Reason #11 – The US News rankings cause institutions to cheat.
Reason #12 – Schools know how to work the system… and they do.
Reason #13 – Rankings lead to higher tuition.
Reason #14 – When students choose colleges based on US News rankings, they’re in trouble.
18.6 8 More Colleges Submitted Incorrect Data for Rankings – InsideHigherEd.com – August 27, 2018
U.S. News will consider them unranked for the rest of this rankings year, but that punishment will last only a few weeks.
Eight of the colleges ranked a year ago submitted information that was sufficiently incorrect that it would have changed their places in the rankings. And so U.S. News has declared them to be "unranked," a punishment that lasts until September 10th, when the new edition comes out.
The eight additional colleges reported to have submitted incorrect data follow other incidents this year, which has seen an uptick in the number of colleges whose rankings were based on incorrect information. In one of those cases -- involving the business school of Temple University -- officials have admitted that incorrect data were submitted intentionally and for several years for several programs. Some of the eight new colleges are blaming human error, not malice, for the flaws in their data. Others are blaming U.S. News.
U.S. News has always said that colleges themselves must audit the data they submit.
18.7 How Much Attention Should You Pay To U.S. News' College Rankings? - Forbes - September 10, 2018
As writer and best-selling author Malcolm Gladwell once pointed out in a scathing critique of U.S. News' rankings in the New Yorker, “There’s no direct way to measure the quality of an institution—how well a college manages to inform, inspire, and challenge its students. So the U.S. News algorithm relies instead on proxies for quality—and the proxies for educational quality turn out to be flimsy at best.”
But perhaps the most damning statement about the U.S. News Rankings comes from U.S. News themselves:
In its 2019 Methodology Statement, the U.S. News wrote:”Hard OBJECTIVE data alone determine each school's rank.
We survey top academics – presidents, provosts, and deans of admissions – asking them to rate the academic quality of peer institutions with which they are familiar on a scale of 1 (marginal) to 5 (distinguished). To get another set of important opinions, U.S. News also surveyed nearly 24,400 counselors at public, private and parochial high schools from all 50 states and Washington, D.C.”
ALL opinion polls (surveys) produce SUBJECTIVE data.
Therefore, their assertion that ”Hard objective data alone determine each school's rank.” is, by definition, deliberately untruthful.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
19. The Problems with the Forbes’ College Rankings
In August 2008, Forbes published their first annual list of "America's Best Colleges". It didn’t take long for the fallout to arrive.
19.1 Forbes college rankings. Can it get any more absurd? – College Admissions Partners – August 14, 2008
Forbes magazine has come out with a new college ranking system in an apparent attempt to compete with US News. The system used by Forbes is in my opinion the most ridiculous system yet devised. I would be more impressed if they just wandered down the street asking people for their opinion on the best colleges.
Here is what the system is based on: 25% of the ranking is based on evaluations from [ulr=https://www.ratemyprofessors.com/]RateMyProfessor[/url], a website that purports to allow students to rate their professors. Of course the website has virtually no control over who posts so anyone, whether a student or not, can go in and post on a particular professor. Even the professor in question can post if they would wish. If you want to post about a professor multiple times, go ahead, they won’t stop you.
The second criteria, also accounting for 25% of the rating, is how many of the college’s alumni (adjusted for enrolment), are listed in Who’s Who in America - published by none other than Forbes magazine.
The final 50% of the ranking is based equally on three factors including the average amount of student debt, the percentage of students graduating in four years, and number of students and faculty (adjusted for enrolment) who have won nationally-competitive awards like Rhodes Scholarships.
Forbes has taken a few important issues and combined them with the least reliable numbers they could find.
19.2 The Sly Logic Behind Forbes Crazy College Rankings - Vanity Fair – August, 2009
Two of the easiest ways for a publication to get attention are to (1) Make a list and (2) Make an outrageous, contrarian argument. Forbes employed both methods in its report on "America's Best Colleges," which it published last week.
Forbes had to work hard to differentiate themselves from the standard-setting U.S. News and World Report best colleges list, and to that effect, they came up with a thoroughly bizarre methodology.
The single most important factor in a college's placement (25 percent) is the average rating of its faculty on [ulr=https://www.ratemyprofessors.com/]RateMyProfessor.ocm[/url], where one of the four categories in which students grade their teachers is "easiness" (there's also a "hotness" category, but that doesn't count toward one's rating).
The second biggest component (20 percent) in the Forbes rankings is the average amount of debt each student incurs. While that is certainly something most prospective students consider, it's an entirely separate issue from the quality of the institution - a fact Forbes implicitly acknowledges. The third most important criterion in Forbes's methodology (16.67 percent) is the percentage of students who graduate in four years.
How can there be no consideration whatsoever given to student satisfaction or quality of life?
Forbes has since addresses this particular criticism – by including Niche’s numbers in Forbes’ Rankings Methodology
In August 2019, U.S. News, Niche, and The Princeton Review will publish their 2020 College Rankings.
In August 2019, Forbes will publish their 2019 College Rankings. Forbes’ College Rankings now lag a year behind their competitors, as Forbes requires Niche’s published 2019 College Rankings as an input to Forbes’ yet-to-be-published 2019 College Rankings.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
20. The Princeton Review Rankings
The Princeton Review Has Laid Off Many Employees. But Revenue Might Not Be Its Only Problem. - Eric Hoover, The Chronicle of Higher Education - August 19, 2018
A thorough search reveals that apparently, no one except the U.S. News has anything bad to say about The Princeton Review’s methodology:
In their Methodology statement U.S. News wrote:Because U.S. News surveys schools directly, much of these data are not readily accessible anywhere else. We do not tour residence halls, chat with recruiters or conduct unscientific student polls for use in our computations.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
21. The Niche College Rankings
A thorough search reveals that apparently, no one except the U.S. News has anything bad to say about Niches’s rankings or methodology. The U.S. News explicitly criticizes the use of student surveys by other college ranking services, while strongly endorsing their own surveys of top academics, presidents, provosts, deans of admissions, and nearly 24,400 high school counsellors. All surveys produce subjective data, but the U.S. News falsely claims ”Hard objective data alone determine each school's rank.”
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
22. Ranking the College Ranking Services
1. Niche (and it’s not even close)
2. The Princeton Review
3. Forbes
4. U.S. News
2020 Best Colleges in America - Niche Best Colleges
2020 Best Catholic Colleges in America - Niche Best Catholic Colleges
.
