New Research on the Role of AI in the College Admissions Process

What does “holistic admissions” mean?

What exactly do colleges mean by the term “holistic admissions?” Research from the University of Michigan’s Center for the Study of Higher and Postsecondary Education has shown that 95% of selective colleges say they embrace a “holistic” practice in which the sum of a student’s life experience, background, and essays are weighed in concert with traditional metrics like grades and test scores.

To achieve this, admissions offices try to discern various traits and qualities that will make the students a good fit for their school. According to a NACAC survey, 70% of admissions officers say that a student’s personal qualities are an important factor in their review of applicants. The reason for this goes beyond the idea that qualities like teamwork or goal pursuit will make those kids nice to have around on campus; longitudinal studies of student outcomes show that such qualities are actually strong predictors of basic success metrics like graduation rate. If you’re admitting a young person to your selective school, making sure they have what it takes to complete the degree is fundamental. 

Unfortunately, appreciating these qualities in an admissions packet takes time that most admissions officers don’t have. The average read time for an entire packet is only 6 minutes, and many counselors tell students they have only a few seconds to hook a reader in an essay. That’s not a lot of runway to convey their life story. The most desired qualities are also often vaguely defined, leading to a prevailing sense amongst students, parents and college counselors that the criteria for admission live within a black box guarded by university administrators.

How Do We Test AI’s ability to identify Human Qualities?

As the application volume to top schools swells each year, how can admissions teams give each candidate the attention they deserve? A study published this October by researchers at the University of Pennsylvania explored whether artificial intelligence (AI)—of late a familiar bogeyman in conversations about college apps—could actually be used to “advance the goals of holistic admissions.” 

UPenn researchers trained an open-source, pre-trained AI model named RoBERTa on a rating system built by 36 college admissions officers who participated in the study. Their assessment criteria, including the seven most desirable personal qualities in applicants, were then coded into the model by manually reading and tagging more than 3,000 student essays submitted through the Common App. Again, these seven traits strongly predict successful graduation within 6 years of matriculation.

The AI model was then asked to recognize these qualities in more than 306,000 de-identified student essays. The result, in lay terms, was that AI analysis correlated with human readers to an extremely high degree. Simply put, the study suggests that AI is a promising means of processing huge amounts of applications with transparent judgment criteria far more efficiently and consistently than human readers can. 

Of course, any innovation that places decisions about the future of individual humans into the hands of machines should be approached with the utmost caution. How concerned should students and their college counselors be about such developments, especially given that this process highlights intimate human experiences? 

To begin, it’s worth remembering that these models are narrowly trained and applied algorithms and not some sentient general artificial intelligence like HAL or the Terminator who intend to lay waste to our children’s dreams. In this case, they’ve been asked to tag keywords in short essays, not render summary judgment on a student application. 

More broadly, as AI researchers will tell you, any biases identified in these models stem from subjective judgments provided by human readers to begin with. In fact, AI may actually offer an advantage here. Because algorithmic models can average the opinions of specific readers into a single institutional view, such AI promises to make those specific human biases less determinative of admissions outcomes. An AI model would in principle provide a fairer shot to two different applicants than they would enjoy from having two different human readers assess each of their essays.

What are the 7 Most Desirable Traits that Predict College Graduation?

Having explored those more general results of the study, the specific personal qualities it used in the process also offer a valuable template for students and counselors as they craft essays. With these qualities as benchmarks, it would be a shrewd idea to sprinkle certain keywords that indicate traits like leadership and intrinsic motivation into their essays regardless of whether a human or AI is reading an admissions essay. Put differently, the categories and vocabularies that human readers had to create to conduct this experiment reveal a lot about they value and therefore provide a roadmap for students to follow as they prepare their applications. 

Below are the 7 different personal traits that the group of 36 AOs from different schools provided. Each is accompanied by a functional definition of the quality and each includes an example of an essay that AOs tagged as representative of that trait. 

Prosocial Purpose

Helping others, wanting to help others, considering the benefits to others, mentioning reasons for helping others, or reflecting on how enjoyable or rewarding it is to help others.

• Example: Every summer for the last 3 years, I worked as camp counselor at a camp for young children from underprivileged families. Helping children realize their hidden talents is one of the most rewarding experiences I have ever had. I’ve been so fulfilled by watching these children develop confidence in their abilities. This experience has been so important to me, and it showed me that a career in education is where I belong.

Leadership

Serving in a leadership role, commenting on what he or she did in his or her capacity as a leader, or discussing the value, meaning, or importance of leadership.

• Example: I was chosen to be cheerleading captain during my senior year. My freshman year captain had a huge impact on my life, and I felt like it was my time to pay it forward. I am so proud of everything I did for the girls: creating a mentorship system, organizing events and fundraisers, and encouraging everyone to work as hard as they could. At the end of the year, a few girls thanked me. I was completely overcome with emotion. I’ve never felt so gratified in my life.

Learning

Improving, learning, or developing knowledge, skills, or abilities.

• I played softball in high school. When I started, I was not a very strong player. When I finally made the varsity team my senior year, I was determined to have a better season. I worked constantly to improve my game - during practice and on my own time. My skills grew so much. Because of my hard work, I finished the year with the best record on my team!

Goal Pursuit

Having a goal or a plan. 

• I have been playing soccer since I was 6 years old. Unfortunately, last year I injured my knee, and it has been a struggle to get back to the level I was playing at before my injury. It has been really challenging, but I’ve been doing physical therapy and practicing everyday so that I can be a varsity starter this year.

Intrinsic Motivation

Describing the activity as enjoyable or interesting. Liking the activity or identifying with it.

• Running track is so much more than a sport to me. It’s a challenge and an adventure, and I put everything I have into it. I love every aspect of it, even the afternoons I spend drenched in sweat in the scorching heat.

Teamwork

Working with or learning from others. Valuing what fellow participants bring to the activity.

• I’ve been on my school’s debate team since my freshman year, and was elected co-captain because of my commitment to the team’s success. My fellow co-captains and I worked together to get our team ready for competitions. We knew that a strong team performance was more important than the successes of a few individuals. We stressed teamwork and cooperation between our teammates. Because we focused on team effort, we earned first place at the state meet.

Perseverance

Persisting in the face of challenge.

• I’ve learned to become a gracious victor and to grow from defeat. Track has helped me overcome my fear of losing, and even helped me put my life in perspective. I’ve learned to keep working and fighting even when the odds seem impossible to beat. There were many times that I found myself lagging, but I pulled ahead at the end because I never gave up. The most important thing I’ve learned is to never let anything stand in my way.

Where do we go from here? 

While the use of these key terms within a college essay would certainly seem to check certain important boxes for essay readers, there are also caveats to consider. For one, the UPenn study invokes Campbell’s law, which states that the more weight given to an assessment in high-stakes decisions, the greater the incentive for distortion. That is, knowing that colleges are looking for certain traits compels more students to mold their essays to highlight those traits, which in turn makes them less useful in the process. If everyone demonstrates basic prosocial or perseverance qualities, then they may cease to provide useful signals to colleges. This distortion holds for either human readers or the AIs they train. 

It’s also clear that even this highly refined model is a blunt instrument. It does a great job of recognizing the right words that connote certain desirable personal qualities, but a bad job of understanding the context in which they’re used. And context is key in holistic admissions practices. 29% of colleges who embrace holistic admissions reach the threshold of “whole context” which goes beyond just what is in the student’s file to account for the privilege (or lack thereof) that their socioeconomic status provides. And the schools most likely to consider the whole context are the most competitive. 

At the risk of a simplistic conclusion, one might say there’s both good news and bad news in this study. The bad news is that students who learn how to game the system by using these keywords may get through the first admissions checkpoint much more creative and qualified students who don’t use those terms. (In fact, there’s ample evidence to show that students with lower socioeconomic status who can’t afford college counselors who show them which terms to use are penalized by admissions offices.) 

The good news is that, for the moment, developments like this one actually won’t change much about student chances in their applications. After all, AI readers will carry the same biases as the humans who trained them, and most of the filters being applied are elemental screening criteria that AOs already use. AI just promises to do it faster, potentially allowing readers to devote more time to the stories of students who do meet minimum thresholds.

To end on a cautiously optimistic note, it may even be better to have an AI read essays on the first found. Since they deliver an “average” assessment of what a given college is looking for, rather than the subjective feelings of the first reader of a student file, AI may be a dispassionate and therefore more equanimous judge in the admissions office.