MIT Webscraper
← Back to Blogs
MIT Webscraper
2026-08-15
When I reached out to MIT they told me their personal blog site had<br>7,270 blog posts. I know now that they have exactly 9,098, as well as<br>every author, classifications of each article, and enough statistics<br>for a blog post. (Hey, I should make one of those!)
This journey started about a month ago. I was researching MIT admissions<br>on their blog site. MIT themselves constantly recommend<br>their blog.<br>This site contains some of the best admissions advice that I can find<br>anywhere, but the vast majority of recent articles seem to not contain<br>any advice at all.
The most recent article on the site is this:
towards the end of october, back in 2024, i was walking back to<br>my dorm and i was about to make the turn from the asphalted<br>footpath that went north from the music building (under<br>construction then) onto vassar st., my thoughts still swimming<br>above me in the elation of starry nights and tall roofs and<br>vents to be crawled over and squeezed through, when i realized<br>that i was halfway through the semester.
it was 3 am then, crickets humming in the grass, and i knew then<br>that i was one-sixteenth of the way through my time at MIT.
but that was a good thing then, because one-sixteenth of my time<br>at MIT had taken a long while. i knew then that i had fifteen<br>more copies of what i had experienced so far,
fifteen times more psets and morning exams in walker memorial<br>and evening square dance with warm bundt cake resting on the<br>table and linguistics essays and midday bluebike errands to<br>chinatown and auroras and books read on the grass at boston<br>commons and leaves along vassar st. turning to the color of the<br>sunset.
i knew that i had fifteen times more of all this.
but now? now i am halfway through. and already i do not have<br>many of those fifteen left.
Where is the admissions advice? For a site that is about MIT admissions<br>this is a pretty poor article.
This is common as well. Looking through the first pages of the site,<br>you can easily see that most of the articles on this site are not about<br>admissions at all. They are blog posts for students already in<br>MIT .
No matter how much I claim that all the blog posts are not admissions<br>focused, I cannot prove it. According to MIT there are more than 7,000<br>articles. I cannot look at all of them!
MIT's Response
Email response from MIT from 7/30/26:
There have been over 7,270
blog entries
posted to MITAdmissions.org. They're all worth reading, but there<br>are a few you really need to read. These are the blog entries that<br>cut deep into our admissions process. Not only what we're looking<br>for, but what we want from you, and how much we care about the work<br>that we do. These blogs will help you "get" what the MIT admissions<br>process is all about.
They include, in no particular order:
It's More Than A Job
by Ben Jones
Applying Sideways
by Chris Peterson
Bleary-Eyed
by Mikey Yang
The Room(s) Where It Happens
by Jessica Ch'ng
There Is No Formula
by Ben Jones
(BTW, those articles listed are some of the only articles that I<br>have found, and people have told me about, that talk about<br>admissions.)
This is also a pretty old statistic as I now know there are exactly<br>9,098 articles on that site.
Building the Scraper
When I was in 7th grade I made a "search engine." To be honest, it kind<br>of sucked and did not work well, but it did teach me how to make a link<br>scraper and scrape websites in Python.
A few weeks beforehand I also made my first thing using AI in Python<br>with Ollama. This gave me an idea: what if I find all the blog article<br>posts, and then use AI to classify them?
That is why I am now writing this blog post here and now.
Starting this project I knew that the challenge would not be to<br>actually do the scraping and analyzing. That is not difficult to make.<br>The challenge would be speed.
Since I am writing this in Python, it was going to be a challenge to<br>make it work quickly and efficiently. Writing this in a language like<br>Go would have been a good idea, but I had no idea how to make a link<br>scraper in it!
Scraping an Article
To start this project out I made a simple program that would go to a<br>blog post, get all of the text, title, date released, etc.
I was worried that getting this data would be difficult, but<br>unironically, they literally have a single div for all of the text in<br>the article and a single div for the header that contains the article<br>name, author and date released.
IT'S LIKE THEY WANTED ME TO SCRAPE THE SITE.
The names of these divs were<br>article__body and<br>page-topper_tittle respectively.<br>THIS MADE THINGS EASY BEYOND BELIEF.
I started out with the article
Applying Sideways<br>which is easily the most referenced article about MIT admissions.<br>It is a good article, but I do not 100% believe it.<br>(That is a story for another day.)
Once I scraped the first article, it should have been obvious:<br>it was a big article. I do not actually need the entire article to<br>classify it.
The beginning...