Today I confess: I am a #webscraper. I use my skills to extract information from websites. But I am also lazy. And you know I don’t like doing the same thing twice if there’s a machine that can do it.

The Statistics

• Category A: 70% of websites are child’s play. Developing a custom Bot is almost an insult to the lazy.
• Category B: 15% of websites are a bit more difficult, requiring Java rendering and thus some more advanced technology. Okay, not for kids, maybe for smart teenagers. Developing a custom Bot is almost entertaining. Almost.
• Category C: 10% of websites have protection systems against DoS attacks (number of requests per IP, request frequency, tokens, header detection, things like that). One must be respectful so that your Bot is not mistaken for an attack and, above all, pay attention to what the owners indicate in the robots.txt. These websites are more entertaining. Finding the right way to extract information is indeed challenging. This is grown-up stuff.
• Category D: the remaining 5%. The most complicated. They encompass all the complexities of categories A, B, and C, and sometimes even more. I give up on these. But studying them is very, very entertaining and fun.

Custom Bots Expire

Yes. It’s a fact. Websites change and sometimes a simple CSS change renders the custom Bot invalid. Imagine a project that collects information from 100 websites. Maintaining 100 operational bots of category A, B, and C can take up the rest of your life. And especially since, according to statistics, 70 of those bots are really simple and the time investment seems excessive.

GPT to Extract Data

What if I directly ask #GPT4 to extract the data from a website? Let’s see… a slightly long prompt… a bit of Python to iterate through the website… this is still custom but I’ll fix it later if GPT gives me the data correctly… and it works!!! GPT gives me the data I requested in perfectly formatted JSON that can be processed. There are only two HUGE drawbacks:

1. The cost of tokens (millions of tokens to process).
2. Processing time. It’s sooo slow!!! almost two seconds per page. Unbearable for a massive environment.

We need to go in another direction.

The Idea

The idea here is not mine. An indexing website of AIs (https://theresanaiforthat.com/) and one of the AIs discovered on that website (https://www.kadoa.com/) put me on the track. #Kadoa uses GPT to generate Python code adapted to a requested website. Of course! There’s no need to waste time on category A websites if a machine can do it! The only drawback of Kadoa is… it has a cost. A lazy person is also stingy. So, garage mode ON.

The Deflation

“I can do this,” I say to myself as I grab the GPT API. But soon I realize the technology is still very green. Generating code that processes the entire volume of HTML of a website using GPT is not easy at the moment. A small-sized website already exceeds the token limit of most current models. The partitioning technique sometimes renders the generated code useless. The generated code does not include the necessary iterations to resolve website paginations, so only the first page is processed. The obtained code sometimes fails due to perfectly controllable errors… phew.

Fortunately, I have many more weapons and ideas. What if we give GPT only the piece of HTML that contains the relevant information? And if we solve the page iteration ourselves?

The Relevant Information

The key is to pass GPT only the piece of HTML that contains the information to extract. Let’s see. This information is usually in lists. Lists are sets of identical elements whose content changes. Aha! Statistics can be used here. So here we go: a bit of #BeautifulSoup, a bit of #Pandas to calculate variables, manual labeling to build a target variable, a bit of #SpaCy to categorize words, a bit of #Scikit-learn with a GBT model for classification, and voila. I already have a weapon that indicates which pieces of HTML are most likely to be the lists I need. What will GPT do with this? Well, a bit of ingenuity with the prompt and I get a perfect piece of Python code having used only a few tokens and whose execution time is exactly the same as a custom Bot. Moreover, having a program write programs for other programs has something of Matrix that seduces me.

The Iteration

Piece of cake. If I have been able to identify the list, create a probabilistic model that identifies it, and ask GPT to build a scraper… iterating over a pagination should be child’s play… And it is. Here I have to use several models for different types of pagination systems because otherwise, the precision decreases. But it’s just a matter of being a bit more patient.

The Program that Executes Programs Written by Another Program

And now it’s just about putting it all together. The result: a generic (and intelligent, hehe) Bot that can process category A pages divinely. Suddenly, I have 70% of the work automated. Well, a bit less, as my generic Bot currently only processes very basic stuff. Let’s say 40%. But it still allows me to go for a beer with the free time. Hooray.

Never underestimate a lazy person.