𝗠𝘆 𝗪𝗲𝗯 𝗦𝗰𝗿𝗮𝗽𝗶𝗻𝗴 𝗡𝗶𝗴𝗵𝘁𝗺𝗮𝗿𝗲 𝗘𝗻𝗱𝗲𝗱 𝗪𝗶𝘁𝗵 𝗟𝗟𝗠𝘀

𝗠𝘆 𝗪𝗲𝗯 𝗦𝗰𝗿𝗮𝗽𝗶𝗻𝗴 𝗡𝗶𝗴𝗵𝘁𝗺𝗮𝗿𝗲 𝗘𝗻𝗱𝗲𝗱 𝗪𝗶𝘁𝗵 𝗟𝗟𝗠𝘀

I built scrapers for years. I used CSS selectors. It worked for one site. It failed for ten sites. Every page had a different layout. Code became a mess.

I tried a new way. I fed raw HTML to an LLM. I told the AI which data I wanted. AI understands text meaning. It ignores tags.

Here is how you do it:

• Use a cheap model like GPT-4o-mini.
• Write a prompt for specific fields.
• Ask for a JSON object.
• Trim HTML to save money.

I improved the process:

• Remove script and style tags.
• Cache results by URL.
• Validate JSON output.
• Use a hybrid system. Try selectors first. Use AI as a backup.

Use this for:

• Sites with shifting layouts.
• Small projects.
• Fast prototypes.

Avoid this for:

• Millions of pages.
• Real time needs.
• Private data.

Try this for job boards or news. Start small. Measure accuracy.

What is your setup? Do you use AI or traditional scrapers?

Source: https://dev.to/__c1b9e06dc90a7e0a676b/my-web-scraping-nightmare-ended-when-i-let-an-llm-read-the-html-1bj4 Optional learning community: https://t.me/GyaanSetuAi