Scraping and Data Extraction

The web is the world's largest database, and most of it has no API. Web scraping lets you extract data from any website — competitor pricing, public records, job listings, market data, news, and more. Used ethically and carefully, it's a legitimate automation tool. Before scraping, always check: does an official API exist? Does the site have a data export feature? Is scraping allowed under the terms of service? For many business use cases (extracting your own data from a tool, monitoring public competitor pricing, academic research), scraping is fine. For building a competing product with scraped data, it's legally murky. Two main approaches: static scraping with requests + BeautifulSoup (for sites that render HTML server-side) and dynamic scraping with Playwright or Puppeteer (for sites that use JavaScript to render content). Start with static — it's 10x simpler and faster. Only reach for dynamic scraping when the content you need isn't in the initial HTML. Basic static scraping with Python: ```python import requests from bs4 import BeautifulSoup response = requests.get('https://example.com/pricing', headers={'User-Agent': 'Mozilla/5.0'}) soup = BeautifulSoup(response.text, 'html.parser') price = soup.select_one('.pricing-amount').text.strip() ``` Always set a User-Agent header to identify your scraper politely. Add delays between requests (time.sleep(1-3)) to avoid hammering servers. Respect robots.txt — check if the path is allowed before scraping it.

Traditional scraping relies on CSS selectors and HTML structure, which breaks when the site redesigns. AI-powered extraction is more resilient — you describe what you want in plain language, and the model extracts it from raw HTML. Approach: fetch the page, pass the HTML (or just the text content) to Claude, and ask it to extract specific information: ```python import anthropic client = anthropic.Anthropic() response = client.messages.create( model='claude-haiku-4-5', max_tokens=1024, messages=[{ 'role': 'user', 'content': f'''Extract all product prices from this HTML. Return ONLY a JSON array: [{{"product": "name", "price": 99.99, "currency": "USD"}}] HTML:\n{html_content[:8000]}''' }] ) prices = json.loads(response.content[0].text) ``` This approach handles minor layout changes without breaking. The tradeoff: it's slower and more expensive per page. For high-volume scraping (thousands of pages), use traditional CSS selectors. For low-volume, high-value extraction (monitoring 20 competitor pages daily), AI extraction is worth the cost. For dynamic sites, Playwright in headless mode is the standard. Install it on your n8n server, write a Python script that launches a headless browser, navigates to the URL, waits for the content to load, and extracts the data. Call this script from n8n's Execute Command node.