Development skill

Web scraping

Updated

Reliable, ethical web scraping with fallback strategies, access-failure handling, and social media extraction.

Works in: Uses the open Agent Skills format. Codex: candidate, adapter review pending.

When to use

What's included

Scraping cascade

Three-tier fallback: Trafilatura (fast) to Requests (HTTP) to Playwright (JavaScript rendering). An access-denied response is a stop signal, not a reason to escalate.

Poison pill detection

Detect paywalls, CAPTCHAs, rate limits, Cloudflare, and login walls with pattern matching.

Undocumented APIs

Find and use hidden APIs via browser dev tools, with examples for autocomplete endpoints.

Social media tools

yt-dlp for YouTube/TikTok, instaloader for Instagram, with metadata extraction and download patterns.

Browser automation by reference

A reference for driving live pages through structured snapshots and element references. Screenshots verify results; they do not pick click coordinates.

Scraping cascade architecture

Try multiple extraction strategies with automatic fallback:

Fast

Trafilatura

Lightweight extraction for standard articles. Best for news sites and blogs.

Medium

Requests + BeautifulSoup

HTTP requests with rotating user agents. Good for static content.

Heavy

Playwright rendering

Full JavaScript rendering for SPAs and other pages that need JavaScript to show authorized content. Run it in an isolated environment with private-network egress blocked.

Poison pill types

Type Detection patterns
Paywall "subscribe to continue", "you've reached your limit"
CAPTCHA "verify you are human", "robot verification"
Rate limit "too many requests", HTTP 429
Cloudflare "checking your browser", "ddos protection"
Login required "sign in to continue", "create an account"

Installation

# Recommended: install the dev-toolkit plugin

/plugin marketplace add jamditis/claude-skills-journalism

/plugin install dev-toolkit@claude-skills-journalism

# Or copy just this skill from the plugin tree

git clone https://github.com/jamditis/claude-skills-journalism.git

mkdir -p ~/.claude/skills ~/.agents/skills

cp -r claude-skills-journalism/dev-toolkit/skills/web-scraping ~/.claude/skills/ # Claude Code

cp -r claude-skills-journalism/dev-toolkit/skills/web-scraping ~/.agents/skills/ # Codex

Or browse this skill in the GitHub repository.

Related skills

Extract what you need, ethically

Cascade architecture, poison pill detection, and social media tools in one skill.

View on GitHub