crawl4ai

<h1><a href="https://trendshift.io/repositories/11716" target="_blank"><img align="right" src="https://trendshift.io/api/badge/repositories/11716" alt="unclecode%2Fcrawl4ai | Trendshift" width="130"/></a>Crawl4AI</h1>

**Crawl4AI Cloud is a hosted web scraping, extraction, and search API for AI agents and data pipelines.** Turn pages into clean Markdown for RAG, extract structured JSON, or search the live web with one API key. Crawl4AI runs the browsers and manages the proxies.

**[Get a free API key](https://crawl4ai.com/?ref=readme)** · 1,000 free credits every month. No credit card required.

[Try a URL in the playground](https://crawl4ai.com/?ref=readme-playground#playground) · [Set up Crawl4AI Cloud in Claude Code, Cursor, or Codex](https://crawl4ai.com/llms.txt?ref=readme-llms)

This repository is home to the Apache-2.0 crawler that powers Crawl4AI Cloud. [Run it yourself](#run-crawl4ai-yourself) with Python or Docker.

<a href="https://crawl4ai.com/?ref=readme-banner">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-dark.svg">
    <img alt="Crawl4AI Cloud is live. 1,000 free credits every month, no card. Get your key." src="https://raw.githubusercontent.com/unclecode/crawl4ai/main/docs/assets/cloud-launch-banner-light.svg" width="960">
  </picture>
</a>

<div align="center">

[![GitHub Stars](https://img.shields.io/github/stars/unclecode/crawl4ai?style=social)](https://github.com/unclecode/crawl4ai/stargazers)
[![PyPI version](https://badge.fury.io/py/crawl4ai.svg)](https://badge.fury.io/py/crawl4ai)
[![Downloads](https://static.pepy.tech/badge/crawl4ai/month)](https://pepy.tech/project/crawl4ai)
[![Discord](https://img.shields.io/badge/Discord-join%20us-5865F2?logo=discord&logoColor=white)](https://discord.gg/jP8KfhDhyN)

</div>

## Cloud quickstart

Hosted API base URL: `https://api.crawl4ai.com`.

[Cloud API reference](https://api.crawl4ai.com/docs) · [Agent setup and MCP](https://crawl4ai.com/llms.txt?ref=readme-llms)

Set `CRAWL4AI_KEY` to your API key, then scrape a page as Markdown:

```bash
curl -s https://api.crawl4ai.com/scrape \
  -H "Authorization: Bearer $CRAWL4AI_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://news.ycombinator.com"}' | jq -r .markdown
```

<details>
<summary>Python 3</summary>

```python
import json
import os
from urllib.request import Request, urlopen

request = Request(
    "https://api.crawl4ai.com/scrape",
    data=json.dumps({"url": "https://news.ycombinator.com"}).encode("utf-8"),
    headers={
        "Authorization": f"Bearer {os.environ['CRAWL4AI_KEY']}",
        "Content-Type": "application/json",
    },
    method="POST",
)

with urlopen(request) as response:
    print(json.load(response)["markdown"])
```

</details>

<details>
<summary>Node.js (18+)</summary>

```javascript
async function main() {
  const key = process.env.CRAWL4AI_KEY;
  if (!key) throw new Error("S