How to Give ChatGPT Web Scraping with MCP Connectors (2026)

ChatGPT can now call your own tools through custom MCP connectors — including web scraping. But there is a catch the marketing pages skip: connectors must be remote servers, so a local tool like CrawlForge cannot be pasted in directly. This is the honest version: what is actually possible, why a wrapper is needed, and the exact bridge to build.

Table of Contents

• What ChatGPT Connectors Are


• Which Plans Can Use Them


• The Transport Catch: Remote Only


• Why CrawlForge Needs a Wrapper


• Build the Bridge


• Add the Connector in ChatGPT


• Auth and Safety


• A Simpler Alternative

What ChatGPT Connectors Are

ChatGPT supports custom MCP connectors — renamed "apps" in December 2025, so the UI now says "Apps & Connectors." Through Developer mode, you connect an external MCP server and ChatGPT calls its tools mid-conversation, asking you to confirm before any write action. Same Model Context Protocol that powers web scraping in Claude — different client. Developer mode is explicitly a beta.

Which Plans Can Use Them

Per OpenAI's plan table, adding a custom MCP connector is available on Plus, Pro, Business, Enterprise, and Edu — not Free or Go. Full write-action support is rolling out most broadly to Business, Enterprise, and Edu. If you only need ChatGPT to read scraped data, the read-only path below is enough.

The Transport Catch: Remote Only

This trips people up. A ChatGPT connector must be a remote MCP server reachable over HTTPS (SSE or Streamable HTTP transport). You paste a URL; you do not point it at a command on your machine. That rules out local stdio servers — the kind you install with npx. To use one, host it publicly or tunnel a local server via ngrok or Cloudflare Tunnel.

There is also a naming rule: ChatGPT's deep research / company-knowledge paths require two read-only tools named search and fetch with a specific schema. Full Developer mode allows arbitrary tools, so that constraint applies only to the deep-research path.

Why CrawlForge Needs a Wrapper

CrawlForge ships as a local stdio MCP server (via npx) plus a REST API at https://www.crawlforge.dev/api/v1/tools/. Neither is a remote MCP URL, and its tools are named search_web, fetch_url, and extract_content — not the search/fetch pair deep research expects. So you cannot paste CrawlForge straight into ChatGPT today. The practical path is a thin remote MCP wrapper — about 30 lines.

Build the Bridge

FastMCP (Python) is the quickest way to stand up a remote MCP server exposing the search + fetch tools ChatGPT wants. Each calls CrawlForge's REST API with your cf_live_ key in the X-API-Key header:

import os

import httpx

from fastmcp import FastMCP



mcp = FastMCP("CrawlForge Bridge")

BASE = "https://www.crawlforge.dev/api/v1/tools"

HEADERS = {"X-API-Key": os.environ["CRAWLFORGE_API_KEY"]}



@mcp.tool()

async def search(query: str) -> list[dict]:

    """Search the web. Returns id/title/url results for ChatGPT."""

    async with httpx.AsyncClient(timeout=30) as client:

        r = await client.post(f"{BASE}/search_web", headers=HEADERS,

                              json={"query": query, "limit": 10})

    results = r.json().get("results", [])

    return [{"id": x["link"], "title": x["title"], "url": x["link"]} for x in results]



@mcp.tool()

async def fetch(id: str) -> dict:

    """Fetch full page content by id (the URL) for ChatGPT."""

    async with httpx.AsyncClient(timeout=30) as client:

        r = await client.post(f"{BASE}/extract_content", headers=HEADERS,

                              json={"url": id})

    data = r.json()

    return {"id": id, "title": data.get("title", id),

            "text": data.get("content", ""), "url": id}



if __name__ == "__main__":

    mcp.run(transport="http", host="0.0.0.0", port=8000)

Run it and expose it over HTTPS. For a quick test, tunnel your local port:

pip install fastmcp httpx

export CRAWLFORGE_API_KEY="cf_live_your_key_here"

python server.py

# in another terminal:

ngrok http 8000

For Developer mode you can skip the search/fetch naming and map tools one-to-one to CrawlForge — expose scrape_structured, stealth_mode, or deep_research directly. Same pattern.

Add the Connector in ChatGPT

1. 
   Settings → Apps & Connectors → Advanced → enable Developer mode.


2. 
   Apps & Connectors → Create.


3. Paste your public HTTPS MCP URL (e.g. your ngrok URL plus /mcp), name it, choose an auth method.


4. Confirm the "I trust this application" checkbox.

Your search and fetch tools appear. In a chat, select the connector and ask ChatGPT to research a topic — it calls search, then fetches the best results through CrawlForge.

Auth and Safety

Connectors authenticate with none (public) or OAuth — there is no API-key-header option in the UI, which is why the wrapper holds your CrawlForge key server-side. ChatGPT confirms before write actions, and you can inspect each call before approving.

Take OpenAI's warnings seriously: only connect servers you trust. Custom connectors increase risk, including prompt injection, and a model mistake on a write action could destroy or leak data. A read-only scraping bridge is low-risk; lock it down with OAuth before sharing.

A Simpler Alternative

If you would rather not host anything, use CrawlForge from code with the OpenAI Agents SDK or Responses API — no remote server required. See CrawlForge with the OpenAI Agents SDK.

Get a Free CrawlForge Key — 1,000 Credits