Connecting Claude Code to Apify: A Practical Guide to AI-Driven Web Scraping & Automation

Author’s note:

Question: What is Apify, and how do i use Claude Code with it?

Context: Context:

I'm using the @apify skills for Claude. Here is the GitHub repository:https://t.co/nMy5kRq3U1

You can install these as a plugin, or you can also build your own skill to do whatever you want.

Alternatively, you can install any of the scrapers using MCP directly in Claude Code. pic.twitter.com/463bQYRad3

— Santiago (@svpino) January 29, 2026

Executive Summary

Integrating Apify with Claude Code bridges the gap between static LLM knowledge and real-time web data. By connecting Claude Code to Apify’s hosted Model Context Protocol (MCP) server, developers can instantly empower their AI agents to scrape websites, automate browser interactions, and access over 5,000 ready-made tools (Actors) without writing custom scraper code.

Key Insights:

• Unified Access: A single MCP connection (https://mcp.apify.com) unlocks the entire Apify marketplace, allowing Claude Code to discover and run Actors dynamically ^{1 2}.
• Performance Boost: Offloading heavy browser rendering (e.g., via Playwright/Puppeteer Actors) to Apify’s cloud infrastructure can reduce local processing latency by up to 70% compared to running browsers locally ^{3 4}.
• Cost Efficiency: Apify’s pricing model, which combines a free tier with credit-based usage (residential proxies at $8/GB), offers a scalable alternative to per-request pricing models like Zyte’s, particularly for complex scraping tasks ^{5 6 7}.
• Enterprise Control: Organizations can enforce security policies by deploying managed-mcp.json files or allowlists, ensuring that AI agents only access approved tools and datasets ^{8 9}.

Introduction: Why AI-Driven Scraping Matters Today

As AI agents evolve from passive chatbots to active assistants, their ability to interact with the live web becomes critical. Static training data is insufficient for tasks like monitoring stock prices, aggregating news, or verifying product availability.

What is Apify?

Apify is a cloud-native web scraping and automation platform that serves as a comprehensive ecosystem for data extraction. It hosts “Actors”—serverless cloud programs that can perform arbitrary computing tasks, from simple HTML parsing to complex headless browser automation ^{1 10}.

Actors & The Apify Store

The core of the platform is the Apify Store, a marketplace containing over 5,000 ready-made Actors. These pre-built tools allow users to scrape popular websites (like Instagram, Google Maps, or Amazon) or perform utility tasks (like converting HTML to PDF) without maintaining the underlying infrastructure ^{1 2}.

Apify SDK: Code-First Development

For developers building custom solutions, Apify provides SDKs in JavaScript and Python. These SDKs manage the complexities of scraping, such as:

• Request Queues: Managing URLs to be crawled ¹⁰.
• Key-Value Stores: Saving unstructured data like screenshots or PDFs ^{10 11}.
• Datasets: Storing structured results (JSON/CSV) for export ¹⁰.

Pricing & Quota Model

Apify operates on a consumption-based model. New users start with a free tier that includes platform credits. Paid plans offer additional resources, such as residential proxies ($8/GB) and datacenter proxies (first 20 IPs free, then $1/IP) ⁶. This contrasts with competitors like Zyte, which often charge per successful request ⁷.

Model Context Protocol (MCP): The Glue Between AI and Tools

The Model Context Protocol (MCP) is an open standard that standardizes how AI models interact with external tools and data. It allows clients like Claude Code to connect to “servers” that expose specific capabilities ¹.

MCP Server Types

Claude Code supports three primary transport modes for MCP servers:

1. Remote HTTP/SSE: Ideal for cloud-hosted services like Apify. It uses Server-Sent Events (SSE) for real-time communication ⁸.
2. Local Stdio: Runs a local process (e.g., a Python script) and communicates via standard input/output. This is useful for local file manipulation or private scripts ⁸.
3. Plugin-Provided: Bundled servers included with Claude Code plugins ⁸.

Apify MCP Server Specifics

Apify provides a hosted MCP server at https://mcp.apify.com. This server acts as a gateway, translating MCP tool calls from Claude Code into API requests that execute Actors on the Apify platform. It supports OAuth for secure authentication and “dynamic tool discovery,” allowing the AI to search for and load new Actors on demand ^{1 2}.

Integrating Claude Code with Apify: Step-by-Step

Connecting Claude Code to Apify is straightforward, involving account setup and a simple configuration command.

1. Prerequisites

Before starting, ensure you have:

• Apify Account: Sign up at Apify.com.
• API Token: Generate a token in the Apify Console (Integrations section) ¹.
• Claude Code: Installed and authenticated on your machine ⁸.

2. Adding the Server

You can add the Apify MCP server to Claude Code using the command line. The recommended method uses the hosted HTTP endpoint.

Command for Remote HTTP (Recommended):

claude mcp add apify -- https://mcp.apify.com

Note: When you first use this connection, Claude Code will initiate an OAuth flow, redirecting you to a browser to authorize access securely ^{1 8}.

Alternative: Manual Configuration via JSON For project-specific setups, you can create or edit a .mcp.json file in your project root. This method is useful for sharing configurations within a team ⁸.

{
 "mcpServers": {
 "apify": {
 "url": "https://mcp.apify.com",
 "type": "sse"
 }
 }
}

3. Authentication Flows

• OAuth (Best for Interactive Use): The default for the hosted server. It avoids storing long-lived API tokens in plain text configuration files ¹.
• Bearer Token (Best for Automation/CI): If you need to bypass the browser flow (e.g., in a headless environment), you can inject the token directly via headers.
• Security Tip: Use environment variable expansion (${APIFY_TOKEN}) rather than hardcoding the token ⁸.

4. First-Run Demo: RAG-Web-Browser

Once connected, you can ask Claude Code to perform tasks that require web access.

Prompt:

“Use the Apify RAG Web Browser actor to scrape the latest headlines from news.ycombinator.com and summarize the top 3 stories.”

What happens behind the scenes:

1. Claude Code identifies the intent and calls the Apify MCP tool.
2. The MCP server triggers the apify-slash-rag-web-browser Actor.
3. The Actor runs in the cloud, renders the page, extracts text, and returns it to Claude.
4. Claude processes the text and generates the summary ^{2 4}.

Advanced Capabilities

Dynamic Tool Discovery

One of the most powerful features of the Apify MCP integration is dynamic discovery. Instead of pre-loading thousands of tools, the server provides an add-actor tool.

• How it works: If you ask Claude to “scrape Instagram comments,” it can use add-actor to search the Apify Store, find the relevant Instagram scraper, and dynamically add it as a usable tool for the current session ^{1 2}.
• Client Support: Full dynamic discovery is supported in VS Code (Genie) and Cursor. Claude Desktop and Claude Code have partial support, sometimes requiring a manual tool reload ².

Enterprise Governance

For organizations, allowing unrestricted access to external tools can be a security risk. Claude Code supports “Managed MCP” configurations.

• Exclusive Control: Administrators can deploy a managed-mcp.json file to system directories (e.g., /etc/claude-code/ on Linux), which overrides user settings and locks down the available servers ⁸.
• Policy-Based Control: Alternatively, admins can use allowlists (allowedMcpServers) or denylists to permit specific servers while blocking others ⁸.

Windows Stdio Pitfall

If you choose to run a local Apify MCP server (using npx) on Windows, you must wrap the command. Windows cannot execute npx directly.

• Error: “Connection closed”
• Fix: Prefix the command with cmd /c.
• Example: cmd /c npx -y @apify/actors-mcp-server ⁸.

Cost & ROI Analysis: Apify vs. Competitors

When selecting a platform for AI-driven scraping, the choice often comes down to Apify, Zyte, or Bright Data.

Feature & Pricing Comparison

Break-Even Scenarios

• Low Volume / Simple API: Zyte’s pricing (starting at $0.06 per 1,000 requests) can be cheaper for simple HTTP requests that don’t require browser rendering ⁷.
• High Volume / Browser Rendering: Apify becomes more cost-effective when tasks require heavy browser automation (Puppeteer/Playwright). Zyte charges significantly more for browser-rendered requests ($1.01+ per 1k requests), whereas Apify charges for compute time, which can be optimized ⁷.

Real-World Use Cases & Code Library

Here are three practical prompts you can use immediately after configuring the Apify MCP server.

1. Competitive Intelligence Monitor

Prompt:

“Check the pricing page of [Competitor URL] using the ‘Web Scraper’ actor. Extract the pricing tiers and compare them to our current pricing table stored in pricing.csv. Alert me if they are undercutting us.”

2. SERP Analysis

Prompt:

“Use the ‘Google Search Results Scraper’ actor to find the top 10 results for ‘AI coding assistants’. Export the titles and descriptions to a JSON dataset and summarize the common keywords in the descriptions.”

3. Documentation Aggregator

Prompt:

“Crawl the documentation at [Docs URL] using the ‘Website Content Crawler’. Download all pages as Markdown and save them to a local Key-Value store for my knowledge base.”

Bottom Line

Integrating Apify with Claude Code transforms the AI from a text generator into a capable web automation agent.

Key Takeaways:

1. Start Simple: Use the hosted MCP server (https://mcp.apify.com) and OAuth for the easiest setup ¹.
2. Leverage the Store: Don’t reinvent the wheel. Use dynamic discovery to find existing Actors for Instagram, Google Maps, or Amazon before writing custom scripts ².
3. Watch Your Scope: Use project-scoped configuration (.mcp.json) for team repositories to ensure everyone has access to the same tools ⁸.
4. Optimize Costs: Use Apify for complex, browser-heavy tasks where its compute-based pricing offers better ROI than per-request alternatives ⁷.

Next Step: Run the claude mcp add command today and try the “RAG-Web-Browser” demo to see the immediate speed difference in your research workflows.

References