Marketplace PluginsWatercrawl

Plugin

Limited listing

Watercrawl

Name: Watercrawl
Author: watercrawl

by watercrawl · v0.1.0

Watercrawl Datasource

1.8k installsUpdated Oct 5, 2025

Publisher information is incomplete

This community listing does not yet include every recommended support, privacy, pricing, and permission disclosure. Review the available package permissions before installing.

Watercrawl Datasource Plugin for Dify

A powerful Dify datasource plugin that integrates with Watercrawl to recursively crawl websites and extract clean, LLM-ready content for AI applications.

🚀 Quick Start

Get API Key: Sign up at Watercrawl and get your API key
Install Plugin: Add the Watercrawl datasource in your Dify workspace
Configure: Enter your API key and start crawling websites
Extract: Get clean, structured content ready for your AI models

✨ Features

Feature	Description
🕷️ Recursive Crawling	Automatically discover and crawl linked pages
⚡ Fast Mode	Skip rendering for 3x faster crawling
📏 Depth Control	Limit crawl depth to control scope
🎯 Smart Extraction	Extract main content, skip navigation/ads
🔍 URL Filtering	Include/exclude specific URL patterns
📊 Progress Tracking	Real-time crawl status monitoring
📝 Clean Output	Markdown-formatted, LLM-optimized content
🏠 Self-Hosted	Use cloud service or your own instance

Setup

Prerequisites

Before using this plugin, you need:

A Watercrawl API key (for cloud service) or self-hosted Watercrawl instance
Target URLs ready for crawling
Understanding of your crawling requirements (depth, limits, patterns)

☁️ Cloud Setup (Recommended)

# 1. Get your API key from Watercrawl dashboard
https://app.watercrawl.dev/dashboard/api-keys

# 2. Configure in Dify
Base URL: https://api.watercrawl.dev  # (or leave empty)
API Key: wc_your_api_key_here

🏠 Self-Hosted Setup

# 1. Deploy your Watercrawl instance
# Follow: https://docs.watercrawl.dev/self-hosted/overview

# 2. Configure in Dify  
Base URL: https://your-watercrawl-instance.com
API Key: any-value  # (required but can be arbitrary)

📖 Usage Examples

📄 Single Page Extraction

Extract content from one specific page:

Start URL: https://example.com/article
Maximum crawl depth: 1
Maximum pages to crawl: 1
Only main content: true

🌐 Full Website Crawl

Crawl an entire website systematically:

Start URL: https://example.com
Maximum crawl depth: 2
Maximum pages to crawl: 50
URL patterns to include: # (optional)
URL patterns to exclude: admin/*, login/*

🎯 Targeted Section Crawl

Focus on specific website sections:

Start URL: https://docs.example.com
Maximum crawl depth: 3
Maximum pages to crawl: 100
URL patterns to include: api/*, guides/*, tutorials/*
URL patterns to exclude: archive/*, changelog/*

⚙️ Configuration Parameters

Parameter	Type	Default	Description
Start URL	`string`	required	Base URL to begin crawling
Ignore rendering	`boolean`	`false`	Skip page rendering for faster crawling
URL patterns to exclude	`string`	-	Comma-separated exclude patterns: `blog/, about/`
URL patterns to include	`string`	-	Comma-separated include patterns: `docs/, api/`
Maximum crawl depth	`number`	`1`	How deep to crawl (1 = start URL only)
Maximum pages to crawl	`number`	`1`	Total page limit for the crawl job
Only main content	`boolean`	`false`	Extract main content, skip nav/footer
Proxy Server Slug	`string`	-	Proxy server identifier (optional)

🎯 Understanding Crawl Depth

Depth 1: [Start URL] → Direct links
Depth 2: [Start URL] → Direct links → Links from those pages
Depth 3: [Start URL] → Direct links → 2nd level → 3rd level pages

🔍 URL Pattern Examples

Include Only Specific Sections:

blog/*, docs/api/*, products/*/specs

Exclude Unwanted Areas:

admin/*, login/*, tag/*, category/*

📤 Output Format

Each crawled page returns structured data:

{
  "source_url": "https://example.com/page",
  "title": "Page Title", 
  "description": "Meta description",
  "content": "# Clean Markdown Content\n\nParagraph text..."
}

🔄 How It Works

Job Creation → Submit crawl request to Watercrawl API
Processing → Watercrawl crawls pages based on your parameters
Monitoring → Plugin polls job status every 5 seconds
Extraction → Content is cleaned and formatted as Markdown
Delivery → Structured results returned to Dify

🎯 Common Use Cases

📚 Documentation Indexing

Perfect for creating AI-powered knowledge bases:

Start URL: https://docs.example.com
Maximum crawl depth: 3
Maximum pages to crawl: 200
URL patterns to include: api/*, guides/*, tutorials/*
URL patterns to exclude: changelog/*, archive/*
Only main content: true

📝 Content Analysis

Extract blog posts and articles for analysis:

Start URL: https://blog.example.com
Maximum crawl depth: 2
Maximum pages to crawl: 100
URL patterns to include: posts/*, articles/*
URL patterns to exclude: tag/*, author/*, comments/*
Only main content: true

🛍️ E-commerce Data

Gather product information systematically:

Start URL: https://shop.example.com/products
Maximum crawl depth: 2
Maximum pages to crawl: 50
URL patterns to include: products/*
URL patterns to exclude: cart/*, checkout/*, account/*
Only main content: true

🔍 Competitor Research

Monitor competitor websites for changes:

Start URL: https://competitor.com
Maximum crawl depth: 2
URL patterns to include: products/*, pricing/*, features/*
URL patterns to exclude: blog/*, news/*, support/*

💡 Best Practices

🧪 Start Small: Test with low limits first (depth: 1, limit: 5)
🎯 Use Filters: Focus crawling with include/exclude patterns
🤖 Respect Robots: Check robots.txt before large crawls
⚡ Fast Mode: Use ignore_rendering: true for speed
📄 Main Content: Enable for cleaner, AI-ready content
📊 Monitor: Watch crawl progress for large jobs
🔄 Iterate: Refine patterns based on results

⚡ Performance Notes

Scenario	Typical Time	Recommendation
Single page	< 5 seconds	Use for quick extractions
Small site (< 20 pages)	30-60 seconds	Good for testing patterns
Medium site (50-100 pages)	2-5 minutes	Monitor progress
Large site (> 200 pages)	5-15 minutes	Use during off-hours

💡 Tip: Deep crawling (depth > 3) can exponentially increase page count. Use cautiously!

🔧 Troubleshooting

❌ Authentication Errors

"API key is required" error:

✅ Verify API key in Dify plugin settings
✅ Check Base URL is correct (https://api.watercrawl.dev)
✅ Ensure API key starts with wc-

🚫 Crawl Failures

"Failed to crawl" error:

✅ Test URL accessibility in browser
✅ Check if site blocks bots (robots.txt)
✅ Verify API key is valid and has credits
✅ Try with ignore_rendering: true

📉 Incomplete Results

Missing expected pages:

✅ Verify URL patterns with pattern tester
✅ Check crawl depth covers target pages
✅ Increase page limit if needed
✅ Review site's internal linking structure

🐌 Performance Issues

Crawling too slow:

✅ Enable ignore_rendering for 3x speed boost
✅ Use more specific URL patterns
✅ Reduce crawl depth and page limits
✅ Check your internet connection

📊 Rate Limits & Usage

☁️ Watercrawl Cloud

Free Plan: 1000 pages/month
Startup Plan: 10,000 pages/month
Pro Plan: 30,000 pages/month
Enterprise: Custom limits
Check pricing for more details
Monitor usage: Dashboard

🏠 Self-Hosted

No external rate limits
Performance depends on your server resources

🔒 Security & Privacy

🔐 Secure: All API calls use HTTPS encryption
🔑 API Keys: Store securely, never in client-side code
🏠 Self-Hosted: Full data control for sensitive content
📋 Review: Always check crawled content before use

📞 Support & Resources

🆘 Get Help

💬 Plugin Issues: support@watercrawl.dev
📖 Documentation: docs.watercrawl.dev
🐛 Bug Reports: GitHub Issues

📚 Learn More

Made with ❤️ by the Watercrawl Team

Website • Documentation • Dashboard

Capabilities

Data sources

Available inside your emploidai workspace after installation.