Recursive Web Crawler · emploidai Marketplace
emploidai Marketplace
Add-onsAppletsPlugins
Search tools, teams, and capabilitiesPublish
MarketplacePluginsRecursive Web Crawler
Plugin
Limited listing

Recursive Web Crawler

by fangyong20062006 · v0.0.2

Recursively crawl web pages from a starting URL and depth, returning structured JSON for each page (title, main content as Markdown, links, and metadata). Supports same-domain restriction, max-page limits, and URL include patterns; tolerant of self-signed intranet certificates.

222 installsUpdated Jun 17, 2026
Publisher information is incomplete

This community listing does not yet include every recommended support, privacy, pricing, and permission disclosure. Review the available package permissions before installing.

Capabilities

Tools

Available inside your emploidai workspace after installation.

Data sources

Available inside your emploidai workspace after installation.

Category

tool

Version

0.0.2fangyong20062006

Requirements

Maximum memory 256MB

Pricing

Not disclosed by publisher

Security & access

Review before installing

CompatibleRequires emploidai 1.0.0+

Permissions

  • Uses tool capability

Dependencies

No additional dependencies

Resources

Privacy policy
emploidai Marketplace

Discover capabilities. Review access. Install inside your workspace.

DocumentationSecuritySupportPrivacyTerms

Recursive Web Crawler

Recursively crawl web pages from a starting URL for use in Dify workflows.

Features

  • Crawl from a starting URL with configurable depth
  • Returns structured JSON per page (title, main content as Markdown, links, metadata)
  • Same-domain restriction, max-page limits, and URL include patterns
  • Tolerant of self-signed intranet certificates

Usage

Add the Web Crawler tool to your Dify workflow, provide a starting URL and crawl depth, and receive structured page data.

Network & Privacy

This plugin sends HTTP requests to the URLs you provide and fetches their content. The fetched web pages are processed within your Dify runtime and are not sent to any third-party service by the plugin. You are responsible for ensuring you have permission to crawl the target sites. See PRIVACY.md for details.