fix: Respect robots.txt by default in website-content-crawler

[jirispilka]

Why

Closes #1574.

apify/website-content-crawler declares respectRobotsTxtFile with default: false (the Console prefills true, which API runs don't get). A tool call that omits the field crawls without consulting robots.txt. The Actor's default stays as is; this server sets it.

What changed

Tool contract change: the WCC input schema now serves default: true for respectRobotsTxtFile.

• The apify--website-content-crawler tool schema and fetch-actor-details for WCC show default: true.
• The direct tool, call-actor and call-actor-widget send respectRobotsTxtFile: true when the caller omits it. An explicit false or true is kept. AJV here does not apply schema defaults, and the platform would fill the Actor's own false, so the value is set when the run starts.
• call-actor is included because it prints the same tool schema in its errors and points to fetch-actor-details for the schema; all of them now agree.
• Other Actors are unchanged. Runs started by schedules or saved tasks still get the Actor's own default.

We don't like this. A per-Actor default table (SERVED_INPUT_DEFAULTS) that the schema step and the run-start step both read is a special case in server code for one Actor's input. We don't see a better way: the Actor's default can't change, AJV here doesn't fill defaults, and turning that on (useDefaults) would make every Actor's defaults explicit in stored INPUT, which is a wider change than this fix. If a cleaner hook turns up, the table is the only thing to move.

Notes for reviewers (human-written)

Proof it works

mcpc against a local stdio build (0.18.3 + this branch).

tools-get apify--website-content-crawler, respectRobotsTxtFile (before: "default": false):

{
  "title": "Respect the robots.txt file",
  "description": "If enabled, the crawler will consult the robots.txt file for the target website before crawling each page. At the moment, the crawler does not use any specific user agent identifier. The crawl-delay directive is also not supported yet.\nExample values: true",
  "type": "boolean",
  "default": true,
  "prefill": true,
  "examples": [true]
}

fetch-actor-details for apify/website-content-crawler shows the same "default": true.

One crawl with the Claude connector tool set, field omitted: apify--website-content-crawler with {"startUrls":[{"url":"https://example.com"}],"maxCrawlPages":1,"crawlerType":"cheerio"} → run gBo139MGZuHIHFUrY, SUCCEEDED, 1 page. Its stored INPUT (apify key-value-stores get-value NzdUXEc9BxZobpfKg INPUT) has "respectRobotsTxtFile": true. That run used an earlier revision of this branch, where only the direct tool filled the value; the direct-tool path is the same in this revision. The call-actor and widget paths are covered by unit tests, including a call by Actor ID.

The new tests failed on master.

AI-assisted: Claude Code wrote the change, tests and this description, and ran the mcpc probes and the crawl.

🤖 Generated with Claude Code

https://claude.ai/code/session_01CPzYK17FoL2F3cuVonqnej

---

Generated by Claude Code

[jirispilka]

Closed in favor of #1585, which combines this change and the input restrictions from #1580 in one declarative direct-tool contract. This keeps the served schema, validation, run input, and annotations consistent.

The agreed scope is the direct WCC tool: respectRobotsTxtFile defaults to true, and an explicit false is preserved. Generic Actor calls, widgets, and Actor details retain the original Actor input contract.

#1586 is a separate PR stacked on #1585 that moves the existing RAG Web Browser and Web Fetch presentation rules into the same registry without changing their behavior.