Skip to main content
Profound raises a $180M in Series D to build your AI Marketer. See it live
Agent Templates
Research

Robots.txt & LLMs.txt Auditor

Enter a domain and the Agent fetches its robots.txt and llms.txt, parses every user-agent rule, and classifies access for 25+ known AI and search crawlers so you know whether a site is actually set up to be discovered and cited by AI answer engines.

Try Agent

Agent node overview

  1. Execute Code

    Write and execute custom Python or JavaScript code to manipulate inputs, format outputs, or build logic specific to your use case.

  2. Web Page Scrape

    Scrape the content from a single webpage.

  3. LLM

    Allows you to use an LLM to generate a response