Knowledge Base - Web Crawler

1. Starting the Crawler

• 1. Navigate to Settings > AI Studio > Knowledge Base.
• 2. Select 'Add URL'.
• [IMAGE PLACEHOLDER]

2. Configuring the Crawl

• 1. Enter your website's main URL.
• 2. Choose whether to crawl just that exact page, or all pages on that domain (up to a limit).
• 3. Click 'Start Crawl'.
• [IMAGE PLACEHOLDER]

3. Reviewing Scraped Data

• 1. Once the crawl is complete, the URLs will appear in your trained data list.
• 2. You can click on any URL to view the exact text the AI extracted.
• 3. Delete any pages (like Privacy Policies) that you don't want the AI using for answers.

Search Tags

• Tags: web crawler, scrape website, train ai urls, knowledge base crawler