Crawl Website


Chatbase offers an API to crawl websites and returns the internal links it found. This works best with server rendered HTML pages.

To do that you need to make a GET request with to with a sourceURL query param. Chatbase will find all the internal links that have the sourceURL as a prefix. For example, if sourceURL is, the API will return ["", "", ""] but not since it doesnt have the sourceURL as a prefix. (basically* )

Example Requests


curl '' \
    --request "GET" \
    -H 'Authorization: Bearer <Your-API-Key>'


const res = await fetch(`/api/fetch-links?sourceURL=${dataSourceUrl}`);
const data = await res.json();

// { fetchedLinks: ["", "", ""], stoppingReason: '' }


{ fetchedLinks: ["", "", ""], stoppingReason: '' }

Stopping reason will sometime not be empty. For example if the site is too big, it will say it is taking more than 60s to find all the links and will return early with the links that it found in 60s.