> ## Documentation Index
> Fetch the complete documentation index at: https://docs.convocore.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# URL scraper

URL scraping is a powerful tool for automatically extracting data from websites to populate your knowledge base.

<CardGroup cols={2}>
  <Card title="" icon="check">
    * Automatically extracts data from websites.
  </Card>

  <Card title="" icon="check">
    * Ensures relevant and updated content.
  </Card>
</CardGroup>

<Steps>
  <Step title="Select 'URL' in the Add Data Source Window">
    Open the `add data source` window and choose the "URL" option.
  </Step>

  <Step title="Enter the URL or Sitemap">
    Input the URL or sitemap you want to scrape. Press `fetch` to retrieve and review the sitemap from the XML file.
  </Step>

  <Step title="Scrape the Data">
    Once the URLs are fetched, scrape the data and save it as individual documents. Scroll through and remove any unwanted information to ensure relevance.
  </Step>
</Steps>

<img class="rounded-md" src="https://mintcdn.com/convocore/45Ecjxi6tZs54Ia5/images/add_URL_shots.png?fit=max&auto=format&n=45Ecjxi6tZs54Ia5&q=85&s=0a8dca477b5e9b268ffadf417cb1b738" width="500" data-path="images/add_URL_shots.png" />

<Warning>
  Using the URL scraper will use X credits per page. Please ensure your account has sufficient credits.
</Warning>

<Tip>
  Regularly check the extracted data for relevance and accuracy to maintain a high-quality knowledge base.
</Tip>