---
title: "Crawl a website directory"
slug: "how-can-i-crawl-a-web-site-directory-1"
updated: 2025-03-12T18:25:12Z
published: 2025-03-12T18:25:12Z
canonical: "help.mozenda.com/how-can-i-crawl-a-web-site-directory-1"
---

> ## Documentation Index
> Fetch the complete documentation index at: https://help.mozenda.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Crawl a website directory

From the **Dashboard** or **Projects** page, 
1. Select the **Create New Robot** button.
2. Select **Crawler**.
3. Enter the required information.

![Screenshot 2021-03-30 at 10.29.39.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/Screenshot%202021-03-30%20at%2010.29.39.png){height="" width=""}

4. Under the **Settings** tab, make any desired configuration changes.

* If you wish to use input data to provide the crawler with URLs to visit, activate the **Dynamic URL?** checkbox.
* To follow the rules of any robots.txt files on the site (recommended), activate the **Respect robots.txt?** checkbox.

![1.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/1%286%29.png){height="" width=""}

5. Under the **Output** tab, create any output fields your project requires. These fields will store the output data generated by the crawler.

![2.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/2%286%29.png){height="" width=""}

6. Under the **Page Processors** tab, configure any page processors required by your project. See [Page processors](/v1/docs/what-should-i-know-about-page-processors) for details.

![3.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/3%282%29.png){height="" width=""}

7. When all necessary page processors are configured, select the blue **Save** button in the top-right of the page to save the crawler.

![4.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/4%281%29.png){height="" width=""}

On the **Projects** page, 
8. Select the crawler.
9. Select the **Create Run** button near the top-right of the page.

![Screenshot 2021-03-30 at 10.33.38.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/Screenshot%202021-03-30%20at%2010.33.38.png){height="" width=""}

10. Enter a name for the run.

![Screenshot 2021-03-30 at 10.33.28.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/Screenshot%202021-03-30%20at%2010.33.28.png){height="" width=""}

11. Select the **New Run**.
12. Select **Open** in the slide-in panel.
13. Under the **Configuration** tab, change settings as needed.

![5.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/5.png){height="" width=""}

14. Under the **Integrations** tab, configure any required integrations.

![6.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/6.png){height="" width=""}

15. Under the **Executions** tab, you may launch the execution when ready, or view existing execution information.

![7.png](https://cdn.document360.io/03076343-9561-4857-8a4e-9f3bfed20ced/Images/Documentation/7.png){height="" width=""}