Design high performing agents

The request blocker autoblock feature allows you to customize your Navigation Auto-Request Blocking in four different levels for faster and more automated scraping usability.
The request blocker autoblock features can be accessed from the Web Console and the Agent Builder. Follow the directions below for each:

From the Web Console

1. Open the Web Console.
2. Click on Agents in the top navigation bar.
3. Click on the agent you’d like to apply request blocking to.
4. Click on the Agent Settings.
5. In the drop-down menu under Settings, click Harvesting.
6. In the Harvesting Settings pop-up window, click Advanced.
7. Under Auto-Block – Automatically block navigation requests choose which level of Request Blocking you’d like to use for your specific Agent. Here, only Levels 1 & 2 are available to avoid scenarios where agents might be ruined by immediately setting them to Level 3 or 4. To access all 4 levels, follow the directions below from the Agent Builder.
8. Click Save.

From the Agent Builder

1. Open the Agent Builder.
2. Click Start New Agent or Open an Existing Agent.
3. Under the File drop-down menu, click on Settings.
4. In the Agent Builder Settings pop-up, under Request Blocking, select the Auto-Block level you’d like to apply.
5. Click Save.

It is important to note that when using request blocking in the Agent Builder you will not be able to concurrently turn on multiple levels of request blocking. To add each level properly you will want to select that level and then save and test. This will allow the agent to process that you turned on this level and it will add the request blocking in the navigation requests.

Who needs request blocker autoblock features?

Anyone who needs to:

• Harvest data from websites with heavy content requests
• Run agents faster
• Cut time from manually blocking requests
• Block CSS content
• Block event-driven content

What is included in the request blocking autoblock feature set?

Mozenda includes 4 different levels of Auto Request Blocking.

• Level 1 - Standard - This will block requests that involve ads, analytics, etc. This level rarely has negative results and is recommended for most agents.
• Level 2 - Third Party - Any request that comes from a different root domain will be blocked.
• Level 3 - Event-Driven - This level doesn't look at the domain or extension of this request, but rather, it looks at when the request occurs. This will stop all requests after the BrowserReadyState is changed to interactive.
• Level 4 - One Request - Blocks all but the document.

The quickest way to get from point A to point B is in a straight line. The same principle applies to web data extraction. If you can just extract the data you seek, without loading all the unnecessary extras like advertisements, images, media files, analytics, CSS, etc. then it makes sense that your web scraping Agents will extract data at a much faster pace.