My agent works in the Agent Builder but not in the Web Console
  • Updated on 01 May 2020
  • 5 minutes to read
  • Contributors
  • Print
  • Share
  • Dark
    Light

My agent works in the Agent Builder but not in the Web Console

  • Print
  • Share
  • Dark
    Light

This article is a compilation of common solutions for agents that work while testing in the Agent Builder, but return an error in the Web Console. These solutions cover a wide range of issues, but we understand that real cases vary. In case the solutions below don't resolve your problem, Mozenda Support is here to help!

Adjust the Error Handling Settings
This setting is useful if the agent in question is collecting blank data and no error is identified. By default, agents are set to attempt to ignore errors. This means that the system will ignore each error type a certain number of times. The system will pause the agent and show an error status only after a particular error is consecutively encountered by the same action too many times.
You can set the agent to stop immediately when it encounters an error, this gives you a chance to fix and rerun your agent.

  1. Open the agent in the web console.
  2. Click the Gear icon and click Harvesting.
  3. Click the Error Handling tab.
  4. Open When an agent error occurs drop-down menu.
  5. Click Stop the job so the agent can be fixed.
  6. Open When a website error occurs drop-down menu.
  7. Click Stop the job so the agent can be fixed.
  8. Click Save.

Adjust advanced settings
The Mozenda harvesters attempt to run every agent as quickly and efficiently as possible. By default, multiple items from the same list are processed at the same time. For more dynamic websites, this may cause unpredictable behavior.

You can make the agent more stable by setting it to process one list item at a time.

  1. Open the agent in the web console.
  2. Click the Gear icon and click Harvesting.
  3. Click the Advanced tab.
  4. Check the Do not process multiple list items concurrently box.
  5. Check the Maximize the browser windows during processing box.

Slow the Agent Down
An agent set up in the Agent Builder runs on your local computer, while an agent run in the Web Console runs on Mozenda's harvesting servers. These servers are more powerful than a standard desktop, and will generally perform actions faster than a personal computer. This can lead to the agent looking for data before the target objects have loaded on the web page.
Adding wait seconds to an agent allows the page to finish loading before the agent attempts to perform actions. It’s a good idea to do this for each page in the agent.

  1. Click the Open an existing agent folder next to Files in the agent builder.
  2. Select the agent.
  3. Right-Click on the top of the page next to Page number.
  4. Click Properties.
  5. Click AFTER LOADING THIS PAGE.
  6. In Wait up to 15 seconds for this page to load, increase the time from 15 to 30.
Note

This does not mean the agent will wait 30 seconds each time, but rather that the agent will wait a maximum of 30 seconds for the page to load.

  1. In Wait 0 seconds before performing the first action on this page, increase the time from 0 to 5.
  2. Check the box next to Wait for AJAX to alter this page when it is done loading.
  3. Click Save.

Pages That Load With AJAX
Some websites use AJAX and other dynamic-content-loading methods to change the page content instead of loading a new page. When scraping these sites, it may be necessary to change the navigation style of the agent so that the agent waits for AJAX instead of loading a new page when a link is clicked. The result is an agent where most or all of the actions occur on the same page.

  1. Right-click the Click Item or Set User Input action which navigates from page 1 to page 2.
  2. Click Properties.
  3. Click AFTER PERFORMING ACTION.
  4. Check the Wait for AJAX to alter the current page box.
  5. Click Save.
  6. Copy the actions from Page 2 and paste them onto Page 1 directly after the Click Item or Set User Input action.
  7. Delete Page 2.
  8. Repeat this process for all the click item actions in the agent until all the actions appear on the same page of actions in the order in which they will be performed.

Order of Actions
Mozenda executes these actions in the order they appear in the Action List. Here are a few things to look for:

  • Ensure that you don’t have two Click Item actions on the same page of actions because they go to different web pages.
  • This may not cause issues in the Agent Builder, but because of processing limitations, this will produce an error in the Web Console.
  • Verify that any Click Item actions are always the last action listed in each page of the Action List.
  • The only actions that should appear after a Click Item that navigates to a new page is an End List or Page List action.
  • Click Item actions that wait for AJAX or don't navigate to a new page, do not need to come at the end of the Action List.

Collection and Field Settings
If the agent has unique fields or contributes to a combined collection that has unique fields, some data may be dropped or ignored. Here are some things to look out for:

  1. Avoiding using a unique field when the field does not contain unique data.
  2. If a unique field encounters a record with data it has encountered before, it will drop the entire record.
  3. If an agent is part of a combined collection, confirm that the agent collection and the combined collection have the same unique fields.

Site Preferences
The servers that Mozenda uses to run an agent are located in the United States. If you are from a location outside the US and the target website changes its content based on location, this may cause problems for your agent. Some websites provide the option to manually select the location, which should be built into the agent when available. Other websites include the location in the URL, which you can modify using the Load Page action.

Was this article helpful?