Change harvesting error behavior
  • 01 May 2020
  • 2 Minutes To Read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Change harvesting error behavior

  • Print
  • Share
  • Dark
    Light

Mozenda agents can produce errors when a website updates. When this happens, you might need to make small changes to your agent to react to these changes and read new information. To change how your agents react to website updates, you can adjust Harvesting settings.

Ignore errors and continue

You can set up an agent to ignore the error and continue to gather data or set it up to stop at an error. By default, Mozenda will ignore errors and try to keep going.
To change this:

  1. In the Web Console, open an agent.
  2. Click .
  3. Click Harvesting.
  4. Select the Error Handling tab.
  5. Select Attempt to ignore the error and continue.
  6. Select Let the system decide what to do.

Stop on error

If 100% accuracy in your data set is essential or if you are looking for one output per input, adjust your settings to the following:

  1. In the Web Console, open an agent.
  2. Click .
  3. Click Harvesting.
  4. Select the Error Handling tab.
  5. Select Stop the job so the agent can be fixed.
  6. Select Stop the job so the agent can be fixed.

The agent will stop when an error occurs so you can fix it. This ensures that your data set is complete and accurate.

Let the system decide

In most cases you can use the Let the system decide what to do option under When a website error occurs. Mozenda has categorized what website errors to ignore and what website errors to fix so you can achieve the best results.

It is important to notice that these errors are usually caused by changes in website format or development. Websites change all of the time, and it often can be the case that once these changes happen, your agent will run into errors. It may be that the website was updated, a format change occurred, or someone slightly altered the code. When this happens, you may need to make small changes to your agent. It’s not that Mozenda doesn’t work, it’s that Mozenda recognizes the change and can’t continue with the original set-up of the agent. Unfortunately, we cannot control what other developers are doing with their sites, but we do the best that we can to help you be aware of the change before it affects your data set.
If 100% accuracy in your data set is essential or if you are looking for one output per input, adjust your settings to the following:

  1. In the Web Console, open an agent.
  2. Click .
  3. Click Harvesting.
  4. Select the Error Handling tab.
  5. Select Stop the job so the agent can be fixed.
  6. Select Stop the job so the agent can be fixed.

Harvest data from a desktop-sized browser window

Some sites change information locations or show different information when they are re-sized for different screen widths or resolutions (like a typical desktop size that is usually more than 1280 pixels wide vs. mobile size that might be as narrow as 450 pixels).

Modern coding practices suggest you can get to the same information on either your phone or desktop, but that's not always true. To be safe, harvest from a desktop-sized resolution to make sure you're getting the most detailed information.

  1. In the Web Console, open an agent.
  2. Click .
  3. Click Harvesting.
  4. On the Advanced tab, click Harvesting Settings.
  5. Select Maximize the browser windows during processing.

Was This Article Helpful?