Webhook Functionality

Prev Next

New Webhook Functionality

Overview

This feature introduces a front-end interface for setting up webhooks (previously available only via REST API calls). It allows users to receive push notifications via HTTP requests when specific events occur in the job pipeline, eliminating the need for constant status polling via API.

How to Configure

To access the new settings:

  1. Navigate to the Harvesting Settings for an agent.

  2. Select the new Web Hooks tab.

  3. Enter your callback URLs in the appropriate fields.

Types of Webhooks

There are three specific triggers available:

Job Status URL Callback: Fires when a web page harvest job changes state (e.g., Running, Done, Paused, Error). This triggers every individual job created by the agent, including concurrent jobs.

Report Refresh Status URL Callback: Fires when the "report refresh" job is complete. This indicates that the bookmark is closed, unique fields are executed, and the data is ready to be viewed or processed.

Combined Collection Status URL Callback: Fires when a downstream combined collection has been successfully rebuilt with the agent's data. This notifies external systems that the combined collection is in a static, complete state, and ready for processing.

URL Configuration & Token Replacements The webhooks make HTTP requests to the URL you define. You can include dynamic information (token replacements) in the URL to pass metadata to your system.

Standard Tokens: It is recommended to always include Job.Status and JobID.

Other Available Tokens: AgentID, CollectionID, job duration, and any custom fields defined for the agent.

Example Format: https://mozenda.mydomain.com?AgentID=%AgentID%&JobID=%JobID%&Status=%Job.Status

Key Behaviors & Rules

• These requests are run on background threads. The system does not wait for or process the response from your server; if your server fails to respond, the system proceeds to avoid instability.

• Settings can be defined at the Account Default level or synced from Agent Group Harvesting Settings.

• If an Agent.Run API call includes specific status URL configurations, those API settings will override the settings defined in the UI for that specific job.

• Webhooks will fire regardless of how the agent is started (Schedule, On Demand, Sequence, or API).

Same functionalities through this API in StatusURLs: Agent.Run - Web Scraping API - Mozenda