Best practices for single accounts & departments

Best practice	Related resources and tools to ensure Compliance
Overall slow account performance (across the web console, agent builder, and/or API)
Use less than 500 GB of storage	- Use the storage dashboard to find out the collections using the most storage and what type of storage (file packages, data, files, etc.) Optimize the biggest storage consumers. - Clear collections after they're exported. - Delete unused agents, collections, and sequences. - Delete File Packages after download. Do not create file packages automatically if receiving downloaded files via publishing.
Use fewer than 2000 agents	- For enterprise users, split into multiple departments. - Determine if certain agents can be combined (similar website structure, same website, etc.) - If available, schedule a training to help optimize your agents.
Have fewer than 10 concurrent users signed in at a time (Concurrent usage includes the web console, agent builder, and the API).	- Split users into different departments. - Have users schedule time so that more than 10 do not overlap in a department (or single account) at one time.
Run fewer than 3000 active jobs (this includes jobs that are queued) at a time	- Limit the number of concurrent jobs on agents (for a single agent it should never exceed 1000). - Schedule agents to run at different times. - If you are unsure about how many active jobs are in your account or department, please contact support.
Slow account performance and/or data loss
Allow no more than a 2 MB/sec transfer rate for any specific agent (There are two methods to measure this: - Look at Items.Queued JobStatistic. If this value stays high for extended periods of time, it is likely not able to save all the data that is captured by the harvesting jobs in a timely manner. - Download the data from an agent. Divide the size by the Duration fields that is stored on the Job(s). This gives you an estimate for the transfer rate).	- Add wait time to an agent. - Do not capture full page HTML. - Run fewer concurrent jobs on the agent.
Allow no more than a 50 MB/sec transfer rate across all jobs running	- Follow the same steps above across your account. - Limit the number of concurrent jobs running across the accounts. - Schedule agents to run at different times. - If possible, divide the workload into multiple departments.
Slow agent/collection
Include fewer than 50 fields per agent/collection	- Create a second agent capturing additional fields. - Prioritize the fields that matter most and only capture those fields. - Capture multiple fields in one capture, e.g. capture a full table instead of each row of the table.
Retain fewer than 50 million raw rows in a collection	- Clear a collection before adding additional rows. - If storing item history, export all items and use publishing settings to export the most recent captures individually (instead of relying on Mozenda's storage). - If an agent is running and collecting beyond 50 million rows, consider pausing the agent, exporting the data captured thusfar, clearing the collection, and then resuming.
Failing or slow API Calls
Avoid making more than 30 API calls per minute	- Find bulk method to accomplish task. - Include wait time between calls.
Avoid repetitive single API calls that can be made in bulk	- Find bulk method of API call.
Slowness in data retrieval and decreased performance with views
Avoid sorting parameters on large views	- On views with over a million rows, sort after exporting the data.
Avoid “Contains” criteria on columns with large amounts of data	- Create filtered view with exported data. - Use RegEx in your agent to mark whether or not a row of data matches a "contains criteria" (e.g. if I wanted all rows that had screen size mentioned in a large table capture, I could use this RegEx to search for it: `.(Screen Size).`). Then apply filter criteria to that row instead of the large text block.
Slow agent groups or combined collections
Include fewer than 1000 agents in an agent group	- Create a new agent group with the same field and move some agents to this new group. - Optimize and combine agents where possible.
Use fewer than 50 agents/collections to source a combined collection	- Combine more complex collections outside of Mozenda (e.g. joining the data in SQL). - Optimize and combine agents where possible.

Best practice

Related resources and tools to ensure Compliance

Overall slow account performance (across the web console, agent builder, and/or API)

Use less than 500 GB of storage

- Use the storage dashboard to find out the collections using the most storage and what type of storage (file packages, data, files, etc.) Optimize the biggest storage consumers.
- Clear collections after they're exported.
- Delete unused agents, collections, and sequences.
- Delete File Packages after download. Do not create file packages automatically if receiving downloaded files via publishing.

Use fewer than 2000 agents

- For enterprise users, split into multiple departments.
- Determine if certain agents can be combined (similar website structure, same website, etc.)
- If available, schedule a training to help optimize your agents.

Have fewer than 10 concurrent users signed in at a time (Concurrent usage includes the web console, agent builder, and the API).

- Split users into different departments.
- Have users schedule time so that more than 10 do not overlap in a department (or single account) at one time.

Run fewer than 3000 active jobs (this includes jobs that are queued) at a time

- Limit the number of concurrent jobs on agents (for a single agent it should never exceed 1000).
- Schedule agents to run at different times.
- If you are unsure about how many active jobs are in your account or department, please contact support.

Slow account performance and/or data loss

Allow no more than a 2 MB/sec transfer rate for any specific agent
(There are two methods to measure this:
- Look at Items.Queued JobStatistic. If this value stays high for extended periods of time, it is likely not able to save all the data that is captured by the harvesting jobs in a timely manner.
- Download the data from an agent. Divide the size by the Duration fields that is stored on the Job(s). This gives you an estimate for the transfer rate).

- Add wait time to an agent.
- Do not capture full page HTML.
- Run fewer concurrent jobs on the agent.

Allow no more than a 50 MB/sec transfer rate across all jobs running

- Follow the same steps above across your account.
- Limit the number of concurrent jobs running across the accounts.
- Schedule agents to run at different times.
- If possible, divide the workload into multiple departments.

Slow agent/collection

Include fewer than 50 fields per agent/collection

- Create a second agent capturing additional fields.
- Prioritize the fields that matter most and only capture those fields.
- Capture multiple fields in one capture, e.g. capture a full table instead of each row of the table.

Retain fewer than 50 million raw rows in a collection

- Clear a collection before adding additional rows.
- If storing item history, export all items and use publishing settings to export the most recent captures individually (instead of relying on Mozenda's storage).
- If an agent is running and collecting beyond 50 million rows, consider pausing the agent, exporting the data captured thusfar, clearing the collection, and then resuming.

Failing or slow API Calls

Avoid making more than 30 API calls per minute

- Find bulk method to accomplish task.
- Include wait time between calls.

Avoid repetitive single API calls that can be made in bulk

- Find bulk method of API call.

Slowness in data retrieval and decreased performance with views

Avoid sorting parameters on large views

- On views with over a million rows, sort after exporting the data.

Avoid “Contains” criteria on columns with large amounts of data

- Create filtered view with exported data.
- Use RegEx in your agent to mark whether or not a row of data matches a "contains criteria" (e.g. if I wanted all rows that had screen size mentioned in a large table capture, I could use this RegEx to search for it: .*(Screen Size).*). Then apply filter criteria to that row instead of the large text block.

Slow agent groups or combined collections

Include fewer than 1000 agents in an agent group

- Create a new agent group with the same field and move some agents to this new group.
- Optimize and combine agents where possible.

Use fewer than 50 agents/collections to source a combined collection

- Combine more complex collections outside of Mozenda (e.g. joining the data in SQL).
- Optimize and combine agents where possible.

A department functions similar to how a single account functions. These tips are for single accounts or departments. Organizations that use multiple departments only need apply these tips within a single department.

Best practices for single accounts & departments

What's Next