Storage FAQ
  • 25 May 2021
  • 4 Minutes to read
  • Contributors
  • Dark
    Light
  • PDF

Storage FAQ

  • Dark
    Light
  • PDF

Article Summary

Extraction is just one step of the data journey. Our online storage offers a waystation to help your vital data get where it's going. Practice exporting, publishing, and connecting to data sources so it can drive your big decisions.

What is a collection?
Collections are table where an agent's data is written and stored. Data in collections can be collected by agents or manually uploaded by the user. Each line or row in a collection is referred to as an item.

Where can I find storage?
An overview of your storage is found on the navigation bar of your Web Console. Select to be brought to your peak storage statistics.
image.png

How is my storage calculated?
Our system records your used storage once a day. Your monthly peak storage equals the highest daily storage on record. For example, if your account storage hits 112 GB on February 14th, your peak storage for the month of February is 112 GB.

Can I change the time frame of the storage statistics?
See a breakdown by day, week, month and quarter, by selecting the drop-down in the far right-hand corner of the screen.

StorageFAQ_Image2

How do I preview and download my statistics?
Preview statistics by selecting an item in the list. Select to download statistic for the selected item .

StorageFAQ_Image4

Statistics fields on storage:

  • All metrics are displayed in megabytes.
  • Database refers to text data.
  • AgentDefinitions refers to the XML file of the agent.
  • Files and Packages refers to any files that have been downloaded during harvesting.

What makes up my storage?

  • Items: The rows within a collection of data - either uploaded by the user to an input collection or collected during harvesting within an agent collection.
Clearing Items

If store item history is enabled and you select items to be deleted, both items and bookmarks will be deleted.

  • Job History: Jobs ran on the agent or collection within the last year.
  • Images and Files: Also referred to as Web files. Long term storage of files downloaded during harvesting including images.
Clearing Images and Files

Clearing Images and Files will delete the files for the agent permanently, which will prevent repackaging until the agent has been run again.

  • File Packages: Short term compressed storage of downloaded images and files used for exporting and publishing. If cleared, this storage can be brought back by repackaging files in the collection.

  • Item history and bookmarks: This is available when store item history is enabled. When item history is enabled, all data and files are kept from previous runs in organization units called "bookmarks". Bookmarks are one complete gathering of the data, i.e. if an agent has been run and has multiple jobs, all data will be stored in a single bookmark. Clearing item history and bookmarks deletes all history and items for the collection.

How is storage billed?
$10 per GB will be charged if your monthly peak storage exceeds your account limit. Any overages accumulated the months before will be added to the next month's invoice. Billing will begin February 1st, 2020.

How do I clear storage?
Select Collections from the navigation bar of the web console. This will bring you to a list of collections in your account.
Select a specific collection > Click on More > Clear collection data.

StorageFAQ_Image5

Difference between File Packages and Images and Files?

Clearing Images and Files will permanently delete the files and images while clearing File Packages will free up temporary storage until the agent has been run again and or the files are repackaged.

How do I download my files?
To download the files and images collected, go to your agent or collection> select > Export > Select the Images & Files tab > Click Package Files to submit your request to repackage any images or files > Select Download.

If you are republishing files then the files and images will be packaged as part of the publishing process.

Repacking images or files

Repacking files may take some time depending on the size and quanity of your images and files. Allow time for the request to be processed, refreshing the page maybe needed.

image.png

Are my files packaged and stored even when I publish the data somewhere else?
When the data is being published, the files/images are packaged and sent to the publishing source. There is a setting that controls if the files packages are stored in the Mozenda Cloud. This setting is found under Harvesting settings in Behavior. Set this on an agent basis or an account basis, for all agents moving forward.

StorageFAQ_Image8

Single Agent Setting Verification:
Go to the agent > select > Harvesting Settings > click Behavior > select Do not create image/file packages.

Global Settings:
From the Summary Page, select Account Settings > Change Harvesting Setting defaults, then check “Do not create image/file packages”.

From that point forward any agent built with file/images will not be packaged.

The only reason why you would need to package files is if you are downloading the files manually from the web console. We recommend that this is selected for all agents.

StorageFAQ_Image9

In sequences, can I set a step to delete my collections?
Yes, depending on your licensing, a step can be added to sequences, or API, to automatically clear a collection. We recommend setting a step to publish the data and clear the collection afterward. Sequences are available for enterprise licenses.

How can I automatically clear my storage?
The best way to automatically clear storage and collections is to use the API, implement a sequence, or clear manually. Access to the API is available for all enterprise licenses and professional subscriptions. Sequences are available for enterprise licenses.


Was this article helpful?