Reference Glossary
Metadata harvesting
The automated process of collecting metadata records from one or more remote repositories over a defined protocol — most notably OAI-PMH — without transferring the underlying asset files themselves.
Why it matters in a DAM
Harvesting is how a DAM or aggregator builds a combined catalog across multiple source systems without duplicating storage: a museum consortium's shared search portal, for example, harvests metadata records from each member institution's own repository on a schedule and indexes them centrally, while the actual image files stay hosted where they originated. Getting the harvesting contract wrong — not tracking deletions, not respecting update timestamps — means the aggregated catalog quietly drifts out of sync with the source systems it's supposed to mirror.
A worked example
Common mistake
A source system exposes a harvesting endpoint but doesn't reliably flag deleted records as deleted (rather than simply omitting them from the next response) — harvesters that only look for new or changed records never learn an asset was removed, so the aggregated catalog keeps showing records for assets that no longer exist at the source.
The dominant standard for metadata harvesting is OAI-PMH — the Open Archives Initiative Protocol for Metadata Harvesting — which emerged from the 1999 Santa Fe Convention bringing together the e-print and digital library communities, incubated at Cornell University. The first version was released in 2001, and the current specification, version 2.0, was published on 14 June 2002. It defines six HTTP-based verbs (Identify, ListMetadataFormats, ListSets, ListIdentifiers, ListRecords, GetRecord) that let a harvester request records incrementally, by date range or by collection, from a compliant repository.
OAI-PMH is deliberately a low-barrier protocol — it doesn’t move the actual files, only structured metadata records (commonly Dublin Core, though repositories can offer richer formats too), which is what makes it practical for building union catalogs and aggregator search portals across institutions that would never agree to centralize their actual digital collections.
In an enterprise DAM context, the same underlying pattern — periodically pulling metadata updates from an external system of record rather than owning that data directly — shows up in integrations with PIMs, ERPs, and rights-management databases, even when the specific protocol isn’t OAI-PMH. The design questions are the same regardless of protocol: how does the harvester know what changed since the last run, and how does it learn about deletions, not just additions.
Frequently asked
What does metadata harvesting mean in a DAM context?
In a DAM, metadata harvesting is the automated, scheduled collection of metadata records from one or more external repositories — a partner system, contributing archive, or federated source — over a defined protocol such as OAI-PMH. Rather than staff re-keying descriptions asset by asset, the DAM pulls updates on a recurring interval and indexes them centrally, while the underlying asset files themselves stay hosted at their original source system.
What kind of DAM use case relies on harvesting?
A museum consortium's shared search portal is a typical example — it harvests metadata records from each member institution's own repository on a schedule and indexes them centrally, while the actual image files stay hosted where they originated.
What goes wrong when a harvesting contract is set up poorly?
A poorly defined harvesting contract creates several failure modes: field schemas between source and destination don't line up, so titles, rights, or creator data land in the wrong properties or get dropped; records harvested more than once without a stable identifier produce duplicates; and mandatory fields go unmapped, leaving required metadata blank. Combined with unhandled deletions and ignored update timestamps, the aggregated catalog quietly drifts out of sync with the source systems it's meant to mirror.
Why do deleted records cause particular problems for harvesters?
A source system may expose a harvesting endpoint but not reliably flag deleted records as deleted, instead simply omitting them from the next response. Harvesters that only look for new or changed records never learn an asset was removed, so the aggregated catalog keeps showing records for assets that no longer exist at the source.
Does metadata harvesting move the actual asset files?
No. Harvesting moves only metadata records — titles, descriptions, rights statements, identifiers, and similar structured fields — not the binary asset files they describe. The files themselves stay stored at their original source system; the harvesting DAM builds a central index that points back to those sources. This keeps storage costs down and avoids duplicating large media, though it also means broken links appear if a source system later removes or relocates a file.
What protocol is commonly associated with metadata harvesting?
OAI-PMH — the Open Archives Initiative Protocol for Metadata Harvesting — is the dominant standard, tracing back to the 1999 Santa Fe Convention. It defines six HTTP-based verbs (Identify, ListSets, ListMetadataFormats, ListIdentifiers, ListRecords, and GetRecord) that let a harvester request records incrementally by date range or collection, typically returning Dublin Core metadata. Deliberately low-barrier, it transfers only structured metadata, never the underlying files, making it well suited to consortium and aggregator catalogs.
Sources
- OAI-PMH emerged from the 1999 Santa Fe Convention, with the first protocol version released in 2001 and version 2.0 published 14 June 2002 checked 2026-08-07 — Open Archives Initiative, OAI-PMH v2.0