- a snapshot: the whole dataset as of that version
- a changes file: what changed since
previousVersion, unless the version hasresyncRequired - its schema and a manifest with every file’s size and checksum
- an id map, only when the version replaced record ids (
recordIdsChanged) - its attachments: the files Ploti holds for the dataset, such as an ordinance’s PDF, a scanned zoning map or a source’s archived copy
GET /v1/feeds/datasets/{dataset}/versions, GET /v1/feeds/datasets/{dataset}/versions/{version} and the dataset’s current, so pulling versions and following events see the same chain. A version’s publishedAt is when it was released, and the updatedSince filter on GET /v1/feeds/datasets reads the same time. A version held for review never appears.
The version chain
Version numbers increase but can skip. A held version that was discarded or replaced by a newer one leaves a gap, and so does a version rebuilt under a new number. Each version names the version its changes apply to inpreviousVersion. Always follow that chain, not the numbers:
- Apply a version’s changes only when its
previousVersionis the version you hold. - When it is any other version, reload that dataset from its current version: read
current.versionfromGET /v1/feeds/datasets/{dataset}, then load that version’s snapshot.
sourceUrl and fetchedAt can be null. counts is null when resyncRequired is true.
Versions released again
Ploti sometimes releases a version of a dataset again, and rebuilds it under a new number. The old number then answers410 expired, with a message that says so.
If subscribers already had the old version, the dataset’s next version sent is a resync_required, even when a newer release arrives before the rebuild. Its reason is previous_unavailable, with these exceptions:
schema_changedorkey_columns_changedwhen it also changes the schemarecord_ids_changedwhen it also changes ids you hold
current can be the version before the one released again.
That version’s records are matched to the key values and content Ploti kept for the version released again (see how records are matched), and its previousVersion names that version. A record it matches keeps the id you hold. When it kept records but none could be compared, the version is held for review rather than sent. If the version released again replaced record ids and had not been sent yet, its rebuild replaces them again and carries its own id map.
Getting a version’s files
GET /v1/feeds/datasets/{dataset}/versions/{version} returns the version with signed links to its files under files:
snapshotandchanges: GeoParquet parts of about 256 MB at mostidMap: Parquet parts, empty unless the version replaced record idsschema,manifestandattachments
files.expiresAt. Ask again for fresh ones at any time, including part way through a large download. Check each file’s sha256.
Attachments
files.attachments lists the files Ploti holds for the dataset as of the version. Each is named by file, the same name GET /v1/datasets/{dataset}/files/{file} takes:
source:<id>: the archived copy of one source in the dataset’s provenance, with its id fromGET /v1/datasets/{dataset}/provenanceoriginal: the document or map file Ploti downloaded for the datasetdocument:<sha256>: a file stored per row, such as a permit’s PDF. The row’s cell holds this name, so join a row to its file by the cell’s value. Several rows naming one document share one attachment.
sha256 is the checksum of the bytes you download.
- Where Ploti’s record of a file names a checksum, Ploti checked the copy against it before delivering it. A file whose copy did not match, or that Ploti could not copy, is left out of that version.
- The same file in several versions has the same
sha256, so you only need to download it again when the checksum changes. - A dataset with no stored files has no attachments.
- A version’s
manifest.jsonlists them underattachments, without apath.
Fresh links to some files
POST /v1/feeds/datasets/{dataset}/versions/{version}/links signs fresh links to the files you name, without reading the whole version again:
file, and any other file by its path in the version’s manifest.json, up to 200 files at a time. The answer has:
files: each file as{"file", "filename", "contentType", "sizeBytes", "sha256", "url"}, withexpiresAtmissing: the names the version has no file for, or none left past retention
GET /v1/feeds/datasets/{dataset}/versions/{version}:
409not_subscribedfor a dataset none of your subscriptions delivers404not_foundfor a version that was never released410expiredfor one released again and rebuilt under a new number
409 not_subscribed, so once a dataset leaves your subscriptions, or you delete the subscription that delivered it, no new links are issued for it. Links already issued keep working until they expire, up to an hour.
Envelope columns
Every snapshot and changes file carries these columns ahead of the dataset’s own:
Changes files add these columns:
See applying changes.
Because every dataset carries the same envelope, you can load many places’ datasets side by side and ask what changed since Tuesday without knowing any dataset’s columns.
Applying changes
A changes file holds one row per change, with the envelope columns and the record as of this version:
Apply every upsert as insert-or-replace by
_ploti_record_id, in keyed and unkeyed datasets alike. A delete row carries only the envelope columns; its record columns are null.
A changed row also carries:
_ploti_changed_columns: the columns whose values differ from the version before, compared over the columns both versions have. A column added by a MINOR schema change is not compared: the row carries its value, and records that did not change get it when you reload._ploti_match: how the record was matched to the version before,key(the same key value),content(otherwise identical content) oragent(Ploti’s matching agent judged it the same record, edited or renumbered)._ploti_match_score: on anagentmatch, the agent’s confidence, from 0.85 to 1; otherwise null.
_ploti_delete_reason removed_at_source: the record was not matched to any record in this version. Before a record is deleted, Ploti tries to match it:
- A record that no key or identical content matched goes to Ploti’s matching agent when a record of the new version is a plausible partner: it shares values with it, or lies within about 50 m of it. It keeps its id when the agent judges them the same record.
- Otherwise it is deleted. An edited record the agent could not match arrives as a delete of the old copy and an
addedrow for the new one. - At most 200 records a version are judged one by one, chosen among the first 20,000 left over, and none when more than 250,000 are left over. The rest of a large change arrives as deletes and adds.
- A version waits for the agent’s matching before it is released, up to 24 hours for each ask. The agent is asked at most twice, so a version can wait up to 48 hours before it is released or held for review.
updated, resync_required and schema_changed events at or below that version. Only loading a snapshot or applying a changes file changes the version you hold. That makes every change safe to apply twice.
When to reload instead
A version withresyncRequired: true has no changes files, so you reload the dataset from its snapshot. A version is one when:
- it is the dataset’s first version
- it makes a MAJOR schema change: a column removed or retyped, or the key changed or dropped
- the snapshot of the version before it is past retention
- a version subscribers already had was released again
- it replaced record ids
- the version has
resyncRequired: true - you receive
feed.dataset.resync_required - the version you need answers
410: it is past retention, or was released again and rebuilt - an
updatednames apreviousVersionother than the one you hold
When record ids are replaced
Record ids carry across versions, so this is rare. A version that changes ids you hold:- has
resyncRequired: trueandrecordIdsChanged: true - arrives as
feed.dataset.resync_requiredwith reasonrecord_ids_changed - lists, in its
files.idMap,_ploti_previous_record_id,_ploti_record_idand_ploti_dataset_idfor every record carried from the version you hold
Retention
File retention counts from when a version was built, and event retention from when each event was sent. A version can be released some time after it was built (one held for review, for example), so its files can expire before its event does, and a gap shorter than 90 days can still need a reload.
Plan on a version past retention answering
410 expired, unless it is the dataset’s newest, or the last of its MAJOR schema and has no changes to apply. On a 410:
- From a version: reload that dataset from its current version. Read
current.versionfromGET /v1/feeds/datasets/{dataset}, load that version’s snapshot, and continue from there. - From the current version too: its files are being rebuilt. Try again in a few minutes.
- From a cursor whose events are gone: reload from the subscription’s manifest.