Skip to main content
Sources change shape: a county adds a column, drops one or changes a type. A schema is the list of a dataset’s columns and their types, plus the columns that identify a record (its key). Data Feeds gives each dataset a schema version, so your pipeline can tell a safe change from one that needs a person.

Format

A schema version is MAJOR.MINOR, for example 2.1. A schema change does not replace record ids. A record matched to the version before keeps its _ploti_record_id across MINOR and MAJOR changes alike. After a MAJOR change fewer records may match, and those that do not get new ids (see keys and schema changes). How each arrives:
  • A MINOR change comes with a version that has changes to apply, compared over the columns both versions have. A changed row carries the new column. Records that did not change keep the value your table gives a new column, usually null, until you reload. Reloading from that version’s snapshot fills the new column for every row, and is optional.
  • A MAJOR change comes with a version that has resyncRequired. Its snapshot carries the new shape for every row, so you reload that dataset from it instead of applying changes.
A new dataset version with the same columns keeps the same schema version. A column renamed at the source is a column removed and a column added, so it is a MAJOR change. Each version’s schema change is measured from the schema of the last version released before it. So a change that a held or discarded version brought is still a change when it next arrives. Schema version numbers only increase along a dataset’s released versions, and a number is never reused for a different shape. When a held or discarded version already took a number for another shape, the new shape takes the next free one.

Holds

Ploti builds some versions but holds them for review instead of sending them:
  • a breaking schema change: a column removed or retyped, or the key changed from one set of columns to another or dropped (a MAJOR bump)
  • key_violation: the dataset’s key has duplicates or nulls in this version
  • a version with fewer than 70% of the last released version’s rows
  • identity_ambiguous: most of the last released version’s records matched no record in this version while the row count barely moved, so record ids would turn over wholesale (see how records are matched)
  • identity_unresolved: Ploti could not decide which of this version’s records continue a previous record, so it holds the version for a person to review instead of giving those records new ids
While a version is held:
  • Its changes, row count and schema version are measured from the last released version, not from other held ones.
  • It is not sent, and neither is anything after it, until Ploti’s team release or discard it. Its subscribers get feed.dataset.delayed once, naming the version that stays current.
  • Staff can release any held version except a key_violation, which can only be discarded.
  • A newer version of the dataset replaces a held one, so a source that is fixed unblocks the feed.
  • It never appears in the versions API. One discarded or replaced leaves a gap in the version numbers.

Types

Reading a schema

In columns, items, precision and geometry appear only on the types that carry them. In changeFromPrevious, every added column and every retyped entry’s from and to carry all three, null where the type has none. Pass ?version= for the schema of an older dataset version. With no released version yet, the answer is 404. GET /v1/feeds/datasets/{dataset}/schemas lists every schema version the dataset’s released versions have had, oldest first.

When it changes

A schema change publishes with the dataset version that brings it, as two events in order:
  1. feed.dataset.schema_changed
  2. That version’s feed.dataset.updated for a MINOR change, or feed.dataset.resync_required for a MAJOR change or when the version needs a reload for another reason
feed.dataset.schema_changed does not change the version you hold. Change your table to match, then apply the updated or load the snapshot when the resync_required arrives. data.schemaChange on the event is that version’s own change. A schema’s changeFromPrevious describes the change the schema brought when it first appeared. A common rule:
  • On a MINOR change, add the column and apply the changes.
  • On a MAJOR change, hold the dataset at its current version until someone on your side has reviewed it.
Snapshots and changes files are kept 90 days from when each version was built, so a long hold on your side ends with a reload from the current snapshot.