Skip to main content
ploti feeds sync downloads every dataset your Data Feeds subscription delivers into a folder, then keeps that folder current. Data Feeds is Ploti’s subscription for keeping a copy of its official county and city records in your own systems. Run sync again, by hand, from cron or when Ploti tells you something changed, and it downloads only the datasets that have a new version. When nothing changed, it changes nothing.
A few words first, since the rest of the page uses them:
  • A subscription is your standing order with Ploti: the places you serve, such as Orange County, Florida, and optionally the topics you want, such as permits or zoning. Its id looks like sub_8f3k2m. Datasets Ploti adds to those places later join it on their own.
  • A dataset is one set of public records, such as a county’s building permits.
  • A version is a numbered release of a dataset. Ploti makes a new one each time the county publishes new or changed records.
  • A snapshot is the whole dataset as of one version, delivered as GeoParquet files: Parquet tables with a geometry column, which DuckDB, GeoPandas, QGIS and most warehouses read directly.
Sync keeps the latest snapshot of each dataset on disk. It does not apply changes record by record; for that, see what sync does not do.

Before you start

Sync needs the Data Feeds permission, which both ploti login and every new API key hold. On your own machine, ploti login is enough. On a server, give the CLI an API key.
1

Create the key

In Ploti, a workspace owner or admin opens Settings, API keys and creates a key. Every new key can read the workspace’s subscriptions.A key belongs to the member who created it and stops working when that member leaves the workspace, so create it from an account that will stay, such as a shared service account.
2

Give it to the CLI

The CLI needs Node.js 20 or later. Run it with npx @ploti/cli, or install it with npm install -g @ploti/cli to get the ploti command.
3

Find your subscription's id

Run it

The first run downloads everything. It prints a line per dataset as it finishes, then a summary:
Later runs download only what moved:
Before it downloads anything, sync checks that the disk has room for the whole run, and stops with nothing changed when it does not.

What ends up in the folder

Each dataset gets a folder named by its dataset id, and that folder always holds the current version. Point your reader at ploti-data/ds_orange_fl_building_permits/snapshot/*.parquet and the path never has to change:
Every row carries _ploti_record_id, an id that stays with the record from version to version, and _ploti_version, the version it came from. Envelope columns lists every column Ploti adds. Files are named as the version’s manifest.json names them, so a path on disk is the same path Ploti’s API uses.

Attachments

Some datasets come with files: the archived copy of each source Ploti read the records from, and the document or map file it downloaded, such as a zoning ordinance’s PDF. A dataset that keeps a file per record, such as a permit’s scanned application, names it in the record’s cell as document:<sha256>. These are the dataset’s attachments. Sync downloads each one into the dataset’s attachments/ folder, named by its sha256 checksum, and checks the bytes against it. Two records citing the same document share one file, and a file already on disk is never downloaded again, even when the dataset moves to a new version. .ploti-sync.json lists each attachment under its dataset with the name Ploti gives it (original, source:<id> or document:<sha256>), its original filename and its path on disk, so you can join a record’s document:<sha256> cell to its file. An attachment that a new version no longer lists stays on disk, marked "listed": false in the state file, until you run sync with --prune.

Options

Each folder keeps one subscription. Sync refuses to write a second subscription into a folder that already holds another.

Run it on a schedule

Sync is safe to run as often as you like, since a run with nothing new downloads nothing.

From cron

Run it every hour, or every night:

When Ploti tells you

A webhook on the subscription has Ploti POST each event to you as it happens, so your copy updates within moments of a new version instead of on the next cron run.
The reply carries the signing secret, shown once. Store it as PLOTI_WEBHOOK_SECRET beside the handler. The handler verifies the POST, answers at once, and runs the sync in the background. The event itself only says that something changed: ploti feeds sync brings every dataset up to its current version, so a missed or repeated POST does no harm.
Ploti retries a failed delivery six times, at growing intervals, over about a day and a half. ploti webhooks test <webhook_id> sends a signed ping to check the handler. Keep the cron job as well, at a slower rate: if the handler is down when a version lands, the next scheduled run still catches up. Two runs on the same folder take turns: the second waits for the first, so a run started by an event never misses the version that event announced.

How sync keeps your copy whole

  • No half versions. A new version downloads into .ploti-sync/ inside the folder, gets checked, and then replaces the dataset’s folder in one step. A reader, or a crash part way, never finds a folder holding half of one version and half of another.
  • Checked bytes. Every file is checked against the sha256 checksum Ploti gives for it, and downloaded again when it does not match.
  • Links that run out. Download links last an hour. When one runs out during a long run, sync asks Ploti for a fresh link to the same file of the same version and carries on.
  • Datasets that leave. A dataset that leaves the subscription, because you changed its places or your plan no longer covers it, keeps its folder and is marked removed in the state file. --prune deletes it.
  • Datasets not ready yet. A dataset whose first version is still being built, or whose files are being rebuilt, is listed as waiting with the reason. What is on disk stays until its next version is ready.

What sync does not do

  • It loads snapshots, not changes. Each new version replaces the dataset’s folder with the full snapshot. If you load records into a database of your own and want only what was added, changed and removed, apply each version’s changes file instead: read the events after the eventCursor that .ploti-sync.json records, with ploti feeds subscriptions events, as Run the loop yourself shows.
  • It writes files, not tables. Sync writes to a folder. To load a database, point your loader at the folder after each run.
  • It syncs the whole subscription. To sync fewer datasets, narrow the subscription itself.

From Python

Ploti’s Python SDK has the same command, and leaves the same folder and state file:
It reads the key from PLOTI_API_KEY too, and takes dry_run=True and prune=True for the options above. It returns the summary --raw prints, as a dict.

Run the loop yourself

To apply change files to your own tables, or to load somewhere the sync does not write, run the same loop with the REST API. Create the key and the subscription as above (see Subscriptions), then:

Load the snapshots

The manifest first brings the subscription’s datasets up to date, then lists every one it still delivers that has a released version and that your workspace’s coverage and plan still open, with links to that version’s snapshot, and the event cursor it is current as of.
Then:
  • Download each snapshot’s parts (GeoParquet) and load them into a table keyed on _ploti_record_id.
  • Record each dataset’s version as the version you hold.
  • Save eventCursor: it is the last event already reflected in these snapshots.
Links expire at expiresAt, an hour after the manifest was made. For fresh links to the same versions, get the manifest again with ?asOf=<eventCursor>, or get each version from GET /v1/feeds/datasets/{dataset}/versions/{version}. attachments lists the files the dataset’s records cite, as on a version. Two kinds of dataset are listed under unavailable instead:
  • Files being rebuilt, with reason rebuilding. Ask for the manifest again in a few minutes to load it. Its events after eventCursor still apply.
  • No released version yet, with no version and its status as the reason: building, held, failed or awaiting_release. Its first version arrives as feed.dataset.resync_required.
A 503 plan_unavailable means Ploti could not read your workspace’s plan just now: try again in a minute.

Apply events

Read the events after your cursor, apply them in order, and save nextCursor once a page is applied. Only loading a snapshot or applying a changes file changes the version you hold.
  • Skip an updated, resync_required or schema_changed for a version at or below the one you hold.
  • Act on a schema_changed by changing your table to match, without recording its version as held. It names the same version as the updated (MINOR change) or resync_required (MAJOR change) that follows it.
  • Apply an updated only when its previousVersion is the version you hold.
Version numbers can skip. When previousVersion is any other version, reload that dataset from its current version instead: read current.version from GET /v1/feeds/datasets/{dataset}, then load that version’s snapshot from GET /v1/feeds/datasets/{dataset}/versions/{version}. You now hold that version, so skip the dataset’s later events at or below it.
Each version’s links come back signed for an hour from GET /v1/feeds/datasets/{dataset}/versions/{version}, which data.links.version names. It signs links only for a dataset one of the workspace’s subscriptions delivers. For any other, it answers 409 not_subscribed. A 409 (not_subscribed, outside_coverage or not_deliverable) or a 404 while you apply older events means the dataset has since left your subscriptions. Skip that dataset’s events up to its feed.dataset.removed. A version past retention answers 410 expired, and so does a version Ploti released again and rebuilt under a new number. Reload that dataset from its current version, as above. Add a webhook for the subscription’s events, as above, and use it as the signal to read the log. The log stays the source of truth. See Events and webhooks.