Spark® API Replication

If you'd like to replicate data and are not sure if you've been provided a key for replication, or find a problem with this outline, please contact api-support@sparkplatform.com.

1. Replication
2. Guidelines
   1. Initial Download
   2. Update Data
   3. Purging Stale Data

Replication

The Spark API endpoint for replicating data is: https://replication.sparkapi.com. Developers replicating data must use this endpoint.

Developers who wish to replicate data are granted Personal Access Tokens with special replication access. These keys are granted a few additional privileges to make replicating data easier. For example:

1. The maximum _limit is raised to 1000 for listings and accounts, and 200 for other resources.
2. The accounts resource does not apply company or office restrictions when searching user records for IDX roles.
3. The CustomFieldsRaw and CustomFieldsExpandedRaw expansions are available for listings when retrieving multiple records.

These keys are otherwise identical to regular API keys, with access to the same resources and governed by one of our data roles

Guidelines

While the listings resource is used in these examples, the same general guidelines also apply to replicating other resources.

1. Initial Download

To kick off the initial download you'll need to make a request to the all listings endpoint. Be sure to use the _skiptoken parameter to paginate through the results.

Optionally, you may include the _select parameter, but no other parameters should be specified.

Important: a ListingKey orderby is applied by default, ensuring that the order of the results is consistent while paginating with _skiptoken.

2. Update Data

Keeping your result set updated involves making requests against the ModificationTimestamp field using the timestamp of your last successful update minus 20 minutes to account for server side caching. Here is a pseudocode example to demonstrate:

query_timestamp = last_successful_retrieval_timestamp - 20 #minutes
?_filter=ModificationTimestamp Gt query_timestamp

Paginating through the update result set is once again accomplished by using _skiptoken.

During this step it is recommended that you also update any listing subresources of interest. Note that our listing data has several timestamps that can be used to determine if the corresponding data needs to be updated: PhotosChangeTimestamp, VideosChangeTimestamp and DocumentsChangeTimestamp.

3. Purging Stale Data

Repeat step 1 with the _select=Id or _select=ListingKey parameter applied, paginate through the result set identically to step 1, and compare all ListingKey values returned with the values in your database. Purge any records that do not match.