Sync Mechanisms: Snapshot vs. Streaming
1. Overview
- Definition: Synchronization mechanism — The method and frequency Mobio uses to retrieve data from a Connector. Mobio provides two main mechanisms: real-time, event-based sync (Streaming) and periodic batch sync (Snapshot).
- Benefits/Value: Understanding and selecting the correct sync mechanism from the start prevents Mobio from becoming overloaded and helps avoid data loss or incorrect overwrites of your current customer data.
- Users: System Administrators, Data Engineers, and IT/Technical Specialists responsible for integrations.
2. Terminology & Configuration Parameters
- Streaming (Real-time Sync) — Data is pushed from your system to Mobio instantly (real-time) as events occur. This mechanism typically applies to API (Webhook) or SDK Data Flows.
- Snapshot (Batch/Periodic Sync) — Mobio pulls a batch of data from your system at fixed time ranges (e.g., 12 AM daily). Snapshot further divides into two methods:
- Full (Complete Sync) — Scans and retrieves all data from start to finish, then overwrites existing records. This places a significant load on Mobio but ensures absolute consistency.
- Incremental (Delta Sync) — Pulls only new or updated data records since the Last Sync. It conserves resources and is extremely fast.
3. Prerequisites
- Previous Post: Overview of Data Flow Operation – Data Source
- Access Permissions: Your Mobio Account Settings must have the RBAC: INTEGRATION Access Permission. For detailed Access Permissions, refer to this post.
- Access Path: Go to Settings → Data Integration → Data Flow.
4. Evaluate and Select a Sync Mechanism
As this is a conceptual guide, there are no direct interface settings to configure. Instead, answer these three questions before configuring your sync rules:
- Where is your source data stored?
- If it is in an internal database, use the Snapshot method.
- If it tracks real-time user engagement on a website, use the Streaming method.
- Does your business require immediate data to trigger real-time interaction campaigns?
- If yes, prioritize connectors that support Streaming.
- For a Snapshot database, does the source data table have a last-modified timestamp column (e.g., updated_at)?
- If yes, we recommend using Incremental Snapshot to optimize performance.
5. Operational Rules
- Profile attributes: Profile data rarely changes and is ideal for Incremental Snapshots (e.g., updating membership card tiers nightly).
- Event data: Interaction events (e.g., page views, purchases) that require immediate response should use Streaming.
[!WARNING]
Avoid high-frequency Full Snapshots on large databases: Running a Full Snapshot on databases with millions of rows at short intervals (e.g., every few hours) can overload both your source database and Mobio.
6. Frequently Asked Questions (FAQ)
- Q: If I configure an Incremental Snapshot, but my database server goes offline, will I lose my data?
- A: No. Mobio records a checkpoint — the last record successfully synced, marked by an order_key value (which can be a number or a timestamp). When your server’s network connection returns, Mobio resumes from that checkpoint and pulls all records that accumulated during the disconnected period.
- Q: Can I change a Full Snapshot to an Incremental Snapshot
- A: Yes. You can edit the sync schedule settings in Data Flow to update this sync configuration.
7. Next Steps
Once you choose a sync mechanism, proceed to create one of the following connector types based on your requirements: