In order to support custom dataflows you have to turn on ‘Data Replication’ and you want to create custom dataflows in order to allow better control over schedules, development, debugging, and insulation against dataflow failures due to changes, data, etc.
That being said you will want to disable incremental updates. Troubleshooting has several times come back to the Replication Object has stale data even though the data was changed hours or days before working the issue with replication and the dataflows running.
Some things to note:
- New Records are picked up immediately
- Changes could take an hour before getting picked up
- Replication counts against the 24 dataflow limit
This might come across as negative but I am not sure why the replication settings give you the option to run hourly if it’s going to cause a limits issue. I have 5 Dataflows plus replication and can only run a refresh once a day.
Use this resource to disable replication: Extract Data Incrementally in Replication
Or basically add this to your dataflows right before the fields: “incremental”: false,