API Limitations
Web service APIs also have practical (in some cases subscription-based) limitations. Each API will have its own limits and rules. Some of the limitations are:
• Batch size
• Call allocation
• Concurrency
Batch Size
Always consider the size of the batches (that is, number of records) being requested from the source and being sent to the target. Application specific map connectors already have a property to account for these limits.
The DataConnect Studio API REST invoker provides complete control to account for limits within the integration:
• If you need more records than the max batch size permits, you must make a subsequent request for the additional records.
• Most endpoints provide the total number of records from the query and the batch size. Therefore, if the total number is greater than the batch, make sure to PAGE through the rest of the records.
• Use the maximum batch size when possible because latency is higher for 200 batches of 1 record than it will be for 1 batch of 200 records. The trade-off considered is the time to wait until you have the full batch.
Call Allocation
Some APIs throttle the number of calls a user is allotted during a 24-hour period (for example, Salesforce). As much as possible, calculate the number of records you will need to access during the period and divide by the number of calls you are allotted during the period. Then, set the batch size accordingly.
Concurrency
The API may have a limit on the number of active sessions an organization or user can have at any given time (example, NetSuite). This adds an implied constraint on how you interact with the API during the session. For example, an error may occur while performing a lookup to the same API you are using for source data. Possible resolutions include:
• Increasing the API limits by upgrading to a higher tier edition.
• Use different users or sessions for each operation.
Note: This may lead to other problems such as data integrity if both users do not have access to the same data at the same time.
• Call the APIs sequentially and stage the data locally or in memory between each call.
Last modified date: 08/02/2023