As businesses continue to scale and manage increasingly large datasets, the need for efficient data synchronization and management tools has never been more critical. Celigo is constantly innovating to meet these needs, and we’re excited to introduce some significant updates to our Salesforce integration capabilities, specifically focusing on the Bulk API 2.0.

In this post, we’ll dive into these new features, explore their benefits, and explain how they can help your organization optimize data handling between Salesforce and other platforms.

Introduction to Salesforce Bulk API 2.0 integration

Historically, Celigo has supported Salesforce’s REST API, which is perfect for typical app-to-app data exchanges, such as syncing customer accounts from Salesforce to an ERP system. However, when dealing with larger volumes of data—such as exporting millions of records into a data warehouse—the REST API can fall short in terms of speed and efficiency.

This is where the Salesforce Bulk API 2.0 comes in. The Bulk API 2.0 is designed to handle high volumes of data by processing records asynchronously, allowing for faster and more efficient data transfers. This makes it ideal for large-scale data migrations, bulk extractions, and other operations that involve extensive data manipulation.

What’s new: Leveraging the Salesforce Bulk API 2.0

With our latest update, Celigo now supports Salesforce’s Bulk API, giving users the ability to synchronize massive datasets between Salesforce and their destination systems more efficiently.

Key features of the Bulk API 2.0 integration

Asynchronous data processing:

Unlike the REST API, which operates synchronously and provides real-time feedback, the Bulk API 2.0 processes data asynchronously. This means that while you won’t get immediate feedback, the overall process is much faster for large datasets.

Improved export times:

Users can expect a 2x to 6x improvement in export times when using the Bulk API 2.0 compared to the REST API. This is particularly beneficial when dealing with millions of records. The larger the data set, the more you’ll notice the performance gains.

Handling large data volumes:

The Bulk API 2.0 is specifically designed to handle large volumes of data, making it the go-to option for exporting data from Salesforce to a data warehouse or object storage. This is crucial for businesses that rely on up-to-date data analytics and reporting.

Customizable page size:

By setting your page size to the maximum, you can avoid chunking results into multiple pages, further optimizing the data transfer process. This feature ensures that your data exports are as efficient as possible.

Limitations and considerations

While the Bulk API 2.0 offers significant advantages, it’s important to be aware of its limitations compared to the REST API. For instance:

SOQL query constraints:

Certain SOQL (Salesforce Object Query Language) queries that work with the REST API may encounter issues with the Bulk API 2.0. For example, attributes like compound fields may not be supported, and some queries might fail if they exceed the Bulk API 2.0’s limitations.

No preview option:

The asynchronous nature of the Bulk API 2.0 means that you won’t have a preview option like you do with the REST API. Therefore, it’s often best to preview your data using the REST API before switching to the Bulk API 2.0 for large-scale data transfers.

We recommend checking Salesforce’s documentation on SOQL query limits for the Bulk API 2.0 to ensure your queries are compatible.

New feature: Syncing archived and deleted records

Another enhancement we’ve introduced is the ability to include archived and deleted records in your data syncs. This feature is particularly useful when maintaining accurate records in a data warehouse or database.

Why this matters

Track deleted records:

Salesforce retains deleted records for about 14 days before permanently removing them. By syncing these records to your database or warehouse, you can maintain a historical log of changes, ensuring that you know what was deleted and when.

Accurate data management:

Including archived and deleted records in your data syncs allows for more accurate data management, particularly in environments where historical data is crucial for compliance or reporting purposes.

What’s next: Importing data to Snowflake and NSAW

Looking ahead, we’re excited to explore the next steps in our product updates, which will focus on importing data to Snowflake and NSAW. These new bulk load features will further enhance your ability to manage large datasets efficiently across different platforms.

In this video tutorial, we’ll walk you through these new bulk load features and demonstrate how they can optimize your data integration workflows.

Conclusion

The introduction of Salesforce Bulk API 2.0 support in Celigo marks a significant enhancement in our platform’s capabilities, particularly for businesses managing large volumes of data. By leveraging these new features, you can dramatically improve data transfer speeds, maintain accurate records, and streamline your overall data management processes.

We encourage you to explore these new features and reach out to our support team if you have any questions or need assistance getting started. With Celigo’s powerful integration tools, managing your Salesforce data has never been easier or more efficient.