Flow building 101: Gathering integration requirements

Creating a flow means you are trying to solve a business problem faced either by yourself or by someone with whom you work. For example, you may be in marketing operations, and you routinely create Event Campaigns in Salesforce and then re-key the same information into Hubspot as a Marketing Event.

You’d like to eliminate the double data entry. How do you do that in the Celigo Platform, integrator.io?

When analyzing this business process, you’re essentially capturing requirements. At Celigo, we advocate for a simple yet effective method of gathering requirements inspired by the story structure from Mike Cohn, a Scrum Alliance co-founder and experienced Scrum Master.

This method is easy to follow and suitable for everyone, whether you’re an integration architect or a DIY enthusiast.

Get our requirements Requirements Document to assist you before you begin the integration process.

Let’s break it down.

What’s your story?

The Scrum Alliance recommends creating a user story structure for requirements where there is an actor, a business function the actor wants to perform, and a business reason the actor wants to perform it, for example:

As a __________ (type of actor) 

I want to _______ (business function)

So that _________ (business reason)

In our example as a marketing ops user, we would break down the story like so:

Now that you have established the actor, the business function, and the business reason for your flow, you can build out the requirements for your flow by answering a set of questions.

When you put your storyline together with the answer to your requirements questions, you’ve got a blueprint requirements document to create your flow.

Create the storyline

What applications do you plan to synchronize?

Identify the applications that you will use and what you plan to synchronize or transform.

What type of connection do you need?

Celigo comes with hundreds of application connectors that can jumpstart your integration. When building your requirements, we recommend carefully reviewing the application connector documentation.

If the out-of-the-box connector doesn’t have all the endpoints needed, you can create a Universal Connection.

You can also explore using prebuilt templates (available at no charge) on the Integration Marketplace. Celigo templates act as pre-configured workflows that automate connection definition, data mapping, data transformation, and data integration between various applications.

What format will the data be in?

A data format is a specific way of arranging and storing data so that a computer program can understand it.

In our flow, the data format will be JSON, a popular format for web APIs.

What web service will you use?

A web service is a software application that exposes functionality over the web using standardized protocols and data formats. It allows applications to communicate and exchange data with each other in a programmatic way, regardless of the underlying programming languages or platforms they are built on. 

In our example, we will be using the most popular format, a REST-based web service.

What transport protocol will be used?

A transport protocol is a communication language that computers use to ensure that data is delivered accurately and efficiently. There are many different transport protocols. Transport protocols are essential for all sorts of network communication, from browsing the web to sending emails to streaming videos. They help to ensure that our data gets where it needs to go quickly and reliably.

In our flow, we’ll be using HTTPS.

What authentication method?

The most common authentication method is OAuth2.0, but other options are available and should be included in your requirements gathering.

What will trigger the integration and how often will the integration run?

Triggers are the sparks that ignite your integration flows. They define when an automation kicks off, ensuring your data flows and tasks run smoothly.

Let’s break down the different types of trigger frequencies:

• Real-time triggers: Real-time triggers fire the flow immediately after the triggering event occurs. For instance, a new customer placing an order triggers an email confirmation to be sent right away.
• Scheduled triggers:  In a scheduled trigger, define a specific time or interval (daily, hourly, etc.), and the flow runs based on that schedule. For example, an inventory report might be automatically generated and sent every Sunday evening.
• Batch triggers:  Batch triggers accumulate a set number of events (a “batch”) before activating the flow. This is useful for processing large volumes of data more efficiently. For example, instead of constantly sending individual orders from an ecommerce system, a batch trigger might wait until 100 orders occur, then send a single consolidated batch file to an ERP.
• Webhooks and Listeners: Proactive triggers
  • Webhooks: Webhooks are a way for external applications to proactively notify the iPaaS when something specific happens (like a new order placed). The iPaaS then listens for these incoming notifications and triggers the appropriate flow.
  • Listeners: These are constantly open channels. The iPaaS actively monitors a specific source (like a database) for changes. When a relevant event occurs (e.g., new data added), the listener detects it and triggers the flow.

Choosing the right trigger

The best trigger frequency depends on your needs. Real-time is ideal for critical actions, while scheduled triggers suit routine tasks. Batch triggers are perfect for handling large datasets without overwhelming the system. Webhooks and Listeners provide a proactive approach to external events or continuous monitoring.

In our example, we synchronize Marketing Events in Hubspot and Campaigns in Salesforce nightly at 9 pm ET.

What APIs and data models will be referenced?

Understanding the APIs available and the data structure of the applications that you plan to synchronize will help you build your flow. Synchronizing data can be tricky because different applications will name objects differently, and you need to understand the nuances. 

For example, in one system, an object that stores information about companies is called “Organizations”; in another system, it’s called “Accounts,” and in a third system, it’s called “Companies.”

We recommend that you review the APIs of the source applications, ERDs, and data models for the flows you build.

In our example, the following APIs and data modules are referenced.

iPaaS requirements business rules

Each set of flows will require thinking through business rules that must be identified, reviewed, and decided upon (perhaps within a requirements session conducted with business users).

This type of session is often called a “requirements workshop.” Our advice is to keep it simple — focus on one flow at a time and be prepared to listen, iterate, document, and listen some more. The discovery of business processes and rules is often a bit messy as it requires understanding how systems are used cross-functionally. It will involve “people, processes, and technology,” with people being the key factor in explaining what needs to be done.

Common considerations to review in a requirements workshop are listed below.

Directionality and source of truth

• Data flow direction: Determine which systems will be the source of truth for specific data elements. This clarifies whether updates flow in one direction (unidirectional) or both directions (bidirectional) between systems.
• Conflict resolution strategy: For bidirectional flows, define how conflicts will be resolved if the same data element is updated in both systems simultaneously. Will the latest update win, or will there be a more complex validation process?
• Error handling: Establish how errors during data transfer will be handled. Should retries be attempted? Will notifications be sent to admins in case of failures?

Primary Identifier

• Unique identifier selection: Identify a common field or combination of fields that uniquely identifies each data element across all connected systems. This ensures data synchronization accuracy.
• Identifier maintenance: Discuss how these identifiers will be maintained. Will a central system manage them, or will each system be responsible for generating unique IDs?

Orchestrating and transforming data

• Data mapping: Define how data elements from one system should map to another. This may involve identifying corresponding fields, handling missing values, and applying data formatting (e.g., converting dates and currency).
• Data aggregation and summarization: It may be advantageous to combine data from multiple sources and transform it into summarized formats (e.g., totals, averages) for further analysis.
• Integration flow (event trigger) – what is the sequence of events for data transfer? This includes specifying triggers (e.g., new record creation) and the actions to be performed (e.g., data creation or update in the target application).
• Data enrichment: Data can be incorporated from external sources (e.g., demographics, weather data) to enrich existing datasets and create a more comprehensive view.
• Data quality requirements: Establish data quality expectations by defining rules to handle errors, missing values, and inconsistencies. This may involve setting thresholds, defining correction methods, or establishing rejection criteria for bad data.

Other common requirements

• Data security: Define the security protocols for data exchange between systems. This includes authentication, authorization, and encryption methods to ensure data privacy and integrity.
• Performance: Discuss the expected data volume and frequency of updates. This helps determine the scalability and performance needs.
• Auditing and compliance: Discuss any auditing or compliance requirements that the flows need to meet. This might involve data lineage tracking or adherence to specific industry regulations such as HIPAA and masked financial information
• Scalability and future needs: Consider the potential for future growth and integration with additional systems. The flow should make business sense and be adaptable to accommodate evolving business needs.

Additional tips:

• Document everything: Clearly document all agreed-upon business rules to ensure everyone involved in the integration process is on the same page.

Gathering requirements for well-planned integrations

Capturing requirements is essential when analyzing business processes. At Celigo, we advocate for a simple yet effective method of gathering requirements. By adopting this approach, you can ensure that your integrations are well-planned, meet user needs, and are primed for success.

As we continue to innovate and refine our methods, embracing such straightforward and inclusive techniques will be key to driving successful integration projects and achieving business excellence.

→Get our example Requirements Document