Solution
Workflow Orchestration: A series of Temporal workflows were created to oversee the data pipeline process so that the developers would have granular control where desired without having to micromanage every aspect of the system. The high-level business logic is centralized within the workflows rather than being spread out in various places. This is significantly less labor-intensive and more efficient than using choreography. Additionally, Temporal is more convenient for development and is less limited than other alternatives like AWS Step Functions.
Segmented Business Rules. Each tier or step in the data pipeline process is configured with distinct business rules. This greatly increases matching accuracy as it contextualizes the data appropriately for each step.
Data Reduction. Multiple avenues were used to condense and reduce data to make it less unwieldy.
Retry Mechanism: Temporal cleanly takes care of retries and ensures sub-steps are not skipped before the next step is initiated.
Visibility. Problems are quickly identified by Temporal and pushed along with a preview of the failures to Slack for real-time visibility.
Results
By deciding in the prototyping phase to pair serverless AWS architecture with Temporal, the new application/data pipeline build process has been smooth, especially given the inherent complexity of this project.
does temporal provide the optics and orchestration layers to build and debug the system, but it also reduces build time as it automatically provides retries and other mechanisms that would otherwise have to be baked into the product.
Bottom line, Implementing Temporal with proper usage of orchestration can make the creation of complex data projects significantly more accessible.