Challenge
More than 400 million athlete records, across sources that never agreed
Athlinks needed a pipeline that could stitch together more than 400 million athlete records held in disparate systems, none of which had a reason to agree on who a given athlete was. Matching at that volume is only half of it. A run that takes hours and dies partway through cannot start again from the beginning, and a failure nobody sees is a data set quietly wrong.
Strategy
Serverless for the scale, durable execution for the failures
The decision taken in the prototyping phase was to pair a serverless AWS architecture with Temporal, so the pipeline could scale horizontally while the orchestration held the state. Business logic was centralised in workflows instead of being spread across choreography, which keeps the rules readable and the failure handling in one place.
Solution
Workflow Orchestration: A series of Temporal workflows were created to oversee the data pipeline process so that the developers would have granular control where desired without having to micromanage every aspect of the system. The high-level business logic is centralized within the workflows rather than being spread out in various places. This is significantly less labor-intensive and more efficient than using choreography. Also, Temporal is more convenient for development and is less limited than other alternatives like AWS Step Functions.
Segmented Business Rules. Each tier or step in the data pipeline process is configured with distinct business rules. This greatly increases matching accuracy as it contextualizes the data appropriately for each step.
Data Reduction. Multiple avenues were used to condense and reduce data to make it less unwieldy.
Retry Mechanism: Temporal cleanly takes care of retries and ensures sub-steps are not skipped before the next step is initiated.
Visibility. Problems are quickly identified by Temporal and pushed along with a preview of the failures to Slack for real-time visibility.
Results
By deciding in the prototyping phase to pair serverless AWS architecture with Temporal, the new application/data pipeline build process has been smooth, especially given the inherent complexity of this project.
does temporal provide the optics and orchestration layers to build and debug the system, but it also reduces build time as it automatically provides retries and other mechanisms that would otherwise have to be baked into the product.
Bottom line, Implementing Temporal with proper usage of orchestration can make the creation of complex data projects significantly more accessible.