Challenge
The current process had several challenges:
API failures sometimes prevented "Closed" status from reaching vendors
- No reliable retry mechanism for failed transmissions
Lack of visibility into any failures on transmission of the status of assignments
These issues exposed the company to the risk of wrongful repos and associated penalties.
Strategy
Make the process durable rather than the API calls reliable
Retries would have improved the odds. They would not have changed the property that mattered, which is that the process had no memory of where it had got to. A close assignment is a business process with legal consequences, and what it needed was state that survives a failure and a history that can be read afterwards.
So the unit of work became the workflow itself, holding the whole lifecycle of a close assignment from receipt through to confirmed transmission, with the failure paths designed in at the start.
Solution
Tech Fabric implemented Temporal's durable execution framework to address these challenges:
Workflow Orchestration: A workflow was designed to handle the entire lifecycle of a close assignment, from receipt to successful transmission to the vendor.
State Management: Temporal's state management capabilities ensured that the status of each assignment was persistently stored and could be recovered in case of system failures.
Retry Mechanism: The workflow incorporated automatic retries (#5) for failed API calls, with configurable backoff periods.
If multiple retries failed, this activity would trigger notifications to the operations team for manual intervention.
Event Sourcing : All actions and state changes were recorded as events, allowing for complete auditability and the ability to reconstruct the state of any assignment.
Real-time Status Tracking : Temporal's Web UI provided a real-time view of all running workflows, allowing operators to quickly identify and troubleshoot any issues.
Debugging Capabilities : Temporal's ability to replay workflows from any point in their execution history significantly improved debugging processes.
Distributed Coordination : Temporal's distributed architecture ensured high availability and fault tolerance, which a 24/7 operation has to have.
Results
The "Closed" status process got reliable. That was the whole point of the engagement, and it is the sentence the operations team cared about.
Wrongful repos became far less likely. Visibility and scalability improved alongside, which nobody had asked for and everybody used.
Durable execution earns its keep in exactly this kind of process, where a missed status update has a legal consequence attached to it and nobody finds out for a week.
Tech Fabric's implementation of Temporal's framework provided the necessary tools to build a reliable, fault-tolerant system that could withstand the challenges of a complex, distributed environment