A redesign exercise of an AWS crypto platform I created a couple years ago which received $7.7 million from my design and an additional $13 million in 2021.
Three application domains defined - Guarantee, Action, Reference.
Guarantee uses AWS SQS queues to guarantee state and execution. We store the original Ethereum-signed message in its own queue keyed by Msg_ID. This is the only place the original message ever exists until it is archived.
Action has a feeder queue which receives Msg_ID. The controller pulls queue entries, references the original message and finds current state. Current state is fed into state machine which determines the next action to execute. When a action executes, it pulls the original message, executes its action, changes Msg_State, and updates the original message. Then it submits the Msg_ID to the feeder queue for the next iteration through the controller.
Reference receives a subset of the message attributes used for reporting/tracking purposes, such as creation_date, client_id, etc. The original message is immutable except for Msg_State. The controller updates Reference with current state before each iteration.
Most complexity is isolated in the State Machine/Rule Engine, so most future changes are there. The rest of the system should be stable, needing few changes except the addition of new actions.
There should have been an additional API which issued Msg_ID as generated by the host system. This would be the initial client call so that the subsequent Msg_ID can be added into the message before it's signed, making it part of the immutable structure.
There's duplicate data between the three domains, Guarantee is the system of record if we get a data mismatch. There's also a need to store intermediate data which may get passed from action to action, so add a Msg_Ext message to the queue. There's a coordination issue we can solve with Json schemas.