🔧 Sourcing: Part 1 – Exploring Event Sourcing in Raku

I’ve been studying a very interesting approach to
building systems called Event Sourcing. In event sourcing,
every change to an application’s state is captured as an
immutable event. Instead of storing only the current state,
you persist a sequence of events that record all the actions
that have taken place. This approach not only allows for an
accurate reconstruction of the system state by replaying
events but also provides a robust audit log and improves
debugging and analysis capabilities.

For my studies, I started writing a framework to easily
implement event sourcing in Raku. It’s called Sourcing.

Sourcing has a few key concepts. In this first part, we’ll discuss three of them:

1. Events
2. Event Store
3. Projections

Events

Events are the core building blocks in an event-sourced system.
Each event represents a fact that has occurred in the past,
such as an order being created or a payment being initiated.
Here are some important points about events:

• Immutability:

Once an event is recorded, it should never change.
This immutability guarantees that the historical
record remains accurate and reliable.

• Past Facts:

Events capture facts that have already happened,
meaning they are a definitive record of the state
transitions within your system.

• Audit Trail:

By recording every event, you create a full history of all actions,
making it easier to trace issues or understand
how a certain state was reached.

Event Store

The Event Store is a specialized storage mechanism for events.
It acts as a single source of truth for all the state changes
in your system by providing these features:

• Append-Only Log:

Events are stored in an append-only manner,
ensuring that once an event is recorded,
it is never modified.

• Order Preservation:

The sequence of events is maintained,
which is critical for reconstructing the state by replaying
the events in the order they occurred.

• Durability and Auditability:

The event store keeps a permanent record of every event,
which can be used for debugging, analytics,
or even recreating the state at any given point in time.

Projections

Projections are responsible for transforming the raw stream
of events into a read model that is easier to query and work with.
Many projections can/should use the same set of events.
They serve several important functions:

• State Aggregation:

Projections listen to events from the Event Store
and update a view or model that represents the current state of the system.

• Multiple Views:

You can create multiple projections for the same set of events,
each tailored to different client needs. For example,
one projection might track the status of a food order
while another aggregates sales data.

• Decoupling:

By separating the write model (events) from the read model (projections),
you can optimize the performance and scalability of your application.

The following code shows how a projection, DeliveryStatus,
is implemented in our framework.
Each projection must define an apply method for every event it wants to handle.
When an event is emitted, the projection’s corresponding
apply method is called to update its state.

Food Delivery Example

To demonstrate the framework, let’s build a simple Food Delivery system.
This system uses a few types of events to represent different
stages in the order lifecycle, an example of one of the events class
wold be:

use Sourcing;
use UUID::V4;

unit event Sourcing::FoodDelivery::Event::OrderCreated;

has Str  $.order-id      = uuid-v4;
has Str  $.delivery-code = (1 ..^ 100)>> .fmt("%02d") .pick;
has UInt $.user-id       is required;
has UInt $.restaurant-id is required;
has Str  %.item{Str};

Projection: DeliveryStatus

Projections consume events to maintain a read model. In this example,
the DeliveryStatus projection tracks the status of an order and stores
the delivery code required for the deliverer.

When a projection object is created, it is registered with a manager
(to be described in a future post) and remains in memory until it is
unloaded (a feature still in development).

Projections don't need to use all events and events don't need to be
used by projections. On this example, I'm defining some Payment events
that are not being used by the projection. That's completely ok. if
in the future, a new projection is created, it will consume those events
(if defined to) and generate new objects using those historical data.

use Sourcing;
use OrderCreated;
use OrderAccepted;
use DeliveryStarted;
use DeliveryCompleted;
use OrderDelivered;

unit projection Sourcing::FoodDelivery::DeliveryStatus;

has Str      $.order-id    is aggregation-id;
has Str      $.status       = "";
has DateTime $.last-status .= now;
has Str      $.delivery-code;

method summary is query{ :sync } {
    %(
        :$!order-id,
        :$!status,
        :$!last-status,
        :$!delivery-code,
    )
}

multi method apply(OrderCreated $_) {
    $!status        = "placed";
    $!last-status   = .timestamp;
    $!delivery-code = .delivery-code;
}

multi method apply(OrderAccepted $_) {
    $!status        = "preparing";
    $!last-status   = .timestamp;
}

multi method apply(DeliveryStarted $_) {
    $!status      = "collected";
    $!last-status = .timestamp;
}

multi method apply(DeliveryCompleted $_) {
    $!status      = "delivered";
    $!last-status = .timestamp;
}

multi method apply(OrderDelivered $_) {
    $!status      = "done";
    $!last-status = .timestamp;
}

The apply Method

Every projection needs an apply method candidate for each
event it intends to handle. When an event is emitted,
the projection automatically calls the corresponding apply
method to update its state based on the event’s data.

The is query Trait

Methods marked with the is query trait are exposed to clients.
When marked with the :sync parameter, these methods first process
all pending events, ensuring that the projection’s state is
up-to-date before returning the result.

Usage Example

Note: Although events should ideally be emitted via dedicated commands,
for this example we emit them directly.

...

my Sourcing::Client $s          = Sourcing::Client.new;
$s.register-class: "DeliveryStatus";
my RedEventStore::Client $store = RedEventStore::Client.new, 
my \DeliveryStatus              = $s.get-class-client("DeliveryStatus");

sub emit-event(Sourcing::Event $event, :$order) {
    say "{ gray $event.timestamp.hh-mm-ss }: emitting the event: { yellow $event.^shortname }";

    $store.add-event: $event;
    given $order.summary {
        say "Current state: STATUS: { green .<status> }; CODE: { green .<delivery-code> }"
    }
    sleep 1
}

sub MAIN(Str :$order-id = uuid-v4) {
    say "Using order-id: { red $order-id }";

    my $order           = DeliveryStatus.new: :$order-id;
    my $user-id         = ^10 .pick;
    my $restaurant-id   = ^10 .pick;
    my $prepare-minutes = 10;
    my $deliverer-id    = ^10 .pick;

    emit-event :$order, OrderCreated.new:      :$order-id, :$user-id, :$restaurant-id;
    emit-event :$order, PaymentInitiated.new:  :$order-id, :payment-data<blablabla>;
    emit-event :$order, PaymentConfirmed.new:  :$order-id;
    emit-event :$order, OrderAccepted.new:     :$order-id, :$restaurant-id, :$prepare-minutes;
    emit-event :$order, DeliveryStarted.new:   :$order-id, :$deliverer-id;
    emit-event :$order, DeliveryCompleted.new: :$order-id, :delivery-code("10");
    emit-event :$order, OrderDelivered.new:    :$order-id;
}

Running this, it will print out something like this:

Projection: UserOrders

Another simple projection we could use here would be a projection
to show all orders of a user. It would be simply consume a single
type of event.
Different from the first projection that aggregates the events
using the order-id, the UserOrders uses the user-id as aggregation-id
to group all event "from" the same user together.
(I plan to, at some point, make it possible to map event attributes
with different named to a aggregation-id on projections).

use Sourcing;
use OrderCreated;

unit projection Sourcing::FoodDelivery::UserOrders;

has Str $.user-id is aggregation-id;
has Str @.orders;

method users-orders is query {
    @!orders
}

multi method apply(OrderCreated $_) {
    @!orders.push: .order-id
}

Conclusion

In this post, we explored the fundamentals of event sourcing
and how to implement a basic framework in Raku using the Sourcing library.
We covered the core concepts of events, the event store, and projections,
and demonstrated these ideas with a practical Food Delivery example.
Event sourcing provides a robust foundation for building systems that are scalable,
auditable, and easier to debug. In future posts, we’ll delve deeper into advanced
topics such as command handling, event replay, and integrating with
production-grade data stores. Stay tuned for more insights into building
reliable and resilient systems with event sourcing!