📓 Event Sourcing & Data Structures

haven’t written anything about the project manager in a hot minute.

Data Structures

my custom written nonsense continues to function. i continue to add scope that means in retrospect i made bad decisions earlier.

e.g. wanting to know when something changed isn’t possible (well, technically it could be scrounging through the git history but that’s unattractive.) storing as json in git wasn’t the optimal choice. (also conflict resolution also would mandate a UI and manually coded resolution for common scenarios because can’t expect anyone doing academic project management to be a git expert. either way, sub-optimal. but i will be riding that “it’s a task to keep my coding skills fresh while i move into a role that requires significantly less coding”)

i should’ve chosen something else to track these changes, probably, i fear, a log structured merge tree. and thus not storing a dict, but, the dict flattened into a KV somehow.

and tracking changes to the object through diffs, turned into LSM Tree key update/deletes.

probably this wll open up other issues, especially if lists are involved, you wouldn’t want to set a.b.1 = 1 and have that be applied to a yet non-existent list, would it result in {a: {b: [null, 1]}} or just error?

outright ban lists? hashes only, and if you need ordering, store a comma separated string of the correct indexing? (i guess that transformation could be automatic. {"k": ['a', 'b', 'c']} could easily become {"k": {"order": "de,ad,be", "de": "a", "ad": "b", "be": "c"})

or is it an xy problem and i’ve got a hammer (LSM Tree) that i’m using by default when really there’s some magical solution out there to “I want a time travel dictionary” where I don’t actually care about the time travel aspect except to figure out when specific values changed and report them in a nice way for the end user.

i wonder how git-mergeable two LSM-shaped tabular files are. that’s. a question. with each line timestamped, surely we won’t get conflicts. surely

probably a thing to attach the account that did the action as well.

Event Sourcing

that’s what it’s called. I knew i wanted a log but, there’s lotsa shapes of log. event sourcing is the thing i want.

refs:

• https://www.geeksforgeeks.org/event-sourcing-pattern/
• https://medium.com/ssense-tech/event-sourcing-a-practical-guide-to-actually-getting-it-done-27d23d81de04
• pyeventsourcing seems like the recommendation but that requires a lot of annotations and I have uhhhh no interest in that

(dog meme, no tracking, only event sourcing!!!)

i want to reconstruct the object without this annoying process.

i guess this is equivalent to “event sourcing but the event is updateObject(obj-goes-here)” as ugly as that is. And, yes, no interest in that precisely, that’s .. my least favourite place to stick it in the api. (it makes sense I guessss but, it’s just ugly since the api interface is so ugly. maybe that’s a sign that should be fixed instead. maybe i should continue down my insane paths.)

Automatic Event Sourcing?

Can we be lazier? it is always my goal in life, can we make it completely automatic, as a set of changes to a dictionary? Given a log that looks something like

can we reconstruct an object out of that? (Spoilers: python says yes.)

the key point to make this potentially automatic is:

• diff objects rather than tracking
• automatically transform lists like [1,2,3] into {“order”: “de,ad,be”, “de”: 1, “ad”: 2, “be”: 3} which can be more safely manipulated.

jsondiff produces sufficiently useful diffs that we could perhaps generate a set of events from that, and emit them.

Testing will be … a thing. hypothesis will be necessary.

Lists

These… lists, yeah. I have fears. subtle fears that say trying to resolve a series of ‘insert x at position y’ will be absolutely dreadful. so converting it to/from a dict based representation maaaaybe makes more sense. we can add an order key and then hash our values. dedupe for free (unlikely to be a benefit.)

{'l': {'7MvIfktc': 3, 'z80ghJXV': 0, '__order': '7MvIfktc|z80ghJXV|7MvIfktc'}, 'a': {'b': {'7MvIfktc': 3, '__order': '7MvIfktc'}}}
{'l': [3, 0, 3], 'a': {'b': [3]}}