Personal Utility, a milestone far short of General Utility


I am finally actively using this myself, namely to work on urn:penemure:5c6880ba-de38-485b-97a9-e78a85beaf88 ¹. It is sufficiently useful for me which is an excellent state. This of course is far, far short of General Utility. I’m not convinced that is a worthwhile goal for me though. An itentionally dangling reference. Maybe someday I’ll make that repo public, and then the issues from it will be included as part of deployment, and that link will become clickable. ↩ Innovations Here are some of the innovative aspects Layering I’d argue the multi repo layering of resources is an innovation. Which naturally leads to “why?” It is reasonable: most people don’t have these needs (disparate set of issues scattered across organisations rather than a central store) most folks need central Auth and ACLs With a centralised server deployed, we can have both, as it were. Storage Innovative is not the right word, storing the documents as folders of json files is in no way innovative. The original plan was to use git-bug which would have the additional benefit of tying change authorship to git identities, at the cost of making the data model significantly more complex and hard to see. I think I’d still like to have that, eventually. Or at minimum to hide the files and commits in the same low visibility way that git-bug does, in order to make it’s use less intrusive. Federation via Git This project is a very, very different take on federated issues than, say, the efforts in forges to do ActivityPub based federation. I look forward to those being generally available. I wanted to actively avoid always online processes (I’m an old fashioned sysadmin after all), and instead to focus on alternative ideas. This approach comes with some compromises, namely that within a repo, there is generally no way to have differing levels of access. That has to either be done through multiple repos, one per ACL role, and issues stored in their appropriate location, cross referencing each other everywhere server side ACL where we implement a non-git based backend for notes, where the server can enforce ACLs, and all requests have to go through http APIs. Non-Innovations some choices were very standard Notion’s Data Model We have essentially reimplemented something very Notion shaped, but I’d argue, poorly. Given how tied I was to the data format (json, on disk, in git repos) it means the resulting system isn’t exactly fast. In fact one of our biggest optimisations was “just shove it all in a sqlite for when you actually have to query”. A lot of the system could be significantly improved by sticking it in one single centralised database with a reasonable schema. Of course that defeats the “federated” ideas where a user can make up whatever subset of repos they like to produce a view customised for them. That could be achieved through a bunch of individual sqlite databases that are overlayed / queried in some fashion. There are options that are only mildly horrifying. Storage is a question then, I still wanted the git repo aspect and blobs in git, conflict resolution… yea nah. So we have a notion like model. At which point it became clear how much of notions usability is in it’s UI and UX. It makes sense why it is so slow and JavaScript-filled, because if you want to be able to, within a note, dynamically add a new field to all of those notes via their (db/template), and then render a nice picker, it makes sense to move that UI calculation into a giant morass of JavaScript that needs to run constantly to render even one letter. Future Work at this point, I am not giving up on this, though I think there are a number of sharp edges to be resolved before anyone else uses it most users probably don’t want to write SQL, that needs to be hidden behind a better UX The conflict resolution story should be better than “then you go into git and fix it” Editable tables is pretty close to a “must” Tracking mentions is mandatory and currently missing. Integrations with external systems would be useful. At bare minimum, replacing URLs with some basic link preview information would be useful. Especially for Google docs, the notion behaviour of replacing a URL with a logo/document title is useful. We need the same for GitHub, also to fetch the open/close state of issues.

I am finally actively using this myself, namely to work on urn:penemure:5c6880ba-de38-485b-97a9-e78a85beaf88 ¹. It is sufficiently useful for me which is an excellent state.

This of course is far, far short of General Utility. I’m not convinced that is a worthwhile goal for me though.

An itentionally dangling reference. Maybe someday I’ll make that repo public, and then the issues from it will be included as part of deployment, and that link will become clickable. ↩

Innovations

Here are some of the innovative aspects

Layering

I’d argue the multi repo layering of resources is an innovation. Which naturally leads to “why?” It is reasonable:

most people don’t have these needs (disparate set of issues scattered across organisations rather than a central store)
most folks need central Auth and ACLs

With a centralised server deployed, we can have both, as it were.

Storage

Innovative is not the right word, storing the documents as folders of json files is in no way innovative. The original plan was to use git-bug which would have the additional benefit of tying change authorship to git identities, at the cost of making the data model significantly more complex and hard to see.

I think I’d still like to have that, eventually. Or at minimum to hide the files and commits in the same low visibility way that git-bug does, in order to make it’s use less intrusive.

Federation via Git

This project is a very, very different take on federated issues than, say, the efforts in forges to do ActivityPub based federation. I look forward to those being generally available. I wanted to actively avoid always online processes (I’m an old fashioned sysadmin after all), and instead to focus on alternative ideas.

This approach comes with some compromises, namely that within a repo, there is generally no way to have differing levels of access.

That has to either be done through

multiple repos, one per ACL role, and issues stored in their appropriate location, cross referencing each other everywhere
server side ACL where we implement a non-git based backend for notes, where the server can enforce ACLs, and all requests have to go through http APIs.

Non-Innovations

some choices were very standard

Notion’s Data Model

We have essentially reimplemented something very Notion shaped, but I’d argue, poorly. Given how tied I was to the data format (json, on disk, in git repos) it means the resulting system isn’t exactly fast. In fact one of our biggest optimisations was “just shove it all in a sqlite for when you actually have to query”.

A lot of the system could be significantly improved by sticking it in one single centralised database with a reasonable schema. Of course that defeats the “federated” ideas where a user can make up whatever subset of repos they like to produce a view customised for them. That could be achieved through a bunch of individual sqlite databases that are overlayed / queried in some fashion. There are options that are only mildly horrifying.

Storage is a question then, I still wanted the git repo aspect and blobs in git, conflict resolution… yea nah.

So we have a notion like model. At which point it became clear how much of notions usability is in it’s UI and UX. It makes sense why it is so slow and JavaScript-filled, because if you want to be able to, within a note, dynamically add a new field to all of those notes via their (db/template), and then render a nice picker, it makes sense to move that UI calculation into a giant morass of JavaScript that needs to run constantly to render even one letter.

Future Work

at this point, I am not giving up on this, though I think there are a number of sharp edges to be resolved before anyone else uses it

most users probably don’t want to write SQL, that needs to be hidden behind a better UX
The conflict resolution story should be better than “then you go into git and fix it”
Editable tables is pretty close to a “must”
Tracking mentions is mandatory and currently missing.
Integrations with external systems would be useful. At bare minimum, replacing URLs with some basic link preview information would be useful. Especially for Google docs, the notion behaviour of replacing a URL with a logo/document title is useful. We need the same for GitHub, also to fetch the open/close state of issues.