8.1 KiB
Title: Functional Spec for the Exocortex Tags: specs
kExocortex is a tool for capturing and retaining knowledge, making it searchable.
This is the initial top-level draft to sort out the high-level vision.
Summary
The more you learn, the harder it is to recall specific things. Fortunately, computers are generally pretty good at remembering things. kExocortex is my attempt at building a knowledge graph for long-term memory.
In addition to having functionality like notetaking systems like Dendron, I'd like to keep track of what I call artifacts. An artifact is a source of some knowledge; it might be a PDF copy of a book, an image, or a URL.
In a perfect world, I would have a local copy of everything with a remote backup. The remote backup lets me restore the exocortex in the event of data loss.
Usage sketches
Research mode
If I am researching a topic, I have a top-level node that contains all the research I'm working on. I can link artifacts to a note, including URLs. One of the reasons it makes sense to attach a URL to a document is that I can reuse them, as well as go back and search URLs based on tags or categories. It would make sense to tag any artifacts with relevant tags from the note once it is saved.
For example, let's say that I am research graphing databases. In Dendron, this
note lives under comp.database.graph. I might find this O'Reilly book on
Neo4J
that discusses graph algorithms. I might link it here, and I might link it
under a Neo4J-specific node. I would store the PDF in an artifact repository,
adding relevant tags (such as "graph-database", "neo4j", "oreilly") and
categorize it under books, PDFs, comp/database/graph/neo4j.
Going forward, if I want to revisit the book, I don't have to find it online again. It's easily accessible from the artifact repository.
The user interface for the knowledge graph should show a list of associated artifacts.
Nodes are also timestamped; I am leaning towards keep track of every time a page was edited (but probably not the edits). If I know I was researching graph databases last week, and I log the URLs I was reading as artifacts, I have a better history of what I was reading.
Reading from a mobile device
Sometimes I'm on my iPad or phone, and I want to save the link I'm reading. I should be able to stash documents, URLs, etc, in the artifact repository. This implies a remote endpoint that I can enter a URL and a tag, and have that entered into the artifact repository later.
Cataloging artifacts
If I've entered a bunch of artifacts, I should be able to see a list of ones that need categorizing or that aren't attached to a node.
Autotagging
The interface should search the text of a note to identify any tags. This brings up an important feature: notes consist of cells, and each cell has a type. The primary use case is to support markdown formatting and code blocks, while not touching the code blocks during autotagging. For example,
---
node: today.2022.02.21
---
I figured out how to get Cayley running in production.
\```
cayleyd --some flag --other flag
\```
The exocortex would see Cayley, identify that as a node, and add the tags for that node to this one. It might see production and add that as a tag, e.g. for ops-related stuff.
Fast capture
I should be able to enter a quick note, which would go under a daily node tree.
Something like quick.2022-02-27.1534.
This would get autotagged. Quick notes might also get a metadata tag indicating whether I went back and integrated them into the rest of the knowledge graph.
One way I could use this might be to text or email a note, or to have a quick capture program on my computer.
Requirements & Assumptions
What should it do? What assumptions are being made about it? What's considered "in scope" and what won't the project try to do?
Does it need to be compatible with any existing solutions or systems?
If it's a daemon, how are you going to manage it?
What are the dependencies that are assumed to be available?
System Design
Major components
The system has two logical components: the artifact repository and the knowledge graph.
Artifact repository
There should be, at a minimum, a local artifact repository. It will have its own tooling and UI for interaction, as well as being linked to the knowledge graph.
Previous prototypes stored artifact metadata in SQLite, and the contents of the artifacts in a blob store. The blob store is a content-addressable system for retrieving arbitrary data. A remote option might use an S3-equivalent like Minio.
Knowledge graph
The knowledge graph stores nodes. The current model stores the graph in SQLite, using an external file sync (e.g. syncthing) to sync the databases across machines.
Data model
Previous prototypes used separate SQLite databases for the artifact repository and the knowledge graph.
Single SQLite database
The concern with a single SQLite database is that it would be accessed by two different systems, causing potential locking issues.
This could be solved by a single unified backend server; this is the preferred approach.
Split SQLite databases
The original prototype split the databases for performance reasons. However, this was based on any empirical evidence.
The major downside to this is that tags and categories are not shared between the artifact repository and the knowledge graph. Categories might make sense for splitting; e.g. an artifact category might be 'PDF' while a node might have the category 'Research'. However, tags should be shared between both systems.
PostgreSQL database
Another option is to to use postgres. This brings a heavy ops cost, while enabling a variety of replication and backup strategies.
Architectural overview
There is a backend server, exod, that will have a gRPC endpoint for
communicating with frontends. The approach allows for a reverse-proxy front end
on a public server over Tailscale for remote devices. It also maintains a local
blob store, the database, and a connection to a remote minio server for backing
up blobs and retrieving missing blobs.
If a standard HTTP API is needed, it can be added in later. One potential use for this is for retrieving blobs (e.g. GET /artifacts/blob/id/...).
Supportability
Failure scenarios
Data corruption
If the data is corrupted locally, a local import from the remote end would restore it. Alternatively, it may be restored from local backups.
If the data is corrupted remotely, a local export to the remote end would restore it.
Platform support
The main program would ideally run on Linux primarily, but I'd like to be able to use it on my Windows desktop too.
Packaging and deployment
Security
The gRPC endpoint should be authenticated. The system is intended to operate over localhost or a local network, so the use of TLS is probably untenable. minica is an option, but then key rotation needs to be built in.
A possible workaround is to only enable authentication (HTTP basic auth will suffice) on the reverse proxy, which will also have TLS.
Project Dependencies
The software should rely on no external sources, except for the software packages that it uses. This can be mitigated with vendoring.
Open Issues
- If I track each time a page was edited, does it make sense to roll this up? e.g. I don't track edits to the second, but maybe to the hour or day.
Milestones
- Specifications a. Write up spec for the artifact repository data structures. b. Write up a spec for the knowledge graph data structures.
- Core systems a. Build the artifact repository server. b. Build the backend for the knowledge graph. c. Build rough CLI interfaces to both.
- Build the user interfaces. a. Simple note taking. b. Artifact upload and searching by tag, content type, title.
Review History
This may not be applicable, but it's usually nice to have someone else sanity check this.
Keep a table of who reviewed the doc and when, for in-person reviews. Consider having at least 1 in-person review.