exo-notes/content/posts/stream-0x03.md

Title: Stream 0x03 Slug: stream-0x03 Date: 2022-03-03 Modified: 2022-03-03 22:03 PST Category: Tags: stream Authors: kyle Summary: Stream notes for tonight's stream.

Tonight's work focused on adding in a mirror between local storage and a remote S3 (Minio) instance. The basic flow goes something like:

1. Fetch the object described
2. Store the object locally
3. Throw the object ID onto the work queue
4. The work queue pulls an item off the queue.
5. Load the object from local storage.
6. If there's no error, go look for the next item.
7. If the job doesn't have a backoff set, initialize a new backoff.
8. Launch a goroutine that waits as long as the backoff says before putting the item back on the work queue.

Next stream (2021/03/08), we'll look at designing the artifact, maybe working on some of the protobuf definitions.

References

• cloudflare/backoff - software I wrote at a previous job.