2022-03-01 05:45:54 +00:00
|
|
|
Title: Stream 0x03
|
|
|
|
|
Slug: stream-0x03
|
2022-03-04 06:02:50 +00:00
|
|
|
Date: 2022-03-03
|
|
|
|
|
Modified: 2022-03-03 22:03 PST
|
2022-03-01 05:45:54 +00:00
|
|
|
Category:
|
|
|
|
|
Tags: stream
|
|
|
|
|
Authors: kyle
|
|
|
|
|
Summary: Stream notes for tonight's stream.
|
|
|
|
|
|
|
|
|
|
Tonight's work focused on adding in a mirror between local storage and
|
|
|
|
|
a remote S3 (Minio) instance. The basic flow goes something like:
|
|
|
|
|
|
|
|
|
|
1. Fetch the object described
|
|
|
|
|
2. Store the object locally
|
|
|
|
|
3. Throw the object ID onto the work queue
|
|
|
|
|
4. The work queue pulls an item off the queue.
|
|
|
|
|
5. Load the object from local storage.
|
|
|
|
|
6. If there's no error, go look for the next item.
|
|
|
|
|
7. If the job doesn't have a backoff set, initialize a new backoff.
|
|
|
|
|
8. Launch a goroutine that waits as long as the backoff says before
|
|
|
|
|
putting the item back on the work queue.
|
|
|
|
|
|
2022-03-04 06:02:50 +00:00
|
|
|
Next stream (2021/03/08), we'll look at designing the artifact, maybe
|
2022-03-01 05:45:54 +00:00
|
|
|
working on some of the protobuf definitions.
|
|
|
|
|
|
|
|
|
|
### References
|
|
|
|
|
|
|
|
|
|
* [cloudflare/backoff](https://github.com/cloudflare/backoff/) - software
|
|
|
|
|
I wrote at a previous job.
|
