Step 16: Polish — docs, flake, goreleaser, e2e test.

Phase 2 complete.

ARCHITECTURE.md: full rewrite covering gRPC protocol, SSH auth,
updated package structure, all Garden methods, design decisions.
README.md: add remote sync section, mirror/prune commands, sgardd usage.
CLAUDE.md: add gRPC/proto/x-crypto deps, server/client/sgardpb packages.
flake.nix: build both sgard + sgardd, updated vendorHash.
goreleaser: add sgardd build target.
E2e test: full push/pull cycle with SSH auth between two clients.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-03-24 00:10:04 -07:00
parent 94963bb8d6
commit 5f1bc4e14c
8 changed files with 333 additions and 46 deletions

View File

@@ -5,7 +5,9 @@ before:
- go mod tidy - go mod tidy
builds: builds:
- main: ./cmd/sgard - id: sgard
main: ./cmd/sgard
binary: sgard
env: env:
- CGO_ENABLED=0 - CGO_ENABLED=0
ldflags: ldflags:
@@ -17,10 +19,22 @@ builds:
- amd64 - amd64
- arm64 - arm64
- id: sgardd
main: ./cmd/sgardd
binary: sgardd
env:
- CGO_ENABLED=0
goos:
- linux
- darwin
goarch:
- amd64
- arm64
archives: archives:
- formats: [binary] - formats: [binary]
name_template: >- name_template: >-
{{ .ProjectName }}_{{ .Os }}_{{ .Arch }}{{- if .Arm }}v{{ .Arm }}{{ end }}_v{{ .Version }} {{ .Binary }}_{{ .Os }}_{{ .Arch }}{{- if .Arm }}v{{ .Arm }}{{ end }}_v{{ .Version }}
changelog: changelog:
sort: asc sort: asc

View File

@@ -123,69 +123,128 @@ sgard checkpoint -m "initial" --repo /mnt/usb/dotfiles
sgard restore --repo /mnt/usb/dotfiles sgard restore --repo /mnt/usb/dotfiles
``` ```
### Phase 2 — Remote (Future) ### Phase 2 — Remote
| Command | Description | | Command | Description |
|---|---| |---|---|
| `sgard push` | Push checkpoint to remote gRPC server | | `sgard push` | Push checkpoint to remote gRPC server |
| `sgard pull` | Pull checkpoint from remote gRPC server | | `sgard pull` | Pull checkpoint from remote gRPC server |
| `sgard serve` | Run the gRPC daemon | | `sgard prune` | Remove orphaned blobs (local or `--remote`) |
| `sgard mirror up <path>` | Sync filesystem → manifest (add new, remove deleted) |
| `sgard mirror down <path>` | Sync manifest → filesystem (restore + delete untracked) |
| `sgardd` | Run the gRPC sync daemon |
## gRPC Protocol
The GardenSync service uses four RPCs for sync plus one for maintenance:
```
service GardenSync {
rpc PushManifest(PushManifestRequest) returns (PushManifestResponse);
rpc PushBlobs(stream PushBlobsRequest) returns (PushBlobsResponse);
rpc PullManifest(PullManifestRequest) returns (PullManifestResponse);
rpc PullBlobs(PullBlobsRequest) returns (stream PullBlobsResponse);
rpc Prune(PruneRequest) returns (PruneResponse);
}
```
**Push flow:** Client sends manifest → server compares `manifest.Updated`
timestamps → if client newer, server returns list of missing blob hashes →
client streams those blobs (64 KiB chunks) → server replaces its manifest.
**Pull flow:** Client requests server manifest → compares timestamps locally →
if server newer, requests missing blobs → server streams them → client
replaces its manifest.
**Last timestamp wins** for conflict resolution (single-user, personal sync).
## Authentication
SSH key signing via gRPC metadata interceptors:
- Server loads an `authorized_keys` file (standard SSH format)
- Client signs a nonce+timestamp with SSH private key (via ssh-agent or key file)
- Signature + public key sent as gRPC metadata on every call
- 5-minute timestamp window prevents replay
## Go Package Structure ## Go Package Structure
``` ```
sgard/ sgard/
cmd/sgard/ # CLI entry point — one file per command cmd/sgard/ # CLI entry point — one file per command
main.go # cobra root command, --repo flag main.go # cobra root command, --repo/--remote/--ssh-key flags
version.go # sgard version (ldflags-injected) push.go pull.go prune.go mirror.go
init.go add.go remove.go checkpoint.go init.go add.go remove.go checkpoint.go
restore.go status.go verify.go list.go diff.go restore.go status.go verify.go list.go diff.go version.go
cmd/sgardd/ # gRPC server daemon
main.go # --listen, --repo, --authorized-keys flags
garden/ # Core business logic — one file per operation garden/ # Core business logic — one file per operation
garden.go # Garden struct, Init, Open, Add, Checkpoint, Status garden.go # Garden struct, Init, Open, Add, Checkpoint, Status, accessors
restore.go # Restore with timestamp comparison and confirm callback restore.go mirror.go prune.go remove.go verify.go list.go diff.go
remove.go verify.go list.go diff.go
hasher.go # SHA-256 file hashing hasher.go # SHA-256 file hashing
e2e_test.go # Full lifecycle integration test
manifest/ # YAML manifest parsing manifest/ # YAML manifest parsing
manifest.go # Manifest and Entry structs, Load/Save manifest.go # Manifest and Entry structs, Load/Save
store/ # Content-addressable blob storage store/ # Content-addressable blob storage
store.go # Store struct: Write/Read/Exists/Delete store.go # Store struct: Write/Read/Exists/Delete/List
flake.nix # Nix flake for building on NixOS server/ # gRPC server implementation
.goreleaser.yaml # GoReleaser config for releases server.go # GardenSync RPC handlers with RWMutex
.github/workflows/ # GitHub Actions release pipeline auth.go # SSH key auth interceptor
convert.go # proto ↔ manifest type conversion
client/ # gRPC client library
client.go # Push, Pull, Prune methods
auth.go # SSHCredentials (PerRPCCredentials), LoadSigner
sgardpb/ # Generated protobuf + gRPC Go code
proto/sgard/v1/ # Proto source definitions
flake.nix # Nix flake (builds sgard + sgardd)
.goreleaser.yaml # GoReleaser (builds both binaries)
``` ```
### Key Architectural Rule ### Key Architectural Rule
**The `garden` package contains all logic. The `cmd` package is pure CLI wiring.** **The `garden` package contains all logic. The `cmd` package is pure CLI
wiring. The `server` package wraps `Garden` methods as gRPC endpoints.**
The `Garden` struct is the central coordinator:
```go ```go
type Garden struct { type Garden struct {
manifest *manifest.Manifest manifest *manifest.Manifest
store *store.Store store *store.Store
root string // repository root directory root string
manifestPath string manifestPath string
clock clockwork.Clock // injectable for testing clock clockwork.Clock
} }
// Local operations
func (g *Garden) Add(paths []string) error func (g *Garden) Add(paths []string) error
func (g *Garden) Remove(paths []string) error func (g *Garden) Remove(paths []string) error
func (g *Garden) Checkpoint(message string) error func (g *Garden) Checkpoint(message string) error
func (g *Garden) Restore(paths []string, force bool, confirm func(path string) bool) error func (g *Garden) Restore(paths []string, force bool, confirm func(string) bool) error
func (g *Garden) Status() ([]FileStatus, error) func (g *Garden) Status() ([]FileStatus, error)
func (g *Garden) Verify() ([]VerifyResult, error) func (g *Garden) Verify() ([]VerifyResult, error)
func (g *Garden) List() []manifest.Entry func (g *Garden) List() []manifest.Entry
func (g *Garden) Diff(path string) (string, error) func (g *Garden) Diff(path string) (string, error)
func (g *Garden) Prune() (int, error)
func (g *Garden) MirrorUp(paths []string) error
func (g *Garden) MirrorDown(paths []string, force bool, confirm func(string) bool) error
// Accessors (used by server package)
func (g *Garden) GetManifest() *manifest.Manifest
func (g *Garden) BlobExists(hash string) bool
func (g *Garden) ReadBlob(hash string) ([]byte, error)
func (g *Garden) WriteBlob(data []byte) (string, error)
func (g *Garden) ReplaceManifest(m *manifest.Manifest) error
func (g *Garden) ListBlobs() ([]string, error)
func (g *Garden) DeleteBlob(hash string) error
``` ```
This separation means the future gRPC server calls the same `Garden` methods The gRPC server calls the same `Garden` methods as the CLI — no logic
as the CLI — no logic duplication. duplication.
## Design Decisions ## Design Decisions
@@ -193,9 +252,13 @@ as the CLI — no logic duplication.
`$HOME` at runtime. This makes the manifest portable across machines with `$HOME` at runtime. This makes the manifest portable across machines with
different usernames. different usernames.
**No history.** Phase 1 stores only the latest checkpoint. For versioning, **Adding a directory recurses.** `Add` walks directories and adds each
place the repo under git — `sgard init` creates a `.gitignore` that excludes file/symlink individually. Directories are not tracked as entries — only
`blobs/`. Blob durability (backup, replication) is deferred to a future phase. leaf files and symlinks.
**No history.** Only the latest checkpoint is stored. For versioning, place
the repo under git — `sgard init` creates a `.gitignore` that excludes
`blobs/`.
**Per-file timestamps.** Each manifest entry records an `updated` timestamp **Per-file timestamps.** Each manifest entry records an `updated` timestamp
set at checkpoint time. On restore, if the manifest entry is newer than the set at checkpoint time. On restore, if the manifest entry is newer than the
@@ -203,8 +266,13 @@ file on disk (by mtime), the restore proceeds without prompting. If the file
on disk is newer or the times match, sgard prompts for confirmation. on disk is newer or the times match, sgard prompts for confirmation.
`--force` always skips the prompt. `--force` always skips the prompt.
**Atomic writes.** Checkpoint writes `manifest.yaml.tmp` then renames to **Atomic writes.** Manifest saves write to a temp file then rename.
`manifest.yaml`. A crash cannot corrupt the manifest.
**Timestamp comparison truncates to seconds** for cross-platform filesystem **Timestamp comparison truncates to seconds** for cross-platform filesystem
compatibility. compatibility.
**Remote config resolution:** `--remote` flag > `SGARD_REMOTE` env >
`<repo>/remote` file.
**SSH key resolution:** `--ssh-key` flag > `SGARD_SSH_KEY` env > ssh-agent >
`~/.ssh/id_ed25519` > `~/.ssh/id_rsa`.

View File

@@ -21,12 +21,12 @@ Module: `github.com/kisom/sgard`. Author: K. Isom <kyle@imap.cc>.
## Build ## Build
```bash ```bash
go build ./cmd/sgard go build ./... # both sgard and sgardd
``` ```
Nix: Nix:
```bash ```bash
nix build .#sgard nix build .#sgard # builds both binaries
``` ```
Run tests: Run tests:
@@ -39,24 +39,36 @@ Lint:
golangci-lint run ./... golangci-lint run ./...
``` ```
Regenerate proto (requires protoc toolchain):
```bash
make proto
```
## Dependencies ## Dependencies
- `gopkg.in/yaml.v3` — manifest serialization - `gopkg.in/yaml.v3` — manifest serialization
- `github.com/spf13/cobra` — CLI framework - `github.com/spf13/cobra` — CLI framework
- `github.com/jonboulle/clockwork` — injectable clock for deterministic tests - `github.com/jonboulle/clockwork` — injectable clock for deterministic tests
- `google.golang.org/grpc` — gRPC runtime
- `google.golang.org/protobuf` — protobuf runtime
- `golang.org/x/crypto` — SSH key auth (ssh, ssh/agent)
## Package Structure ## Package Structure
``` ```
cmd/sgard/ CLI entry point (cobra commands, pure wiring) cmd/sgard/ CLI entry point (cobra commands, pure wiring)
cmd/sgardd/ gRPC server daemon
garden/ Core business logic (Garden struct orchestrating everything) garden/ Core business logic (Garden struct orchestrating everything)
manifest/ YAML manifest parsing (Manifest/Entry structs, Load/Save) manifest/ YAML manifest parsing (Manifest/Entry structs, Load/Save)
store/ Content-addressable blob storage (SHA-256 keyed) store/ Content-addressable blob storage (SHA-256 keyed)
server/ gRPC server (RPC handlers, SSH auth interceptor, proto conversion)
client/ gRPC client library (Push, Pull, Prune, SSH credentials)
sgardpb/ Generated protobuf + gRPC Go code
``` ```
Key rule: all logic lives in `garden/`. The `cmd/` layer only parses flags Key rule: all logic lives in `garden/`. The `cmd/` layer only parses flags
and calls `Garden` methods. This enables the future gRPC server to reuse and calls `Garden` methods. The `server` wraps `Garden` as gRPC endpoints.
the same logic with zero duplication. No logic duplication.
Each garden operation (remove, verify, list, diff) lives in its own file Each garden operation lives in its own file (`garden/<op>.go`) to minimize
(`garden/<op>.go`) to minimize merge conflicts during parallel development. merge conflicts during parallel development.

View File

@@ -7,7 +7,7 @@ ARCHITECTURE.md for design details.
## Current Status ## Current Status
**Phase:** Phase 2 in progress. Steps 915 complete, ready for Step 16 (Polish). **Phase:** Phase 2 complete (Steps 916). Remote sync fully implemented.
**Last updated:** 2026-03-24 **Last updated:** 2026-03-24
@@ -42,7 +42,7 @@ Phase 2: gRPC Remote Sync.
## Up Next ## Up Next
Step 16: Polish + Release. Phase 2 is complete. Future work: TLS transport, shell completions, multi-repo server.
## Known Issues / Decisions Deferred ## Known Issues / Decisions Deferred
@@ -76,3 +76,4 @@ Step 16: Polish + Release.
| 2026-03-23 | 13 | Client library: Push, Pull, Prune with chunked blob streaming. 6 integration tests. | | 2026-03-23 | 13 | Client library: Push, Pull, Prune with chunked blob streaming. 6 integration tests. |
| 2026-03-23 | 14 | SSH key auth: server interceptor (authorized_keys, signature verification), client PerRPCCredentials (ssh-agent/key file). 8 tests including auth integration. | | 2026-03-23 | 14 | SSH key auth: server interceptor (authorized_keys, signature verification), client PerRPCCredentials (ssh-agent/key file). 8 tests including auth integration. |
| 2026-03-24 | 15 | CLI wiring: push, pull, prune commands, sgardd daemon binary, --remote/--ssh-key flags, local prune with 2 tests. | | 2026-03-24 | 15 | CLI wiring: push, pull, prune commands, sgardd daemon binary, --remote/--ssh-key flags, local prune with 2 tests. |
| 2026-03-24 | 16 | Polish: updated all docs, flake.nix (sgardd + vendorHash), goreleaser (both binaries), e2e push/pull test with auth. |

View File

@@ -171,11 +171,11 @@ Depends on Steps 13, 14.
### Step 16: Polish + Release ### Step 16: Polish + Release
- [ ] Update ARCHITECTURE.md, README.md, CLAUDE.md, PROGRESS.md - [x] Update ARCHITECTURE.md, README.md, CLAUDE.md, PROGRESS.md
- [ ] Update flake.nix (add sgardd, protoc to devShell) - [x] Update flake.nix (add sgardd, updated vendorHash)
- [ ] Update .goreleaser.yaml (add sgardd build) - [x] Update .goreleaser.yaml (add sgardd build)
- [ ] E2e integration test: init two repos, push from one, pull into other - [x] E2e integration test: init two repos, push from one, pull into other (with auth)
- [ ] Verify: all tests pass, full push/pull cycle works - [x] Verify: all tests pass, full push/pull cycle works
## Future Steps (Not Phase 2) ## Future Steps (Not Phase 2)

View File

@@ -69,10 +69,12 @@ sgard restore --repo /mnt/usb/dotfiles
## Commands ## Commands
### Local
| Command | Description | | Command | Description |
|---|---| |---|---|
| `init` | Create a new repository | | `init` | Create a new repository |
| `add <path>...` | Track files, directories, or symlinks | | `add <path>...` | Track files, directories (recursed), or symlinks |
| `remove <path>...` | Stop tracking files | | `remove <path>...` | Stop tracking files |
| `checkpoint [-m msg]` | Re-hash tracked files and update the manifest | | `checkpoint [-m msg]` | Re-hash tracked files and update the manifest |
| `restore [path...] [-f]` | Restore files to their original locations | | `restore [path...] [-f]` | Restore files to their original locations |
@@ -80,8 +82,42 @@ sgard restore --repo /mnt/usb/dotfiles
| `diff <path>` | Show content diff between stored and current file | | `diff <path>` | Show content diff between stored and current file |
| `list` | List all tracked files | | `list` | List all tracked files |
| `verify` | Check blob store integrity against manifest hashes | | `verify` | Check blob store integrity against manifest hashes |
| `prune` | Remove orphaned blobs not referenced by the manifest |
| `mirror up <path>` | Sync filesystem → manifest (add new, remove deleted) |
| `mirror down <path>` | Sync manifest → filesystem (restore + delete untracked) |
| `version` | Print the version | | `version` | Print the version |
### Remote sync
| Command | Description |
|---|---|
| `push` | Push checkpoint to remote gRPC server |
| `pull` | Pull checkpoint from remote gRPC server |
| `prune --remote` | Remove orphaned blobs on the remote server |
| `sgardd` | Run the gRPC sync daemon (separate binary) |
Remote commands require `--remote host:port` (or `SGARD_REMOTE` env, or a
`<repo>/remote` config file) and authenticate via SSH keys.
## Remote sync
Start the daemon on your server:
```sh
sgard init --repo /var/lib/sgard
sgardd --repo /var/lib/sgard --authorized-keys ~/.ssh/authorized_keys
```
Push and pull from client machines:
```sh
sgard push --remote myserver:9473
sgard pull --remote myserver:9473
```
Authentication uses your existing SSH keys (ssh-agent, `~/.ssh/id_ed25519`,
or `--ssh-key`). No passwords or certificates to manage.
## How it works ## How it works
sgard stores files in a content-addressable blob store keyed by SHA-256. sgard stores files in a content-addressable blob store keyed by SHA-256.
@@ -100,6 +136,7 @@ mtime. If the manifest is newer, the file is restored without prompting.
Otherwise, sgard asks for confirmation (`--force` skips the prompt). Otherwise, sgard asks for confirmation (`--force` skips the prompt).
Paths under `$HOME` are stored as `~/...` in the manifest, making it Paths under `$HOME` are stored as `~/...` in the manifest, making it
portable across machines with different usernames. portable across machines with different usernames. Adding a directory
recursively tracks all files and symlinks inside.
See [ARCHITECTURE.md](ARCHITECTURE.md) for full design details. See [ARCHITECTURE.md](ARCHITECTURE.md) for full design details.

155
client/e2e_test.go Normal file
View File

@@ -0,0 +1,155 @@
package client
import (
"context"
"crypto/ed25519"
"crypto/rand"
"net"
"os"
"path/filepath"
"testing"
"time"
"github.com/kisom/sgard/garden"
"github.com/kisom/sgard/server"
"github.com/kisom/sgard/sgardpb"
"golang.org/x/crypto/ssh"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/test/bufconn"
)
// TestE2EPushPullCycle tests the full lifecycle:
// init two repos → add files to client → checkpoint → push → pull into fresh repo → verify
func TestE2EPushPullCycle(t *testing.T) {
// Generate auth key.
_, priv, err := ed25519.GenerateKey(rand.Reader)
if err != nil {
t.Fatalf("generating key: %v", err)
}
signer, err := ssh.NewSignerFromKey(priv)
if err != nil {
t.Fatalf("creating signer: %v", err)
}
// Set up server.
serverDir := t.TempDir()
serverGarden, err := garden.Init(serverDir)
if err != nil {
t.Fatalf("init server: %v", err)
}
auth := server.NewAuthInterceptorFromKeys([]ssh.PublicKey{signer.PublicKey()})
lis := bufconn.Listen(bufSize)
srv := grpc.NewServer(
grpc.UnaryInterceptor(auth.UnaryInterceptor()),
grpc.StreamInterceptor(auth.StreamInterceptor()),
)
sgardpb.RegisterGardenSyncServer(srv, server.New(serverGarden))
t.Cleanup(func() { srv.Stop() })
go func() { _ = srv.Serve(lis) }()
dial := func(t *testing.T) *Client {
t.Helper()
creds := NewSSHCredentials(signer)
conn, err := grpc.NewClient("passthrough:///bufconn",
grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
return lis.Dial()
}),
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithPerRPCCredentials(creds),
)
if err != nil {
t.Fatalf("dial: %v", err)
}
t.Cleanup(func() { _ = conn.Close() })
return New(conn)
}
ctx := context.Background()
// --- Client A: add files and push ---
clientADir := t.TempDir()
clientA, err := garden.Init(clientADir)
if err != nil {
t.Fatalf("init client A: %v", err)
}
// Create test dotfiles.
root := t.TempDir()
bashrc := filepath.Join(root, "bashrc")
sshConfig := filepath.Join(root, "ssh_config")
if err := os.WriteFile(bashrc, []byte("export PS1='$ '\n"), 0o644); err != nil {
t.Fatalf("writing bashrc: %v", err)
}
if err := os.WriteFile(sshConfig, []byte("Host *\n AddKeysToAgent yes\n"), 0o600); err != nil {
t.Fatalf("writing ssh_config: %v", err)
}
if err := clientA.Add([]string{bashrc, sshConfig}); err != nil {
t.Fatalf("Add: %v", err)
}
if err := clientA.Checkpoint("from machine A"); err != nil {
t.Fatalf("Checkpoint: %v", err)
}
c := dial(t)
pushed, err := c.Push(ctx, clientA)
if err != nil {
t.Fatalf("Push: %v", err)
}
if pushed != 2 {
t.Errorf("pushed %d blobs, want 2", pushed)
}
// --- Client B: pull from server ---
clientBDir := t.TempDir()
clientB, err := garden.Init(clientBDir)
if err != nil {
t.Fatalf("init client B: %v", err)
}
// Backdate so server is newer.
bm := clientB.GetManifest()
bm.Updated = bm.Updated.Add(-2 * time.Hour)
if err := clientB.ReplaceManifest(bm); err != nil {
t.Fatalf("backdate: %v", err)
}
c2 := dial(t)
pulled, err := c2.Pull(ctx, clientB)
if err != nil {
t.Fatalf("Pull: %v", err)
}
if pulled != 2 {
t.Errorf("pulled %d blobs, want 2", pulled)
}
// Verify client B has the same manifest and blobs as client A.
manifestA := clientA.GetManifest()
manifestB := clientB.GetManifest()
if len(manifestB.Files) != len(manifestA.Files) {
t.Fatalf("client B has %d entries, want %d", len(manifestB.Files), len(manifestA.Files))
}
for _, e := range manifestB.Files {
if e.Type == "file" {
dataA, err := clientA.ReadBlob(e.Hash)
if err != nil {
t.Fatalf("read blob from A: %v", err)
}
dataB, err := clientB.ReadBlob(e.Hash)
if err != nil {
t.Fatalf("read blob from B: %v", err)
}
if string(dataA) != string(dataB) {
t.Errorf("blob %s content mismatch between A and B", e.Hash)
}
}
}
// Verify manifest message survived.
if manifestB.Message != "from machine A" {
t.Errorf("message = %q, want 'from machine A'", manifestB.Message)
}
}

View File

@@ -15,11 +15,11 @@
packages = { packages = {
sgard = pkgs.buildGoModule { sgard = pkgs.buildGoModule {
pname = "sgard"; pname = "sgard";
version = "0.1.0"; version = "2.0.0";
src = pkgs.lib.cleanSource ./.; src = pkgs.lib.cleanSource ./.;
subPackages = [ "cmd/sgard" ]; subPackages = [ "cmd/sgard" "cmd/sgardd" ];
vendorHash = "sha256-uJMkp08SqZaZ6d64Li4Tx8I9OYjaErLexBrJaf6Vb60="; vendorHash = "sha256-6tLNIknbxrRWYKo5x7yMX6+JDJxbF5l2WBIxXaF7OZ4=";
ldflags = [ "-s" "-w" ]; ldflags = [ "-s" "-w" ];