Step 16: Polish — docs, flake, goreleaser, e2e test.

Phase 2 complete. ARCHITECTURE.md: full rewrite covering gRPC protocol, SSH auth, updated package structure, all Garden methods, design decisions. README.md: add remote sync section, mirror/prune commands, sgardd usage. CLAUDE.md: add gRPC/proto/x-crypto deps, server/client/sgardpb packages. flake.nix: build both sgard + sgardd, updated vendorHash. goreleaser: add sgardd build target. E2e test: full push/pull cycle with SSH auth between two clients. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-24 00:10:04 -07:00
parent 94963bb8d6
commit 5f1bc4e14c
8 changed files with 333 additions and 46 deletions
--- a/.goreleaser.yaml
+++ b/.goreleaser.yaml
@@ -5,7 +5,9 @@ before:
    - go mod tidy

 builds:
-  - main: ./cmd/sgard
+  - id: sgard
+    main: ./cmd/sgard
+    binary: sgard
    env:
      - CGO_ENABLED=0
    ldflags:
@@ -17,10 +19,22 @@ builds:
      - amd64
      - arm64

+  - id: sgardd
+    main: ./cmd/sgardd
+    binary: sgardd
+    env:
+      - CGO_ENABLED=0
+    goos:
+      - linux
+      - darwin
+    goarch:
+      - amd64
+      - arm64
+
 archives:
  - formats: [binary]
    name_template: >-
-      {{ .ProjectName }}_{{ .Os }}_{{ .Arch }}{{- if .Arm }}v{{ .Arm }}{{ end }}_v{{ .Version }}
+      {{ .Binary }}_{{ .Os }}_{{ .Arch }}{{- if .Arm }}v{{ .Arm }}{{ end }}_v{{ .Version }}

 changelog:
  sort: asc
--- a/ARCHITECTURE.md
+++ b/ARCHITECTURE.md
@@ -123,69 +123,128 @@ sgard checkpoint -m "initial" --repo /mnt/usb/dotfiles
 sgard restore --repo /mnt/usb/dotfiles
 ```

-### Phase 2 — Remote (Future)
+### Phase 2 — Remote

 | Command | Description |
 |---|---|
 | `sgard push` | Push checkpoint to remote gRPC server |
 | `sgard pull` | Pull checkpoint from remote gRPC server |
-| `sgard serve` | Run the gRPC daemon |
+| `sgard prune` | Remove orphaned blobs (local or `--remote`) |
+| `sgard mirror up <path>` | Sync filesystem → manifest (add new, remove deleted) |
+| `sgard mirror down <path>` | Sync manifest → filesystem (restore + delete untracked) |
+| `sgardd` | Run the gRPC sync daemon |
+
+## gRPC Protocol
+
+The GardenSync service uses four RPCs for sync plus one for maintenance:
+
+```
+service GardenSync {
+  rpc PushManifest(PushManifestRequest) returns (PushManifestResponse);
+  rpc PushBlobs(stream PushBlobsRequest) returns (PushBlobsResponse);
+  rpc PullManifest(PullManifestRequest) returns (PullManifestResponse);
+  rpc PullBlobs(PullBlobsRequest) returns (stream PullBlobsResponse);
+  rpc Prune(PruneRequest) returns (PruneResponse);
+}
+```
+
+**Push flow:** Client sends manifest → server compares `manifest.Updated`
+timestamps → if client newer, server returns list of missing blob hashes →
+client streams those blobs (64 KiB chunks) → server replaces its manifest.
+
+**Pull flow:** Client requests server manifest → compares timestamps locally →
+if server newer, requests missing blobs → server streams them → client
+replaces its manifest.
+
+**Last timestamp wins** for conflict resolution (single-user, personal sync).
+
+## Authentication
+
+SSH key signing via gRPC metadata interceptors:
+- Server loads an `authorized_keys` file (standard SSH format)
+- Client signs a nonce+timestamp with SSH private key (via ssh-agent or key file)
+- Signature + public key sent as gRPC metadata on every call
+- 5-minute timestamp window prevents replay

 ## Go Package Structure

 ```
 sgard/
  cmd/sgard/              # CLI entry point — one file per command
-    main.go               # cobra root command, --repo flag
-    version.go            # sgard version (ldflags-injected)
+    main.go               # cobra root command, --repo/--remote/--ssh-key flags
+    push.go pull.go prune.go mirror.go
    init.go add.go remove.go checkpoint.go
-    restore.go status.go verify.go list.go diff.go
+    restore.go status.go verify.go list.go diff.go version.go
+
+  cmd/sgardd/             # gRPC server daemon
+    main.go               # --listen, --repo, --authorized-keys flags

  garden/                 # Core business logic — one file per operation
-    garden.go             # Garden struct, Init, Open, Add, Checkpoint, Status
-    restore.go            # Restore with timestamp comparison and confirm callback
-    remove.go verify.go list.go diff.go
+    garden.go             # Garden struct, Init, Open, Add, Checkpoint, Status, accessors
+    restore.go mirror.go prune.go remove.go verify.go list.go diff.go
    hasher.go             # SHA-256 file hashing
-    e2e_test.go           # Full lifecycle integration test

  manifest/               # YAML manifest parsing
    manifest.go           # Manifest and Entry structs, Load/Save

  store/                  # Content-addressable blob storage
-    store.go              # Store struct: Write/Read/Exists/Delete
+    store.go              # Store struct: Write/Read/Exists/Delete/List

-  flake.nix               # Nix flake for building on NixOS
-  .goreleaser.yaml        # GoReleaser config for releases
-  .github/workflows/      # GitHub Actions release pipeline
+  server/                 # gRPC server implementation
+    server.go             # GardenSync RPC handlers with RWMutex
+    auth.go               # SSH key auth interceptor
+    convert.go            # proto ↔ manifest type conversion
+
+  client/                 # gRPC client library
+    client.go             # Push, Pull, Prune methods
+    auth.go               # SSHCredentials (PerRPCCredentials), LoadSigner
+
+  sgardpb/                # Generated protobuf + gRPC Go code
+  proto/sgard/v1/         # Proto source definitions
+
+  flake.nix               # Nix flake (builds sgard + sgardd)
+  .goreleaser.yaml        # GoReleaser (builds both binaries)
 ```

 ### Key Architectural Rule

-**The `garden` package contains all logic. The `cmd` package is pure CLI wiring.**
-
-The `Garden` struct is the central coordinator:
+**The `garden` package contains all logic. The `cmd` package is pure CLI
+wiring. The `server` package wraps `Garden` methods as gRPC endpoints.**

 ```go
 type Garden struct {
    manifest     *manifest.Manifest
    store        *store.Store
-    root         string              // repository root directory
+    root         string
    manifestPath string
-    clock        clockwork.Clock     // injectable for testing
+    clock        clockwork.Clock
 }

+// Local operations
 func (g *Garden) Add(paths []string) error
 func (g *Garden) Remove(paths []string) error
 func (g *Garden) Checkpoint(message string) error
-func (g *Garden) Restore(paths []string, force bool, confirm func(path string) bool) error
+func (g *Garden) Restore(paths []string, force bool, confirm func(string) bool) error
 func (g *Garden) Status() ([]FileStatus, error)
 func (g *Garden) Verify() ([]VerifyResult, error)
 func (g *Garden) List() []manifest.Entry
 func (g *Garden) Diff(path string) (string, error)
+func (g *Garden) Prune() (int, error)
+func (g *Garden) MirrorUp(paths []string) error
+func (g *Garden) MirrorDown(paths []string, force bool, confirm func(string) bool) error
+
+// Accessors (used by server package)
+func (g *Garden) GetManifest() *manifest.Manifest
+func (g *Garden) BlobExists(hash string) bool
+func (g *Garden) ReadBlob(hash string) ([]byte, error)
+func (g *Garden) WriteBlob(data []byte) (string, error)
+func (g *Garden) ReplaceManifest(m *manifest.Manifest) error
+func (g *Garden) ListBlobs() ([]string, error)
+func (g *Garden) DeleteBlob(hash string) error
 ```

-This separation means the future gRPC server calls the same `Garden` methods
-as the CLI — no logic duplication.
+The gRPC server calls the same `Garden` methods as the CLI — no logic
+duplication.

 ## Design Decisions

@@ -193,9 +252,13 @@ as the CLI — no logic duplication.
 `$HOME` at runtime. This makes the manifest portable across machines with
 different usernames.

-**No history.** Phase 1 stores only the latest checkpoint. For versioning,
-place the repo under git — `sgard init` creates a `.gitignore` that excludes
-`blobs/`. Blob durability (backup, replication) is deferred to a future phase.
+**Adding a directory recurses.** `Add` walks directories and adds each
+file/symlink individually. Directories are not tracked as entries — only
+leaf files and symlinks.
+
+**No history.** Only the latest checkpoint is stored. For versioning, place
+the repo under git — `sgard init` creates a `.gitignore` that excludes
+`blobs/`.

 **Per-file timestamps.** Each manifest entry records an `updated` timestamp
 set at checkpoint time. On restore, if the manifest entry is newer than the
@@ -203,8 +266,13 @@ file on disk (by mtime), the restore proceeds without prompting. If the file
 on disk is newer or the times match, sgard prompts for confirmation.
 `--force` always skips the prompt.

-**Atomic writes.** Checkpoint writes `manifest.yaml.tmp` then renames to
-`manifest.yaml`. A crash cannot corrupt the manifest.
+**Atomic writes.** Manifest saves write to a temp file then rename.

 **Timestamp comparison truncates to seconds** for cross-platform filesystem
 compatibility.
+
+**Remote config resolution:** `--remote` flag > `SGARD_REMOTE` env >
+`<repo>/remote` file.
+
+**SSH key resolution:** `--ssh-key` flag > `SGARD_SSH_KEY` env > ssh-agent >
+`~/.ssh/id_ed25519` > `~/.ssh/id_rsa`.
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -21,12 +21,12 @@ Module: `github.com/kisom/sgard`. Author: K. Isom <kyle@imap.cc>.
 ## Build

 ```bash
-go build ./cmd/sgard
+go build ./...                   # both sgard and sgardd
 ```

 Nix:
 ```bash
-nix build .#sgard
+nix build .#sgard                # builds both binaries
 ```

 Run tests:
@@ -39,24 +39,36 @@ Lint:
 golangci-lint run ./...
 ```

+Regenerate proto (requires protoc toolchain):
+```bash
+make proto
+```
+
 ## Dependencies

 - `gopkg.in/yaml.v3` — manifest serialization
 - `github.com/spf13/cobra` — CLI framework
 - `github.com/jonboulle/clockwork` — injectable clock for deterministic tests
+- `google.golang.org/grpc` — gRPC runtime
+- `google.golang.org/protobuf` — protobuf runtime
+- `golang.org/x/crypto` — SSH key auth (ssh, ssh/agent)

 ## Package Structure

 ```
 cmd/sgard/    CLI entry point (cobra commands, pure wiring)
+cmd/sgardd/   gRPC server daemon
 garden/       Core business logic (Garden struct orchestrating everything)
 manifest/     YAML manifest parsing (Manifest/Entry structs, Load/Save)
 store/        Content-addressable blob storage (SHA-256 keyed)
+server/       gRPC server (RPC handlers, SSH auth interceptor, proto conversion)
+client/       gRPC client library (Push, Pull, Prune, SSH credentials)
+sgardpb/      Generated protobuf + gRPC Go code
 ```

 Key rule: all logic lives in `garden/`. The `cmd/` layer only parses flags
-and calls `Garden` methods. This enables the future gRPC server to reuse
-the same logic with zero duplication.
+and calls `Garden` methods. The `server` wraps `Garden` as gRPC endpoints.
+No logic duplication.

-Each garden operation (remove, verify, list, diff) lives in its own file
-(`garden/<op>.go`) to minimize merge conflicts during parallel development.
+Each garden operation lives in its own file (`garden/<op>.go`) to minimize
+merge conflicts during parallel development.
--- a/PROGRESS.md
+++ b/PROGRESS.md
@@ -7,7 +7,7 @@ ARCHITECTURE.md for design details.

 ## Current Status

-**Phase:** Phase 2 in progress. Steps 9–15 complete, ready for Step 16 (Polish).
+**Phase:** Phase 2 complete (Steps 9–16). Remote sync fully implemented.

 **Last updated:** 2026-03-24

@@ -42,7 +42,7 @@ Phase 2: gRPC Remote Sync.

 ## Up Next

-Step 16: Polish + Release.
+Phase 2 is complete. Future work: TLS transport, shell completions, multi-repo server.

 ## Known Issues / Decisions Deferred

@@ -76,3 +76,4 @@ Step 16: Polish + Release.
 | 2026-03-23 | 13 | Client library: Push, Pull, Prune with chunked blob streaming. 6 integration tests. |
 | 2026-03-23 | 14 | SSH key auth: server interceptor (authorized_keys, signature verification), client PerRPCCredentials (ssh-agent/key file). 8 tests including auth integration. |
 | 2026-03-24 | 15 | CLI wiring: push, pull, prune commands, sgardd daemon binary, --remote/--ssh-key flags, local prune with 2 tests. |
+| 2026-03-24 | 16 | Polish: updated all docs, flake.nix (sgardd + vendorHash), goreleaser (both binaries), e2e push/pull test with auth. |
--- a/PROJECT_PLAN.md
+++ b/PROJECT_PLAN.md
@@ -171,11 +171,11 @@ Depends on Steps 13, 14.

 ### Step 16: Polish + Release

- [ ] Update ARCHITECTURE.md, README.md, CLAUDE.md, PROGRESS.md
- [ ] Update flake.nix (add sgardd, protoc to devShell)
- [ ] Update .goreleaser.yaml (add sgardd build)
- [ ] E2e integration test: init two repos, push from one, pull into other
- [ ] Verify: all tests pass, full push/pull cycle works
+- [x] Update ARCHITECTURE.md, README.md, CLAUDE.md, PROGRESS.md
+- [x] Update flake.nix (add sgardd, updated vendorHash)
+- [x] Update .goreleaser.yaml (add sgardd build)
+- [x] E2e integration test: init two repos, push from one, pull into other (with auth)
+- [x] Verify: all tests pass, full push/pull cycle works

 ## Future Steps (Not Phase 2)

--- a/README.md
+++ b/README.md
@@ -69,10 +69,12 @@ sgard restore --repo /mnt/usb/dotfiles

 ## Commands

+### Local
+
 | Command | Description |
 |---|---|
 | `init` | Create a new repository |
-| `add <path>...` | Track files, directories, or symlinks |
+| `add <path>...` | Track files, directories (recursed), or symlinks |
 | `remove <path>...` | Stop tracking files |
 | `checkpoint [-m msg]` | Re-hash tracked files and update the manifest |
 | `restore [path...] [-f]` | Restore files to their original locations |
@@ -80,8 +82,42 @@ sgard restore --repo /mnt/usb/dotfiles
 | `diff <path>` | Show content diff between stored and current file |
 | `list` | List all tracked files |
 | `verify` | Check blob store integrity against manifest hashes |
+| `prune` | Remove orphaned blobs not referenced by the manifest |
+| `mirror up <path>` | Sync filesystem → manifest (add new, remove deleted) |
+| `mirror down <path>` | Sync manifest → filesystem (restore + delete untracked) |
 | `version` | Print the version |

+### Remote sync
+
+| Command | Description |
+|---|---|
+| `push` | Push checkpoint to remote gRPC server |
+| `pull` | Pull checkpoint from remote gRPC server |
+| `prune --remote` | Remove orphaned blobs on the remote server |
+| `sgardd` | Run the gRPC sync daemon (separate binary) |
+
+Remote commands require `--remote host:port` (or `SGARD_REMOTE` env, or a
+`<repo>/remote` config file) and authenticate via SSH keys.
+
+## Remote sync
+
+Start the daemon on your server:
+
+```sh
+sgard init --repo /var/lib/sgard
+sgardd --repo /var/lib/sgard --authorized-keys ~/.ssh/authorized_keys
+```
+
+Push and pull from client machines:
+
+```sh
+sgard push --remote myserver:9473
+sgard pull --remote myserver:9473
+```
+
+Authentication uses your existing SSH keys (ssh-agent, `~/.ssh/id_ed25519`,
+or `--ssh-key`). No passwords or certificates to manage.
+
 ## How it works

 sgard stores files in a content-addressable blob store keyed by SHA-256.
@@ -100,6 +136,7 @@ mtime. If the manifest is newer, the file is restored without prompting.
 Otherwise, sgard asks for confirmation (`--force` skips the prompt).

 Paths under `$HOME` are stored as `~/...` in the manifest, making it
-portable across machines with different usernames.
+portable across machines with different usernames. Adding a directory
+recursively tracks all files and symlinks inside.

 See [ARCHITECTURE.md](ARCHITECTURE.md) for full design details.
--- a/client/e2e_test.go
+++ b/client/e2e_test.go
@@ -0,0 +1,155 @@
+package client
+
+import (
+	"context"
+	"crypto/ed25519"
+	"crypto/rand"
+	"net"
+	"os"
+	"path/filepath"
+	"testing"
+	"time"
+
+	"github.com/kisom/sgard/garden"
+	"github.com/kisom/sgard/server"
+	"github.com/kisom/sgard/sgardpb"
+	"golang.org/x/crypto/ssh"
+	"google.golang.org/grpc"
+	"google.golang.org/grpc/credentials/insecure"
+	"google.golang.org/grpc/test/bufconn"
+)
+
+// TestE2EPushPullCycle tests the full lifecycle:
+// init two repos → add files to client → checkpoint → push → pull into fresh repo → verify
+func TestE2EPushPullCycle(t *testing.T) {
+	// Generate auth key.
+	_, priv, err := ed25519.GenerateKey(rand.Reader)
+	if err != nil {
+		t.Fatalf("generating key: %v", err)
+	}
+	signer, err := ssh.NewSignerFromKey(priv)
+	if err != nil {
+		t.Fatalf("creating signer: %v", err)
+	}
+
+	// Set up server.
+	serverDir := t.TempDir()
+	serverGarden, err := garden.Init(serverDir)
+	if err != nil {
+		t.Fatalf("init server: %v", err)
+	}
+
+	auth := server.NewAuthInterceptorFromKeys([]ssh.PublicKey{signer.PublicKey()})
+	lis := bufconn.Listen(bufSize)
+	srv := grpc.NewServer(
+		grpc.UnaryInterceptor(auth.UnaryInterceptor()),
+		grpc.StreamInterceptor(auth.StreamInterceptor()),
+	)
+	sgardpb.RegisterGardenSyncServer(srv, server.New(serverGarden))
+	t.Cleanup(func() { srv.Stop() })
+	go func() { _ = srv.Serve(lis) }()
+
+	dial := func(t *testing.T) *Client {
+		t.Helper()
+		creds := NewSSHCredentials(signer)
+		conn, err := grpc.NewClient("passthrough:///bufconn",
+			grpc.WithContextDialer(func(context.Context, string) (net.Conn, error) {
+				return lis.Dial()
+			}),
+			grpc.WithTransportCredentials(insecure.NewCredentials()),
+			grpc.WithPerRPCCredentials(creds),
+		)
+		if err != nil {
+			t.Fatalf("dial: %v", err)
+		}
+		t.Cleanup(func() { _ = conn.Close() })
+		return New(conn)
+	}
+
+	ctx := context.Background()
+
+	// --- Client A: add files and push ---
+	clientADir := t.TempDir()
+	clientA, err := garden.Init(clientADir)
+	if err != nil {
+		t.Fatalf("init client A: %v", err)
+	}
+
+	// Create test dotfiles.
+	root := t.TempDir()
+	bashrc := filepath.Join(root, "bashrc")
+	sshConfig := filepath.Join(root, "ssh_config")
+	if err := os.WriteFile(bashrc, []byte("export PS1='$ '\n"), 0o644); err != nil {
+		t.Fatalf("writing bashrc: %v", err)
+	}
+	if err := os.WriteFile(sshConfig, []byte("Host *\n  AddKeysToAgent yes\n"), 0o600); err != nil {
+		t.Fatalf("writing ssh_config: %v", err)
+	}
+
+	if err := clientA.Add([]string{bashrc, sshConfig}); err != nil {
+		t.Fatalf("Add: %v", err)
+	}
+	if err := clientA.Checkpoint("from machine A"); err != nil {
+		t.Fatalf("Checkpoint: %v", err)
+	}
+
+	c := dial(t)
+	pushed, err := c.Push(ctx, clientA)
+	if err != nil {
+		t.Fatalf("Push: %v", err)
+	}
+	if pushed != 2 {
+		t.Errorf("pushed %d blobs, want 2", pushed)
+	}
+
+	// --- Client B: pull from server ---
+	clientBDir := t.TempDir()
+	clientB, err := garden.Init(clientBDir)
+	if err != nil {
+		t.Fatalf("init client B: %v", err)
+	}
+	// Backdate so server is newer.
+	bm := clientB.GetManifest()
+	bm.Updated = bm.Updated.Add(-2 * time.Hour)
+	if err := clientB.ReplaceManifest(bm); err != nil {
+		t.Fatalf("backdate: %v", err)
+	}
+
+	c2 := dial(t)
+	pulled, err := c2.Pull(ctx, clientB)
+	if err != nil {
+		t.Fatalf("Pull: %v", err)
+	}
+	if pulled != 2 {
+		t.Errorf("pulled %d blobs, want 2", pulled)
+	}
+
+	// Verify client B has the same manifest and blobs as client A.
+	manifestA := clientA.GetManifest()
+	manifestB := clientB.GetManifest()
+
+	if len(manifestB.Files) != len(manifestA.Files) {
+		t.Fatalf("client B has %d entries, want %d", len(manifestB.Files), len(manifestA.Files))
+	}
+
+	for _, e := range manifestB.Files {
+		if e.Type == "file" {
+			dataA, err := clientA.ReadBlob(e.Hash)
+			if err != nil {
+				t.Fatalf("read blob from A: %v", err)
+			}
+			dataB, err := clientB.ReadBlob(e.Hash)
+			if err != nil {
+				t.Fatalf("read blob from B: %v", err)
+			}
+			if string(dataA) != string(dataB) {
+				t.Errorf("blob %s content mismatch between A and B", e.Hash)
+			}
+		}
+	}
+
+	// Verify manifest message survived.
+	if manifestB.Message != "from machine A" {
+		t.Errorf("message = %q, want 'from machine A'", manifestB.Message)
+	}
+}
--- a/flake.nix
+++ b/flake.nix
@@ -15,11 +15,11 @@
        packages = {
          sgard = pkgs.buildGoModule {
            pname = "sgard";
-            version = "0.1.0";
+            version = "2.0.0";
            src = pkgs.lib.cleanSource ./.;
-            subPackages = [ "cmd/sgard" ];
+            subPackages = [ "cmd/sgard" "cmd/sgardd" ];

-            vendorHash = "sha256-uJMkp08SqZaZ6d64Li4Tx8I9OYjaErLexBrJaf6Vb60=";
+            vendorHash = "sha256-6tLNIknbxrRWYKo5x7yMX6+JDJxbF5l2WBIxXaF7OZ4=";

            ldflags = [ "-s" "-w" ];