Instrument mc-proxy with prometheus/client_golang. New internal/metrics/ package defines counters, gauges, and histograms for connection totals, active connections, firewall blocks by reason, backend dial latency, bytes transferred, L7 HTTP status codes, and L7 policy blocks. Optional [metrics] config section starts a scrape endpoint. Firewall gains BlockedWithReason() to report block cause. L7 handler wraps ResponseWriter to record status codes per hostname. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
716 lines
31 KiB
Markdown
716 lines
31 KiB
Markdown
# ARCHITECTURE.md
|
|
|
|
mc-proxy is a TLS proxy and router for Metacircular Dynamics services. It
|
|
operates in two modes on a per-route basis:
|
|
|
|
- **L4 (passthrough):** Reads the SNI field from the TLS ClientHello and
|
|
proxies the raw TCP stream to the matched backend without terminating TLS.
|
|
This is the default mode and the original behavior.
|
|
- **L7 (terminating):** Terminates the client TLS connection, then reverse
|
|
proxies HTTP/2 (and HTTP/1.1) traffic to the backend. This enables
|
|
HTTP-level features: header inspection, user-agent blocking, request
|
|
routing, and `X-Forwarded-For` injection.
|
|
|
|
A global firewall evaluates every connection before routing. PROXY protocol
|
|
support enables multi-hop deployments where a public edge mc-proxy forwards
|
|
through a private network (e.g., Tailscale) to an origin mc-proxy while
|
|
preserving the real client IP.
|
|
|
|
## Table of Contents
|
|
|
|
1. [System Overview](#system-overview)
|
|
2. [Connection Lifecycle](#connection-lifecycle)
|
|
3. [PROXY Protocol](#proxy-protocol)
|
|
4. [Firewall](#firewall)
|
|
5. [Routing](#routing)
|
|
6. [L7 Proxying](#l7-proxying)
|
|
7. [gRPC Admin API](#grpc-admin-api)
|
|
8. [Configuration](#configuration)
|
|
9. [Storage](#storage)
|
|
10. [Deployment](#deployment)
|
|
11. [Security Model](#security-model)
|
|
12. [Future Work](#future-work)
|
|
|
|
---
|
|
|
|
## System Overview
|
|
|
|
```
|
|
┌──────────────────────────────────────────────┐
|
|
│ mc-proxy │
|
|
Clients ──────┐ │ │
|
|
│ │ ┌───────┐ ┌──────┐ ┌───┐ ┌─────┐ │
|
|
├────▶│ │ PROXY │─▶│ Fire │─▶│SNI│─▶│Route│──┐ │
|
|
│ │ │ proto │ │ wall │ │ │ │Table│ │ │
|
|
├────▶│ └───────┘ └──────┘ └───┘ └─────┘ │ │
|
|
│ │ │ │ RST │ │ │ │
|
|
Clients ──────┘ │ │ ▼ │ │ │ │
|
|
│ (optional) (blocked) │ │ │ │
|
|
│ │ │ │ │
|
|
│ ┌──────────────────────┘ │ │ │
|
|
│ │ L4 route │ │ │
|
|
│ ▼ │ │ │ ┌────────────┐
|
|
│ ┌──────────┐ │ │ │────▶│ Backend A │
|
|
│ │ Raw TCP │───────────────────│────│─────│ │ :8443 (TLS)│
|
|
│ │ Relay │ │ │ │ └────────────┘
|
|
│ └──────────┘ │ │ │
|
|
│ │ │ │
|
|
│ ┌─────────────────────────┘ │ │
|
|
│ │ L7 route │ │
|
|
│ ▼ │ │ ┌────────────┐
|
|
│ ┌──────────┐ ┌──────────┐ │ │────▶│ Backend B │
|
|
│ │ TLS Term │─▶│ HTTP/2 │──────────│─────│ │ :8080 (h2c)│
|
|
│ │ (certs) │ │ Rev Proxy│ │ │ └────────────┘
|
|
│ └──────────┘ └──────────┘ │ │
|
|
└──────────────────────────────────────────────┘
|
|
|
|
Listener 1 (:443) ─┐
|
|
Listener 2 (:8443) ─┼─ Each runs the same pipeline; L4/L7 is per-route
|
|
Listener N (:9443) ─┘
|
|
```
|
|
|
|
Key properties:
|
|
|
|
- **Dual mode, per-route.** Each route is independently configured as L4
|
|
(passthrough) or L7 (terminating). Both modes coexist on the same listener.
|
|
- **TLS-only on ingest.** Non-TLS connections are not supported. If the first
|
|
bytes of a connection are not a TLS ClientHello (after any PROXY protocol
|
|
header), the connection is reset.
|
|
- **Multiple listeners.** A single mc-proxy instance binds to one or more
|
|
ports. Each listener runs the same pipeline.
|
|
- **Global firewall.** Firewall rules apply to all listeners uniformly.
|
|
There are no per-route firewall rules.
|
|
- **PROXY protocol.** Listeners can accept PROXY protocol v1/v2 headers to
|
|
learn the real client IP from an upstream proxy. Routes can send PROXY
|
|
protocol v2 headers to downstream backends or proxies.
|
|
- **No authentication.** mc-proxy is pre-auth infrastructure. It sits in
|
|
front of services that handle their own authentication via MCIAS.
|
|
|
|
---
|
|
|
|
## Connection Lifecycle
|
|
|
|
Every inbound connection follows this sequence. Steps 1-5 are common to all
|
|
connections. The route's mode determines whether the L4 or L7 path is taken.
|
|
|
|
```
|
|
1. ACCEPT Listener accepts TCP connection.
|
|
2. PROXY PROTOCOL If listener has proxy_protocol enabled:
|
|
Parse PROXY v1/v2 header to extract real client IP.
|
|
If malformed or missing → RST, done.
|
|
Otherwise: use TCP source IP directly.
|
|
3. FIREWALL Check source IP (real, from step 2) against blocklists:
|
|
a. Per-IP rate limit check.
|
|
b. IP/CIDR block check.
|
|
c. GeoIP country block check.
|
|
If blocked → RST, done.
|
|
4. SNI EXTRACT Read the TLS ClientHello (without consuming it).
|
|
Extract the SNI hostname.
|
|
If no valid ClientHello or no SNI → RST, done.
|
|
5. ROUTE LOOKUP Match SNI hostname against the route table.
|
|
If no match → RST, done.
|
|
Route determines mode (L4 or L7).
|
|
|
|
── L4 path (passthrough) ──────────────────────────────────────────────
|
|
|
|
6a. BACKEND DIAL Open TCP connection to the matched backend.
|
|
If dial fails → RST, done.
|
|
7a. PROXY HEADER If route has send_proxy_protocol enabled:
|
|
Send PROXY v2 header with real client IP to backend.
|
|
8a. FORWARD Write the buffered ClientHello to the backend.
|
|
9a. RELAY Bidirectional byte copy: client ↔ backend.
|
|
10a. CLOSE Either side closes → half-close propagation → done.
|
|
|
|
── L7 path (terminating) ──────────────────────────────────────────────
|
|
|
|
6b. TLS HANDSHAKE Complete TLS handshake with the client.
|
|
The buffered ClientHello is replayed into crypto/tls
|
|
via a prefixConn wrapper. The server certificate is
|
|
selected from the route's configured cert/key pair
|
|
using tls.Config.GetCertificate keyed on SNI.
|
|
If handshake fails → close, done.
|
|
7b. HTTP PARSE Read HTTP/2 (or HTTP/1.1) request from the TLS conn.
|
|
8b. L7 POLICY Apply HTTP-level policies:
|
|
- User-agent blocking (future)
|
|
- Header inspection (future)
|
|
If blocked → HTTP error response, done.
|
|
9b. BACKEND DIAL Open connection to the matched backend.
|
|
If backend_tls: TLS connection to backend.
|
|
Otherwise: plaintext (h2c or HTTP/1.1).
|
|
If dial fails → HTTP 502, done.
|
|
10b. PROXY HEADER If route has send_proxy_protocol enabled:
|
|
Send PROXY v2 header with real client IP to backend.
|
|
11b. REVERSE PROXY Forward HTTP request to backend, stream response to
|
|
client. 1:1 connection mapping (no pooling).
|
|
12b. CLOSE Connection closed normally or on error.
|
|
```
|
|
|
|
### SNI Extraction
|
|
|
|
The proxy peeks at the initial bytes of the connection without consuming
|
|
them. It parses just enough of the TLS record layer and ClientHello to
|
|
extract the `server_name` extension.
|
|
|
|
- **L4 routes:** The full ClientHello (including the SNI) is forwarded to
|
|
the backend so the backend's TLS handshake proceeds normally.
|
|
- **L7 routes:** The buffered ClientHello is replayed into `crypto/tls.Server`
|
|
via a `prefixConn` wrapper (a `net.Conn` that returns the buffered bytes
|
|
before reading from the underlying connection). The TLS library sees the
|
|
complete ClientHello and performs a normal handshake.
|
|
|
|
If the ClientHello spans multiple TCP segments, the proxy buffers up to
|
|
16 KiB (the maximum TLS record size) before giving up.
|
|
|
|
### Bidirectional Copy (L4)
|
|
|
|
After the backend connection is established, the proxy runs two concurrent
|
|
copy loops (client->backend and backend->client). When either direction
|
|
encounters an EOF or error:
|
|
|
|
1. The write side of the opposite direction is half-closed.
|
|
2. The remaining direction drains to completion.
|
|
3. Both connections are closed.
|
|
|
|
Timeouts apply to the copy phase to prevent idle connections from
|
|
accumulating indefinitely (see [Configuration](#configuration)).
|
|
|
|
---
|
|
|
|
## PROXY Protocol
|
|
|
|
mc-proxy supports [PROXY protocol](https://www.haproxy.org/download/2.9/doc/proxy-protocol.txt)
|
|
for multi-hop deployments where the real client IP must be preserved across
|
|
proxy boundaries.
|
|
|
|
### Use Case: Edge-to-Origin
|
|
|
|
A common deployment has a public-facing mc-proxy on a VPS forwarding
|
|
through Tailscale to a private-network mc-proxy:
|
|
|
|
```
|
|
Client ──→ [edge mc-proxy] ──PROXY v2 + TLS──→ Tailscale ──→ [origin mc-proxy] ──→ backend
|
|
VPS, :443 private network
|
|
proxy_protocol = false proxy_protocol = true
|
|
send_proxy_protocol = true send_proxy_protocol = false
|
|
```
|
|
|
|
Without PROXY protocol, the origin mc-proxy sees the Tailscale peer IP as
|
|
the client address. With it, the real client IP is preserved for firewall
|
|
evaluation, logging, and `X-Forwarded-For` injection on L7 routes.
|
|
|
|
### Receiving (Listener Side)
|
|
|
|
When `proxy_protocol = true` on a listener:
|
|
|
|
1. After accepting the TCP connection, read the PROXY protocol header
|
|
**before** any TLS data.
|
|
2. Accept both v1 (text) and v2 (binary) formats.
|
|
3. Extract the real client IP and port from the header.
|
|
4. Use the real client IP for all subsequent processing (firewall, logging,
|
|
`X-Forwarded-For`).
|
|
5. If the header is malformed, missing, or the connection times out waiting
|
|
for it, reset the connection.
|
|
|
|
When `proxy_protocol = false` (default), the TCP source IP is used directly.
|
|
No PROXY header is expected or tolerated — a connection that starts with a
|
|
PROXY header will fail SNI extraction (it's not a TLS ClientHello) and be
|
|
reset. This prevents a non-PROXY listener from accidentally trusting
|
|
attacker-supplied headers.
|
|
|
|
### Sending (Route Side)
|
|
|
|
When `send_proxy_protocol = true` on a route:
|
|
|
|
1. After connecting to the backend, send a PROXY protocol v2 header before
|
|
any other data.
|
|
2. The header contains the real client IP (from the PROXY header on the
|
|
listener, or the TCP source IP if the listener is not PROXY-enabled).
|
|
3. For L4 routes: the v2 header is sent, then the buffered ClientHello,
|
|
then raw relay.
|
|
4. For L7 routes: the v2 header is sent on the backend connection before
|
|
HTTP traffic begins.
|
|
|
|
mc-proxy always sends v2 (binary format) when `send_proxy_protocol` is
|
|
enabled. v2 is more compact, supports IPv6 natively, and is the modern
|
|
standard.
|
|
|
|
### Header Timeout
|
|
|
|
A hardcoded 5-second timeout applies to reading the PROXY protocol header.
|
|
If the header is not fully received within this window, the connection is
|
|
reset. This prevents slowloris-style attacks on PROXY-enabled listeners.
|
|
|
|
---
|
|
|
|
## Firewall
|
|
|
|
The firewall is a global, ordered rule set evaluated on every new
|
|
connection before SNI extraction. Rules are evaluated in definition order;
|
|
the first matching rule determines the outcome. If no rule matches, the
|
|
connection is **allowed** (default allow -- the firewall is an explicit
|
|
blocklist, not an allowlist).
|
|
|
|
The source IP used for firewall evaluation is the **real client IP** --
|
|
either from a PROXY protocol header (if the listener is PROXY-enabled) or
|
|
the TCP source address.
|
|
|
|
### Rule Types
|
|
|
|
| Config Field | Match Field | Example |
|
|
|--------------|-------------|---------|
|
|
| `blocked_ips` | Source IP address | `"192.0.2.1"` |
|
|
| `blocked_cidrs` | Source IP prefix | `"198.51.100.0/24"` |
|
|
| `blocked_countries` | Source country (ISO 3166-1 alpha-2) | `"KP"`, `"CN"`, `"IN"`, `"IL"` |
|
|
|
|
### Blocked Connection Handling
|
|
|
|
Blocked connections receive a TCP RST. No TLS alert, no HTTP error page, no
|
|
indication of why the connection was refused. This is intentional -- blocked
|
|
sources should receive minimal information.
|
|
|
|
### GeoIP Database
|
|
|
|
mc-proxy uses the [MaxMind GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data)
|
|
free database for country-level IP geolocation. The database file is
|
|
distributed separately from the binary and referenced by path in the
|
|
configuration.
|
|
|
|
The GeoIP database is loaded into memory at startup and can be reloaded
|
|
via `SIGHUP` without restarting the proxy. If the database file is missing
|
|
or unreadable at startup and GeoIP rules are configured, the proxy refuses
|
|
to start.
|
|
|
|
---
|
|
|
|
## Routing
|
|
|
|
Each listener has its own route table mapping SNI hostnames to backend
|
|
addresses. A route entry consists of:
|
|
|
|
| Field | Type | Default | Description |
|
|
|-------|------|---------|-------------|
|
|
| `hostname` | string | *(required)* | Exact SNI hostname to match (e.g. `metacrypt.metacircular.net`) |
|
|
| `backend` | string | *(required)* | Backend address as `host:port` (e.g. `127.0.0.1:8443`) |
|
|
| `mode` | string | `"l4"` | `"l4"` for passthrough, `"l7"` for TLS-terminating reverse proxy |
|
|
| `tls_cert` | string | `""` | Path to TLS certificate file (PEM). Required for L7 routes. |
|
|
| `tls_key` | string | `""` | Path to TLS private key file (PEM). Required for L7 routes. |
|
|
| `backend_tls` | bool | `false` | If true, connect to backend with TLS (re-encrypt). If false, plaintext h2c/HTTP. |
|
|
| `send_proxy_protocol` | bool | `false` | If true, send PROXY protocol v2 header to backend. |
|
|
|
|
Routes are scoped to the listener that accepted the connection. The same
|
|
hostname can appear on different listeners with different backends and
|
|
different modes.
|
|
|
|
### Match Semantics
|
|
|
|
- Hostname matching is **exact** and **case-insensitive** (per RFC 6066,
|
|
SNI hostnames are DNS names and compared case-insensitively).
|
|
- Wildcard matching is not supported in the initial implementation.
|
|
- If duplicate hostnames appear within the same listener, the proxy refuses
|
|
to start.
|
|
- If no route matches an incoming SNI hostname, the connection is reset.
|
|
|
|
### Route Table Source
|
|
|
|
Route tables are persisted in the SQLite database. On first run, they are
|
|
seeded from the TOML configuration. On subsequent runs, the database is
|
|
the source of truth. Routes can be added or removed at runtime via the
|
|
gRPC admin API.
|
|
|
|
### Validation
|
|
|
|
- L7 routes must have `tls_cert` and `tls_key` set and pointing to
|
|
readable files. The proxy validates certificate/key pairs at startup and
|
|
refuses to start if any are invalid.
|
|
- L4 routes must not have `tls_cert` or `tls_key` set (they are ignored
|
|
and produce a warning if present).
|
|
- `backend_tls` is only meaningful for L7 routes. On L4 routes it is
|
|
ignored (the raw TLS stream passes through regardless).
|
|
|
|
---
|
|
|
|
## L7 Proxying
|
|
|
|
L7 routes terminate TLS at the proxy and reverse proxy HTTP traffic to the
|
|
backend. This enables HTTP-level features that are impossible in L4 mode.
|
|
|
|
### TLS Termination
|
|
|
|
The proxy uses `crypto/tls.Server` with a `GetCertificate` callback that
|
|
selects the certificate based on the SNI hostname. Since SNI has already
|
|
been extracted in step 4 of the connection lifecycle, the route (and its
|
|
cert/key) is known before the handshake begins.
|
|
|
|
The `prefixConn` wrapper replays the buffered ClientHello bytes, then reads
|
|
from the underlying `net.Conn`. From `crypto/tls`'s perspective, it is
|
|
reading a normal TCP connection.
|
|
|
|
TLS configuration:
|
|
|
|
- Minimum version: TLS 1.2 (TLS 1.3 preferred).
|
|
- Cipher suites: Go's default secure set (no manual override).
|
|
- ALPN: `h2` and `http/1.1` advertised. The negotiated protocol determines
|
|
whether the connection uses HTTP/2 or HTTP/1.1.
|
|
|
|
### HTTP/2 Reverse Proxy
|
|
|
|
After the TLS handshake, the proxy serves HTTP using Go's `net/http` server
|
|
with HTTP/2 enabled. Each request is forwarded to the backend using a
|
|
reverse proxy.
|
|
|
|
**Backend transports:**
|
|
|
|
| `backend_tls` | Protocol | Description |
|
|
|----------------|----------|-------------|
|
|
| `false` | h2c | HTTP/2 over plaintext TCP. Used for localhost/trusted-network backends. Supports gRPC and standard HTTP. |
|
|
| `true` | h2 (TLS) | HTTP/2 over TLS to backend. Used when backend is remote or requires encryption. |
|
|
|
|
The reverse proxy handles:
|
|
|
|
- **HTTP/2 and HTTP/1.1**: Protocol negotiated via ALPN on the client side.
|
|
Backend connections use h2c (or h2 if `backend_tls`). If the backend only
|
|
speaks HTTP/1.1, Go's transport handles the protocol translation.
|
|
- **gRPC**: gRPC is HTTP/2 with `content-type: application/grpc`. The
|
|
reverse proxy forwards HTTP/2 frames, including **trailers** (critical
|
|
for gRPC status codes and error details).
|
|
- **Streaming**: Server-side, client-side, and bidirectional gRPC streaming
|
|
work through the reverse proxy. Request and response bodies are streamed,
|
|
not buffered.
|
|
- **Header injection**: `X-Forwarded-For`, `X-Forwarded-Proto`, and
|
|
`X-Real-IP` are set from the real client IP (PROXY protocol or TCP source).
|
|
- **Hop-by-hop headers**: `Connection`, `Transfer-Encoding`, and other
|
|
hop-by-hop headers are stripped per RFC 7230.
|
|
|
|
### Connection Model
|
|
|
|
L7 uses 1:1 connection mapping: each client connection maps to one backend
|
|
connection. There is no connection pooling. This keeps the implementation
|
|
simple and the connection lifecycle predictable. Pooling may be added later
|
|
if backend connection setup becomes a bottleneck.
|
|
|
|
### Error Handling
|
|
|
|
| Condition | Client Response |
|
|
|-----------|----------------|
|
|
| TLS handshake failure | Connection closed (TCP RST) |
|
|
| Backend dial failure | HTTP 502 Bad Gateway |
|
|
| Backend timeout | HTTP 504 Gateway Timeout |
|
|
| Backend returns error | Error response forwarded as-is |
|
|
|
|
---
|
|
|
|
## gRPC Admin API
|
|
|
|
The admin API is optional (disabled if `[grpc]` is omitted from the config).
|
|
It listens on a Unix domain socket for security -- access is controlled via
|
|
filesystem permissions (0600, owner-only). The API provides runtime management
|
|
of routes and firewall rules without restarting the proxy.
|
|
|
|
### RPCs
|
|
|
|
| RPC | Description |
|
|
|-----|-------------|
|
|
| `ListRoutes` | List all routes for a given listener |
|
|
| `AddRoute` | Add a route to a listener (write-through to DB) |
|
|
| `RemoveRoute` | Remove a route from a listener (write-through to DB) |
|
|
| `GetFirewallRules` | List all firewall rules |
|
|
| `AddFirewallRule` | Add a firewall rule (write-through to DB) |
|
|
| `RemoveFirewallRule` | Remove a firewall rule (write-through to DB) |
|
|
| `GetStatus` | Return version, uptime, listener status, connection counts |
|
|
| `grpc.health.v1.Health` | Standard gRPC health check (Check, Watch) |
|
|
|
|
### Input Validation
|
|
|
|
The admin API validates all inputs before persisting:
|
|
|
|
- **Route backends** must be valid `host:port` tuples.
|
|
- **Route mode** must be `"l4"` or `"l7"`.
|
|
- **L7 routes** must include valid `tls_cert` and `tls_key` paths.
|
|
- **IP firewall rules** must be valid IP addresses (`netip.ParseAddr`).
|
|
- **CIDR firewall rules** must be valid prefixes in canonical form.
|
|
- **Country firewall rules** must be exactly 2 uppercase letters (ISO 3166-1 alpha-2).
|
|
|
|
### Security
|
|
|
|
The gRPC admin API has no MCIAS integration -- mc-proxy is pre-auth
|
|
infrastructure. Access control relies on Unix socket filesystem permissions:
|
|
|
|
- Socket is created with mode `0600` (read/write for owner only)
|
|
- Only processes running as the same user can connect
|
|
- No network exposure -- the API is not accessible over TCP
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
TOML configuration file, loaded at startup. The proxy refuses to start if
|
|
required fields are missing or invalid.
|
|
|
|
```toml
|
|
# Database. Required.
|
|
[database]
|
|
path = "/srv/mc-proxy/mc-proxy.db"
|
|
|
|
# Listeners. Each has its own route table (seeds DB on first run).
|
|
[[listeners]]
|
|
addr = ":443"
|
|
proxy_protocol = false # accept PROXY protocol headers (default: false)
|
|
|
|
# L4 route: passthrough, no TLS termination.
|
|
[[listeners.routes]]
|
|
hostname = "metacrypt.metacircular.net"
|
|
backend = "127.0.0.1:18443"
|
|
mode = "l4"
|
|
|
|
# L7 route: terminate TLS, reverse proxy HTTP/2.
|
|
[[listeners.routes]]
|
|
hostname = "api.metacircular.net"
|
|
backend = "127.0.0.1:8080"
|
|
mode = "l7"
|
|
tls_cert = "/srv/mc-proxy/certs/api.crt"
|
|
tls_key = "/srv/mc-proxy/certs/api.key"
|
|
|
|
# L4 route with PROXY protocol to downstream proxy.
|
|
[[listeners.routes]]
|
|
hostname = "mcias.metacircular.net"
|
|
backend = "10.0.0.5:443"
|
|
mode = "l4"
|
|
send_proxy_protocol = true
|
|
|
|
[[listeners]]
|
|
addr = ":8443"
|
|
|
|
[[listeners.routes]]
|
|
hostname = "metacrypt.metacircular.net"
|
|
backend = "127.0.0.1:18443"
|
|
|
|
# gRPC admin API. Optional -- omit or leave addr empty to disable.
|
|
# Listens on a Unix socket; access controlled via filesystem permissions.
|
|
[grpc]
|
|
addr = "/var/run/mc-proxy.sock"
|
|
|
|
# Firewall. Global blocklist, evaluated before routing. Default allow.
|
|
[firewall]
|
|
geoip_db = "/srv/mc-proxy/GeoLite2-Country.mmdb"
|
|
blocked_ips = ["192.0.2.1"]
|
|
blocked_cidrs = ["198.51.100.0/24"]
|
|
blocked_countries = ["KP", "CN", "IN", "IL"]
|
|
rate_limit = 20 # max new connections per IP per rate_window
|
|
rate_window = "10s"
|
|
|
|
# Proxy behavior.
|
|
[proxy]
|
|
connect_timeout = "5s" # Timeout for dialing backend
|
|
idle_timeout = "300s" # Close connections idle longer than this
|
|
shutdown_timeout = "30s" # Graceful shutdown drain period
|
|
|
|
# Logging.
|
|
[log]
|
|
level = "info" # debug, info, warn, error
|
|
```
|
|
|
|
### Environment Variable Overrides
|
|
|
|
Configuration values can be overridden via environment variables using the
|
|
prefix `MCPROXY_` with underscore-separated paths:
|
|
|
|
```
|
|
MCPROXY_LOG_LEVEL=debug
|
|
MCPROXY_PROXY_IDLE_TIMEOUT=600s
|
|
```
|
|
|
|
Environment variables cannot define listeners, routes, or firewall rules --
|
|
these are structural and must be in the TOML file.
|
|
|
|
---
|
|
|
|
## Storage
|
|
|
|
### SQLite Database
|
|
|
|
Listeners, routes, and firewall rules are persisted in a SQLite database
|
|
(WAL mode, foreign keys enabled, busy timeout 5000ms). The pure-Go driver
|
|
`modernc.org/sqlite` is used (no CGo).
|
|
|
|
**Startup behavior:**
|
|
|
|
1. Open the database at the configured path. Run migrations.
|
|
2. If the database is empty (first run): seed from the TOML config.
|
|
3. If the database has data: load from it. TOML listener/route/firewall
|
|
fields are ignored.
|
|
|
|
The TOML config continues to own operational settings: proxy timeouts,
|
|
log level, gRPC config, GeoIP database path.
|
|
|
|
**Write-through pattern:** The gRPC admin API writes to the database first,
|
|
then updates in-memory state. If the database write fails, the in-memory
|
|
state is not modified.
|
|
|
|
### Schema
|
|
|
|
```sql
|
|
CREATE TABLE listeners (
|
|
id INTEGER PRIMARY KEY,
|
|
addr TEXT NOT NULL UNIQUE,
|
|
proxy_protocol INTEGER NOT NULL DEFAULT 0
|
|
);
|
|
|
|
CREATE TABLE routes (
|
|
id INTEGER PRIMARY KEY,
|
|
listener_id INTEGER NOT NULL REFERENCES listeners(id) ON DELETE CASCADE,
|
|
hostname TEXT NOT NULL,
|
|
backend TEXT NOT NULL,
|
|
mode TEXT NOT NULL DEFAULT 'l4' CHECK(mode IN ('l4', 'l7')),
|
|
tls_cert TEXT NOT NULL DEFAULT '',
|
|
tls_key TEXT NOT NULL DEFAULT '',
|
|
backend_tls INTEGER NOT NULL DEFAULT 0,
|
|
send_proxy_protocol INTEGER NOT NULL DEFAULT 0,
|
|
UNIQUE(listener_id, hostname)
|
|
);
|
|
|
|
CREATE TABLE firewall_rules (
|
|
id INTEGER PRIMARY KEY,
|
|
type TEXT NOT NULL CHECK(type IN ('ip', 'cidr', 'country')),
|
|
value TEXT NOT NULL,
|
|
UNIQUE(type, value)
|
|
);
|
|
```
|
|
|
|
### Data Directory
|
|
|
|
```
|
|
/srv/mc-proxy/
|
|
├── mc-proxy.toml Configuration
|
|
├── mc-proxy.db SQLite database
|
|
├── mc-proxy.sock Unix socket for gRPC admin API
|
|
├── certs/ TLS certificates for L7 routes
|
|
│ ├── api.crt
|
|
│ └── api.key
|
|
├── GeoLite2-Country.mmdb GeoIP database (if using country blocks)
|
|
└── backups/ Database snapshots
|
|
```
|
|
|
|
L4 routes do not use any TLS material -- the raw TLS stream passes through.
|
|
L7 routes require certificate/key pairs in the `certs/` directory (or any
|
|
path specified in the route config). The gRPC admin API uses a Unix socket.
|
|
|
|
---
|
|
|
|
## Deployment
|
|
|
|
### Binary
|
|
|
|
Single static binary, built with `CGO_ENABLED=0`. No runtime dependencies
|
|
beyond the configuration file, optional GeoIP database, and TLS
|
|
certificates for any L7 routes.
|
|
|
|
### Container
|
|
|
|
Multi-stage Docker build:
|
|
|
|
1. **Builder**: `golang:<version>-alpine`, static compilation.
|
|
2. **Runtime**: `alpine`, non-root user (`mc-proxy`), port exposure
|
|
determined by configuration.
|
|
|
|
### systemd
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `mc-proxy.service` | Main proxy service |
|
|
| `mc-proxy-backup.service` | Oneshot database backup (VACUUM INTO) |
|
|
| `mc-proxy-backup.timer` | Daily backup timer (02:00 UTC, 5-minute jitter) |
|
|
|
|
The proxy binds to privileged ports (443) and should use `AmbientCapabilities=CAP_NET_BIND_SERVICE`
|
|
in the systemd unit rather than running as root.
|
|
|
|
Standard security hardening directives apply per engineering standards
|
|
(`NoNewPrivileges=true`, `ProtectSystem=strict`, etc.).
|
|
|
|
### Multi-Hop Deployment
|
|
|
|
A typical production topology uses two mc-proxy instances:
|
|
|
|
```
|
|
Internet ──→ [edge mc-proxy] ──Tailscale──→ [origin mc-proxy] ──→ backends
|
|
VPS, :443 private network
|
|
```
|
|
|
|
**Edge mc-proxy** (public VPS):
|
|
- Listeners with `proxy_protocol = false` (clients connect directly).
|
|
- All routes are L4 passthrough with `send_proxy_protocol = true`.
|
|
- Firewall rules for GeoIP and IP blocking applied here.
|
|
|
|
**Origin mc-proxy** (private network):
|
|
- Listeners with `proxy_protocol = true` (receives real client IP from edge).
|
|
- Mix of L4 and L7 routes depending on the backend service.
|
|
- L7 routes terminate TLS and reverse proxy HTTP/2 to local backends.
|
|
- Firewall rules applied using the real client IP from the PROXY header.
|
|
|
|
### Graceful Shutdown
|
|
|
|
On `SIGINT` or `SIGTERM`:
|
|
|
|
1. Stop accepting new connections on all listeners.
|
|
2. Wait for in-flight connections to complete (up to `shutdown_timeout`).
|
|
3. Force-close remaining connections.
|
|
4. Exit.
|
|
|
|
On `SIGHUP`:
|
|
|
|
1. Reload the GeoIP database from disk.
|
|
2. Continue serving with the updated database.
|
|
|
|
Routes and firewall rules can be modified at runtime via the gRPC admin API
|
|
(write-through to SQLite). Listener changes (adding/removing ports) require
|
|
a full restart. TOML configuration changes (timeouts, log level, GeoIP path)
|
|
also require a restart.
|
|
|
|
---
|
|
|
|
## Security Model
|
|
|
|
mc-proxy is infrastructure that sits in front of authenticated services.
|
|
It has no authentication or authorization of its own.
|
|
|
|
### Threat Mitigations
|
|
|
|
| Threat | Mitigation |
|
|
|--------|------------|
|
|
| SNI spoofing | L4: Backend performs its own TLS handshake -- spoofed SNI fails certificate validation at the backend. L7: The proxy validates the certificate against the SNI during its own handshake, so a mismatched SNI results in a handshake failure before reaching the backend. |
|
|
| TLS private key compromise (L7) | Keys are stored on the filesystem with restrictive permissions. The proxy process runs as a dedicated non-root user. Keys are loaded into memory at startup; the file can be made read-only after startup. Consider integration with metacrypt for key storage (future). |
|
|
| Client IP spoofing via PROXY protocol | PROXY protocol headers are only trusted on listeners explicitly configured with `proxy_protocol = true`. Non-PROXY listeners reject connections that start with a PROXY header (they fail SNI extraction). Edge listeners facing the public internet should never enable `proxy_protocol`. |
|
|
| Resource exhaustion (connection flood) | Idle timeout closes stale connections. Per-IP rate limiting. Per-listener connection limits (future). |
|
|
| GeoIP evasion via IPv6 | GeoLite2 database includes IPv6 mappings. Both IPv4 and IPv6 source addresses are checked. |
|
|
| GeoIP evasion via VPN/proxy | Accepted risk. GeoIP blocking is a compliance measure, not a security boundary. Determined adversaries will bypass it. |
|
|
| Slowloris / slow ClientHello | Hardcoded 10-second timeout on the SNI extraction phase. 5-second timeout on PROXY protocol header parsing. |
|
|
| Backend unavailability | Connect timeout prevents indefinite hangs. L4: connection reset. L7: HTTP 502 response. |
|
|
| Information leakage (L4) | Blocked connections receive only a TCP RST. No version strings, no error messages, no TLS alerts. |
|
|
| Information leakage (L7) | Error responses are minimal (502/504 with no detail). The proxy does not expose its version or software name in HTTP headers. |
|
|
| HTTP request smuggling (L7) | Go's `net/http` server handles HTTP parsing. The reverse proxy uses `httputil.ReverseProxy` which strips hop-by-hop headers. HTTP/2 framing prevents classic smuggling vectors. |
|
|
|
|
### Security Invariants
|
|
|
|
1. **L4 routes never terminate TLS.** The proxy cannot read application-layer
|
|
traffic on L4 routes. It never modifies the byte stream.
|
|
2. **L7 routes terminate TLS at the proxy.** The proxy can read HTTP headers
|
|
and request metadata on L7 routes. It cannot read encrypted backend
|
|
traffic when `backend_tls = true`.
|
|
3. Firewall rules are always evaluated before any routing decision.
|
|
4. PROXY protocol is only parsed on explicitly enabled listeners.
|
|
5. The proxy never logs request or response bodies -- only metadata (source
|
|
IP, SNI hostname, backend, timestamps, bytes transferred, HTTP status
|
|
codes on L7 routes).
|
|
|
|
---
|
|
|
|
## Future Work
|
|
|
|
Items are listed roughly in priority order:
|
|
|
|
| Item | Description |
|
|
|------|-------------|
|
|
| **ACME integration** | Automatic certificate provisioning via Let's Encrypt for L7 routes, removing the need for manual cert management. |
|
|
| **MCP integration** | Wire the gRPC admin API into the Metacircular Control Plane for centralized management. |
|
|
| **Connection pooling** | Pool backend connections for L7 routes to reduce connection setup overhead under high request volume. |
|
|
| **Metacrypt key storage** | Store L7 TLS private keys in metacrypt rather than on the filesystem. |
|