Skip to content

SETUP.md

Latest commit

 

History

History
449 lines (394 loc) · 27.4 KB

SETUP.md

File metadata and controls

449 lines (394 loc) · 27.4 KB
Raw

Setup and configuration

This file includes everything needed to build, configure, and run nullnet.

For what nullnet is and why, see the README; for how the pieces fit together, see docs/architecture.md.

This repository is a Cargo workspace holding the three binaries that make up the architecture plus the shared gRPC interface.

Layout

.
├── members/
│   ├── nullnet-client/      # runs on each host, exposes local services to the control plane
│   ├── nullnet-server/      # control plane: orchestrates VLAN/VXLAN setup and tears them down
│   ├── nullnet-proxy/       # ingress proxy: maps `service_name:80` requests to the right host
│   └── nullnet-grpc-lib/    # shared gRPC interface + generated types (proto + build.rs live here)
├── ebpf/                    # eBPF program loaded by nullnet-client (nightly toolchain)
└── xtask/                   # builds the eBPF program + the client userspace

Prerequisites

  • install Rust 
    curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

The repository should be cloned under /root so the provided setup-*.sh scripts and .service units work without changes. Run the setup scripts as root. The supplied systemd services also run as root; privileged commands execute directly without sudo.

Usage

nullnet-server

  • set environment variables (in members/nullnet-server/.env)

    NET_TYPE=VXLAN
CERT_ENCRYPTION_KEY=<32 raw bytes or 64 hex chars>
JWT_SIGNING_KEY=<32 raw bytes or 64 hex chars>
MFA_ENCRYPTION_KEY=<32 raw bytes or 64 hex chars>
DATABASE_URL=/var/nullnet/data/nullnet.db   # SQLite db path; default shown
ADMIN_BOOTSTRAP_USERNAME=admin              # only used the first time the users table is empty
ADMIN_BOOTSTRAP_PASSWORD=<a real password>  # ditto — defaults to admin/admin if either is unset
PROXY_IP=192.168.1.100
ENCRYPTION_ENABLED=true
INGRESS_ALLOW_TCP_PORTS=22,8080   # inbound TCP listeners every node accepts
INGRESS_ALLOW_UDP_PORTS=          # inbound UDP listeners (e.g. Swarm gossip 7946)
EGRESS_ALLOW_TCP_PORTS=           # outbound TCP dsts (e.g. 80,443 for updates)
EGRESS_ALLOW_UDP_PORTS=53,123     # outbound UDP dsts (DNS, NTP)

    CERT_ENCRYPTION_KEY is required — the server refuses to start without it. It encrypts TLS certificate private keys (and the DNS-provider credentials of ACME-issued certs) at rest; keep it stable, since rotating it makes existing encrypted data undecryptable. Generate one with openssl rand -hex 32.

    JWT_SIGNING_KEY and MFA_ENCRYPTION_KEY are also required — the server refuses to start without them. JWT_SIGNING_KEY signs the admin UI's session access tokens; MFA_ENCRYPTION_KEY encrypts TOTP secrets at rest. Each is deliberately a separate key from CERT_ENCRYPTION_KEY (no shared blast radius between secret classes) — generate each independently with openssl rand -hex 32 and keep them stable, for the same reason as CERT_ENCRYPTION_KEY.

    DATABASE_URL is the path to the server's SQLite database (users, sessions, service config, etc.), created on first start along with any parent directories. Defaults to /var/nullnet/data/nullnet.db if unset.

    ADMIN_BOOTSTRAP_USERNAME/ADMIN_BOOTSTRAP_PASSWORD create the first admin account the one time the users table is empty — irrelevant on every subsequent start. If either is unset, the server falls back to admin/admin and prints a loud warning; change that password immediately after first login if the admin UI is reachable by anyone else.

    PROXY_IP is the IP of the host running nullnet-proxy (the egress gateway). It is required to enable egress brokering: when a registered service reaches out to the internet the server builds a per-initiator egress edge to this host. If unset, egress is disabled (the trigger is rejected with "PROXY_IP is not configured") — ingress still works.

    ENCRYPTION_ENABLED toggles per-tunnel VLAN/VXLAN encryption (AES-256-GCM for VLAN, XFRM/MACsec for VXLAN) and defaults to true — omit it to keep encryption on. Set it to false/0/no to run tunnels unencrypted instead (a bare vxlan/veth link, no XFRM SA/policy or MACsec).

    The four {INGRESS,EGRESS}_ALLOW_{TCP,UDP}_PORTS lists are the global host-NIC firewall allowlist — decided here once (single point of decision) and delivered to every node in its NetworkType response at startup. They apply uniformly to all clients (matched on destination port). Put 22 in INGRESS_ALLOW_TCP_PORTS or every strict node loses SSH the moment its client starts. A node that needs name resolution / time sync needs EGRESS_ALLOW_UDP_PORTS=53,123; DHCP renewal needs 67 (and inbound 68 for broadcast replies). The gateway node (the one whose IP equals PROXY_IP) is switched to gateway posture automatically — all outbound allowed and tracked — so no per-node flag is needed; inbound there still obeys these lists, so include 80,443.

  • the admin web UI now requires login (JWT-based, cookie sessions). Roles are admin (all permissions) and user (explicit per-resource scopes, assigned from the Users page). MFA (TOTP) is optional per-account — an account without it configured sees a persistent banner prompting setup (QR code + confirm step), until it's done. See ADMIN_BOOTSTRAP_USERNAME above for how the first admin account is created.

  • Set optional server env UI_TLS_DOMAIN=control.example.com to serve the UI on https://control.example.com:8080 with its matching managed certificate (exact name preferred, then a single-label wildcard). Point that hostname at the server. Until a usable certificate is installed, the UI uses a self-signed setup certificate and reports this in Events. For first setup, use https://<server-ip>:8080 and accept the certificate warning; an existing HSTS policy can block this on the hostname. Install the certificate from the Certificates page, then use the hostname: the UI switches automatically and reloads renewals without a restart. Failed replacements or removal retain the last loaded certificate in memory and report an event; it still expires normally. After a restart without a usable stored certificate, setup mode returns. Leaving UI_TLS_DOMAIN unset preserves self-signed UI TLS.

  • TLS certificates are issued from Let's Encrypt via a DNS-01 challenge (UI: Certificates page). Each cert stores its DNS-provider credentials encrypted at rest and is renewed automatically before expiry. The renewal scan is tunable via optional env vars (defaults shown):

    CERT_RENEWAL_CHECK_INTERVAL_SECS=43200   # how often to scan (12h)
CERT_RENEWAL_DAYS_BEFORE=30              # renew when expiring within N days
CERT_RENEWAL_DNS_PROPAGATION_SECS=30     # wait after writing the TXT record

  • events (UI: Events page) are persisted to the SQLite database rather than kept in a bounded in-memory buffer, so a burst of routine events can no longer evict a warning/error before anyone sees it. They're pruned automatically past a retention window, tunable via optional env vars (defaults shown):

    EVENT_RETENTION_DAYS=7                       # how long an event is kept
EVENT_RETENTION_SWEEP_INTERVAL_SECS=3600     # how often the deletion sweep runs (1h)

    Writes use a bounded queue and batched transactions; normal events appear after commit. Storage failures retain accepted events and retry, reporting failure/recovery in Events and stderr. A full queue drops new events instead of blocking routing; Events reports the dropped count when storage catches up. Shutdown drains for up to five seconds; a longer outage, forced kill or power loss can lose events still queued in memory.

  • sessions (UI: Sessions page) are persisted the same way: every ingress session and every external destination reached over an egress edge is stored, so the page shows the full history and not just what is active right now. Filter it by status (active/ended), service, direction (ingress/egress), and egress policy verdict (allowed/blocked). Ended sessions are pruned on the same sweep as events:

    SESSION_RETENTION_DAYS=30                    # how long an ended session is kept

    An active session is never pruned, however old. Sessions still marked active are closed at startup, since the state they described died with the previous process. History writes run independently of networking, in ordered batches. Database failures retain accepted updates for retry; Events reports failures and recovery. If the bounded buffer fills, new history updates are dropped until the accepted prefix drains, and affected open rows become history interrupted (their actual end time is unknown). Normal shutdown drains for up to five seconds; forced termination can lose queued history.

  • Upgrade server, proxy and clients together: VXLAN backend/egress packet release uses an acknowledged NetReady command after both endpoints finish setup. Nullnet reserves Linux interface groups 0x4e000000–0x4e400000 for tunnel cleanup; do not assign unrelated interfaces to that range. Failed setup is not acknowledged as ready. Failed teardown keeps its ID reserved; after correcting the reported error, restart that client to purge stale state and release the reservation. Application containers do not need restarting.

  • the gRPC control channel itself (nullnet-client/nullnet-proxy ↔ nullnet-server) is TLS-only, authenticated by a private CA. On first boot the server generates its own CA (members/nullnet-server/grpc-tls/ca-cert.pem + ca-key.pem, created once and never regenerated) and signs its leaf cert with it. Copy ca-cert.pem to every client/proxy host and set CONTROL_SERVICE_CA_CERT there (see below) if it isn't at the default path (the repo root, i.e. nullnet/ca-cert.pem) — clients pin the channel to that CA and do full standard chain validation, so only a leaf actually signed by it is accepted. Because clients trust the stable CA root rather than the leaf, rotating the leaf later needs no client-side changes. Client authentication (mTLS) remains a further follow-up.

    Validation includes hostname matching, so the leaf's SAN must cover whatever host/IP clients use as CONTROL_SERVICE_ADDR. It's derived in this order: CONTROL_SERVICE_TLS_SAN (comma-separated, if set) → the server's own CONTROL_SERVICE_ADDR (if set — often already the same address clients are told to connect to) → localhost. Set CONTROL_SERVICE_TLS_SAN explicitly if the server's address isn't in its own .env or differs from what clients use.

  • service configuration is split per stack and lives in the server's SQLite database as normalized rows (stacks/services/service_triggers/service_dependencies/routes — one service's triggers/dependency branches are child rows of its own row, keyed by an autoincrement id). Edit it through the admin UI's Config page (per-service widgets), or directly via GET/POST/DELETE /api/service-config/{stack} (services, structured JSON) and GET/POST /api/routes/{stack} (routes) — both validate and apply live, no restart needed. The Config page's Export/Import buttons (GET/POST /api/service-config/{stack}/export/ import) round-trip a stack as a single TOML file — a hand-editable backup/version-history format outside the DB, validated through the exact same path as the widget UI on import.

    Upgrading a host still on pre-v0.2 file-based config? There is no automatic migration — ./services/<stack>.toml files are no longer read at all, by anything, once this build starts. For each stack, create it (Config page → name it → "Create stack") and use Import from TOML to paste/upload that stack's existing .toml file before relying on this build in production, or the stack comes up with zero services/routes until you do.

    The TOML shown below is that Export/Import format, and the clearest way to document the field set, which the JSON wire format (and the widget UI) mirrors field-for-field. For example, this defines a stack called my-app:

    [[services]]                 # http entry point, backed by a Docker container
name = "color.com"
timeout = 0
docker_container = "my-app_color"
port = 3001
proxy_dependencies = [["fs.color.com"]]
backends = ["ts.color.com"]

[[services]]                 # backend-only dep of color.com
name = "fs.color.com"
docker_container = "my-app_fs"
port = 8080

[[services]]                 # backend target — its port is used by the trigger
name = "ts.color.com"
docker_container = "my-app_ts"
port = 5555

[[services]]                 # host (non-Docker) service, matched by process
name = "metrics.com"
timeout = 0
process_path = "/usr/local/bin/metrics-exporter"
port = 9090

[[services]]                 # raw tcp — proxy binds listen_port and forwards
name = "redis.internal"
timeout = 0
docker_container = "my-app_redis"
port = 6379
protocol = "tcp"
listen_port = 6379

[[services]]                 # traffic filters (egress + ingress)
name = "api.internal"
timeout = 0
docker_container = "my-app_api"
port = 8000
egress_filter = { block = { groups = [
  [{ field = "country", condition = "equal", values = ["RU", "CN"] }],
  [{ field = "dst_ip", condition = "contains", values = ["203.0.113.0/24"] }],
] } }
ingress_filter = { allow = { groups = [
  [{ field = "country", condition = "equal", values = ["US", "IT"] }],
] } }

  • a service is hostable when it declares a match key plus a port (the backend port replicas are reached on). Clients hold no service file: each reports its raw local observations and the server matches them against these keys. A container/process may match several services across stacks; every match registers a replica:

    • docker_container — the Swarm service label (com.docker.swarm.service.name) or, standalone, the container name; matched against a running container
    • process_path — a listening process's exe path (/proc/<pid>/exe); matched against a host (non-Docker) service

  • host_ip optionally limits either match to one node's control-channel IPv4 address (for example, host_ip = "192.168.1.103" for that host's SSH service). Omit it to match all hosts.

  • pausable = true opts a Docker service into pausing when idle (the Config checkbox). It defaults to false, including existing DB rows. Backend services follow the same setting. Live chains and egress sessions keep containers running; disabling pause starts an asynchronous resume of a paused replica. If several declarations share a container, all must opt in. Paused initiators cannot start work themselves; use this only when incoming traffic can wake them.

  • timeout controls proxy-reachability: when present the service is a proxy-reachable entry point with that per-client idle timeout in seconds (0 disables the timeout); omit it to keep the service off the proxy (backend-only)

  • max_networks is an optional hard cap on dedicated ingress networks for a proxy-reachable service. Independently of that setting, encrypted VXLAN ingress switches new clients to multiplexing when the proxy's RPC backlog and slot-wait time grow for consecutive windows. A host-scoped controller first pauses dedicated setups, then additively admits one more probe per healthy window; renewed congestion immediately returns the rate to zero. Stable queues do not activate it, and only sustained successful probes under traffic remove the limit (idle time does not count as recovery). Multiplexed clients get one policy generation per client IP over a shared VXLAN. Cross-host groups share a persistent host-pair IPsec transport; packet metadata and endpoint maps preserve per-client isolation. Same-host multiplexing uses the same isolation policy with direct eBPF redirection and no VXLAN hairpin.

  • proxy_dependencies is a list of independent dep chains walked when the service is reached via a Proxy RPC from nullnet-proxy; each inner array is one linear branch and all branches are brought up in parallel

  • backends = ["peer-a", "peer-b"] lists services this service may trigger. Each peer must declare a nonzero port; that port identifies its trigger. One source cannot target different peers on the same port. A trigger opens only the direct connection, without following the peer's dependencies.

  • service names must be globally unique across every stack, and dependency chains stay intra-stack.

  • protocol selects how a proxy-reachable service is exposed: http (the default — routed by Host header on the shared 80/443 listeners) or tcp/udp, which each require listen_port — the external port nullnet-proxy binds directly and forwards raw traffic from. listen_port must be globally unique per protocol across every stack (the server refuses to start, or rejects a hot-reload, if two services claim the same protocol/listen_port pair)

  • TCP/UDP listen_port values are automatically allowed through the proxy host's eBPF ingress firewall at startup and refreshed within the client's 10-second service-report interval. Backend port values are not opened; explicit firewall allowlists still apply.

  • egress_filter/ingress_filter restrict traffic with an AND/OR combination of conditions, evaluated via rpn-predicate-interpreter (the same postfix-expression engine appguard-server/src/firewall/ uses). groups is OR-of-ANDs: every condition within a group must match (AND), and groups are OR'ed together. block denies a match (no match → allow); allow permits only a match (no match → deny) — same either way if the filter is omitted (no policy). Matchable fields: country/org (condition = equal/not_equal, values = ISO alpha-2 codes / ASN organization names, matched case-insensitively — resolved server-side, from one shared geo cache; the org is what the topology and Sessions views display), and src_ip/dst_ip (condition = contains/not_contains, values = CIDRs/addresses the peer IP is checked against). src_ip only applies to ingress filters, dst_ip only to egress — the server rejects the other. Two directions:

    • egress — where a service may reach on the internet (destination country/org/IP). Enforced at the initiator's nullnet-client: the first packet of each new external flow is held and verdicted, denied destinations show a BLOCKED chip in the topology UI, and editing the filter at runtime tears down already-established flows the new filter forbids.
    • ingress — which external clients may reach a proxy-reachable service (client source country/org/IP). Enforced server-side at the nullnet-proxy chokepoint: HTTP denials get a 403, raw tcp/udp denials close the connection. Only valid on a service with a timeout (an entry point) — the server rejects an ingress filter on a backend-only service.

  • [[route]] blocks add NGINX-location-style HTTP dispatch on top of [[services]], matching by host + prefix path (defaults to /) rather than by Host header alone:

    [[route]]                    # forward to a backend, stripping the matched prefix
host = "ops.example.com"
path = "/api"
service = "api.internal"     # must be an in-stack, http-protocol, proxy-reachable service
strip_prefix = true          # "api" sees "/users", not "/api/users"

[[route]]                    # redirect, no backend needed
host = "old.example.com"
path = "/old"
redirect_to = "/new"
redirect_status = 301        # optional, defaults to 301; one of 301/302/307/308
preserve_path = true         # /old/x?foo=bar -> /new/x?foo=bar
preserve_query = true

    service and redirect_to are mutually exclusive (exactly one required per route); strip_prefix only applies to service routes, preserve_path/preserve_query only to redirect_to routes — each is rejected on the wrong kind. All four flags default to false, so a stack with no [[route]] blocks keeps today's plain per-service Host-header routing. A given (host, path) pair must be claimed by exactly one route across every stack — a collision (even within the same stack) is a hard config error, reported as a route_conflict event. Configurable from the admin UI's Routes page as well as by editing the TOML directly.

  • run the project as a daemon (from the repo root)

    ./setup-server.sh

  • the server regularly renders one Graphviz file per stack under members/nullnet-server/graphs/<stack>.dot

nullnet-proxy

  • set environment variables (in members/nullnet-proxy/.env; set CONTROL_SERVICE_ADDR to the IP of nullnet-server)

    CONTROL_SERVICE_ADDR=192.168.1.100
CONTROL_SERVICE_PORT=50051
CONTROL_SERVICE_CA_CERT=../../ca-cert.pem   # optional, defaults to the repo root's ca-cert.pem
HSTS_ENABLED=true                         # optional, defaults to true; false for development

    CONTROL_SERVICE_CA_CERT points at a copy of the server's own grpc-tls/ca-cert.pem (see the server section above), used to pin and authenticate the control channel. If unset it defaults to ../../ca-cert.pem — the repo root's ca-cert.pem, relative to the service's members/nullnet-<component> working directory; either way, startup fails if the file is missing or isn't a valid cert.

  • run the project as a daemon (from the repo root)

    ./setup-proxy.sh

  • the proxy listens on port 80 (requests in the form service_name:80) and, for hosts that have a TLS certificate, on port 443 — HTTP requests to those hosts get a 301 redirect to HTTPS

  • HTTPS responses use Strict-Transport-Security: max-age=31536000, without includeSubDomains or preload. HSTS_ENABLED=false suppresses the header, including backend-supplied policies; restart the proxy after changing it. Disabling does not clear policies already cached by browsers. Use separate development hostnames: HSTS covers every port on a hostname, including a self-signed admin UI on :8080, and prevents bypassing certificate errors.

  • for services declared with protocol = "tcp" or "udp" in the server's stack config, the proxy also opens a raw listener on each listen_port and forwards traffic to the matching service — no Host header involved. This table is pushed live by the server, so listeners open and close as services/<stack>.toml changes, without a proxy restart

nullnet-client

  • set environment variables (in members/nullnet-client/.env; set CONTROL_SERVICE_ADDR to the IP of nullnet-server). The uplink interface is auto-detected from the host's default route.

    CONTROL_SERVICE_ADDR=192.168.1.100
CONTROL_SERVICE_PORT=50051
CONTROL_SERVICE_CA_CERT=../../ca-cert.pem   # optional, defaults to the repo root's ca-cert.pem

    CONTROL_SERVICE_CA_CERT points at a copy of the server's own grpc-tls/ca-cert.pem (see the server section above), used to pin and authenticate the control channel. If unset it defaults to ../../ca-cert.pem — the repo root's ca-cert.pem, relative to the service's members/nullnet-<component> working directory; either way, startup fails if the file is missing or isn't a valid cert.

    ⚠️ The client attaches a default-deny eBPF firewall to the uplink NIC on startup. It permits only the nullnet control plane (gRPC to the server), data plane (VXLAN to peers), established returns, ICMP (always, both directions — echo + PMTUD), and the port allowlist. That allowlist is no longer set here — it is decided globally on nullnet-server (the four {INGRESS,EGRESS}_ALLOW_{TCP,UDP}_PORTS variables) and delivered to the client in its NetworkType response at startup, so there is a single point of decision. Make sure 22 is in the server's INGRESS_ALLOW_TCP_PORTS before starting a client over SSH, or the session dies. The gateway-vs-strict posture is likewise derived server-side from PROXY_IP — no per-client flag.

  • run the project as a daemon (from the repo root)

    ./setup-client.sh

    Setup applies and persists a conntrack limit of at least 1,048,576 entries, preserving a higher existing limit. No manual sysctl configuration is needed. It also keeps NetworkManager and udev hotplug helpers from managing Nullnet-owned interfaces, avoiding competing network-configuration work during tunnel bursts.

The client also filters owned-device kernel uevents and link/address/route/neighbor notifications before delivery to udev, NetworkManager, OVS, Avahi and the desktop portal. Docker/foreign devices, netlink replies and dumps pass. This early socket filtering is separate from udev rules, which run after event queueing. Root needs CAP_SYS_PTRACE to duplicate consumer sockets, alongside its existing BPF privileges. The client mounts bpffs if needed and pins its filters under /sys/fs/bpf/nullnet-device-events-v1. It reconciles new consumer sockets every second; existing foreign filters are preserved and reduced coverage appears in Events. A three-second lease makes filters pass all notifications after client exit; pinned filters resume on client restart. Inactive filters remain attached until their consumer closes its socket, so shutdown never removes a replacement filter installed by that consumer. Set NULLNET_DEVICE_EVENT_BYPASS=false in the client .env to disable filtering. Other subscribers and namespaces are untouched.

On upgrades, deploy the server, proxy and all clients from the same revision. Client startup cleans stale Nullnet network state while preserving Docker interfaces; routine container restarts are unnecessary. Containers already damaged by an older client's cleanup need a one-time restart (Swarm: docker service update --force SERVICE).

Observation mode

Open Observation below Config and confirm the checkbox to enable all directed backend links within the selected stack. Activation lists any port conflicts; every destination must have a declared nonzero port. Proxy dependencies stay unchanged. The saved configuration is read-only until observation stops.

The service-only matrix counts backend sessions since activation (including ones already open). Gray borders mark saved backend links. Counts persist across restarts and session retention; SESSION_RETENTION_DAYS still controls session details. Stop observation to restore the saved config and review added/unused backends. Applying updates only backend lists. Completed observations remain available; results cannot overwrite a configuration edited since activation.

VXLAN edges create fresh dedicated interfaces and use fixed TC redirects; Nullnet shares UDP 47890, separate from Docker Swarm’s UDP 4789. Egress policy rules use priorities 1000–1015 and protocol 242; reserve these for Nullnet. Reserve TC blocks 7801–7832 and interface names nn-tc-0 through nn-tc-15 for Nullnet's shared classifiers. These down dummy interfaces carry no traffic; each edge still has dedicated interfaces and encryption keys. Startup removes stale holders using their reserved name, group, and device type before rebuilding the classifiers. br_<id>_<side> is the gateway interface name, not a Linux bridge. Startup keeps routed overlay traffic allowed by owned interface groups while preserving the host's existing FORWARD policy. Pooling and policy multiplexing are deferred.

The client and proxy services set LimitNOFILE=524288: fresh endpoints retain netlink sockets, and each proxied connection needs downstream and upstream sockets. Update both installed units and run systemctl daemon-reload when upgrading.