Skip to content

Add PubSub and BigTable support for Central hosted network controllers#2526

Open
glimberg wants to merge 247 commits into
devfrom
gl/ctl-pubsub
Open

Add PubSub and BigTable support for Central hosted network controllers#2526
glimberg wants to merge 247 commits into
devfrom
gl/ctl-pubsub

Conversation

@glimberg

@glimberg glimberg commented Oct 6, 2025

Copy link
Copy Markdown
Contributor

This is a big one that goes along with other internal upcoming changes.

  • Refactors existing Redis & Postgres NOTIFY message passing systems into the NotificationListener interface, and adds a 3rd method for message passing via GCP PubSub. Which one to use is configurable by local.conf controller settings
  • Adds GCP BigTable as a 3rd option for writing member status updates along side Postgres and Redis. Also configurable via local.conf
  • Set up CMake for building Central controllers and extended for the regular agent build as well (optional use threre. existing build methods still exist).
  • scripts/bootstrap-deps.sh for bootstrapping external dependencies (scripts/bootstrap-deps.ps1 on Windows)
  • CMake for external dependency management instead of dumping everything in ext/. This is likely more useful for Central controller builds than end user builds for the time being.
  • Controller log formatting standardization

glimberg added 30 commits August 5, 2025 15:52
…stance for the whole library, add init/shutdown functions for it exposed to C
match other things already written
Tests currently need to be run with --test-threads=1.  Seems like the instances of the pubsub emulator stomp on each other without that
…s in new CentralDB class

This allows us to interchangeably use different listeners (pgsql, redis, pubsub) depending on configuration values passed into the constructor.
PubSub allows us to do schema validation, however it only allows one top level message at a time.  Move other sub-message declarations under the main message declaration so that we can enable schema validation in the pubsub stream directly
Muuuuch easier to use external dependencies now

Also tried out conan and vcpkg.   Ran into dependency issues when solving for packages to install with conan.

vcpkg is just obtuse as all hell to install and not easy to integrate
glimberg added 30 commits June 12, 2026 16:23
* Remove now dead separate CV1/CV2 implementations.  Consolidated into
  CentralDB.
* Remove smeeclient.  No longer needed.
* Brewfile for homebrew dependencies on macos
* Pull & build google-cloud-cpp/otel via bootstrap script rather than
  keeping it all in the ext/ subfolder.
* Update controller CI build with new steps
Also fixed a ZT_NONFREE issue
removing openssl dependency (except on Linux where it's still necessary)
upgrade cpp-httplib
new cmake-presets
lock -> notification, and double notify on erase
More use of scoped RAII guard for connections where appropriate.
Also add presets for the BSDs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants