Skip to content

Integrate new p2panda high-level API#218

Draft
adzialocha wants to merge 13 commits into
mainfrom
adz/p2panda-high-level-api
Draft

Integrate new p2panda high-level API#218
adzialocha wants to merge 13 commits into
mainfrom
adz/p2panda-high-level-api

Conversation

@adzialocha
Copy link
Copy Markdown
Member

@adzialocha adzialocha commented Mar 18, 2026

This patch integrates the new p2panda crate into the "backend" of Reflection.

p2panda is the new all-in-one high-level Node API which should make usage of p2panda features easier, removing a bunch of code in reflection-node which is now handled in p2panda directly, such as:

  • Encoding / Decoding of operations and their message payloads
  • Make sure changes to the append-only log are strictly serialized, to avoid breaking consistency / integrity of data due to race conditions
  • Re-playing all operations on demand, this is required by Reflection to re-materialize all state for a document
  • Manage header extensions to include log ids and prune flags
  • Process local and remotely incoming operations ("ingest", "pruning")
  • SQLite storage handling for logs and operations
  • Mapping between logs and topics
  • De-duplicate messages in ephemeral streams by adding timestamp
  • Add public key / signatures to give provenance to ephemeral messages

Reflection Node

Some features implemented in reflection-node stay as they're adding functionality on top of p2panda. Some of them might be moved there as well in the future, such as:

Unfortunately we need to remove one feature for now which is to change "connection modes", causing the network to be disabled / enabled. reflection-node handled that well in the past by only disabling the networking layer while keeping the event processing layer intact. We will have to do something similar in p2panda to offer this feature as well since the new API surface will not allow such low-level access anymore. See related issue: p2panda/p2panda#1093

Breaking Changes

These changes introduce breaking changes to the core operation format (previous field was removed) and the header extensions changed which means that all previously created, persisted operations will not be compatible anymore.

THIS RESULTS IN LOOSING ALL PREVIOUSLY CREATED DOCUMENTS.

Since p2panda is not stable yet this might happen again in the near future, though this was the most important and largest update and we're much closer to a stable API and protocol now.

Final remarks

Some minor changes in reflection-doc where also necessary, I specifically decided to make p2panda-core a concrete dependency there since too much functionality is used to justify re-exports from reflection-node.

@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch 12 times, most recently from 84af564 to e3aa553 Compare March 20, 2026 10:56
@adzialocha adzialocha force-pushed the adz/fix-tests branch 7 times, most recently from 87b3021 to 8d83882 Compare March 20, 2026 11:47
@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch from e3aa553 to a79f048 Compare March 20, 2026 11:52
@adzialocha adzialocha force-pushed the adz/fix-tests branch 3 times, most recently from 49c10ba to 9adc524 Compare March 20, 2026 12:13
@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch from a79f048 to 6e34c9e Compare March 20, 2026 12:14
@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch from 6e34c9e to a45d239 Compare March 24, 2026 10:36
@adzialocha adzialocha changed the title WIP: Use new p2panda high-level API Integrate new p2panda high-level API Mar 24, 2026
@jsparber
Copy link
Copy Markdown
Collaborator

Only had a quick look. Can you split code changes to sensible commits, e.g. creating the new database.rs file. Following the commit style https://handbook.gnome.org/development/commit-messages.html

@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch from a45d239 to 93cf54f Compare May 21, 2026 08:16
@adzialocha
Copy link
Copy Markdown
Member Author

Only had a quick look. Can you split code changes to sensible commits, e.g. creating the new database.rs file. Following the commit style https://handbook.gnome.org/development/commit-messages.html

Yeah, the commits are a bit unfortunate. Not sure I'm able to do this as this refactor kinda requires me to touch upon everything at the same time. I'll try to split them up a bit but no guarantee they'll compile in each patch.

@adzialocha adzialocha changed the base branch from adz/fix-tests to main May 21, 2026 09:42
@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch 5 times, most recently from d3bec3d to be98707 Compare May 21, 2026 10:37
adzialocha added 13 commits May 22, 2026 18:55
This patch integrates the new `p2panda` crate into the "backend" of
Reflection.

`p2panda` is the new all-in-one high-level Node API which should make
usage of p2panda features easier, removing a bunch of code in
`reflection-node` which is now handled in p2panda directly, such as:

* Encoding / Decoding of operations and their message payloads
* Make sure changes to the append-only log are strictly serialized, to
  avoid breaking consistency / integrity of data due to race conditions
* Re-playing all operations on demand, this is required by Reflection to
  re-materialize all state for a document
* Manage header extensions to include log ids and prune flags
* Process local and remotely incoming operations ("ingest", "pruning")
* SQLite storage handling for logs and operations
* Mapping between logs and topics
* De-duplicate messages in ephemeral streams by adding timestamp
* Add public key / signatures to give provenance to ephemeral messages

Some features implemented in `reflection-node` stay as they're adding
functionality _on top_ of `p2panda`. Some of them might be moved there
as well in the future, such as:

* Author Presence: Indicate which authors have contributed to which
  topic and if they're currently online.
  Related issue: p2panda/p2panda#1002
* Persisted Topics: Store to persist all previously used topics

Unfortunately we need to remove one feature for now which is to change
"connection modes", causing the network to be disabled / enabled.
`reflection-node` handled that well in the past by only disabling the
networking layer while keeping the event processing layer intact. We
will have to do something similar in `p2panda` to offer this feature as
well since the new API surface will not allow such low-level access
anymore.

See related issue: p2panda/p2panda#1093

BREAKING CHANGES

These changes introduce breaking changes to the core operation format
(`previous` field was removed) and the header extensions changed which
means that all previously created, persisted operations will _not_ be
compatible anymore.

THIS RESULTS IN LOOSING ALL PREVIOUSLY CREATED DOCUMENTS.

Since `p2panda` is not stable yet this might happen again in the near
future, though this was the most important and largest update and we're
much closer to a stable API and protocol now.

Final remarks:

Some minor changes in `reflection-doc` where also necessary, I
specifically decided to make `p2panda-core` a concrete dependency there
since too much functionality is used to justify re-exports from
`reflection-node`.
@adzialocha adzialocha force-pushed the adz/p2panda-high-level-api branch from 4682087 to d2352de Compare May 22, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants