Integrate new p2panda high-level API#218
Draft
adzialocha wants to merge 13 commits into
Draft
Conversation
84af564 to
e3aa553
Compare
87b3021 to
8d83882
Compare
e3aa553 to
a79f048
Compare
49c10ba to
9adc524
Compare
a79f048 to
6e34c9e
Compare
9adc524 to
074a7f1
Compare
6e34c9e to
a45d239
Compare
Collaborator
|
Only had a quick look. Can you split code changes to sensible commits, e.g. creating the new database.rs file. Following the commit style https://handbook.gnome.org/development/commit-messages.html |
a45d239 to
93cf54f
Compare
Member
Author
Yeah, the commits are a bit unfortunate. Not sure I'm able to do this as this refactor kinda requires me to touch upon everything at the same time. I'll try to split them up a bit but no guarantee they'll compile in each patch. |
d3bec3d to
be98707
Compare
This patch integrates the new `p2panda` crate into the "backend" of
Reflection.
`p2panda` is the new all-in-one high-level Node API which should make
usage of p2panda features easier, removing a bunch of code in
`reflection-node` which is now handled in p2panda directly, such as:
* Encoding / Decoding of operations and their message payloads
* Make sure changes to the append-only log are strictly serialized, to
avoid breaking consistency / integrity of data due to race conditions
* Re-playing all operations on demand, this is required by Reflection to
re-materialize all state for a document
* Manage header extensions to include log ids and prune flags
* Process local and remotely incoming operations ("ingest", "pruning")
* SQLite storage handling for logs and operations
* Mapping between logs and topics
* De-duplicate messages in ephemeral streams by adding timestamp
* Add public key / signatures to give provenance to ephemeral messages
Some features implemented in `reflection-node` stay as they're adding
functionality _on top_ of `p2panda`. Some of them might be moved there
as well in the future, such as:
* Author Presence: Indicate which authors have contributed to which
topic and if they're currently online.
Related issue: p2panda/p2panda#1002
* Persisted Topics: Store to persist all previously used topics
Unfortunately we need to remove one feature for now which is to change
"connection modes", causing the network to be disabled / enabled.
`reflection-node` handled that well in the past by only disabling the
networking layer while keeping the event processing layer intact. We
will have to do something similar in `p2panda` to offer this feature as
well since the new API surface will not allow such low-level access
anymore.
See related issue: p2panda/p2panda#1093
BREAKING CHANGES
These changes introduce breaking changes to the core operation format
(`previous` field was removed) and the header extensions changed which
means that all previously created, persisted operations will _not_ be
compatible anymore.
THIS RESULTS IN LOOSING ALL PREVIOUSLY CREATED DOCUMENTS.
Since `p2panda` is not stable yet this might happen again in the near
future, though this was the most important and largest update and we're
much closer to a stable API and protocol now.
Final remarks:
Some minor changes in `reflection-doc` where also necessary, I
specifically decided to make `p2panda-core` a concrete dependency there
since too much functionality is used to justify re-exports from
`reflection-node`.
4682087 to
d2352de
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This patch integrates the new
p2pandacrate into the "backend" of Reflection.p2pandais the new all-in-one high-level Node API which should make usage of p2panda features easier, removing a bunch of code inreflection-nodewhich is now handled in p2panda directly, such as:Reflection Node
Some features implemented in
reflection-nodestay as they're adding functionality on top ofp2panda. Some of them might be moved there as well in the future, such as:Unfortunately we need to remove one feature for now which is to change "connection modes", causing the network to be disabled / enabled.
reflection-nodehandled that well in the past by only disabling the networking layer while keeping the event processing layer intact. We will have to do something similar inp2pandato offer this feature as well since the new API surface will not allow such low-level access anymore. See related issue: p2panda/p2panda#1093Breaking Changes
These changes introduce breaking changes to the core operation format (
previousfield was removed) and the header extensions changed which means that all previously created, persisted operations will not be compatible anymore.THIS RESULTS IN LOOSING ALL PREVIOUSLY CREATED DOCUMENTS.
Since
p2pandais not stable yet this might happen again in the near future, though this was the most important and largest update and we're much closer to a stable API and protocol now.Final remarks
Some minor changes in
reflection-docwhere also necessary, I specifically decided to makep2panda-corea concrete dependency there since too much functionality is used to justify re-exports fromreflection-node.