2026 Storage overhaul

[traumschule]

Context

• v2 Storage and Distribution System Specification #2224
• Gateway Specification #2089
• CIJTF Meeting Tracker Issue #4455
• How to do data fetching in Joystream applications #1096
• Distributor node - current progress and future plans #2553

Requirements

• A new infra release: Origo  #4766
  • Make Argus/Colossus/Apps(Orion) aware of the Operational status of the storage/distributior-nodes #4816
  • Add support to mark storage-node/distrubtion-node as under maintenance by the Node operators #4765
  • Improving Storage/Distribution related integration tests #4815
  • Argus/Colossus QueryNode result caching to improve latency #4424
• Metaprotocol replacement of storage pallet #4940
  • Is DataObject Accepted status necessary? #5058
  • Remove unique handle constraint enforcement from runtime #2912
• QN/ Mataptorocol Backwards Compatibility #4699
• Content directory smart contracts: Splitting the logic #1922

Scope

The general goal is to improve the content storage system independently from runtime constrains.
Prototype colossus / argus changes to implement listed features using a flexible DAO hosted API.
Develop meta-protocols to reflect necessary communication on chain for apps to be able to interoperate using preferably chain data to avoid SPOF.

Error reporting and tracking

• Lead infrastructure (error) monitoring #4452
• Proof-of-storage #2555
• Lighthouse - An automated storage interrogation service #4270
• QN: Improve Metaprotocol transactions error handling #4573

Update Access Policies

• Verified status for channels #2586 Channel Verification #4487 Explore channels as interoperable NFT assets #2925
• Standardise Argus access policy signals in metadata #4332
• Uploader authentication #2815 Object Upload fails but still set as accepted on chain #5097 [Colossus] Resumable uploads #5043
• Runtime enforcement of channel storage limits #4819
• Argus: content delivery authentication #3152
• Argus: screen non-public content #4325
• Review: How Top CDN manage content authentication? #4438
• Standardise Argus access policy signals in metadata #4332
• Argus: timed non-publicness #4331
• Argus & Colossus private blacklists #4588
• Infrastructure authentication #4414
• Argus: content access authentication scheme #4415
• Authentication between Argus and Colossus nodes #4420

Colossus / Argus refactor

• Make sure the apps don't exceed GraphQL request size limit #4624
• [Colossus] Add timeout enhancements to superagent requests based on the object sizes #4999
• Colossus state api re-factor #5059
• Minimise use and processing of large arrays #5051
• [Colossus] Trying to hash File not found response #5045
• colossus: support specifying multiple endpoints in operator metadata #4982
• New storage buckets analytics command #4444
• Update reverse proxy recommendations for Argus and Colossus to support HTTP/2 #3829 Question: HTTP/2 for Argus & Colossus #3808

Enable third-party actors to enrich (on-chain) content

• WIP: Probabilistic Intra-Infrastructure Payments #2757
• Delayed video publishing metaprotocol accomodation #4558
• Unlock third party post-processing for uploaders #2788
• Create Adaptive Streaming Specification #4720
• Store videos in different qualities, audio-only version #3806
• Separate audo & video tracks #4702
• QN Subtitles #3246

Enable viewers to publish playlists

• Requirements for playlists from Atlas perspective #3417
• QN Playlists #3245
• Idea: Paid Playlists (WIP) #2055

Update tooling

• Joystream SDK: RFC / draft #5198
• CLI command wishlist #276