mTCP Chaining and Zero-Copy RX/TX Pipeline by ameer-hamza0046 · Pull Request #7 · networkedsystemsIITB/flash

ameer-hamza0046 · 2026-07-02T05:28:00Z

Description:

Overview
This PR introduces the core Zero-Copy packet semantics and L7 application chaining capabilities into the mTCP network stack over FLASH.

The goal of this architecture is to minimize memory copies between the NIC, the mTCP stack, and the L7 application.

Key Features Implemented:

Flash ID Trailer (MTCP_FLASH_ID_TRAILER)
- Modified the packet flow to append and parse 2-byte flash IDs [src_flash_id][dst_flash_id] at the end of payloads to enable stateful routing between chained NFs.
Zero-Copy RX Data Path (MTCP_RX_ZERO_COPY)
- Implemented mtcp_recv_zc, mtcp_read_zc, and batching variants.
- Built a lockless SPSC ring (zc_rx_free_ring) to safely reclaim UMEM buffers. Note that this design may have scalability issues that can be optimized in the future.
Zero-Copy TX Data Path (MTCP_TX_ZERO_COPY) (WIP)
- Implemented mtcp_get_zc_wptr, mtcp_write_zc, and batching variants.
- Built zc_tx_free_ring to pass available AF_XDP TX descriptors up to the application thread.

Known Issues & Caveats (Work In Progress)
While the RX data path is stable in terms of correctness, there are some known bugs in the TX path and chaining.

Throughput Collapse: Under heavy load, the TX zero-copy throughput slowly degrades and collapses over time.
AF_XDP Backend Deadlock Risk: Chaining using MTCP_FLASH_ID_TRAILER requires the lower-level API functions (flash__allocmsg, flash__sendmsg, __reserve_tx) to be non-blocking. If these functions use while loops to block indefinitely waiting for resources, they freeze the single-threaded mTCP event loop. This blocks RX polling, causing the stack to deadlock.
Current Workaround: During my experiments, I patched this by replacing the infinite blocking loops with a random number of retries to approximate non-blocking behavior. A pure fail-fast architecture is needed for production.

- added flash_id in tcp stream metadata - added mtcp-utils: mtcp-enter and mtcp-exit - now, mtcp_create_context also takes in nic queue id as an argument - now mtcp_connect also takes in dst_flash_id as an argument - added pps stats in flash_module.c - modified afxdp_get_wptr to batch requests to flash__allocmsg

- added api mtcp_recv_zc, mtcp_read_zc, mtcp_zc_free, mtcp_read_zc_batch, mtcp_zc_free_batch - SPSC ring to reclaim buffers, inefficient but correct, can optimize it in future - added zero copy APIs for TCP Ring Buffer

- added api mtcp_get_zc_wptr, mtcp_write_zc, mtcp_get_zc_wptr_batch, mtcp_write_zc_batch - updated tcp_send_buffer to support zero copy tx. - NOTE: This is a work in progress. The TX data path currently has a critical bug causing throughput to slowly collapse over time.

Copilot

Pull request overview

This PR integrates FLASH-specific chaining metadata and introduces zero-copy RX/TX plumbing into the mTCP-over-FLASH stack to reduce NIC↔stack↔application copies and enable L7/NF chaining.

Changes:

Adds FLASH ID trailer support (MTCP_FLASH_ID_TRAILER) to carry [src_flash_id][dst_flash_id] and thread it through stream state and TX.
Implements RX zero-copy receive semantics (MTCP_RX_ZERO_COPY) via new ring-buffer fragment paths and an SPSC reclaim ring.
Introduces a TX zero-copy path (MTCP_TX_ZERO_COPY, WIP) including ZC send lists, ZC write-pointer APIs, and AF_XDP backend support for enqueueing ZC buffers.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
lib/include/flash_defines.h	Adds `zc_tracker` in config for TX ZC buffer ownership tracking.
lib/flash/nf/flash_txrx.c	Uses `zc_tracker` to decide whether TX completion should return buffers to the pool.
lib/flash/nf/flash_nf.c	Allocates `zc_tracker` during NF configuration for TX ZC.
lib/flash/mtcp/timer.c	Hooks retransmission scheduling into the ZC send list when ZC buffers are pending.
lib/flash/mtcp/tcp_stream.c	Propagates flash ID into streams; ensures ZC send-list cleanup; updates RBFree call signature.
lib/flash/mtcp/tcp_send_buffer.c	Adds ZC send-buffer queueing and ACK-based release (`SBPut_zc/SBRemove_zc/SBClear_zc`).
lib/flash/mtcp/tcp_ring_buffer.c	Adds RX ZC fragment path + buffer reclaim into `RBFree` and new ZC APIs.
lib/flash/mtcp/tcp_rb_frag_queue.c	Adds a ZC fragment free-queue implementation for RX ZC fragments.
lib/flash/mtcp/tcp_out.c	Adds TX ZC flush path and ZC send list management; adjusts payload copy behavior.
lib/flash/mtcp/tcp_in.c	Threads RX ZC behavior through payload processing and alters packet-freeing semantics.
lib/flash/mtcp/ip_out.c	Updates flash context dst id on output; adjusts header placement for TX ZC.
lib/flash/mtcp/ip_in.c	Adjusts ICMP handling return semantics for RX ZC freeing.
lib/flash/mtcp/include/tcp_stream.h	Adds ZC send-list bookkeeping and per-stream dst flash id.
lib/flash/mtcp/include/tcp_send_buffer.h	Adds ZC send-buffer structures/constants and ZC API declarations.
lib/flash/mtcp/include/tcp_ring_buffer.h	Extends ring buffer with ZC fragment lists and ZC APIs; changes `RBFree` signature.
lib/flash/mtcp/include/tcp_rb_frag_queue.h	Exposes the ZC fragment queue types/APIs.
lib/flash/mtcp/include/tcp_out.h	Exposes ZC send-list APIs to other mTCP modules.
lib/flash/mtcp/include/mtcp.h	Adds flash context (queues, trailer ids, ZC rings) and ZC send queue state.
lib/flash/mtcp/include/mtcp_api.h	Extends public API for NIC queue selection and connect-time dst flash id; adds ZC read/write APIs.
lib/flash/mtcp/include/io_module.h	Extends I/O module interface for TX ZC (`get_zc_wptr`, `enqueue_zc_buf`).
lib/flash/mtcp/flash_module.c	Adds AF_XDP backend batching, trailer parsing/appending, ZC TX enqueueing, and descriptor throttling/stats.
lib/flash/mtcp/eth_out.c	Enqueues ZC buffers for TX when header headroom is provided.
lib/flash/mtcp/eth_in.c	Changes packet-free return semantics to support RX ZC freeing.
lib/flash/mtcp/core.c	Adds RX ZC reclaim + TX ZC replenish rings; integrates ZC sendq/send-list handling; changes RX free logic.
lib/flash/mtcp/api.c	Adds RX ZC recv/free APIs and TX ZC get-wptr/write APIs (single + batch).
examples/mtcp/epserver.c	Updates example to new `mtcp_create_context(core, queue)` signature.
examples/mtcp-utils/mtcp-exit.c	Adds a new utility NF that strips the trailer before forwarding out.
examples/mtcp-utils/mtcp-enter.c	Adds a new utility NF that appends the trailer before forwarding in.
examples/mtcp-utils/meson.build	Adds build targets for `mtcp-enter` and `mtcp-exit`.
examples/meson.build	Registers the new `mtcp-utils` examples directory.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

rickydebojeet · 2026-07-03T11:53:35Z

+	/* if no space, return -2 */
+	if (buf->available_zc_slots <= 0) {
+		/* should not happen */
+		return -2;
+	}


will this be a problem?

rickydebojeet · 2026-07-03T11:42:33Z

+#ifdef MTCP_TX_ZERO_COPY
+size_t SBRemove_zc(mtcp_manager_t mtcp, struct tcp_send_buffer *buf, size_t acked_len);
+size_t SBPut_zc(sb_manager_t sbm, struct tcp_send_buffer *buf, const void *data, size_t len, uint64_t flash_addr);
+void SBClear_zc(mtcp_manager_t mtcp, struct tcp_send_buffer *buf);
+#endif /* MTCP_TX_ZERO_COPY */


will this be a problem?

rickydebojeet · 2026-07-03T11:52:28Z

+#ifdef MTCP_TX_ZERO_COPY
+	// h-> need to check mss length for pkt? for now i am experimenting with pkt sizes upto 1024 B
+	if (mtcp->ctx->flash_ctx.tx_zc == 0 && payloadlen + optlen > cur_stream->sndvar->mss) {
+		TRACE_ERROR("Payload size exceeds MSS\n");
+		return ERROR;
+	}
+	if (mtcp->ctx->flash_ctx.tx_zc && payloadlen > 0) {
+		// data for IP and TCP header will be written in place
+		// h-> the header len is hardcoded for (TCP_OPT_TIMESTAMP_ENABLED = 1, TCP_OPT_SACK_ENABLED = 1,), where SACK flag is never set
+		assert(optlen == (TCP_OPT_TIMESTAMP_LEN + 2));
+		mtcp->ctx->flash_ctx.data = payload - (TCP_HEADER_LEN + optlen);
+	} else {
+		mtcp->ctx->flash_ctx.data = NULL;
+	}
+#else
+	if (payloadlen + optlen > cur_stream->sndvar->mss) {
+		TRACE_ERROR("Payload size exceeds MSS\n");
+		return ERROR;
+	}
+#endif /* MTCP_TX_ZERO_COPY */


this should be checked.

rickydebojeet · 2026-07-03T11:39:54Z

 int mtcp_core_affinitize(int cpu);

-mctx_t mtcp_create_context(int cpu);
+mctx_t mtcp_create_context(int cpu, int flash_nic_queue);


this definitely needs to be fixed. I would say make a wrapper function so that backward compatibility is not broken.

rickydebojeet · 2026-07-03T11:40:49Z

 int mtcp_init_rss(mctx_t mctx, in_addr_t saddr_base, int num_addr, in_addr_t daddr, in_addr_t dport);

-int mtcp_connect(mctx_t mctx, int sockid, const struct sockaddr *addr, socklen_t addrlen);
+int mtcp_connect(mctx_t mctx, int sockid, const struct sockaddr *addr, socklen_t addrlen, uint8_t dst_flash_id);


again here compatibility issues may be there. wrapper will fix it.

rickydebojeet · 2026-07-03T10:43:45Z

+	cfg->zc_tracker = (uint8_t *)calloc(cfg->umem->size / cfg->umem->frame_size, sizeof(uint8_t));
+	if (!cfg->zc_tracker) {
+		log_error("ERROR: Memory allocation failed for zero-copy tracker");
+		goto out_error;
+	}


@ameer-hamza0046 I think a memory leak is here. I think we should patch this up.

rickydebojeet

@ameer-hamza0046 I have tried to go through the code. I think some of the changes may break normal execution of tcp applications too.

Can you please go through the review and do the required changes?
I think with ifdef we can conditionally toggle the zero-copy without any compatibility issues.
Also can you also mention if there are any changes required during compilation?

rickydebojeet · 2026-07-03T10:43:45Z

+	cfg->zc_tracker = (uint8_t *)calloc(cfg->umem->size / cfg->umem->frame_size, sizeof(uint8_t));
+	if (!cfg->zc_tracker) {
+		log_error("ERROR: Memory allocation failed for zero-copy tracker");
+		goto out_error;
+	}


@ameer-hamza0046 I think a memory leak is here. I think we should patch this up.

rickydebojeet · 2026-07-03T11:25:57Z

+	"-D <num>\tDestination NF ID (default: 0)",
+    NULL
+};
+// clang-format on


@ameer-hamza0046 can we have some description about how the flash trailer should be setup and how will it work?

rickydebojeet · 2026-07-03T11:33:04Z

What is the use of this nf?

rickydebojeet · 2026-07-03T11:39:54Z

 int mtcp_core_affinitize(int cpu);

-mctx_t mtcp_create_context(int cpu);
+mctx_t mtcp_create_context(int cpu, int flash_nic_queue);


this definitely needs to be fixed. I would say make a wrapper function so that backward compatibility is not broken.

rickydebojeet · 2026-07-03T11:40:49Z

 int mtcp_init_rss(mctx_t mctx, in_addr_t saddr_base, int num_addr, in_addr_t daddr, in_addr_t dport);

-int mtcp_connect(mctx_t mctx, int sockid, const struct sockaddr *addr, socklen_t addrlen);
+int mtcp_connect(mctx_t mctx, int sockid, const struct sockaddr *addr, socklen_t addrlen, uint8_t dst_flash_id);


again here compatibility issues may be there. wrapper will fix it.

rickydebojeet · 2026-07-03T11:49:34Z

 	case IPPROTO_ICMP:
-		return ProcessICMPPacket(mtcp, iph, ip_len);
+		ProcessICMPPacket(mtcp, iph, ip_len);
+		return FALSE; // false to free up the packet in core.c for RX ZC.


i think a ifdef should be here right for backward compatibility

rickydebojeet · 2026-07-03T11:51:55Z

there are lot of changes in this file:

from normal function calls to direct returns.

changing FALSE to TRUE and vice versa.

Will all of this changes not affect non-zc setups?

rickydebojeet · 2026-07-03T11:52:28Z

+#ifdef MTCP_TX_ZERO_COPY
+	// h-> need to check mss length for pkt? for now i am experimenting with pkt sizes upto 1024 B
+	if (mtcp->ctx->flash_ctx.tx_zc == 0 && payloadlen + optlen > cur_stream->sndvar->mss) {
+		TRACE_ERROR("Payload size exceeds MSS\n");
+		return ERROR;
+	}
+	if (mtcp->ctx->flash_ctx.tx_zc && payloadlen > 0) {
+		// data for IP and TCP header will be written in place
+		// h-> the header len is hardcoded for (TCP_OPT_TIMESTAMP_ENABLED = 1, TCP_OPT_SACK_ENABLED = 1,), where SACK flag is never set
+		assert(optlen == (TCP_OPT_TIMESTAMP_LEN + 2));
+		mtcp->ctx->flash_ctx.data = payload - (TCP_HEADER_LEN + optlen);
+	} else {
+		mtcp->ctx->flash_ctx.data = NULL;
+	}
+#else
+	if (payloadlen + optlen > cur_stream->sndvar->mss) {
+		TRACE_ERROR("Payload size exceeds MSS\n");
+		return ERROR;
+	}
+#endif /* MTCP_TX_ZERO_COPY */


this should be checked.

rickydebojeet · 2026-07-03T11:53:35Z

+	/* if no space, return -2 */
+	if (buf->available_zc_slots <= 0) {
+		/* should not happen */
+		return -2;
+	}


will this be a problem?

rickydebojeet · 2026-07-03T11:54:28Z

 	int max_outstanding_tx;
+
+#if defined(MTCP_TX_ZERO_COPY)
+	uint8_t *zc_tracker;


what's the use of this. can you please give me a short description of this?

ameer-hamza0046 added 4 commits June 29, 2026 00:57

feat: zero copy RX path in mtcp

fe67516

- added api mtcp_recv_zc, mtcp_read_zc, mtcp_zc_free, mtcp_read_zc_batch, mtcp_zc_free_batch - SPSC ring to reclaim buffers, inefficient but correct, can optimize it in future - added zero copy APIs for TCP Ring Buffer

fix: build failure when MTCP_RX_ZERO_COPY is disabled

1e51173

rickydebojeet requested a review from Copilot July 2, 2026 08:52

Copilot started reviewing on behalf of rickydebojeet July 2, 2026 08:53 View session

rickydebojeet self-requested a review July 2, 2026 08:53

Copilot AI reviewed Jul 2, 2026

View reviewed changes

rickydebojeet requested changes Jul 3, 2026

View reviewed changes

Uh oh!

Conversation

ameer-hamza0046 commented Jul 2, 2026

Description:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rickydebojeet left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants