Hybrid Adaptive Multi-modal Fusion for Brain Tumor Segmentation with Missing Modalities

Environment

Tested with:

• Python 3.12
• CUDA 12.8
• torch==2.8.0+cu128
• GPU: NVIDIA RTX 5090 (32 GB VRAM)

Installation

pip install -r requirements.txt
bash install_mamba.sh

Code Structure

Component	Location
Model (HAMFuse)	HAMFuse.py
Backend (Inference API)	api.py
Frontend	frontend/

Dataset

Training and evaluation use the BraTS 2023 glioma segmentation data from the MICCAI challenge. Obtain the dataset from the official Synapse page:

BraTS 2023 Challenge on Synapse

Case folders should contain the standard BraTS NIfTI modalities and segmentation.

Preprocessing

Set src_path and tar_path at the top of preprocess.py, then run:

python preprocess.py

This writes cropped, normalized volumes and labels as .npy files for --datapath in training.

Running

Backend

CHECKPOINT_PATH=/path/to/best.pth USE_SAF=true USE_HYBRID_ENCODER=true uvicorn api:app --host 0.0.0.0 --port 8000

Frontend

cd frontend
npm install
npm run dev

Training

python train_poly.py \
  --datapath /workspace/data/BRATS2023_Training_preprocessed \
  --dataname BRATS2023 \
  --savepath /data/checkpoints/A0_baseline \
  --hamfuse \
  --use_saf \
  --gate_loss_weight 0.01 \
  --batch_size 1 \
  --num_epochs 200 \
  --no_early_stopping

Module Flags

Two key modules can be toggled independently:

Module	Enable	Disable
Hybrid Encoder	(on by default)	--no_hybrid_encoder
Spatially Adaptive Fusion (SAF)	--use_saf	(off by default)