Support for 2 GX10s cluster with 128GB each - support tensor parallelism

[rmunad]

Any work been done to provide that support?

[mlow]

I believe this project focuses on single-node inference, which is a great thing to focus on (it's a niche that needs filled).

There is promising effort to get the full-fat DSV4 Flash running across 2 sparks over at VLLM though (at ~1000 t/s PP, >30 t/s TG)

[CobraSoftware]

I would love to see this for macs. I could try to make pr if the maintainers are willing.