Mesh LLM: distributed AI computing on iroh

Mesh LLM enables organizations to pool their existing GPU resources across multiple machines and expose them through a single OpenAI-compatible API. Rather than relying on third-party providers, users gain "control over when models change, where your data goes, and what hardware runs your workloads."

Key features

The system operates through three distribution strategies:

• Local execution on the requesting machine's GPU
• Routing to peer nodes that already have models loaded
• Pipeline splitting for models too large for individual hardware

The architecture supports 40+ models, ranging from compact half-billion-parameter variants suitable for laptops to massive 235B mixture-of-experts systems. Its "split mode" partitions large models by layer ranges across multiple nodes, allowing modest machines to collectively run models none could handle independently.

Network infrastructure

Built on iroh, Mesh LLM eliminates central servers through peer-to-peer connectivity using authenticated QUIC connections. The system handles NAT traversal and hole-punching automatically, with two iroh relays providing fallback paths for nodes unable to connect directly.

Communication uses tagged QUIC streams carrying gossip announcements, routing queries, inference requests, and plugin channels, all demultiplexed by a single leading byte identifier.

Getting started

Users can install the lightweight software (about 18 MB) and either join the public mesh or configure private deployments. The system presents itself as localhost:9337/v1 to any standard OpenAI client.