I built an AI supercomputer with 5 Mac Studios

Clustering Five Mac Studios to Run a 405B AI Model

This video follows NetworkChuck as he attempts an ambitious experiment: connecting five Mac Studios into a local AI cluster using the new EXO Labs software. His goal is to run the massive Llama 3.1 405B parameter model, a task typically reserved for enterprise-grade cloud servers. The video documents the entire process, from unboxing the hardware to testing performance and overcoming significant bottlenecks.

The Hardware & The Goal

• The Setup: Five Mac Studios, each with an M2 Ultra chip and 64GB of unified memory, are connected to form a single, powerful AI cluster.
• The Target Model: The primary goal is to run the Llama 3.1 405B model, which normally requires around one terabyte of VRAM.
• Why Mac Studios? The key advantage is their unified memory architecture, which allows the entire 64GB of RAM to be used as GPU memory. This gives the cluster a combined 320GB of usable memory, far exceeding what a single consumer GPU like an RTX 4090 (24GB) can offer.

Understanding AI Model Requirements

The video breaks down the relationship between model size, parameters, and hardware requirements.

• Parameters are the learned knowledge of an AI model. More parameters generally mean a smarter, more capable model.
• VRAM (Video RAM) is the critical resource for running AI models, as it's the memory on your GPU.
• Quantization is a technique used to shrink model sizes by reducing precision, allowing them to fit on consumer hardware. The video focuses on using a 4-bit quantized version of the 405B model.

Building the Cluster with EXO Labs

• Software: EXO Labs is a new, beta software that allows you to cluster any type of computer hardware to run AI models collaboratively.
• Installation: The process involves installing Python 3.12, the MLX (Machine Learning Acceleration) library for Apple Silicon, and cloning the EXO repository on each Mac.
• Discovery: Once running, the Macs automatically discover each other on the network and form a cluster, with a web GUI for interaction.

Performance Testing & Bottlenecks

• Single Mac Baseline: A single Mac Studio ran a small 1B model at an impressive 117 tokens per second.
• The Network Bottleneck: When clustering all five Macs over 10-gigabit Ethernet, performance for the same small model dropped to 29 tokens per second. The communication overhead between machines became the primary limiting factor.
• Thunderbolt Improvement: Switching to a Thunderbolt bridge improved performance, but the cluster still couldn't match the speed of a single machine.
• Running the 70B Model: The cluster successfully ran the Llama 3.3 70B model at a usable 15 tokens per second.
• The Big Test: The cluster was able to load and run the Llama 3.1 405B model, but at a painfully slow 0.8 tokens per second. While it worked, it was not practical for real-time use.

The Verdict

• EXO Labs is a promising and easy-to-use tool for creating AI clusters.
• MLX performance on Macs is not yet fully optimized, and the network connection is the biggest bottleneck.
• The experiment proved that it is possible to run a 405B model on local hardware, but the speed is currently too slow for practical use.
• For a single Mac, running a 70B model with a tool like Ollama is a much more efficient and practical experience.

Key Takeaway

The video demonstrates that while clustering consumer hardware to run massive AI models is technically possible, it is not yet a practical solution. The primary challenge is the network bottleneck, which severely limits performance. However, the experiment highlights the rapid innovation in local AI and the potential for software like EXO to evolve and make such clusters more viable in the future.