My Local AI Is Now Much Faster! But …

Local AI Progress: Speed, Costs, and Practical Viability

This video documents the creator’s six-month journey running local AI on a Beelink Strix Halo (AMD AI Max 395, 128 GB shared RAM). The focus is on privacy — avoiding cloud AI prompt transport — and on the dramatic speed improvements achieved through software changes alone, despite steep hardware costs.

💰 Cost Reality and Cloud Alternatives

• The Strix Halo machine cost roughly $2,000 in February, but now sells for over $4,000, making the upgrade path impractical for most users.
• The creator advises most learners to stay with cloud AI for now, with Ollama ($20/month) as the least-bad privacy compromise.
• Major cloud models (ChatGPT, Claude, Gemini, etc.) are not private; local AI remains best for sensitive tasks.

⚡ Measured Performance Gains

• Initial setup: Ollama hosting GPT-OSS 120B delivered only ~20 output tokens/sec, plus frequent crashes caused by a bug in high-speed Intel network drivers.
• Software fix #1: Switching from AMD’s ROCm to the Vulkan stack allowed full 96 GB VRAM allocation, boosting speed to ~32 tokens/sec.
• Software fix #2: An OpenClaw agent automatically compiled llama.cpp and loaded GPT-OSS 120B in GGUF format, reaching ~53 output tokens/sec and ~150 input tokens/sec.

🤖 Real-World Public Deployment

• The model runs live as Braxbot on the creator’s site; a 16K-context prompt is read in ~2 minutes, with responses typically delivered in 5 minutes or less.
• The bot has survived hundreds of prompt injection attacks; guardrails, layered bots, and access controls are essential.
• A smaller dense model (Nematron 30B) performed worse on security judgment and was slower (~30 TPS) than the larger mixture-of-experts model.

🏢 Business Potential

• The setup can handle email triage, tech support, and sensitive planning (financial, tax, retirement) with strong privacy guarantees.
• RAG (retrieval-augmented generation) gives the bot searchable domain knowledge, while the agent handles low-level setup tasks automatically.

Key Takeaway

Local AI is now genuinely usable for serious business tasks — but the hardware cost remains prohibitive, and most speed gains came from software improvements, not new components.

For privacy-critical work, a tuned local setup is a viable investment; for most individuals, cloud-based Ollama remains the pragmatic starting point.