I Tested Ornith's New 35b MoE on Mac — Here's How It Goes

Testing Orith 1.5: A Comprehensive Evaluation of Vision, Code Generation, and Reasoning

This video provides an in-depth, hands-on evaluation of Orith 1.5, a newly released multimodal large language model, with a focus on the 35-billion-parameter mixture-of-experts (MoE) variant. The creator tests the model's vision capabilities, code generation, and reasoning through a series of one-shot tasks, while also comparing the impact of prompt engineering, thinking modes, and quantization levels. The entire workspace, including prompts and results, is shared on GitHub for reproducibility.

Model Overview and Setup

Orith 1.5 is available in three sizes: a 9B dense model, a 35B MoE (the primary test subject), and a 397B variant. The testing environment consisted of an M3 Ultra Mac Studio running the model via the MLX engine with 4-bit OQ4E quantization. Key performance metrics include:

• Decode speed: 120 tokens/second (4-bit with MTP), 107 tokens/second (8-bit with MTP), and 83 tokens/second (4-bit without MTP)
• Memory footprint: ~43 GB total for model weights (21.6 GB) plus a 260k-token conversation context (21.5 GB)
• Context scaling: Decode speed degrades gracefully from 120 to 76 tokens/second as context grows to 120k tokens

Vision Capabilities: Invoice Extraction

The vision test involved extracting text from three invoices of increasing information density. The results were impressive:

• Basic invoice: 15/16 fields correctly extracted (missed a line-item description)
• Medium invoice: 32/32 fields correct
• Dense invoice: 82/84 fields correct (98% accuracy)

The two errors were minor: a single misread character in a SWIFT code and a digit transposition in a product number. The creator notes that while this is strong performance, production systems would still benefit from fallback mechanisms like mechanical parsers or secondary LLM verification.

Code Generation: Tower Defense Game

The creator tested both basic prompts (simple description) and upgraded prompts (requiring planning, implementation, and self-verification), with thinking mode enabled or disabled:

• Basic prompt + no thinking: Produced a polished, fully functional game with smooth animations
• Basic prompt + thinking: Functionally similar but with a slightly more premium look; minor background issues prevented full completion
• Upgraded prompt + thinking: Resulted in a more basic UI but fully working gameplay

Interestingly, the upgraded prompt produced 200+ fewer lines of code, suggesting the model may have optimized for correctness over visual polish when explicitly asked to verify its work.

Spreadsheet Replication

The spreadsheet test required building a functional calculator with formulas:

• Basic prompt + no thinking: Partial functionality—cells were clickable but couldn't accept numeric input
• Basic prompt + thinking: Fully functional spreadsheet with working formulas (e.g., =A1+B1 correctly returned 30)
• Upgraded prompt + thinking: Also functional, with improved cell highlighting and selection feedback

The creator notes that Orith 1.5 outperformed both Qwen 3.8 and DeepSeek in comparable tests from previous videos, particularly in the no-thinking scenario.

3JS Game Development

The most challenging task was building a first-person 3JS game. All one-shot attempts failed due to trivial errors:

• Basic + no thinking: Duplicate function declaration
• Basic + thinking: Missing full stop
• Upgraded + thinking: Comment inside a JavaScript module
• 8-bit + thinking: Variable used before function definition

A manual patch of a single full stop resolved the issue, demonstrating that the model was "1% off" from working code. The creator emphasizes that these are exactly the kind of errors that would be caught by a single additional iteration.

Comparative Analysis and Observations

• Orith 1.5 outperformed Qwen 3.8 27B across most tasks, particularly in vision and spreadsheet generation
• Thinking mode had inconsistent effects—sometimes improving quality, sometimes introducing unnecessary complexity
• The upgraded prompt (with self-checking) did not consistently improve outcomes and sometimes reduced code quality
• Vision tasks were remarkably accurate at 98%+, even with information-dense invoices

"The creativity of building something that looks nice was actually consumed by possibly the creativity needed to produce this plan and then to produce the test cases as well."

Key Takeaway

Orith 1.5 represents a significant step forward for locally deployable multimodal LLMs, delivering near-state-of-the-art vision accuracy and impressive code generation for a 35B-parameter model. However, the persistence of trivial syntax errors in complex, multi-file tasks highlights the continued need for iterative refinement or external validation layers in production systems. The model's speed (120 tokens/second) and accuracy make it a compelling option for developers, provided they account for its occasional "one-character-off" failures.