I've been running local AI on my workstation for a couple of years now with Ollama and various small models for sub-tasks. But about five months ago, everything changed.
When Minimax 2.7 weights dropped in April, I set up the unsloth quantized UD-Q5_K_XL with llama.cpp. Suddenly I had Sonnet 4.6-level intelligence sitting in my office — amazing for coding, writing, and running agents.
Since then, I've been monitoring this class of models in the 150 GB to 300 GB range, and this summer felt like Christmas with new releases:
I jumped on DeepSeek V4 Flash when it came out. These are quite experimental models and need some tweaking when deploying them.
I set it up with a 460,000 token context window for some large coding and agent work. Unfortunately, the model failed. It seems like they're still experimenting with these models — it takes time for inference providers like vLLM, llama.cpp, and SGLang to catch up and implement all the new features.
It was a pretty rough experience. The failure produced an infinite output loop that was impossible to stop. If you've never seen an AI go crazy, it's impressive and quite scary — like a token vomit. Turns out, pushing past about 120,000 tokens in the context window makes these models start getting unstable.
So I switched back to Minimax 2.7. If anyone has had success with the other two models, let me know. If not, I'll test them later this month once the pull requests have been merged into llama.cpp.
Conclusion
These models are amazing. The 200+ GB size is significantly better than the small 30–40 GB models we get from Google, OpenAI, and Meta. However, they're very experimental, and it takes a lot of trial and error to find the right quantization and configuration to get them to perform well.
When we get models this capable running locally, it's worth investing in GPUs — you'll see a return on your investment very quickly.
Feel free to reach out if you need guidance getting started, or ask an AI. :)