shoehorn is a tool that automatically quantizes language models to fit your exact available memory by solving a per-tensor mixed-precision assignment problem rather than picking from preset quantization levels. You give it your hardware budget (8GB Mac, 16GB GPU, etc.), it measures your inference overhead, then produces a custom GGUF file that uses nearly all remaining capacity—the example shows 99.998% utilization with 13 KB slack. The workflow is: install via Homebrew with llama.cpp as backend, run `shoehorn fit <model> --serve`, get a custom quantized model optimized for your machine, then chat. It includes a browser-based model finder that ranks available models by quality-per-megabyte for your specific setup and context window needs. The quantizer is Rust from scratch (not llama.cpp code), outputs standard GGUF v3, so downstream tools can load the result. No training or fine-tuning—purely a packing problem solver. One concrete claim worth testing: whether the per-tensor optimization actually preserves quality better than preset quantizations at the same filesize.
reply