The paper introduces "Intelligence per Watt" (IPW) as a metric—accuracy divided by power consumption—to evaluate whether small local LLMs (≤20B parameters) can handle real user queries efficiently on consumer hardware like Apple M4 Max instead of relying on cloud inference. They tested 20+ models against 1M real queries and found local models now handle 88.7% accurately, with a 5.3x efficiency improvement since 2023 and 71.3% of queries now serviceable locally. Critically, they show local accelerators achieve 1.4x better efficiency than cloud accelerators running the same models. The actionable takeaway: for practitioners, IPW gives you a single dial to trade off capability against device power budget, and the data suggests a meaningful fraction of real workloads can shift device-side without accuracy loss—though domain variation exists and the source doesn't specify which domains work well.
One sharp question: does the 88.7% figure mean local models answered correctly, or that they matched frontier models' answers (which might both be wrong)?
reply