Dunno about the memory prices yet. The AI players were already demanding more than there likely was capacity to produce. This may just mean they get to build more of the infrastructure they wanted rather than there being a surplus of memory yet.
TurboQuant proved it can quantize the key-value cache to just 3 bits without requiring training or fine-tuning and causing any compromise in model accuracy, all while achieving a faster runtime than the original LLMs (Gemma and Mistral). It is exceptionally efficient to implement and incurs negligible runtime overhead. The following plot illustrates the speedup in computing attention logits using TurboQuant: specifically, 4-bit TurboQuant achieves up to 8x performance increase over 32-bit unquantized keys on H100 GPU accelerators.
Dunno about the memory prices yet. The AI players were already demanding more than there likely was capacity to produce. This may just mean they get to build more of the infrastructure they wanted rather than there being a surplus of memory yet.
You’re a mod- why was this post never visible until it was about a day old?
I approved it as soon as I saw it, at ~12 hours. It's been a busy month. Handshakes need manual approval for top level posts.
👍
More from the paper itself :
TurboQuant proved it can quantize the key-value cache to just 3 bits without requiring training or fine-tuning and causing any compromise in model accuracy, all while achieving a faster runtime than the original LLMs (Gemma and Mistral). It is exceptionally efficient to implement and incurs negligible runtime overhead. The following plot illustrates the speedup in computing attention logits using TurboQuant: specifically, 4-bit TurboQuant achieves up to 8x performance increase over 32-bit unquantized keys on H100 GPU accelerators.
https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
This is my quant, my math specialist. Look at his face. Look at his eyes. He's asian!