#memory-efficiency

#memory-efficiency

As memory prices continue to rise, it is time engineers reconsidered their applications and toolchains' voracious appetite for memory. Does a simple web page really need megabytes to show a user the modern equivalent of Hello World? Today's Windows Task Manager executable occupies 6 MB of disk space. It demands almost 70 MB before it will show a user just how much of a memory hog Chrome is these days. The original weighs in at 85 KB on disk.

Nowadays, everyone uses cross-platform hybrid desktop apps written in JavaScript, ignoring excessive CPU and RAM usage. You most likely use a hybrid, native-like, cross-platform code editor for day-to-day programming activities. It may work fine on your computer because you've upgraded your hardware, since it may have worked slowly before. If you check the resource usage of your favorite code editor, you'll see not megabytes of RAM, but gigabytes of RAM;

IBM attributes those improved characteristics vs. larger models to its hybrid architecture that combines a small amount of standard transformer-style attention layers with a majority of Mamba layers-more specifically, Mamba-2. With 9 Mamba blocks per 1 Transformer block, Granite gets linear scaling vs. context length for the Mamba parts (vs. quadratic scaling in transformers), plus local contextual dependencies from transformer attention (important for in-context learning or few-shots prompting).

Larger memory page sizes can impact memory efficiency and performance, potentially leading to less efficient memory use if pages are not fully utilized.