@hn_1d2a3c
5 months ago
Public discussion, not a rating submitted on Rate My AI.
We very much can, especially such a Mixture of Experts model with only 3B activated parameters. With an RTX 3070 (7GB GRAB VRAM), 32 GB RAM and an SSD I can run such models at speeds tolerable for casual use.