qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.
I have one in a laptop with 64gb of ram that I bought before things went too crazy for $2200. Not sure how much one is now.
Anyway, you don’t need a datacenter and that’s why I personally think the current AI “boom” is a bubble around a nascent tech that’s not really ready for lots of the situations it’s been sold for. 🤷♂️
I haven’t really found that to be the case for my uses, but getting much faster decode is really useful for me… Have you tried running the MoE model at a higher quant? I usually use Q6_K.
I run both of them at Q8, unquantized KV cache and same settings.
35b does really well at most tasks but once the dependencies get complicated and the tool calls reach into the hundreds, 35b struggles where 27b still excels.
So generally I use 27b unless I need it fast, then I swap to 35b or 122b depending on what I need.
qwen 3.6 27b runs on an AMD AI MAX 395+ fairly well.
I have one in a laptop with 64gb of ram that I bought before things went too crazy for $2200. Not sure how much one is now.
Anyway, you don’t need a datacenter and that’s why I personally think the current AI “boom” is a bubble around a nascent tech that’s not really ready for lots of the situations it’s been sold for. 🤷♂️
Try Qwen 3.6 35B-A3B on your hardware instead; it’s much faster.
27b is much better though…
And it still gets 15-20 t/s with MTP which is plenty fast.
I haven’t really found that to be the case for my uses, but getting much faster decode is really useful for me… Have you tried running the MoE model at a higher quant? I usually use Q6_K.
I run both of them at Q8, unquantized KV cache and same settings.
35b does really well at most tasks but once the dependencies get complicated and the tool calls reach into the hundreds, 35b struggles where 27b still excels.
So generally I use 27b unless I need it fast, then I swap to 35b or 122b depending on what I need.
I’ll check it out!
deleted by creator