Help with EXL2/3 pls
I am trying to run EXL2/3 models using Silly Tavern. Normally I am running GGUF but I wanted to see if EXL2/3 format could provide a better lore coherence than Q4 quants.
My rig is running a 4060 with 16GB.
As an API provider I tried TabbyAPI (know a better one for EXL2/3?).
I tried it with this template (and various adjustments, tempereture etc) https://huggingface.co/Nitral-AI/Violet\_Magcap-12B/blob/main/ST%20Presets/ChatML\_Master-Import.json
But it generates gibberish only. Sometimes it runs halfway ok but there will still be grammatical errors, half words and sometimes loops (doesn't get a stop token), often it's just entire word salad.
Now wtf am I doing wrong? How do I run EXL2 or 3 locally?
Screenshot is default bot's response to "Hello".
https://preview.redd.it/k6q7nbylcauh1.jpg?width=1159&format=pjpg&auto…