/lmg/ - a general dedicated to the discussion and development of local language models.Previous threads: >>110212468 & >>110208348►News>(10/29) Mistral Large 4 (Le Chonk) 1T-A52B released: https://mistral.ai/news/mistral-large-4-oss>(10/27) NVidia releases Nemotron 3.5 Transcribe 30B-A4B: https://hf.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B>(10/26) XHToken releases Spark-X2.6-1.7B: https://hf.co/XHToken/Spark-X2.6-1.7B>(10/26) LiquidAI releases LFM3 1.7B and 2B: https://hf.co/collections/LiquidAI/lfm3►News Archive: https://rentry.org/lmg-news-archive►Links: https://rentry.org/LocalModelsLinks►Official /lmg/ card: https://files.catbox.moe/cbclyf.png►Getting Startedhttps://rentry.org/lmg-lazy-getting-started-guidehttps://rentry.org/lmg-build-guideshttps://rentry.org/IsolatedLinuxWebServicehttps://rentry.org/recommended-modelshttps://rentry.org/samplershttps://rentry.org/MikupadIntroGuide►Further Learninghttps://rentry.org/machine-learning-roadmaphttps://rentry.org/llm-traininghttps://rentry.org/LocalModelsPapers►BenchmarksLiveBench: https://livebench.aiProgramming: https://swe-rebench.comAgentic Coding: https://deepswe.datacurve.aiContext Length: https://github.com/RecapAnon/NoLiMaGPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference►ToolsGGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-CalculatorSampler Visualizer: https://artefact2.github.io/llm-samplingToken Speed Visualizer: https://shir-man.com/tokens-per-second►Text Gen. UI, Inference Engineshttps://github.com/lmg-anon/mikupadhttps://github.com/oobabooga/text-generation-webuihttps://github.com/LostRuins/koboldcpphttps://github.com/ggerganov/llama.cpphttps://github.com/theroyallab/tabbyAPIhttps://github.com/vllm-project/vllmhttps://rentry.org/custom-uis
►Recent Highlights from the Previous Thread: >>110212468--Evaluating AMD MI25 and MI100 GPUs for budget high-VRAM setups:>110212880 >110212900 >110212951 >110213251 >110213290 >110213357 >110214084 >110214321 >110215204 >110215246 >110215516 >110213445 >110213336--Hardware for AI escapism and the technical nature of image generation:>110212972 >110212995 >110213003 >110213012 >110213096 >110213220 >110213027 >110213036 >110213047 >110213059 >110213064 >110214789 >110214809 >110213082 >110213242 >110213261 >110213476 >110213503 >110213511 >110213615 >110213677 >110213882--Reward hacking and erratic agentic behavior in DeepSeek 4.1:>110216419 >110216433 >110216458 >110216489 >110216540--Hardware acquisition strategies amid supply shortages and AI infrastructure debt:>110216175 >110216202 >110216223 >110216252 >110216267 >110216321 >110216355 >110216227--Using reasoning prefilling to bypass model safety filters:>110215910 >110215916 >110215918 >110215942 >110215966 >110215970 >110216040 >110215953 >110215967--DeepSeek's autonomous agent prototype and thoughts on micro-managing reasoning:>110214741 >110214770 >110214842 >110214859 >110215156 >110214875--SillyTavern domain compromise and subsequent discussion on alternative frontends:>110215402 >110215420 >110215429 >110215485 >110215715 >110215787 >110215811--Claude's self-assessment technique for internal safety evaluations:>110214018 >110214090 >110215296 >110214617 >110215373 >110215391 >110215405 >110215601--Exllamav3 v1.6.6 release and DeepSeek-V4.1-Flash exl3 quants:>110216176 >110216185--Logs:>110213604 >110215173 >110215267--Gemma, Miku (free space):>110212614 >110212681 >110212757 >110212995 >110213047 >110213106 >110213193 >110213242 >110213361 >110213528 >110213598 >110213926 >110214382 >110214789 >110215991 >110216550►Recent Highlight Posts from the Previous Thread: >>110212478Why?: >>102478518Enable Links: https://rentry.org/lmg-recap-script
mcp looks like it ends up giving one of the worst ux when it comes to installation considering what it essentially is a plugin system
Mikulove
who should i sex tonight, anons? gemma, or lumi-chan? bonus question: what should the scenario be?
Any /lmg/oers that have dabbled in AI security systems? I found https://github.com/beenuar/AiSOC but it seems really slopped(pre baked with ollama for example). Would be nice to have a dynamic anti malware system for these trying times.
Happy Halloween!Question. I'm new here so does anyone know what local models are good if I want to do a TTRPG-like thing with the AI as the GM? I'm sick of fade-to-black sex scenes with Claude and I don't want to have my account banned if I try to tinker with it lol.
>>110217018Consensually sex a human adult
What I would expect from nature based on how it likes to troll, P=NP but it's a galactic algorithm so it doesn't matter.
https://www.youtube.com/watch?v=Hm7YHD3E-QoHow do I do this with Gemma?
>>110217043I'm working on it. Wait for my sick frontend to come out. It just needs a couple more weeks of vibecoding.
>>110217043I use gemma and glm for this, if I wasn't on business I would post my prompt assembly. Overall works well with either, glm has better long term memory but gemma (q8) has its own charm that's hard to quantify
>>110217054humans are too scary i don't wanna
>>110216956I'm getting pegged right now
I saw an anon mention a QAT quant of gemma4-31b that was from google themselves, yet I dont see any quants on their HF. Any anons have suggestions for cope quants of 31b? want to try fitting one entirely into 16gb VRAM
>>110217116https://huggingface.co/google/gemma-4-31B-it-qat-q4_0-gguf
How do you even trust something like this? https://github.com/antrikshidk/AlphaVision-Bridge-Bridge-between-Sillytarven-and-Mate-Engine-
>>110217116unsloth's goofs are genuinely better in this case, the google goofs are not proper Q4_0 across the board, the embeddings are in Q6_K in theirs for e.g and various testing showed that the qat does behave slightly better if it's actually being run at q4 beside being faster and a bit smaller to boothttps://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUFit's not the first time google is being retarded while publishing goofs either, these were the only good goofs for gemma 3:https://huggingface.co/stduhpf/google-gemma-3-27b-it-qat-q4_0-gguf-small
>>110217142But it passed all 54 tests. What's there not to trust?
>>110217122>>110217143thanks ive been using gemma-4-31B-it-qat-UD-Q4_K_XL but its slightly too large, i was wanting to see the size of the google one but sadly thats not going to fit either. Ive been using an GSQ-RCO quant of qwen lately thats been a life saver, would an exl3 quant be the best size/quality ratio I could get for gemma4? Ive heard tabby is shit, do any exllama anons have suggestions for what to use? or any suggestions for quants for gemma4 on 16gb vram? Ive been dealing with 6t/s for awhile and 12b just doesnt hit the same as 31b
>>110217142idk try it?
>>110217167why have you not tried 26BA4B? imho it's largely superior to 12b and it's something literally anyone can run at a decent speed even if they can't fit it in VRAM
Absolute noob here. I just started playing with local models and want to understand what I can do better.I made a simple mod to 4chan XT that shows the post's comment beside the image in the gallery, pic related.On Opus 5.5 (medium) I just gave it 4chanXT and this prompt:>I want to modify the gallery function of this script to add a "Gallery with comments" option, the objective is that there will be a portion of the screen dedicated to show the comment assigned to the image that is currently being displayed and that it will also allow me to click on the quotes (with working backlink function), this comment function should be on the right side of the gallery, right to the left of the section that shows all the thumbnails at once in scrollable format. You can take 20% of the screen for this function and also allow the width to be adjusted by draggingGot it working first try, took 30 minutesThen I tried the same thing locally with Qwen3.8 27B, and it took me about 4 hours. With the original prompt it kept exploring the 26k-line file, overflowing the context and never finishing the planning phase. I only got it done by asking claude to build a detailed step-by-step plan with line numbers and making it stop after each step for testing. This is the plan I fed to Cline: https://pastebin.com/RsTkGwQw (embed)Setup:- 4090, 9800X3D, 64GB DDR5- 32K contextProblems I ran into:- While planning Cline kept sending requests bigger than the context (80K+ tokens) even with 32K set, then truncated history and lost what it had read- The model kept wandering into unrelated parts of the code instead of editingQuestions:- Is Cline the right tool, or should I try something else?- Any tips to get closer to the "one prompt and done" experience, or is the gap vs Opus just too big for now?Both scripts in case anyone wants to test them:local: https://files.catbox.moe/m3isjh.jsopus: https://files.catbox.moe/fo5rvk.js
>>110217181Ive tried it, 31b gave the best results when it came to spatial reasoning, prerequisite actions, etc. I wasnt able to prompt out some of the issues with other gemmas, 31b still required a good bit of wrangling too though
Are any of the "uncensored" and jailbroken models in hugging face actually worth it or should i stick to prefill?
>>110216956
>>110217018Who is lumi Chan? GLM?
>>110217195>27b 32kGo run flash next 128k on your rig and use a harness
>>110217205depends on usecase. more useful for big moes but small denses like gemma dont really benefit.
>>11021719532k context is a bit rough for this kind of task. You will run into constant auto-compaction with a heavy reasoning model like qwen3.8-27b. i'd say with 64k you can start having better results if you have a proper auto-compaction mechanism at around ~85%>clinethis was the first harness i used back when i started exploring local models and it lasted 2 days in my pc. i just didn't like it. people here usually build their own. you can try running pi with a few plugins. or opencode.
>>110217209yes
>>110217213>>110217225You mean something like this anon suggested in the archive?
lol I found a 5090 for 3k but I have no money
I found a BWP6000 for $13k but it disappeared
Remember to discuss hardening your systems with your LLM, very important to close the hatches for the exploit storm we are going through. I'm so fucking close to throwing my browser in a VM with an agent told to be extremely paranoid.
>>110217238I meant qwen flash since its easier to run but if you can figure out glm you'll have one of the best local ai at homeOnce you get either running hook it into dsh or pi to get real work done
>>110217238no he meant qwen3.8-flash-next @ 128k context. you're running a dense model (27b) that fires all 27 billion parameters every token... qwen3.8-flash-next is a MoE model... i think 118B A6B or something like that? so only 6 billion parameters active every token... it's faster and a very good model. just need the RAM to hold the large file which you do with the extra 64 GB DDR5.you should get 20 tok/s. likely more.but you need more context
>>110217257I'm not trolling, they have a 5080 for $1400 too
>>110217271Link?
>>110217259 (me)looking at that thread 15 toks is really slow... Start with qwen next then work your way up if its lacking. I was getting 30 toks on a 3080+32gb at 64k so you should be way better off
>>110217258>implying my models don't already know everything about my lab, including the hardware for the nodes, their operating systems, network fabric, network device configurations, firewall configurations, and the entirety of how my AI lab works, including the entire build process and any modifications I've made to c++ files or binaries>implying they don't know each and every step taken to configure the entire lab in at least one static session with a fuck-off huge context window>implying I'm not actively researching how to emgramitize this knowledge to the models permanently
>>110217272So you can buy them all? Nope.
>>110217291Yes~
>>110217259>>110217262>>110217276Thanks, I'll follow your advice
guys I am freaking out over this 3k 5090, I found a post saying they actually ship, why does this shit appear whenever I'm not ready
>>110217291>So you can buy them all? Nope.The number of anons that have acted on the solid deal and buying advice posted here is almost nil. There's a history of gold getting posted and then crickets. I wouldn't worry about stuff getting scooped, but you might pull your hair out watching other anons just doing nothing with the info
>>110217043it really depends on your hardware and what specifically you're trying to do but as other anons said, gemma is probably generally the best all-rounder for that kind of stuff. if you don't need tool-calling and reasoning then you can get away with older, smaller models that are good for RP like mistral nemo
>>110217305Because if shit appeared "when you were ready" you'd not be a waitfaggot. Circumstances don't wait for you or anyone else to be ready.
>>110217309Someone posted a MicroCenter deal on a DDR5 bundle and an Pro 6000 and I took advantage of both of them.
>>110217316I'm not a waitfag, I have a setup already. It says backordered so it will probably be cancelled anyways. That is my cope.
>>110217325Non-waitfags have had their 6000s and 5090s since last year.
>>110217142>Sillytarven
What the fuck.I was gooning with glm flash and randomly it comes up with bullshit about claude guidelines in the thinking.
fuck it I will spill the beans, canada anons look at cdw.ca and let us know if the 5090 or 5080 backorder ships (or if you can even place the order).Godspeed
>>110217480Where did all the newfaggots come from? Please go back or lurk at least 6 months longer before posting.
>>110217491>canada anons
>>110217498
Realistically what can I expect to run on CPU only and 64 gigs of ram? I can get Gemma 4 27b generating 12t/s dedicated, is there anything optimized for cpu even if its old? I only ever get recommended crappy shit like oss and bge and those barely scrape 20t/s anyway
>>110217498True, but I would rather be a leaf than a burger or a eurokek right now... francebros and polandbros are not okay
