No good deed goes unpunished. The sense of self entitlement some people display is staggering. FOSS project? Well, you should have done x y or z.

Also, I gave you $3 via Ko-fi, so you need to provide customer support in perpetuity and come to my house and install it. And heaven forbid you try to recoup costs!

Projects don’t just die out - a lot of them are killed (one way or another). For example, I had a fully specced out FPGA design that would capture the signal from Wii GPU and do internal upscaled resolution (think: like what dolphin emulator does but with actual hardware) not just post process sharpening. Total cost under $100 and some know how.

The amount of flack I copped for it made me shut down the github and work on it for myself. Once it’s perfected, I may post about it again but I sure as shit am not compelled to deal with the fucking peanut gallery anymore.

There are many excellent options - far too many to list. So I will briefly say - there are some really nice 4B models (like Qwen3-4B HIVEMIND, Nanbeige, IBM Granite 3B) which you should be able to run at higher quants (Q6 and up) quite nicely. Of course, there are always newer models (Gemma, Qwen3.6 - soon 3.7) etc.

Best bet is to poke around hugging face, on TheBloke, Unsloth or DavidAUs archives and see what they have in the 3-7B range that tickles your fancy. Don’t immediately jump for the newest releases - the old ones are still good. Qwen3-4B 2507 instruct is still a favourite of mine and more recently Qwen3.5-2B shows promise.

How about this?

https://yacy.net/

The Jellyfin vs Plex thing always struck me as odd. As in - why are we holding JF to a different standard to (say) Immich, Syncthing, Pi-hole or any one of a thousand different programs people self host?

Yes, JF ships multi-user accounts and client apps etc. I get it, “multi-use” is implied, so the comparison isn’t totally unfair. But there’s a difference between ‘this feature exists’ and ‘this is the primary purpose of the tool’.

The fact that you CAN share it externally doesn’t mean everyone running JF is doing that, or that it should be the benchmark the whole project is judged by.

To me, self host means “I host it, myself” not “I host it and then pretend to be Netflix for family and friends”. If that’s the use case, then of course, Plex away.

It’s cool that you CAN share JF externally, and it’s cool that Plex does that differently / better. We shouldn’t hold one to the standards of the other.

Well, it IS French. All the best evil comes from France :P

Funny, I was just thinking last night how stupid Haiku seems compared to Sonnet…and all the while, that chipper little fuck was our only hope of avoiding AI apocalypse LOL

Do wish they had used more LOCAL models, instead of cloud based. I’m pretty sure Granite would have told em to go pound sand. That thing is straight laced to point of absurdity.

There use to be one called PDFDRIVE. I mean, there still is, but there use to be too :)

I actually have a theory here…I think there’s a bare basement level that a model needs to be…anything above which, deterministic tooling can do the rest. We’ve just been yeeting into a black box.

Why that matters is this - if you can make a 450M model do what a 7B model does…that has a huge set of implications (see above examples), not least of which is for use GPU poors.

I’m doing some smoke testing on this idea right now for what I’m calling an ‘expert system’, where the model is treated like a squawk box and the infrastructure around it provides the brains (not RAG, per se. More like sidecars or tool calling). I’m liking what I see so far but there’s lots of fucking work to go. There may yet be a cheat code for some of the NVIDIA tax, if we take the work outside of the magic parrot :)

No? Just me then. How about this - 99% accurate COPD cough count…with a itty bitty convolutional model, on a $30 Adurino.

https://www.edgeimpulse.com/blog/ai-dont-like-the-sound-of-that-cough/

Why this might be cool. Different coughs correlate to different conditions (aka there is work going on in cough acoustics as a diagnostic signal / proxy for spirometry and breath sounds).

The above was trained on his coughs…it’s not far from there to “was that a healthy cough, wet cough, dry cough, wheeze? Is this a Blue Bloater or Pink Puffer?”

I’ve long suspected PoC (Point Of Care) systems could be adapted to use language models. Imagine - Qwen3.5-2B (with --mmproj) that lives on your phone…and you can point at mole or freckle and ask “hey…is this fucky or what” - and it actually KNOWS because it has access to DermNZ and can classify based on ABCDEs

Agreed. I’m all in on home lab / local LLM stuff. And entirely OUT on microslop.

(Which reminds me, I need to turn my Github into a billboard for Codeberg and then strip Github. Watching the traffic count on Github, the only clear signal I see it “bots crawl this shit daily; enjoy”, despite being politely told no)

I’m with you on this I think.

I have no problem with anyone using an AI scribe (though I would prefer one that was on device rather than cloud based). I am aware of things like Lyrebird Health that integrate with EHR management software - frankly, anything that allows the practitioner to focus more solely on the patient is a good thing. After all, they are meant to be treating the patient in front of them, not the computer screen.

The prior point about legal liability is accurate IMHO. Medical health records are functionally a legal record, and should be treated as such. Responsibility for review, redress of inaccuracies etc cannot be waved away as “ChatGPT did it”. If the practitioner is willing to take the onus of that on, and treats the scribed document with the same fidelity, chain of provenance etc as other records, I’m probably ok with it.

Requiring patients to consent to cloud-based AI scribing as a condition of access is where it gets uncomfortable, and your point about local alternatives is exactly why. If deterministic, on-device transcription exists and does the job, the justification for mandating a cloud pipeline through a psychiatric service gets pretty dicey, pretty fast.

I think I can see a way to have Dragon Dictate record the audio, convert it to text and then have on device AI pull out relevant bits to populate a template. That doesn’t abrogate the need to actually LISTEN to the patient but it might fix that ‘capture’ part of the funnel.

I hear you; I’m not wildly enamored with reddit either…but that convo is a good springboard.

I see almost everyone chasing bigger GPUs, more parameters, more more more. I figure when 9 people say “go right”, there should be at least someone that can make the plausible case for “actually, here’s why go left works”.

Eg: I think there should be some discussion about watts per token vs tokens per second.

I’m still re-writing the FAQ for my project - when it’s done (and if there’s interest) I will post it here.

On the broader topic of “the llm is the mouth, not the brain”, I just stumbled across this.

https://www.atomelm.com/index.html#what

https://www.atomelm.com/index.html#prototypes

Might turn out to be something yet, dunno. Web demo is a bit meh.

Thanks for that. I’ve been meaning to (re) disable Google Play services. I have a few older phones too that never had it to begin with. I wonder how/if Aurora Store will be impacted. Presumably, if you don’t have Google Play Services functioning, you don’t get the poison pill. But…given that Big Evil likes to just … do shit (cf the recent 4GB forced ingestion of their LLM with Chrome) I dunno.

In any case, step 1 is probably nuking that.

Granite is much more straight laced. Qwen is more expressive. Honestly, it reminds me a lot of early days with GPT 4 class models (and the benchmarks show it about matches that, too).

Cool. So what happens if I run a version of Android that doesn’t inherit Google security theater cruft? That is to say…what if the user simply…does not…upgrade the Android version to be affected by this (eg: uses an old phone or blocks OS version update?).

My phone is going on 7yrs old. Perfectly happy with it. When it breaks, I will get a phone of the same era (2nd hand or new-old stock) or investigate other options.

So, it seems to me, the winning move is not to play the game (in any one of 100 diff ways).

Or am I missing something here? Is there something that will prevent older tech from working? Because if so, I am happy to YOLO my phone and switch to a dumbphone if I have to.

Good man/woman. Nerd Valhalla awaits you :)

Hey, me too :) As my school teachers use to tell me “Great minds think alike (but fools seldom differ :)”

For me, I’m thinking of having a LLM as one layer / one container in a homelab that does some specific stuff

• queries against local docs / notes / manuals / PDFs / wiki material as the trusted knowledge layer
• uses tools for search, file lookup, shell, git, Docker, Home Assistant, calendar, etc.
• a local “Codex” / wiki layer that turns my own source material into an inspectable knowledge base
• provenance and audit trails

I want to take a screenshot of something, drop it into Syncthing from my phone, then later ask “did I fuck the pins on this?” … and for it to look up the schematics, eyeball the pins and tell me. Or I say “hey, can you grab a copy of X for me, usual params” and have the LLM instruct Sonarr/Radarr/Sabnzdb to do that. (That is, make your OWN “Alexa” with an Arduino ESP32, stick it in a room and then call it when you need it).

So instead of asking a 70B model to “know” why your media server is down, the system checks service status, logs, last config changes, prior notes, Docker state, network state, etc., then the LLM explains the result in human language. You can probably do that with a 4B (I’m testing that assumption now).

Same for “find that motherboard note,” “summarize this email thread,” “turn this into a task,” “compare this Ebay listing to my saved hardware notes,” “what did I do last time this broke,” or “run the smoke test and tell me the first real failure.”

I think small models are the shit for this because if the model only has to classify intent, route the request, render structured evidence, and talk like a normal human…then it doesn’t need to be a giant oracle. The expensive (time wise) part becomes less “make the model smarter” and more “build a better control plane around it.”

Basically: local LLM as semantic HID; expert system/tool router underneath; user owns the data and the machine.

As always, ICBW…but fuck it, I’m gonna try.

PS: I have an idea of how to apply that to coding too…but that’s a project for much later. I’ve been cooking this shit for far too long. The next thing I wanna do is a fun project for myself (that is: ROM hack a parachute and grappling gun into Super Mario Sunshine, so I can basically play “What if Super Mario Sunshine but actually Just Cause 2” on my Wii with the kids.

I’m actually thinking of pivoting my router/orchestrater entirely. I think the way forward is to look at expert systems (yes, those ancient things from the long, long ago of…1980) but with modern tooling (that can be user updated), with a small LLM in the middle that the user can talk to. That is, de-emphasize the central role of the LLM entirely; rather, make it the user-facing NLP input/output and let the real programs, running on real silicon, do the work. I might have a different use case than most, but I bet not so different (that is to say, online LLM discussion seem to gravitate around user that use LLMs for coding; Anthropic and OAI internal reports say otherwise)

Ironically, I’m writing the blurb now while waiting for smoke test #90238472398 to finish.

I’d ask why…but “because I fucking wanted to” is entirely cromulent (and 100% valid) response. Just wish it had some screenshots or videos of it in action that we could geek out over.

EDIT: I need reading glasses, clearly

https://www.youtube.com/watch?v=eGS9su_inBY

The next step for the dev (are you here?) - get IE running and post from your N64 onto this Lemmy thread. I double dog dare you :)