Oops, realized I didn’t answer your question about actually crawling, dig into common crawl documentation, they provide a bunch of technical data and stats that show you the scale…2-4billion pages per month
And note CC just does a sample of the pages it finds. So the more monthly dumps don’t contain all of the data afaik
And the number above are for one of the monthly dumps
https://commoncrawl.github.io/cc-crawl-statistics/plots/crawlermetrics
The index (i.e just the list of urls) of common crawl is 250gb. The actual crawl is 500tb ish. Then pull out just the text you get down to 30tb ish. And then you need that all to be in a fts index… (something like opensearch/elasticsearch) idk how big the cluster you would need for that is… maybe 2tb per data node max. So 15 pretty beefy vms (8cpu w/64gb memory) with 2tb nvme for data and a few more as controller nodes
Common crawl numbers pulled from this paper
https://aclanthology.org/anthology-files/anthology-files/pdf/L/L18/L18-1286.pdf
Cluster sizing numbers pulled from tea leaf divination
Are you the llm though…this is a mangled rephrasing of the punchline… was your prompt something like rephrase a common from a similar thread
Better Nate than lever?
Is this an llm bugging out by rephrasing?
Nice :)
Nice, ok hmm… sounds like you are using the router for internal DNS, did you set all the services to point at caddy?
Eg you could try a quick hack in hosts (assuming the caddy ip is 192.168.10.10)
192.168.10.10 grafana.domain.example
192.168.10.10 prometheus.domain.example
192.168.10.10 jellyfin.domain.example
And then navigate to one of the services over https
Main thing is all your internal DNS records need to point at caddy, and then it forwards to the real container IP
Not strictly required but could be worthwhile to get a dedicated internal dns server up… I use technitium, or there’s pi hole or a few other options …
I’m trying to highlight the absurdity of tieing buggyness to sloppyness, they are two distinct axis, not a spectrum. There are plenty of problems with LLMs but bugginess is not one of them. Idk if you’ve been watching the news lately, bug squashing in old code is up big time, can you guess why? And I wonder have you ever worked on a real codebase? Because the only perfect code is the code you wrote 5minites ago… go look at Linux kernel code… arguably the most used but of software out there, find me anything perfect in there… like what is going on here in this artisanal, hand rolled, organic code?! You’re arguing devs need more time to work on security, well guess what’s giving them more time to work on security…
Ps. Sorry to the Bluetooth dev, didn’t mean for you to catch strays, I love your code anyway!
If you are doing a dns challenge then You need to give caddy access to your domain to perform the dns challenge. Depending on domain provider that is easy or hard. It will briefly change some records that tells let’s encrypt you own the domain and then issue the cert
Or you can do http challenge, this requires you to manually configure your domain to point to a publicly accessible caddy endpoint which is probably not what you want, maybe there is a way to make it work through a cloud flare tunnel
Who is your domain provider?
Its so annoying that all code has been perfect and bug free until these silly LLM people came and made buggy code
Me: spruke gimp to artist friends
Them: gimp? Like a gimp suit?
End of conversation
Dunno why you’re getting downvotes
Envoy is what i use, or squid is an option
They can be used for training…
That’s all software, I don’t think anyone is doing back prop in hardware. Your are right about cuda, but z.ai just demonstrated that moat is evaporating quick
Idk anything about this…but here is my Google results
Don’t think golden dome is actually implemented, though just some trump delusion…but apparently this Redhawk Linux is used in…
The article states 1.5 million liters… and the paper says 1.55 x 10^6 m³
Referring to sqlite is “some database format” feels wrong
DANE provides user friction…but as OP mentioned TLS doesn’t work
Very old article, they are absolutely just using llms nowdays
Edit: this is the paper https://arxiv.org/pdf/1611.04558 the article is about
So this would work for you?
“She floats, so she’s a witch, and all witches are guilty so she’s guilty”