Wikimedia has seen a 50 percent increase in bandwidth used for downloading multimedia content since January 2024 due to AI crawlers taking its content to train generative AI models. It has to find a way to address the problem, because it could slow down actual readers' access to its pages and assets.
  • Admiral Patrick@dubvee.orgEnglish
    38·
    1 day ago

    Fuckin’ crawlers, man.

    There’s not even a reason to crawl Wikipedia. You can literally download dumps and feed them in, and those dumps are mirrored as to not DDoS Wikipedia itself.

    • The Quuuuuill@slrpnk.netMEnglish
      38·
      1 day ago

      when you’re a moron ai tech bro with a hammer, everything looks like a nail

      also

      i think they want to ddos wikipedia. that way if we want info we have to ask their chat bots

  • Churbleyimyam@lemm.ee
    8·
    1 day ago

    Is there an easy way to download Wikipedia for offline use and periodically update it? I realise it will be a lot of data.

    • atro_city@fedia.io
      121·
      1 day ago

      That what these AI crawler builders should be doing. They should be downloading the wikipedia backup or whatever it is and running their own wikipedia locally. They can download an update once a day or however often the backup is updated. Wouldn’t surprise me if it’s some poor intern had to implement a bot, was fired or moved on, and it’s just running with nobody maintaining it. All the while the C-suits are shovelling money into their pockets.

      • miguel@fedia.io
        10·
        1 day ago

        Why would an AI company hire someone when they can just tell an ai prompt to write a script to download wikipedia, and run it without even checking?