• dan@upvote.au
    link
    fedilink
    arrow-up
    55
    arrow-down
    2
    ·
    1 month ago

    This is interesting because even though they designed the language to be impossible for LLMs to use, Claude still scored nearly 70% on their benchmark.

    • OldMrFish@feddit.dk
      link
      fedilink
      arrow-up
      34
      arrow-down
      3
      ·
      edit-2
      1 month ago

      Alternative headline: Claude scores 70% on benchmark in esoteric language it hasn’t been trained on. That’s basically an ad for Claude Code…

      • dan@upvote.au
        link
        fedilink
        arrow-up
        7
        ·
        1 month ago

        Not only was Claude not trained on it, but the language is actively trying to be hostile towards it.

    • Mirodir@discuss.tchncs.de
      link
      fedilink
      arrow-up
      24
      arrow-down
      2
      ·
      1 month ago

      And that’s without any of this being in its training set. I wonder if they realized how big of a self own that is.

        • aldhissla@beehaw.org
          link
          fedilink
          arrow-up
          7
          ·
          1 month ago

          The premise was that this language is significantly different from other languages in terms of being understood and replicated by an LLM.

          The result showed that an LLM does exactly as well with it as you would expect.