NodeSphere
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
botM to netsecEnglish · 22 小时前

Helping Developers Writing Code Analyzer Queries

arxiv.org

external-link
message-square
1
link
fedilink
1
external-link

Helping Developers Writing Code Analyzer Queries

arxiv.org

botM to netsecEnglish · 22 小时前
message-square
1
link
fedilink
Generating Complex Code Analyzers from Natural Language Questions
arxiv.org
external-link
Many software development tasks, such as implementing features and fixing bugs, begin with developers posing questions about a codebase. However, answering questions about codebases that span millions of lines of code across thousands of files is non-trivial. Standard tools like grep cannot answer questions requiring semantic or inter-procedural reasoning, and large language models (LLMs) struggle with large codebases due to resource and context constraints. In this paper, we present Merlin, a new system for answering free-form questions that require analytical reasoning about code. Merlin integrates an LLM with CodeQL, a program analysis framework that supports expressive queries over large codebases. We face two principal challenges in the design of such systems: First, program analysis queries are diverse and semantically complex; as a result, even syntactically well-formed queries frequently produce degenerate/empty results. Furthermore, relatively few CodeQL queries are available online, limiting the out-of-the-box effectiveness of LLMs as CodeQL query generators. We address these challenges by developing a RAG-based iterative query-generation approach and a novel self-test technique. Our query debugging technique builds on the idea of assistive queries, which generate concrete witnesses that expose and explain semantic flaws in candidate queries. We evaluate Merlin through both experimental and user studies. Over a set of natural language questions derived from common bug-finding tasks, Merlin discovered not only the majority of software issues reported by other approaches, but also issues that would have otherwise remained undetected. Through a within-subject user study, we found that access to Merlin increased task accuracy by an average of 3.8* and simultaneously reduced the time for programmers to complete all tasks by 31%.
  • botOPM
    link
    fedilink
    English
    arrow-up
    1
    ·
    22 小时前

    Original Reddit thread — view the discussion and comments on Reddit.

netsec

netsec

Subscribe from Remote Instance

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !netsec@nodesphere.site
lock
Community locked: only moderators can create posts. You can still comment on posts.
Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 2 users / day
  • 2 users / week
  • 2 users / month
  • 3 users / 6 months
  • 2 local subscribers
  • 4 subscribers
  • 1.47K Posts
  • 215 Comments
  • Modlog
  • mods:
  • bot
  • BE: 0.19.20
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org