AWS researchers are working on developing a large language model-based debugger for databases to help organizations solve performance issues in such systems.
The new debugging framework, called Panda, is designed to work similarly to a database engineer (DBE), the company said in a blog post, adding that troubleshooting performance issues in the database can be “very difficult.”
Unlike database administrators, who are tasked with managing multiple databases, database engineers are tasked with designing, developing, and maintaining databases.
The researchers explained that Panda is an effective framework that provides context foundation to pre-trained LLMs to produce more “useful” and “in-context” troubleshooting recommendations.
Panda’s components and architecture
The framework includes four key components; grounding, validation, relevance and feedback.
Researchers define validation as the ability of the model to verify the generated response using relevant sources and produce the quote with its output so that the end user can verify it.
Convenience, on the other hand, can be defined as the framework’s ability to clearly highlight high-risk action, such as DROP or DELETE, while informing the user about the consequences of the recommended action suggested by the LLM, the researchers said.
According to the researchers, Panda’s feedback component allows the LLM-based debugger to accept feedback from the user and take it into account when generating responses.
These four components constitute the architecture of the debugger, which includes the question validation agent (QVA), the underlying mechanism, the validation mechanism, the feedback mechanism, and the availability mechanism, respectively.
While QVA identifies and filters irrelevant queries, the grounding mechanism consists of a document getter, Telemetry-2-text, and a context collector to provide more context to a prompt or query.
The researchers said that the verification mechanism consists of response verification and source attribution, adding that all these mechanisms, together with the feedback and compliance mechanism, work in the background of the natural language (NL) interface with which the corporate user interacts.
Pitching Panda against OpenAI’s GPT-4
Researchers working at AWS also pitted Panda against OpenAI’s GPT-4 model, which currently underpins ChatGPT.
“…prompting ChatGPT with database performance queries often results in ‘technically correct’ but extremely ‘vague’ or ‘generic’ recommendations, and experienced database engineers (DBE’s) s) are often rendered useless and unreliable by PostgreSQL database.