How might we incorporate LLMs to summarize the data presented to the user?

[jayantbh]

The UI offers neat ways to understand your four-keys and relative score based on DORA Quickcheck. But what are some ways we can come up with to create some kind of a report powered by any LLMs?
Perhaps the LLM could also be self-hosted within the container? Thus making even that kind of a thing fully self-contained in terms of where your data goes?

[instinxt]

LLMs like gemini 1.5 that is currently free for devs with a good enough code interpretation benchmark can be helpful in finding code changes issues that occur and can be reflected in reports with CFR(change failure rate), maybe we can also find RCA for the code issues using LLMs and make sure they don't occur again.

Hope this is also helpful for your question @jayantbh

[jayantbh]

The app does not access code level data right now simply out of concerns for not storing code related data in the DB.
RCA's, while a golden idea, might need a level of access a lot of people would be uncomfortable with.

What do you think? How do you feel about it?

[instinxt]

If we have this limitation, then LLM can take the issue description as an input and provide a checklist for the developer to go through for fixing the problem, thats the minimal integration that can be done. The checklist can be verified/modified by the maintainer further for certainity.

[jayantbh]

Sounds like an idea, but I would imagine that most devs would not diligently add/maintain such checklists on their PRs. So either this needs a mechanism to enforce such checklists, or such PRs will get excluded from the reports/RCA (unless some other way is used).

Ideally having a way to analyse the code would be amazing... but yeah, security concerns are too great.