Limiting Description and Commit Messages Length

[okotek]

PR Type:

Enhancement

---

PR Description:

This PR introduces a method to limit the length of the description and commit messages to a maximum number of tokens. This is achieved by adding a new function 'clip_tokens' that clips the number of tokens in a string to a maximum number. The function is then used to limit the length of 'description' and 'commit_messages_str' in the PR reviewer.

---

PR Main Files Walkthrough:

• pr_agent/algo/pr_processing.py: A new function 'clip_tokens' is added. This function takes a string and a maximum number of tokens as input and returns the string clipped to the maximum number of tokens.
• pr_agent/tools/pr_reviewer.py: The 'clip_tokens' function is imported and used to limit the length of 'description' and 'commit_messages_str'.

[okotek]

/describe

[github-actions]

PR Analysis

• 🎯 Main theme: Adding a function to clip the number of tokens in a string to a maximum limit and applying it to PR descriptions and commit messages
• 📌 Type of PR: Enhancement
• 🧪 Relevant tests added: No
• ✨ Focused PR: Yes, the PR is focused as all changes are related to the implementation of the token clipping functionality
• 🔒 Security concerns: No, the changes made in this PR do not introduce any apparent security concerns.

PR Feedback

• General suggestions: The implementation of the token clipping function seems to be efficient and well-documented. However, it would be beneficial to add unit tests to ensure the function behaves as expected under different scenarios.

• 🤖 Code feedback:

  • relevant file: pr_agent/algo/pr_processing.py
    suggestion: Consider using a more efficient method to find the end position. For example, you could use a generator expression with islice from itertools to avoid enumerating all words. This would be more efficient for large texts. [important]
    relevant line: for i, token in enumerate(words):

  • relevant file: pr_agent/algo/pr_processing.py
    suggestion: The heuristic of 2.5 tokens per word might not be accurate for all languages or types of text. Consider making this a configurable parameter, or better yet, consider a more robust method of estimating tokens. [medium]
    relevant line: max_words = max_tokens // 2.5

  • relevant file: pr_agent/tools/pr_reviewer.py
    suggestion: The magic number 500 is used here for the maximum number of tokens. It would be better to define this as a constant or a configurable setting, to improve maintainability and flexibility. [important]
    relevant line: self.vars["description"] = clip_tokens(self.vars["description"], 500)

How to use

To invoke the PR-Agent, add a comment using one of the following commands:
/review [-i]: Request a review of your Pull Request. For an incremental review, which only considers changes since the last review, include the '-i' option.
/describe: Modify the PR title and description based on the contents of the PR.
/improve: Suggest improvements to the code in the PR.
/ask <QUESTION>: Pose a question about the PR.

To edit any configuration parameter from 'configuration.toml', add --config_path=new_value
For example: /review --pr_reviewer.extra_instructions="focus on the file: ..."
To list the possible configuration parameters, use the /config command.

[mrT23]

i really don't like the 2.5 factor

[mrT23]

far better is to encode the original, and estimate the tokens-to-chars ratio

[mrT23]

why only for reviewer ? what about other tools

better to edit 'self.git_provider.get_pr_description(),'
'self.git_provider.get_commit_messages()'