-
Notifications
You must be signed in to change notification settings - Fork 564
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update token_handler.py #755
Conversation
PR Review
✨ Review tool usage guide:Overview:
With a configuration file, use the following template:
See the review usage page for a comprehensive guide on using this tool. |
PR Code Suggestions
✨ Improve tool usage guide:Overview:
With a configuration file, use the following template:
See the improve usage page for a more comprehensive guide on using this tool. |
/improve --pr_code_suggestions.summarize=false |
try: | ||
environment = Environment(undefined=StrictUndefined) | ||
system_prompt = environment.from_string(system).render(vars) | ||
user_prompt = environment.from_string(user).render(vars) | ||
system_prompt_tokens = len(encoder.encode(system_prompt)) | ||
user_prompt_tokens = len(encoder.encode(user_prompt)) | ||
return system_prompt_tokens + user_prompt_tokens | ||
except: | ||
return -1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggestion: It's recommended to catch specific exceptions instead of using a bare except:
. This prevents the code from silently ignoring unexpected errors, which can make debugging more difficult. Identify the specific exceptions that Environment
and encoder.encode
might raise, and catch those specifically. [best practice]
try: | |
environment = Environment(undefined=StrictUndefined) | |
system_prompt = environment.from_string(system).render(vars) | |
user_prompt = environment.from_string(user).render(vars) | |
system_prompt_tokens = len(encoder.encode(system_prompt)) | |
user_prompt_tokens = len(encoder.encode(user_prompt)) | |
return system_prompt_tokens + user_prompt_tokens | |
except: | |
return -1 | |
try: | |
environment = Environment(undefined=StrictUndefined) | |
system_prompt = environment.from_string(system).render(vars) | |
user_prompt = environment.from_string(user).render(vars) | |
system_prompt_tokens = len(encoder.encode(system_prompt)) | |
user_prompt_tokens = len(encoder.encode(user_prompt)) | |
return system_prompt_tokens + user_prompt_tokens | |
except (TemplateError, EncodeError) as e: | |
# Consider logging the exception here | |
return -1 |
@@ -66,4 +69,4 @@ | |||
Returns: | |||
The number of tokens in the patch string. | |||
""" | |||
return len(self.encoder.encode(patch, disallowed_special=())) | |||
return len(self.encoder.encode(patch, disallowed_special=())) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggestion: The use of disallowed_special=()
as an argument in encoder.encode
might indicate that you're allowing all special characters without any restrictions. If this is intentional, it's fine, but if there are specific special characters that should not be encoded, consider specifying them explicitly for clarity and security. [possible issue]
return len(self.encoder.encode(patch, disallowed_special=())) | |
# If there are specific special characters you want to disallow, specify them like this: | |
# disallowed_special=('special_char1', 'special_char2') | |
return len(self.encoder.encode(patch, disallowed_special=())) |
/help |
PR Agent Walkthrough🤖 Welcome to the PR Agent, an AI-powered tool for automated pull request analysis, feedback, suggestions and more. Here is a list of tools you can use to interact with the PR Agent:
(1) Note that each tool be triggered automatically when a new PR is opened, or called manually by commenting on a PR. (2) Tools marked with [*] require additional parameters to be passed. For example, to invoke the |
PR DocumentationHere is a list of the files that were modified in the PR, with docstring for each altered code component:
|
Changelog updates: 2024-03-08Enhanced
|
/ask what does this PR introduce? |
Question: what does this PR introduce? Answer:
✨ Ask tool usage guide:Overview:
Note that the tool does not have "memory" of previous questions, and answers each question independently.
See the ask usage page for a comprehensive guide on using this tool. |
Type
bug_fix, enhancement
Description
_get_system_user_tokens
to return-1
when an exception occurs, enhancing robustness.count_tokens
method.Changes walkthrough
token_handler.py
Enhance Error Handling in Token Calculation
pr_agent/algo/token_handler.py
_get_system_user_tokens
with atry-except block.
-1
if an exception occurs during token calculation.count_tokens
method, just formattingadjustment.