Gitlab - ERROR:root:Failed to parse AI prediction: while scanning a simple key #373

myvo · 2023-10-13T09:57:52Z

ERROR:root:Failed to parse AI prediction: while scanning a simple key
  in "<unicode string>", line 14, column 5:
        ...
        ^
could not find expected ':'
  in "<unicode string>", line 15, column 5:
        relevant lines start: 290
        ^
INFO:root:Successfully parsed AI prediction after removing 107 lines
INFO:root:Pushing PR review...
INFO:root:Pushing inline code suggestions...

I got this error when I run improve command in Gitlab MR with 49 changed files.

The text was updated successfully, but these errors were encountered:

myvo · 2023-10-13T10:27:41Z

And this error when I run improve command with some MR has only 1 changed file

ERROR:root:Could not publish code suggestion:
suggestion: {'body': "**Suggestion:** Remove unused code in the 'created' lifecycle hook\n```suggestion\n    // Remove unused code in the 'created' lifecycle hook\n    // ...\n```", 'relevant_file': 'myfil.vue', 'relevant_lines_start': 387, 'relevant_lines_end': 392}
error: 400: 400 Bad request - Note {:line_code=>["can't be blank", "must be a valid line code"]}

mrT23 · 2023-10-13T22:50:12Z

@myvo
There is no 100% success when working with AI models.
Sometimes the model fails to follow the instructions it is given (produce valid yaml, with valid MR lines).
Hence some suggestions may not be published. This is not necessarily a "bug" that can be fixed
We still provided informative log messages.

In any case, you have not provided any reproduction code. Does it always happen on a specific MR ?

myvo · 2023-10-18T08:15:23Z

Almost my MRs contains the Unicode characters. Perhaps, the error came from yaml.load at

This function didn't support Unicode characters.
Do you have any idea to fix? @mrT23

mrT23 · 2023-10-18T10:06:55Z

without a proper way to debug it, i cannot solve.
log message is not enough

try to provide a reproduction code.

mrT23 · 2023-10-24T13:45:10Z

closing. welcome to share more details, and can re-open

myvo · 2023-11-10T08:39:21Z

With the latest code of pr-agent, I have the trace log now.

2023-11-10 15:33:53.620 | INFO     | pr_agent.tools.pr_code_suggestions:_prepare_prediction:84 - Getting PR diff...
2023-11-10 15:33:53.736 | INFO     | pr_agent.tools.pr_code_suggestions:_prepare_prediction:91 - Getting AI prediction...
2023-11-10 15:34:07.969 | INFO     | pr_agent.algo.ai_handler:chat_completion:120 - AI response
2023-11-10 15:34:07.972 | INFO     | pr_agent.tools.pr_code_suggestions:run:76 - Pushing PR review...
2023-11-10 15:34:08.048 | INFO     | pr_agent.tools.pr_code_suggestions:run:78 - Pushing inline code suggestions...
2023-11-10 15:34:09.607 | ERROR    | pr_agent.git_providers.gitlab_provider:publish_code_suggestions:243 - Could not publish code suggestion:
suggestion: {'body': "**Suggestion:** Add error handling for failed API call\n```suggestion\n      if (res.err) {\n        this.$router.push(APP_ROUTES.AUTH_LOGIN);\n        this.$alertError(this.$t('page/auth/CompleteYourAccount/error/linkExpired'));\n      } else {\n        // Handle success case\n      }\n```", 'relevant_file': 'src/pages/Auth/CompleteYourAccount/CompleteYourAccount.vue', 'relevant_lines_start': 273, 'relevant_lines_end': 276}
error: 400: 400 Bad request - Note {:line_code=>["can't be blank", "must be a valid line code"]}
Traceback (most recent call last):

  File "/home/myvv/.local/lib/python3.10/site-packages/gitlab/exceptions.py", line 336, in wrapped_f
    return f(*args, **kwargs)
           │  │       └ {}
           │  └ (<gitlab.v4.objects.discussions.ProjectMergeRequestDiscussionManager object at 0x7f148c44d9c0>, {'body': "**Suggestion:** Add...
           └ <function CreateMixin.create at 0x7f148cff6200>
  File "/home/myvv/.local/lib/python3.10/site-packages/gitlab/mixins.py", line 300, in create
    server_data = self.gitlab.http_post(path, post_data=data, files=files, **kwargs)
                  │    │      │         │               │           │        └ {}
                  │    │      │         │               │           └ {}
                  │    │      │         │               └ {'body': "**Suggestion:** Add error handling for failed API call\n```suggestion:-0+3\n      if (res.err) {\n        this.$rou...
                  │    │      │         └ '/projects/896/merge_requests/3347/discussions'
                  │    │      └ <function Gitlab.http_post at 0x7f14bf9484c0>
                  │    └ <gitlab.client.Gitlab object at 0x7f148c632890>
                  └ <gitlab.v4.objects.discussions.ProjectMergeRequestDiscussionManager object at 0x7f148c44d9c0>
  File "/home/myvv/.local/lib/python3.10/site-packages/gitlab/client.py", line 1021, in http_post
    result = self.http_request(
             │    └ <function Gitlab.http_request at 0x7f14bf9481f0>
             └ <gitlab.client.Gitlab object at 0x7f148c632890>
  File "/home/myvv/.local/lib/python3.10/site-packages/gitlab/client.py", line 794, in http_request
    raise gitlab.exceptions.GitlabHttpError(
          │      │          └ <class 'gitlab.exceptions.GitlabHttpError'>
          │      └ <module 'gitlab.exceptions' from '/home/myvv/.local/lib/python3.10/site-packages/gitlab/exceptions.py'>
          └ <module 'gitlab' from '/home/myvv/.local/lib/python3.10/site-packages/gitlab/__init__.py'>

gitlab.exceptions.GitlabHttpError: 400: 400 Bad request - Note {:line_code=>["can't be blank", "must be a valid line code"]}

Thanks for your support!

myvo · 2023-11-10T08:42:20Z

And here is the log of my first issue with unicode:

2023-11-10 15:40:11.681 | INFO     | pr_agent.tools.pr_reviewer:_prepare_prediction:148 - Getting PR diff...
2023-11-10 15:40:11.826 | INFO     | pr_agent.tools.pr_reviewer:_prepare_prediction:150 - Getting AI prediction...
2023-11-10 15:40:22.343 | INFO     | pr_agent.algo.ai_handler:chat_completion:120 - AI response
2023-11-10 15:40:22.343 | INFO     | pr_agent.tools.pr_reviewer:run:114 - Preparing PR review...
2023-11-10 15:40:22.345 | ERROR    | pr_agent.algo.utils:load_yaml:294 - Failed to parse AI prediction: mapping values are not allowed here
  in "<unicode string>", line 14, column 32:
          relevant line: +  isReady: false,
                                   ^
2023-11-10 15:40:22.366 | INFO     | pr_agent.algo.utils:try_fix_yaml:305 - Successfully parsed AI prediction after removing 15 lines
2023-11-10 15:40:22.367 | INFO     | pr_agent.tools.pr_reviewer:run:118 - Pushing PR review...

The other log:

2023-11-10 15:52:49.062 | INFO     | pr_agent.tools.pr_reviewer:_prepare_prediction:148 - Getting PR diff...
2023-11-10 15:52:49.188 | INFO     | pr_agent.tools.pr_reviewer:_prepare_prediction:150 - Getting AI prediction...
2023-11-10 15:52:57.937 | INFO     | pr_agent.algo.ai_handler:chat_completion:120 - AI response
2023-11-10 15:52:57.937 | INFO     | pr_agent.tools.pr_reviewer:run:114 - Preparing PR review...
2023-11-10 15:52:57.938 | ERROR    | pr_agent.algo.utils:load_yaml:294 - Failed to parse AI prediction: mapping values are not allowed here
  in "<unicode string>", line 18, column 103:
     ...  isSelectCurrentPageMode ? true : selectedItemIds.length > 0;
                                         ^
2023-11-10 15:52:57.947 | INFO     | pr_agent.algo.utils:try_fix_yaml:305 - Successfully parsed AI prediction after removing 7 lines
2023-11-10 15:52:57.948 | INFO     | pr_agent.tools.pr_reviewer:run:118 - Pushing PR review...

Many thanks, @mrT23 !

mrT23 · 2023-11-10T16:54:34Z

Hi @myvo

The issue you encountered has nothing to do with 'Unicode characters'. It is a limitation of LLM, that don't always obey commands.

We specifically ask the LLM to output yaml with block scalar format.

Most of the time the LLM does that. Sometimes not.
When not, we experience problems with parsing their answers.

The error mechanism we have worked well - they disqualified just the suggestions (with the bad parsing), but output everything before. And we provide logs. The fact that the log says 'error' is just for emphasis. In practice, the error was catched and mitigated. I do believe you received an output.

Anyway, I open a PR with another fallback mechanism for your specific type of error:
#444
if you are able to reproduce the issue, try to run with the PR, I think it will work.

In any case, remember - no 100% success all the time is guaranteed when working with LLMs.

myvo · 2023-11-11T07:37:42Z

Got it. Let me try to run the PR #444
Many thanks @mrT23 !

mrT23 added the clarification needed label Oct 18, 2023

mrT23 closed this as completed Oct 24, 2023

mrT23 reopened this Nov 10, 2023

mrT23 added solved and removed clarification needed labels Nov 16, 2023

mrT23 closed this as completed Nov 19, 2023

mrT23 mentioned this issue Nov 19, 2023

Pr-agent /improve ERROR", "no": 40, Failed to parse AI prediction #461

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gitlab - ERROR:root:Failed to parse AI prediction: while scanning a simple key #373

Gitlab - ERROR:root:Failed to parse AI prediction: while scanning a simple key #373

myvo commented Oct 13, 2023

myvo commented Oct 13, 2023

mrT23 commented Oct 13, 2023 •

edited

Loading

myvo commented Oct 18, 2023

mrT23 commented Oct 18, 2023

mrT23 commented Oct 24, 2023

myvo commented Nov 10, 2023

myvo commented Nov 10, 2023 •

edited

Loading

mrT23 commented Nov 10, 2023 •

edited

Loading

myvo commented Nov 11, 2023

Gitlab - ERROR:root:Failed to parse AI prediction: while scanning a simple key #373

Gitlab - ERROR:root:Failed to parse AI prediction: while scanning a simple key #373

Comments

myvo commented Oct 13, 2023

myvo commented Oct 13, 2023

mrT23 commented Oct 13, 2023 • edited Loading

myvo commented Oct 18, 2023

mrT23 commented Oct 18, 2023

mrT23 commented Oct 24, 2023

myvo commented Nov 10, 2023

myvo commented Nov 10, 2023 • edited Loading

mrT23 commented Nov 10, 2023 • edited Loading

myvo commented Nov 11, 2023

mrT23 commented Oct 13, 2023 •

edited

Loading

myvo commented Nov 10, 2023 •

edited

Loading

mrT23 commented Nov 10, 2023 •

edited

Loading