Console logging output can not display non-ASCII charactors correctly

[ghost]

Type of JetBrains IDE (IntelliJ, PyCharm, etc.) and version

pycharm pro 2021.2.3

Operating System

Windows11 Pro 21H2 Chinese version ()

TeXiFy IDEA version

0.7.12

What I did (steps to reproduce)

Minimal example to reproduce the problem

file 1. main.tex

\documentclass{article}
\usepackage{ctex} % the package ctex is for chinese output
\begin{document}
    \cite{测试test} % add a cite with non-ASCII charactor
    \bibliographystyle{plain}
    \bibliography{bibfile}
\end{document}

file 2. bibfile.bib

@phdthesis{测试test,
  title  = {测试test}
}

use engine latexmk with -xelatex to compile, and all file is in utf-8 encoding.

Expected behavior

the warning should be show like LaTeX Warning: Citation `测试test' on page 1 undefined on input line 4.

Actual behavior

But it shows LaTeX Warning: Citation `娴嬭瘯test' on page 1 undefined on input line 4.

and 娴嬭瘯 is 测试 display in error encoding, it should display in utf-8, but actually it display in GBK.

What's more

It's a encoding error, Windows in Chinese use GBK as its default encoding, but I set UTF-8 in pycharm and Windows


set UTF-8 as default encoding with cmd and programs

Use cmd or powershell to complie the tex file, the logging will not display in error encoding.
And others in pycharm like python output non-ASCII charactors in console is correct.

[PHPirates]

Thanks for the extensive report! I will have a look, but in order to make sense of your information I think I first need to install some fonts (I'm on Linux):

[ghost]

Thank you to reply me so fast!
I recommand you to use these font:
https://github.com/be5invis/Sarasa-Gothic/releases
or
https://github.com/adobe-fonts/source-han-sans/

[ghost]

and I guess linux or macOS will not meet the bug, they don't have confusion system file encoding.(en_US.UTF-8 or zh_CN.UTF-8) But
Windows is different :(

I guess the method that this plugin to call IDE console cause this problem.

(not sure, maybe is a problem of jetbrain IDE or Windows)

[PHPirates]

Thanks, I installed Sarasa Gothic and I think I understand the problem. However, as you guessed indeed on Linux I cannot reproduce the problem:

I will try to reproduce it on Windows. Indeed, it sounds like an IntelliJ bug, but not 100% sure.

[PHPirates]

Okay I installed the kaiti and simhei fonts and can reproduce the problem on Windows, and for me it looks even worse:

but I think I found a solution: go to Help > Edit custom VM options, add -Dfile.encoding=UTF-8 and restart PyCharm. Does that solve it for you as well?

If it works, we should definitely add this to the wiki.

[ghost]

Yes! It works!
I added -Dconsole.encoding=UTF-8 to the VM options before, not worked.
But your solution -Dfile.encoding=UTF-8 works!
and I think 测试 is 测试 display in Windows-1252 encoding?

import java.io.UnsupportedEncodingException;

public class main {
    public static void main(String[] args) throws UnsupportedEncodingException {
        String a = "测试";
        String b = new String(a.getBytes("Windows-1252"), "UTF-8");
        // https://en.wikipedia.org/wiki/Windows-1252
        // Windows-1252 used by default in the legacy components of Microsoft Windows for English
        System.out.println(b);
        // output: 测试
    }
}

This problem will happen if

1. use Windows
2. use a language has some strange charactors, like Chinese, Japanese, Korean.

and add -Dfile.encoding=UTF-8 can solve it.
Thank you!
谢谢!

[PHPirates]

used by default in the legacy components of Microsoft Windows

I didn't know Windows had any components that are not legacy 😆

Added it to the wiki: https://github.com/Hannah-Sten/TeXiFy-IDEA/wiki/Unicode

别客气!