Code coverage in editor

[fenuks]

What?
Seeing which lines in currently opened buffer are covered by tests. Many IDEs provide that. Below are few examples:

• Intellij
• Gitlab code review
• Visual Studio

Why?
It helps tremendously during writing tests. Most coverage plugins allow exporting report as HTML, but it's tedious to navigate and switch context.

Potential existing implementations:

• https://github.com/ruanyl/coverage.vim
• https://github.com/mgedmin/coverage-highlight.vim
• https://github.com/google/vim-coverage
• https://github.com/alepez/vim-llvmcov
• https://github.com/dsych/blanket.nvim (written in Lua, JaCoCo only)
• https://github.com/rafaelsq/nvim-goc.lua (written in Lua, Go only)
• https://github.com/deepdivr/solargraph_test_coverage (written in Ruby as LS plugin)

Potential pitfalls:
In the simplest scenario, plugin should be able to show which lines are covered. Some coverage tools provide additional information, e.g. branch coverage, which is more difficult to present (I saw this reported with yellow background colour, and perhaps popup could be displayed with further details on CursorHold). As for visualization, I think plugin should at least allow choosing between highlighting whole lines and using signs. Perhaps virtual text could be also used as third method. There are few report types, from my own experience many tools know how to generate report in cobertura XML format.

[matu3ba]

Unfortunately there is no standard for coverage reports or interface to generate those similar (or reusing) the debug adapter protocol.

The most reliable solution is using kcov by parsing the coverage.json and visualising it accordingly. One could use lunajson to do it in lua, which looks to me like the most performant json parser in lua.

The biggest unsolved question for a very convenient experience is the missing build system integration for emitting/lookup of debug symbols of projects and their build+link dependencies, which need per-project setup. However this does in no way block parsing and visualising the coverage.json.

[fenuks]

Unfortunately there is no standard for coverage reports or interface to generate those similar (or reusing) the debug adapter protocol.

I think cobertura report is semi-standard. I would say that most tools I am familiar with can generate it.

• pytest
• kcov
• Gcovr script produces number of formats, cobertura included.

On top of that Gitlab accepts it. Jenkins has plugin for it. For a fairly comprehensive list of other formats in the wild, CodeCov documentation lists several several of them.

I'd go for it, although it's an XML format, so no vim native parser is available.

I think that it shouldn't be plugin's job to emit coverage. For me, it would very much acceptable if I just could configure plugin to expect coverage report is some place, and read it whenever it is (re)generated. I can see how generation being managed by plugin would be convenient, but it's a serious undertaking. Perhaps coverage plugin should be companion to full-blown testing plugin like https://github.com/vim-test/vim-test.

Another feature besides visualization I didn't mention before is an option to open list of covered files in quickfix/Telescope/other solution for quick navigation over files included in report.

[disrupted]

I think cobertura report is semi-standard.

Not familiar with it but looks like the project is dead. Last official release was in 2015. cobertura/cobertura#400, cobertura/cobertura#418

Or were you just referring to the format used for the coverage report?

[fenuks]

Yes, I was talking about format they created. For testing java code I was once using jacoco, which admirably didn't support generating cobertura format at the time, but I found python script that could convert between jacoco own XML format into cobertura format that could be processed further.

[CKolkey]

If its of any help, I wrote an LSP plugin for Solargraph (ruby lsp) that provides realtime line-by-line coverage:
https://github.com/deepdivr/solargraph_test_coverage

Might serve as inspiration.

[fenuks]

Thank you, I've added it to the list.