coveralls support

[gbrownmozilla]

Support for coveralls needs attention.

We generate coverage data for each script via tox and then call coveralls to submit coverage data to coverage.io, but that's all legacy code that doesn't quite work currently. It appears that this was originally set up when each script was in a separate github repo, and using TravisCI. Each script's README points to coveralls.io badges for the old repos (like mozilla-releng/balrogscript, rather than mozilla-releng/scriptworker-scripts/balrogscript); of course those are all dormant currently.

As far as I can tell, coveralls.io assumes a one-to-one correspondence between a coverage project and a github repo: We can monitor coverage for mozilla-releng/scriptworker-scripts, but not mozilla-releng/scriptworker-scripts/balrogscript on its own.

Currently each scriptworker-scripts 'coveralls' command fails with the error message:

coveralls.exception.CoverallsException: Not on TravisCI. You have to provide either repo_token in .coveralls.yml or set the COVERALLS_REPO_TOKEN env var.

This does not cause a job failure (because of the leading hyphens in the main tox.ini), but you can see the tox taskcluster logs for details. The repo token can be found on coveralls.io. The Balrog project provides an example of specifying COVERALLS_REPO_TOKEN in a taskcluster project:

https://github.com/mozilla-releng/balrog/blob/main/.taskcluster.yml

Open questions:

• Can we continue to call coveralls for each script, or do we need to call coveralls just once for scriptworker-scripts after each script has updated its coverage data?

• Is there a way to track coverage per script, or do we just get one badge and report per repo? Can we live with that?

[ahal]

Just noticed this.. Perhaps we should move to CodeCov like we do in most other repos these days.