Azure DevOps pipelines running out of memory

[lukelloydagi]

Is anyone else having issues with Azure DevOps pipelines running out of memory recently?

I'm trying to figure out if it's MegaLinter that's grown or underlying MS infrastructure (which we've seen a lot of lately).

Would reducing CPU usage with PARALLEL_PROCESS_NUMBER reduce memory usage too?

[lukelloydagi]

Well...that answers that, the PARALLEL_PROCESS_NUMBER setting made no difference 😞

[lukelloydagi]

Tried using the megalinter-dotnet image to see if a smaller image size would make a difference....it did not 😞

[nvuillam]

@lukelloydagi Are you using custom self-hosted Azure Runners ?

I have many project using Azure, and have not detected similar issues :/

[lukelloydagi]

@nvuillam, I've reverted to v8.1 and I don't seem to have the issue, I only have the issue with v8.2+ using either default or dotnet flavor. Is there anything in those releases that's likely to be the cause? If not I will have to raise an issue with MS 😭

[nvuillam]

@lukelloydagi just to see if it is dotnet related, please can you try to run it with DOCUMENTATION or CI_LIGHT flavor ?

[lukelloydagi]

@lukelloydagi just to see if it is dotnet related, please can you try to run it with DOCUMENTATION or CI_LIGHT flavor ?

@nvuillam, I can do but none of those flavors contain arm-ttk linters so would not lint my codebase.

It's odd as it seems to only have the issue with v8.2+ on one of my bigger ARM template repos.
It's only configured to lint changed files so size of repo shouldn't matter?
Also I would have thought size of repo (and image for that matter) would only effect disk space not memory?

[lukelloydagi]

@nvuillam, if I reduce the linters (by only specifying a list using the ENABLE configuration variable) then I also don't get the issue. It therefore clearly relates to the linters that are running against the repo and maybe it's the linter(s) that are scanning the entire repo.

Has anything changed between v8.2+ in how the linters are running/consuming memory?

[nvuillam]

@lukelloydagi If a linter cli_lint_mode is in project mode, it does not lint only new files, but all files

By playing with enabling/disabling linters, did you identify the linter(s) that makes your CI job fail ?

I don't think we changed anything memory related between 8.2 & 8.3 :/ What changed are the linter versions, that have been upgraded, so maybe one of them has new performance issues or a memory leak :/