Fix Azure Devops detector

[rgmz]

Description:

This fixes #3680.

Checklist:

• Tests passing (make test-community)?
• Lint passing (make lint this requires golangci-lint)?

[rosecodym]

@rgmz thanks for this! Would you mind providing a brief summary of the changes you made to the logic? The simultaneous code reorganization has made me unable to find a useful diff. (Or maybe you could just tell me the right buttons to push to get one :/ )

[rgmz]

It seems that GitHub isn't smart enough to track the rename; oddly, Git seems to track it just fine.

TLDR

• Fix the keywords
• Fix the pattern (both key and organization)
• Filter obviously invalid matches
• Fix unhandled status codes
• Add extradata
• Fix test cases

[rosecodym]

It seems that GitHub isn't smart enough to track the rename; oddly, Git seems to track it just fine.

Huh, weird. It looks like pushed a new version that renamed the package but not the files - was that intentional?

Regardless, thanks for looking into this. It looks like there's some good cleanup in here. I have two specific concerns, though:

• The invalid org cache introduces state to the detector, which in turn suggests that someday it might need lifecycle management that nobody's going to realize. To be clear, I don't think this necessarily going to cause an acute problem now, but I'm worried about a hidden maintenance cost. How much does this cache speed detection up? My experience is that verification in general is rarely/never the bottleneck on a scan, so I'm wondering if it's possible to remove the cache in order to simplify the implementation without a performance cost.
• Changing Raw or RawV2 "orphans" secrets that are tracked externally. Do you have any guesses about why the original implementation was the way it was? Specifically, can we get away with just making a V2 of this detector without feeling ashamed of ourselves?

[rosecodym]

Actually, I think that we can retain the existing Raw/RawV2 behavior by just:

• Making dev\.azure\.com/ non-capturing in its regex
• Reverting to the existing Raw/RawV2 concatenation logic

[rgmz]

• Specifically, can we get away with just making a V2 of this detector without feeling ashamed of ourselves?

This might be the ideal change. The existing RawV2 is a hiderance for OSS users because of reasons mentioned in #3634 (comment).

[rgmz]

The invalid org cache introduces state to the detector, which in turn suggests that someday it might need lifecycle management that nobody's going to realize. To be clear, I don't think this necessarily going to cause an acute problem now, but I'm worried about a hidden maintenance cost. How much does this cache speed detection up? My experience is that verification in general is rarely/never the bottleneck on a scan, so I'm wondering if it's possible to remove the cache in order to simplify the implementation without a performance cost.

In my view, we should not be flooding services with requests for resources we know are invalid. Doing so can result in providers blocking TruffleHog or legitimate requests failing due to ratelimiting / captcha (this frequently occurs for BrowserStack). It also hampers throughput; my home Internet is DSL and TruffleHog is largely unusable because of how noisy it is.

Perhaps it'll be pointless once granular result caching is introduced. For now, the pros heavily outweigh potential cons IMO.