Documenting technical decisions re: what files go into the repository

[jayofdoom]

It seems like a recent hot topic has been whether or not to commit vendored go modules or generated protobufs files into our repositories.

I know there's already been some offline discussion about this, and maybe a conclusion reached -- the goal of this discussion is to find consensus if one doesn't exist, or document it if one does, regarding what things should be in armada repos and what things should be generated.

[richscott]

Committing third-party Go modules to a repo has the advantages of:

• fastest build performance (no possible network latency)
• the exact version and contents are available forever
• the expectation of increased security due to untampered modules

The downsides are:

• repository bloat over time
• a minor module version change (even just a single-digit change) or new module introduction causes commit diffs to become gigantic

My preference is to not commit modules ("vendoring") as the module management in Go 1.17+ has very strong guarantees about security and correctness, and fails to meet only the second advantage listed above, but I don't think that matters that much. The generated go.sum file contains expected cryptographic hashes of all the modules, including indirectly referenced modules - this ensures that all future downloads of these particular module references must be exactly as what was used when this repository was last built - go mod verify will tell you if something was mangled/altered. A global checksum db is hosted by Google at https://sum.golang.org/ and a global default proxy is at https://proxy.golang.org/ - you can ignore certain module by using GONOPROXY, and/or use GOPRIVATE to specify lists of modules that are private/proprietary.

The security is very solid on Go modules, the only scenario this does not cover is when a developer pulls their module from their repo source (and all known mirrors, etc). However, Go caches modules locally for all builds - all Go devs have a local module cache at $HOME/go/pkg plus there are numerous public global caches (and there is no equivalent to npm unpublish with a single registry). So the probability that a legacy published module would disappear and be unavailable forever is very low, I think. See more info about the various commands for using/ managing/verifying modules at https://go.dev/ref/mod and details about the global module mirror, index and checksum db at https://proxy.golang.org/

[dgl]

Currently I am on the side of checking in generated code (such as protoc's .pb.go) -- in the open source world giving people an out of the box development experience where go build just works is very valuable. However I think this is an area where tooling improvements could in the future help change that opinion (for internal codebases I very much have the opposite opinion).

One potentially interesting tool for this is https://buf.build -- it can deal with the code generation and then provide a go module based on the proto file. I haven't used it enough yet to form a full opinion (it's fairly new, and the go module proxy is considered alpha) but I see it particularly useful for gRPC clients where a language boundary is being crossed -- i.e. for Armada Python clients, etc. as it can avoid the need to generate and manage stub code for multiple languages (sadly it doesn't appear to support the dynamic generation for .NET yet).

[severinson]

I'm somewhat averse to committing auto-generated code. The reason is that we have to approve PRs and inspect code as part of our high-integrity requirements (to avoid a contributor sneaking in malware). The more code there is in the repo, the more work to inspect it. Dependencies and auto-generated code create an enormous amount of code. This has previously been an issue.

I'd be happy to bundle auto-generated code with releases, however. In this way, someone can easily download a release to get the auto-generated files and go from there. Combined with nicer tooling (any suggested changes to the make proto command?), that sounds like a clean and easy solution.