>If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency.
iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in your binary. I know this because those AWS / K8S / Google Cloud libraries cause MASSIVE problems with their constant breaking changes without major version changes, and their importing other libraries that also have frequent breaking changes without major version changes, even if the dependency those unused subpackages include (and therefore raise the minimum version) is only used by some other module that needs a lower version (iirc). It's quite a headache sometimes, and could be rather easily solved if you could set upper bounds and not just lower. Or if those giga-projects would stop doing such obviously bad things.
The version-affecting behavior is kinda unavoidable afaict. If they didn't include those unused version constraints, `go build ...` or just importing a new package within existing modules could cause your build to fail, forcing you to rerun version resolution. That'd probably just lead people to feel like "go is broken UGH", and go leans awfully hard towards avoiding that kind of thing. Mostly for the better, but not quite always / not in all ways.
The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution.
This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust.
If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.
I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods.
Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.
I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.
I like your comparison about Firefox not being a great example of "good" C/C++.
I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.
Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods).
They write it in space shuttle mode. Which means that it’s all got too many components from too many makers all kludged together and then one piece falls off and the whole thing blows up in production.
I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.
You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability.
For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this.
But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager.
Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature
It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.
I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.
This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor".
[1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.
I'll throw in a +1, afaik vendoring is a complete and reliable solution here.
Go's `mod`-related commands have had quite a large number of breaking changes in behavior through the years, so I won't say it's a stable solution (aside from completely disabling modules, that has always worked fine from my use). I've had to fix build scripts at least once or twice a year due to that, e.g. when they started validating that the vendor folder was unmodified (a very reasonable thing to do, but still breaking). But once that is overcome it has always worked fine.
> Instead, just look at go.mod. It lists the precise version at which all dependencies are built.
No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such.
That I'm reading this blog post at all suggests there is a "market" for a single checksum/version manifest, which data is currently housed in go.sum . This is sad, but, Hyrum's Law and all that.
> No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such.
No?
All dependencies - direct and indirect - are listed in your go.mod. Your module - as is - depends on nothing else. And those exact versions will be used to build it, if yours is the main module.
If your module is used as a dependency of another module, then yes, your module may be built with a newer version of those dependencies. But that version will be listed in that module's go.mod.
There's no way to use different versions without them being listed in some go.mod.
go.sum to only maps between versions and hashes, and may contain hashes for multiple versions of modules.
If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes.
If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mechanisms that all ensure that a tag is immutable once it is used:
- The Go CLI tools will of course use the go.sum file to validate that a given published version of a module can never change (at least since it was depended on, but it also is complementary with the features below as well, so it can be even better than that.)
- Let's say you rm the go.sum file. That's OK. They also default to using the Go Sum DB to verify that a given published version of a module can never change. So if a module has ever been `go get`'d by a client with the Sum DB enabled and it's publicly accessible, then it should be added to the Sum DB, and future changes to tags will cause it to be rejected.
- And even then, the module proxy is used by default too, so as soon as a published version is used by anyone, it will wind up in the proxy as long as its under a suitable license. Which means that even if you go and overwrite a tag, almost nobody will ever actually see this.
The downside is obviously all of this centralized infrastructure that is depended on, but I think it winds up being the best tradeoff; none of it is a hard dependency, even for the "dependencies should be immutable" aspect thanks to go.sum files. Instead it mostly helps dependency resolution remain fast and reproducible. Most language ecosystems have a hard dependency on centralized infrastructure, whether it is a centralized package manager service like NPM or a centralized repository on GitHub, whereas the centralized infrastructure with Go is strictly complementary and you can even use alternative instances if you want.
But digression aside, because of that, you can trust the version numbers in go.mod.
> If you wanted to verify the contents of a dependency, you would want to check go.sum
You're right, but also TFA says "There is truly no use case for ever parsing it outside of cmd/go". Since cmd/go verifies the contents of your dependencies, the point generally stands. If you don't trust cmd/go to verify a dependency, then you have a valid exception to the rule.
Agreed. Arguably, though, it would be much more reasonable to trust cmd/go to verify a dependency than it would to trust your own code. A lot more effort is put into it and it has a proper security process established. So I think the point is, if you find yourself actually needing to verify the go.sum, not by using cmd/go, you are very likely doing something wrong.
A local cache of sums are also stored in (iirc) $GOCACHE, so even if you delete go.sum from the project, the local toolchain should still be able to verify module versions previously seen without needing to call out to the Sum DB.
Probably unpopular, but I just use Bazel and pick the versions of software I use.
I know the current attitude is to just blindly trust 3rd party libraries (current and all future versions) and all of their dependencies, but I just can't accept that. This is just unsustainable.
Go MVS does not require you to blindly trust 3rd party libraries. Certainly not "current and all future versions". Go modules also offer hermetic and reproducible dependency resolution by default.
Conversely, I can say that an hash being in go.sum doesn't mean it will be used for anything.
Only that if the corresponding version does get used, and the hash doesn't match, you get an error. But you can have multiple versions of the same dep in your go.sum - or none at all - and this has no bearing on what version gets picked when you build your module.
The version that does get picked is the one in go.mod of the main module, period; go.sum, if it exists, assists hash verification.
Yes, if you want a lockfile in the npm sense, you need both.
But a Go module does not get built with new transitive dependencies (as was claimed) unless they're listed in some go.mod; go.sum is irrelevant for that.
Although he doesn't spell it out, I suspect this is the primary misunderstanding that drove Filo to open with "I need everyone to stop looking at go.sum, especially to analyze dependency graphs". I've had more than one code reviewer ding me on a module showing up in go.sum. Usually it's a situation where a dependency has tests for compatibility with some other module so that other module gets added to go.sum. Given Filo is a professional open source maintainer, any annoyance I've run into he's probably experienced 100x.
As explained in the post, if a transitive dependency asks for a later version than you have in go.mod, that’s an error if -mod is readonly (the default for non-get non-tidy commands).
I encourage you to experiment with it!
This is exactly how the “stricter” commands of other package managers work with lockfiles.
I'm not deeply familiar with this, but from reading the `go mod tidy` manual[1], it seems that running `go mod tidy` loads all packages imported from the main module (including transitive dependencies) and records them with their precise versions back to `go.mod`, which should prevent them from being substituted with later versions. Am I understanding this correctly?
go.mod will always match whatever versions are being used directly, as far as I know. But it's not possible to lock them using go.mod. Like if you wanted to bump one version only in go.mod, you're then stumped for actually doing that. Because _probably_ the only reasonable way to get that to build is to do `go mod tidy` after doing that, which will modify go.mod itself. And you can't _really_ go back in and undo it unless you just manually do all of go.mod and go.sum yourself.
Running `go mod tidy` months apart with no other changes to your module will not change your go.mod. It certainly won't update dependencies.
You run that when you've made manual changes (to go.mod or to your Go code), or when you want to slim down your go.sum to the bare minimum needed for the current go.mod.
And that's one common way to update a dependency: you can edit your go.mod manually. But there are also commands to update dependencies one by one.
go always requires a dependency graph that is consistent with all the declared requirements.
Which means if you wanted to update one version, it might bump up the requirements on its dependencies, and that's all the changes you see from running go mod tidy afterwards.
Manually constructing an inconsistent dependency graph will not work.
The MVS choices will be encoded into the go.mod; you may have been correct in the past, but as the post mentions transitive dependencies have been incorporated since Go 1.17. So yes, really: the only point of go.sum is to enable checking the integrity of dependencies, as a nice double-check against the sumdb itself.
Just to clarify, this will download the entirety of all of the dependencies in order to find out what versions are resolved? And then those versions aren't actually locked unless you keep the vendored dependencies around indefinitely?
My understanding is that the point of a lockflle is that you don't need to do that.
> I'm not a go developer and must be misunderstanding something...
I think it's coz not EVERY language's lockfile comes with checksum
So, Go's go.mod is functionally equivalent Ruby Gem lockfile (that doesn't have checksum) but need to get go.sum to be equivalent to npm's (that does come with checksum)
Author just compared it to languages where lockfile means just version lock
Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that the people that expect the former commit under the v1.0.0 tag will actually get that and not the other (malicious) commit?
its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy).
go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums.
i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if even one user saw the previous version (and that one user is probably you, as you probably fetched it through the proxy to see the mistake)
This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanently.
> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module
Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb).
Like any code host, the proxy does not guarantee that the code for a module will be available forever, since code may have to be removed for legal reasons.
But you absolutely can rely on the checksum being preserved and thus you can be sure you'll never be given different code for a particular version.
Ah, my mistake. I had read in the FAQ that it does not guarantee that data is stored forever, but overlooked the part about preserving checksums specifically.
To be very pedantic, there are two separate services: The module proxy (proxy.golang.org) serves cached modules and makes no guarantees about how long cache entries are kept. The sum database (sum.golang.org) serves module checksums, which are kept forever in a Merkle tree/transparency log.
Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.
The checksums are published in a transparency log, which uses a Merkle Tree[1] to make the attack you describe detectable. Source Spotter, which is unaffiliated with Google, continuously verifies that the log contains only one checksum per module version.
If Google were to present you with a different view of the Merkle Tree with different checksums in it, they'd have to forever show you, and only you, that view. If they accidentally show someone else that view, or show you the real view, the go command would detect it. This will eventually be strengthened further with witnessing[2], which will ensure that everyone's view of the log is the same. In the meantime, you / your coworker can upload your view of the log (in $GOPATH/pkg/sumdb/sum.golang.org/latest) to Source Spotter and it will tell you if it's consistent with its view:
$ curl --data-binary "@$(go env GOPATH)/pkg/sumdb/sum.golang.org/latest" https://gossip.api.sourcespotter.com/sum.golang.org
consistent: this STH is consistent with other STHs that we've seen from sum.golang.org
For the question “is the data in the checksum database immutable” you can trust people like the parent, who double checks what Google is doing.
For the question “is it the same data that can be downloaded directly from the repos” you can skip the proxy to download dependencies, then do it again with the proxy, and compare.
So I'd say you don't need to trust Google at all in this case.
ok, I guess I was wrong about the cache, but not the checksums. I was somewhat under the impression that it was forever due to the getting rid of vendoring. Getting rid of vendoring (to me) only makes sense if its cached forever (otherwise vendoring has significant value).
Go modules did not get rid of vendoring. You can do 'go mod vendor' and have been able to do so since Go modules were first introduced.
How long the google-run module cache (aka, module proxy or module mirror) at https://proxy.golang.org caches the contents of modules is I think slightly nuanced.
That page includes:
> Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package
But that page also discusses how modules might need to be removed for legal reasons or if a module does not have a known Open Source license:
> proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.
If interested, there's a good overview of how it all works in one of the older official announcement blog posts (in particular, the "Module Index", "Module Authentication", "Module Mirrors" sections there):
ok, 1) so would it be fair to modify my statement that it basically tries to cache forever unless its can't determine that its legally allowed to cache forever?
2) you're right, glanced at kubernetes (been a long time since I worked on it) and they still have a vendor directory that gets updated regularly.
You are not misunderstanding anything, I use Go and Rust/TypeScript in my daily work and you are correct - it is the OP that does not understand why people use lockfiles in CI (to prevent minor updates and changes in upstream through verifying a hash signature).
They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile
> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages
Thanks for that link.
Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other languages, but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
This subthread to me was a seemingly prototypical exchange there:
Someone in that subthread tells Filippo (FiloSottile) that he is misunderstanding cargo behavior, but Filippo then reiterates which behavior he is talking about (add vs. install), Filippo does a simple test to illustrate his point, and some others seem to agree that he is correct in what he originally said.
That said, YMMV, and that overall discussion does certainly seem to have some confusion and people seemingly talking past each other (e.g., some people mixing up "dependents" vs. "dependencies", etc.).
> but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
I don't get this impression. Rather, as you say, I get the impression that people are talking past each other, a property which also extends to the author, and the overall failure to reach a mutual understanding of terms only contributes to muddying the waters all around. Here's a direct example that's still in the OP:
"The lockfile (e.g. uv.lock, package-lock.json, Cargo.lock) is a relatively recent innovation in some ecosystems, and it lists the actual versions used in the most recent build. It is not really human-readable, and is ignored by dependents, allowing the rapid spread of supply-chain attacks."
At the end there, what the author is talking about has nothing to do with lockfiles specifically, let alone when they are applied or ignored, but rather to do with the difference between minimum-version selection (which Go uses) and max-compatible-version selection.
Here's another one:
"In other ecosystems, package resolution time going down below 1s is celebrated"
This is repeating the mistaken claims that Russ Cox made years ago when he designed Go's current packaging system. Package resolution in e.g. Cargo is almost too fast to measure, even on large dependency trees.
The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file.
While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range.
Go also strictly follows semantic versioning. Thus the implicit exclusive upper bound is the next major version. This assumes that all minor and patch releases are backwards compatibile and not breaking.
Dependency resolution in Go uses minimum version selection. That means the minimum requirements of all dependencies are evaluated and highest minimums are selected. In principle, this minimum version selection should be time invariant since the oldest versions of the compatible dependencies are used
While the minimum versions specified in go.mod are not necessarily the version of the dependencies used, they can be resolved to the versions used irrespective of time or later versions of dependencies being released.
Other languages do not use minimum version selection. Their package resolution often tries to retrieve the latest compatible dependency. Thus a lock file is needed.
Python packages in particular do not follow semantic versioning. Thus ranges are critical in a pyproject.toml.
In summary, the "manifests" files that the original author describes are configuration files. In some languages, or more accurately their package management schemes, they can also be lock files, true manifests, due to version semantics. If those semantics are absent, then lock files are necessary for compatibility.
That’s very interesting. Most systems I know would pick the highest versions allowed by the ranges. In maven and gradle, for example, at least by default they choose the highest versions allowed. Even if no version range is used, it picks the highest choice even across major versions, which I always thought was completely broken. What does go do if you have two transitive dependency versions whose allowed major is different?
In some sense, Go does not allow you to change the major version. Packages with the same name but different major versions are treated as different packages.
nice, if a module like github.com/foo/bar v1.2.3 is later modified or compromised upstream, the checksum in go.sum will cause the build to fail unless the exact same content is fetched.
Correct, but it's been ages and the default actions/setup-go github action still uses go.sum instead. I see that someone already commented on the longstanding issue to reference this post, and that there is some hope that they'll update it!
The private repo mention in the comments there is kinda a good one, unfortunately. If someone runs a private gosum/goproxy (relatively common) and amends a tag (hopefully very uncommon but I have personally seen it happen at least three times) then the cache could be wrong because go.mod didn't have to change. Which is Bad™ but it depends on what edge cases they want to handle automatically, vs optimization for the majority case.
For well-behaving/stable/consistent setups I fully agree though, go.mod is both sufficient and better, and those other cases can probably just key off both instead. I think I've seen go.mod to change without go.sum changes (change an unused transitive dependency into a direct dependency), which can lead to your build needing something that wasn't cached because it was pruned in the previous version.
From a relatively naive outside perspective, it sounds like this would be pretty much only ever a self-inflicted probably (where "self" might be an org rather than an individual). What you're describing sounds almost like a force push to a private repo; if you're doing that, you might break things for anyone using it, so the risks should probably rest with you for asking so. "This breaks my setup if I modify history in a way that's expected to be immutable" isn't a super compelling argument for everyone behaving well to have to continue dealing with a suboptimal status quo.
Two very similar things are presented as though they are different (go.mod and lockfiles, not go.sum) for the purpose of sneering at one of them, when both are essentially the same. 'Ignored by downstream dependents' is not any less true of go.mod than of lockfiles. In both cases a later version can be demanded, overriding the earlier version, potentially breaking your code.
>If the main module imports example.com/mod1/pkg1 and a separate example.com/mod1/pkg2 imports example.com/mod2, there is no way for example.com/mod2 to affect the build or run code on the developer’s machine, so you don’t need to consider it a dependency.
iirc (I do not have a test setup at the moment to verify) it does affect your dependency resolution, and therefore your build, though its code does not exist in your binary. I know this because those AWS / K8S / Google Cloud libraries cause MASSIVE problems with their constant breaking changes without major version changes, and their importing other libraries that also have frequent breaking changes without major version changes, even if the dependency those unused subpackages include (and therefore raise the minimum version) is only used by some other module that needs a lower version (iirc). It's quite a headache sometimes, and could be rather easily solved if you could set upper bounds and not just lower. Or if those giga-projects would stop doing such obviously bad things.
The version-affecting behavior is kinda unavoidable afaict. If they didn't include those unused version constraints, `go build ...` or just importing a new package within existing modules could cause your build to fail, forcing you to rerun version resolution. That'd probably just lead people to feel like "go is broken UGH", and go leans awfully hard towards avoiding that kind of thing. Mostly for the better, but not quite always / not in all ways.
It's tricky, to the point that I made a little playground to explore it.
https://github.com/FiloSottile/mostly-harmless/tree/main/dep...
The example.com/mod2 go.mod does not in fact affect version resolution, because it's not even fetched. However, it affects the example.com/mod1 go.mod, and the example.com/mod1 go.mod affects version resolution.
This doesn't help with the problem you are describing, but it still has value from a security point of view, because example.com/mod2 truly doesn't matter except to the extent that was already checked into example.com/mod1, which you do need to trust.
If you try to "go build" or "go test" something in example.com/mod2, you actually do get an error since Go 1.17, as if it was not in your dependency tree at all. You need to "go get" it like any new dependency.
These libraries are honestly so bad that whenever I have to interact with them I just split those dependencies into a separate binary :)
I dabbled in Kubernetes at work a few years ago. I was shocked at the state of their go.mod files. Especially because I had heard Kubernetes used as an example of great Go software. Right off the bat I needed to copy a dozen or so `replace` statements into my go.mod. I was sure I just didn’t understand something and spent an hour or so looking for the “right way” only to discover other open source controllers that had the same replaces in their go.mods.
Maybe that was a transitory stage and it’s straightened out now. I certainly hope so.
I think a lot of people assume Kubernetes must be a good Go example, because it's so big and successful. But it started life as a Java project and was ported into Go, and that shows in some of the architecture, Also, in general across all languages, picking the absolute largest projects you can is often not a great idea in terms of copying design unless you too are going to be that large, e.g., I wouldn't suggest using Firefox as an example of C++ necessarily. Such projects always end up developing solutions to problems you will never have, and solving problems we don't have is one of the most common mistakes software developers make.
I like your comparison about Firefox not being a great example of "good" C/C++.
I'd personally consider myself comfortable in C/C++. I've built Wayland compositors, H264 backends for live-streaming, and built Chromium occasionally for testing. Despite being a die-hard Firefox (zen) user - I still have not been able to compile Firefox! To be fair, this was pre-Firefox quantum days so I hope their SVM and build tools have improved.
I remember early on the Go people complaining about the K8S Go code
Definitely do not look at Kubernetes as a good example of go code and especially not how it handles deps (btw, it only relatively recently switched to go mods).
Not to say it is a bad project. Not at all.
They write it in space shuttle mode. Which means that it’s all got too many components from too many makers all kludged together and then one piece falls off and the whole thing blows up in production.
I just write my own, small, focused client for these AWS services since the SDKs are generally just so unergonomic. With vibe coding becoming a thing it's become even easier to do that.
You may already know these things, but for others who may not: the AWS SDK for Go suffers from shockingly bad discoverability.
For example, take the S3 library, github.com/aws/aws-sdk-go-v2/service/s3. If you have an s3.Client and you look at e.g. the ListObjectsV2 method, you might have no idea that there is a ListObjectsV2Paginator which makes it much easier to use, because nowhere in the method docs is it mentioned. Indeed, most operations that paginate have more ergonomic paginators, but none of them tell you this.
But that isn't even the worst of it. Say you want to download or upload a file to S3. If you haven't worked with AWS for other languages, you might think that you just do GetObject and PutObject. And yes, for small files, that's generally fine. But for large files you want to use resumable downloads and multipart uploads. So you look and lo, there is no simple way to do this in the AWS SDK for Go. But actually, there is! It's in a totally unrelated and unlinked package, called github.com/aws/aws-sdk-go-v2/feature/s3/manager.
Now you're getting some religion, so you ask "what are the other so-called 'feature' packages?" and you try to browse pkg.go.dev at the github.com/aws/aws-sdk-go-v2/feature level but nope, that's not a Go module so there's nothing to see there. In fact, you can't even browse the other s3 features, never mind find out what other services have features. Fortunately, you can browse their GitHub repo at least: https://github.com/aws/aws-sdk-go-v2/tree/main/feature
It's quite clear that they use poorly thought-out cross-language codegen for this, which partly explains the lack of ergonomics, but also shows that they don't much care whether you use their stuff properly.
Yeah the AWS PHP SDK is similarly bad. The Java SDK is okay but still completely lacking in examples for anything outside of the basics.
AWS seems to optimise for an SDK that be completely generated but not for an SDK that tells you what you want to know.
godep and vendoring completely solved this problem.
Go modules relies too heavily on dependencies being good citizens, which is a very naive approach to dependency management.
You can still do vendoring with Go modules, using the `go mod vendor` command.
I'm a bit out of date now, but 'go mod vendor' did not in fact vendor all dependencies when I last used it. It seemed more like a CI caching mechanism then actual complete dependency vendoring.
This is not the experience that I have. I use vendoring heavily because it's nice to be able to review all the relevant code changes when Renovatebot updates a dependency [1], and to get a feel for how heavy or lightweight certain dependencies are. If vendoring was incomplete, I would see it trying to download missing dependencies at compile time, but the "go: downloading $MODULE $VERSION" lines only show up when I expect them too, e.g. during "go get -u" or "go mod tidy/vendor".
[1] Before you ask, I'm not reading the full diff on something like x/sys. Mostly on third-party dependencies where I find it harder to judge the reliability of the maintainers.
I'll throw in a +1, afaik vendoring is a complete and reliable solution here.
Go's `mod`-related commands have had quite a large number of breaking changes in behavior through the years, so I won't say it's a stable solution (aside from completely disabling modules, that has always worked fine from my use). I've had to fix build scripts at least once or twice a year due to that, e.g. when they started validating that the vendor folder was unmodified (a very reasonable thing to do, but still breaking). But once that is overcome it has always worked fine.
> Instead, just look at go.mod. It lists the precise version at which all dependencies are built.
No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such.
That I'm reading this blog post at all suggests there is a "market" for a single checksum/version manifest, which data is currently housed in go.sum . This is sad, but, Hyrum's Law and all that.
> No, it does not. Minimum version selection means that the libraries will at least be that version, but it could be substituted for a later version if a transient dependency asks for such.
No?
All dependencies - direct and indirect - are listed in your go.mod. Your module - as is - depends on nothing else. And those exact versions will be used to build it, if yours is the main module.
If your module is used as a dependency of another module, then yes, your module may be built with a newer version of those dependencies. But that version will be listed in that module's go.mod.
There's no way to use different versions without them being listed in some go.mod.
go.sum to only maps between versions and hashes, and may contain hashes for multiple versions of modules.
But a “version” as listed in go.mod may have different hashes over time, if tags are changed. That’s the issue.
This doesn't make any sense to me.
If you wanted to verify the contents of a dependency, you would want to check go.sum. That's what it is there for, after all. So if you wanted to fetch the dependencies, then you would want to use it to verify hashes.
If all you care about the is the versions of dependencies, you really can (and should) trust go.mod alone. You can do this because there are multiple overlapping mechanisms that all ensure that a tag is immutable once it is used:
- The Go CLI tools will of course use the go.sum file to validate that a given published version of a module can never change (at least since it was depended on, but it also is complementary with the features below as well, so it can be even better than that.)
- Let's say you rm the go.sum file. That's OK. They also default to using the Go Sum DB to verify that a given published version of a module can never change. So if a module has ever been `go get`'d by a client with the Sum DB enabled and it's publicly accessible, then it should be added to the Sum DB, and future changes to tags will cause it to be rejected.
- And even then, the module proxy is used by default too, so as soon as a published version is used by anyone, it will wind up in the proxy as long as its under a suitable license. Which means that even if you go and overwrite a tag, almost nobody will ever actually see this.
The downside is obviously all of this centralized infrastructure that is depended on, but I think it winds up being the best tradeoff; none of it is a hard dependency, even for the "dependencies should be immutable" aspect thanks to go.sum files. Instead it mostly helps dependency resolution remain fast and reproducible. Most language ecosystems have a hard dependency on centralized infrastructure, whether it is a centralized package manager service like NPM or a centralized repository on GitHub, whereas the centralized infrastructure with Go is strictly complementary and you can even use alternative instances if you want.
But digression aside, because of that, you can trust the version numbers in go.mod.
> If you wanted to verify the contents of a dependency, you would want to check go.sum
You're right, but also TFA says "There is truly no use case for ever parsing it outside of cmd/go". Since cmd/go verifies the contents of your dependencies, the point generally stands. If you don't trust cmd/go to verify a dependency, then you have a valid exception to the rule.
Agreed. Arguably, though, it would be much more reasonable to trust cmd/go to verify a dependency than it would to trust your own code. A lot more effort is put into it and it has a proper security process established. So I think the point is, if you find yourself actually needing to verify the go.sum, not by using cmd/go, you are very likely doing something wrong.
A local cache of sums are also stored in (iirc) $GOCACHE, so even if you delete go.sum from the project, the local toolchain should still be able to verify module versions previously seen without needing to call out to the Sum DB.
Probably unpopular, but I just use Bazel and pick the versions of software I use.
I know the current attitude is to just blindly trust 3rd party libraries (current and all future versions) and all of their dependencies, but I just can't accept that. This is just unsustainable.
I guess I'm old or something.
Go MVS does not require you to blindly trust 3rd party libraries. Certainly not "current and all future versions". Go modules also offer hermetic and reproducible dependency resolution by default.
Conversely, I can say that an hash being in go.sum doesn't mean it will be used for anything.
Only that if the corresponding version does get used, and the hash doesn't match, you get an error. But you can have multiple versions of the same dep in your go.sum - or none at all - and this has no bearing on what version gets picked when you build your module.
The version that does get picked is the one in go.mod of the main module, period; go.sum, if it exists, assists hash verification.
Yes, if you want a lockfile in the npm sense, you need both.
But a Go module does not get built with new transitive dependencies (as was claimed) unless they're listed in some go.mod; go.sum is irrelevant for that.
Although he doesn't spell it out, I suspect this is the primary misunderstanding that drove Filo to open with "I need everyone to stop looking at go.sum, especially to analyze dependency graphs". I've had more than one code reviewer ding me on a module showing up in go.sum. Usually it's a situation where a dependency has tests for compatibility with some other module so that other module gets added to go.sum. Given Filo is a professional open source maintainer, any annoyance I've run into he's probably experienced 100x.
No.
As explained in the post, if a transitive dependency asks for a later version than you have in go.mod, that’s an error if -mod is readonly (the default for non-get non-tidy commands).
I encourage you to experiment with it!
This is exactly how the “stricter” commands of other package managers work with lockfiles.
Minimum version selection happens when the go.mod file is updated, so it contains the minimum versions already.
It doesn't happen only later at build time.
For example:
- `go get x@v1.0.0` => Your go.mod contains `x v1.0.0`
- `go get y@v1.0.0` with y having x v1.0.1 as dep => Your go.mod is already updated with the resolved minimum selected version: `x v1.0.1`
This requires using Go commands to manage the go.mod file. If you edit it in a text editor then a final `go mod tidy` will help.
I'm not deeply familiar with this, but from reading the `go mod tidy` manual[1], it seems that running `go mod tidy` loads all packages imported from the main module (including transitive dependencies) and records them with their precise versions back to `go.mod`, which should prevent them from being substituted with later versions. Am I understanding this correctly?
[1]: https://go.dev/ref/mod#go-mod-tidy
go.mod will always match whatever versions are being used directly, as far as I know. But it's not possible to lock them using go.mod. Like if you wanted to bump one version only in go.mod, you're then stumped for actually doing that. Because _probably_ the only reasonable way to get that to build is to do `go mod tidy` after doing that, which will modify go.mod itself. And you can't _really_ go back in and undo it unless you just manually do all of go.mod and go.sum yourself.
Running `go mod tidy` months apart with no other changes to your module will not change your go.mod. It certainly won't update dependencies.
You run that when you've made manual changes (to go.mod or to your Go code), or when you want to slim down your go.sum to the bare minimum needed for the current go.mod.
And that's one common way to update a dependency: you can edit your go.mod manually. But there are also commands to update dependencies one by one.
go always requires a dependency graph that is consistent with all the declared requirements.
Which means if you wanted to update one version, it might bump up the requirements on its dependencies, and that's all the changes you see from running go mod tidy afterwards.
Manually constructing an inconsistent dependency graph will not work.
The MVS choices will be encoded into the go.mod; you may have been correct in the past, but as the post mentions transitive dependencies have been incorporated since Go 1.17. So yes, really: the only point of go.sum is to enable checking the integrity of dependencies, as a nice double-check against the sumdb itself.
This comment suggests that there is a "market" for confidently incorrect, contrarian HN comments. This is sad, but, Hyrum's Law and all that.
The correct answer is `go mod vendor && cat vendor/modules.txt`
Just to clarify, this will download the entirety of all of the dependencies in order to find out what versions are resolved? And then those versions aren't actually locked unless you keep the vendored dependencies around indefinitely?
My understanding is that the point of a lockflle is that you don't need to do that.
A lock file, in my world, contains a cryptographic hash of dependencies. go.mod does not, it only lists tags, which are (in git) movable references.
If go.sum has "no observable effect on builds", you don't know what you're building and go can download and run unverified code.
I'm not a go developer and must be misunderstanding something...
> I'm not a go developer and must be misunderstanding something...
I think it's coz not EVERY language's lockfile comes with checksum
So, Go's go.mod is functionally equivalent Ruby Gem lockfile (that doesn't have checksum) but need to get go.sum to be equivalent to npm's (that does come with checksum)
Author just compared it to languages where lockfile means just version lock
Now I understand :) thanks for clarifying
By default, all go mod downloads go through the golang proxy (https://proxy.golang.org/). That is part of the verification process.
Let's assume I publish a github repo with some go code, and tag a particular commit with tag v1.0.0. People start using it and put v1.0.0 into their go.mod file. They use the golang proxy to fetch the code (and that proxy does the "verification", according to your comment). Now I delete the v1.0.0 tag and re-create the tag to point to different (malicious) commit. Will the golang proxy notice? How does it verify that the people that expect the former commit under the v1.0.0 tag will actually get that and not the other (malicious) commit?
yes.
From my understanding
its stored forever in the proxy cache and your new tag will never be fetched by users who go through the language's centralized infrastructure (i.e. proxy).
go can also validate the checksums (go.sum) against the languages central infrastructure that associates version->checksums.
i.e. if you cut a release, realize you made a mistake and try to fix it quitely, no user will ever see it if even one user saw the previous version (and that one user is probably you, as you probably fetched it through the proxy to see the mistake)
> its stored forever in the proxy cache
This is mistaken. The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module. From the outside, we would expect that their policy is to only ever delete checksums for modules that haven't been fetched in a long time. But in general, you should not base your security model on the notion that these checksums are stored permanently.
> The Go module proxy doesn't make any guarantee that it will permanently store the checksum for any given module
Incorrect. Checksums are stored forever, in a Merkle Tree, meaning if the proxy were to ever delete a checksum, it would be detected (and yes, people like me are checking - https://sourcespotter.com/sumdb).
Like any code host, the proxy does not guarantee that the code for a module will be available forever, since code may have to be removed for legal reasons.
But you absolutely can rely on the checksum being preserved and thus you can be sure you'll never be given different code for a particular version.
Here's another person auditing the checksum database: https://raphting.dev/posts/gosumdb-live-again/
Ah, my mistake. I had read in the FAQ that it does not guarantee that data is stored forever, but overlooked the part about preserving checksums specifically.
To be very pedantic, there are two separate services: The module proxy (proxy.golang.org) serves cached modules and makes no guarantees about how long cache entries are kept. The sum database (sum.golang.org) serves module checksums, which are kept forever in a Merkle tree/transparency log.
Ok. So to answer the question whether the code for v1.0.0 that I downloaded today is the same as I downloaded yesterday (or whether the code that I get is the same as the one my coworker is getting) you basically have to trust Google.
The checksums are published in a transparency log, which uses a Merkle Tree[1] to make the attack you describe detectable. Source Spotter, which is unaffiliated with Google, continuously verifies that the log contains only one checksum per module version.
If Google were to present you with a different view of the Merkle Tree with different checksums in it, they'd have to forever show you, and only you, that view. If they accidentally show someone else that view, or show you the real view, the go command would detect it. This will eventually be strengthened further with witnessing[2], which will ensure that everyone's view of the log is the same. In the meantime, you / your coworker can upload your view of the log (in $GOPATH/pkg/sumdb/sum.golang.org/latest) to Source Spotter and it will tell you if it's consistent with its view:
[1] https://research.swtch.com/tlog[2] https://github.com/C2SP/C2SP/blob/main/tlog-witness.md
Not really.
For the question “is the data in the checksum database immutable” you can trust people like the parent, who double checks what Google is doing.
For the question “is it the same data that can be downloaded directly from the repos” you can skip the proxy to download dependencies, then do it again with the proxy, and compare.
So I'd say you don't need to trust Google at all in this case.
ok, I guess I was wrong about the cache, but not the checksums. I was somewhat under the impression that it was forever due to the getting rid of vendoring. Getting rid of vendoring (to me) only makes sense if its cached forever (otherwise vendoring has significant value).
Go modules did not get rid of vendoring. You can do 'go mod vendor' and have been able to do so since Go modules were first introduced.
How long the google-run module cache (aka, module proxy or module mirror) at https://proxy.golang.org caches the contents of modules is I think slightly nuanced.
That page includes:
> Whenever possible, the mirror aims to cache content in order to avoid breaking builds for people that depend on your package
But that page also discusses how modules might need to be removed for legal reasons or if a module does not have a known Open Source license:
> proxy.golang.org does not save all modules forever. There are a number of reasons for this, but one reason is if proxy.golang.org is not able to detect a suitable license. In this case, only a temporarily cached copy of the module will be made available, and may become unavailable if it is removed from the original source and becomes outdated.
If interested, there's a good overview of how it all works in one of the older official announcement blog posts (in particular, the "Module Index", "Module Authentication", "Module Mirrors" sections there):
https://go.dev/blog/modules2019#module-index
ok, 1) so would it be fair to modify my statement that it basically tries to cache forever unless its can't determine that its legally allowed to cache forever?
2) you're right, glanced at kubernetes (been a long time since I worked on it) and they still have a vendor directory that gets updated regularly.
This makes the assumption that everyone uses the default proxy, which is not the case
Does this mean, that when you change the proxy, you lose all guarantees?
Only if you change checksum servers https://sum.golang.org/ Note that the default one uses data from https://proxy.golang.org/
You are not misunderstanding anything, I use Go and Rust/TypeScript in my daily work and you are correct - it is the OP that does not understand why people use lockfiles in CI (to prevent minor updates and changes in upstream through verifying a hash signature).
I would hazard a guess that the (former) head of the Go security team at Google (OP) _does_ in fact understand.
They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages. See the previous discussion here: https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile
> They may be an expert in Go, but from their writing they appear to be misunderstanding (or at least misrepresenting) how things work in other languages
Thanks for that link.
Based on reading through that whole discussion there just now and my understanding of the different ecosystems, my conclusion is that certainly people there are telling Filippo Valsorda that he is misunderstanding how things work in other languages, but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
This subthread to me was a seemingly prototypical exchange there:
https://lobste.rs/s/exv2eq/go_sum_is_not_lockfile#c_d26oq4
Someone in that subthread tells Filippo (FiloSottile) that he is misunderstanding cargo behavior, but Filippo then reiterates which behavior he is talking about (add vs. install), Filippo does a simple test to illustrate his point, and some others seem to agree that he is correct in what he originally said.
That said, YMMV, and that overall discussion does certainly seem to have some confusion and people seemingly talking past each other (e.g., some people mixing up "dependents" vs. "dependencies", etc.).
> but then AFAICT Filippo or others chime in to explain how he is in fact not misunderstanding.
I don't get this impression. Rather, as you say, I get the impression that people are talking past each other, a property which also extends to the author, and the overall failure to reach a mutual understanding of terms only contributes to muddying the waters all around. Here's a direct example that's still in the OP:
"The lockfile (e.g. uv.lock, package-lock.json, Cargo.lock) is a relatively recent innovation in some ecosystems, and it lists the actual versions used in the most recent build. It is not really human-readable, and is ignored by dependents, allowing the rapid spread of supply-chain attacks."
At the end there, what the author is talking about has nothing to do with lockfiles specifically, let alone when they are applied or ignored, but rather to do with the difference between minimum-version selection (which Go uses) and max-compatible-version selection.
Here's another one:
"In other ecosystems, package resolution time going down below 1s is celebrated"
This is repeating the mistaken claims that Russ Cox made years ago when he designed Go's current packaging system. Package resolution in e.g. Cargo is almost too fast to measure, even on large dependency trees.
The pyproject.toml, package.json, and Cargo.toml are declarative project configuration files. While the Rust community refers to Cargo.toml as a manifest, it is not a comprehensive and detailed list of a build. That is the lock file.
While go.mod does not allow for explicit version ranges, the versions given are the minimum versions. In other words, the versions given are the lower bound of the compatibility range.
Go also strictly follows semantic versioning. Thus the implicit exclusive upper bound is the next major version. This assumes that all minor and patch releases are backwards compatibile and not breaking.
Dependency resolution in Go uses minimum version selection. That means the minimum requirements of all dependencies are evaluated and highest minimums are selected. In principle, this minimum version selection should be time invariant since the oldest versions of the compatible dependencies are used
While the minimum versions specified in go.mod are not necessarily the version of the dependencies used, they can be resolved to the versions used irrespective of time or later versions of dependencies being released.
Other languages do not use minimum version selection. Their package resolution often tries to retrieve the latest compatible dependency. Thus a lock file is needed.
Python packages in particular do not follow semantic versioning. Thus ranges are critical in a pyproject.toml.
In summary, the "manifests" files that the original author describes are configuration files. In some languages, or more accurately their package management schemes, they can also be lock files, true manifests, due to version semantics. If those semantics are absent, then lock files are necessary for compatibility.
> While the minimum versions specified in go.mod are not necessarily the version of the dependencies used
This has not been true since Go 1.17 with the default -mod=readonly, which is why go.mod is a reliable lockfile.
That’s very interesting. Most systems I know would pick the highest versions allowed by the ranges. In maven and gradle, for example, at least by default they choose the highest versions allowed. Even if no version range is used, it picks the highest choice even across major versions, which I always thought was completely broken. What does go do if you have two transitive dependency versions whose allowed major is different?
In some sense, Go does not allow you to change the major version. Packages with the same name but different major versions are treated as different packages.
nice, if a module like github.com/foo/bar v1.2.3 is later modified or compromised upstream, the checksum in go.sum will cause the build to fail unless the exact same content is fetched.
Correct, but it's been ages and the default actions/setup-go github action still uses go.sum instead. I see that someone already commented on the longstanding issue to reference this post, and that there is some hope that they'll update it!
https://github.com/actions/setup-go/issues/478
At this point I'm just assuming that all of the official actions under https://github.com/actions are unmaintained garbage until proven otherwise, given that high-impact issues like this can sit for years without as much as an initial triage: https://github.com/actions/delete-package-versions/issues/16...
The private repo mention in the comments there is kinda a good one, unfortunately. If someone runs a private gosum/goproxy (relatively common) and amends a tag (hopefully very uncommon but I have personally seen it happen at least three times) then the cache could be wrong because go.mod didn't have to change. Which is Bad™ but it depends on what edge cases they want to handle automatically, vs optimization for the majority case.
For well-behaving/stable/consistent setups I fully agree though, go.mod is both sufficient and better, and those other cases can probably just key off both instead. I think I've seen go.mod to change without go.sum changes (change an unused transitive dependency into a direct dependency), which can lead to your build needing something that wasn't cached because it was pruned in the previous version.
From a relatively naive outside perspective, it sounds like this would be pretty much only ever a self-inflicted probably (where "self" might be an org rather than an individual). What you're describing sounds almost like a force push to a private repo; if you're doing that, you might break things for anyone using it, so the risks should probably rest with you for asking so. "This breaks my setup if I modify history in a way that's expected to be immutable" isn't a super compelling argument for everyone behaving well to have to continue dealing with a suboptimal status quo.
It's basically a force-push, yeah. Which does happen, like with git, so you do kinda need a way to deal with it even if it's not automatic.
Which it sounds like they have (planned), which seems like a good improvement.
Two very similar things are presented as though they are different (go.mod and lockfiles, not go.sum) for the purpose of sneering at one of them, when both are essentially the same. 'Ignored by downstream dependents' is not any less true of go.mod than of lockfiles. In both cases a later version can be demanded, overriding the earlier version, potentially breaking your code.
npm's package.json and package-lock.json get out of sync constantly on my team. at least go only has one file to mess up
how is this possible? isn't everyone using the same node version?
and checking in lockfile changes
I am today years old to find this...