> In my opinion, every commerical software development team using Go should be using custom domains for namespacing their internal libraries and packages.
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
I'm confused by the article, and this comment, treating these URLs as difficult to replace.
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
I've gone through such a migration. Huge Go code base distributed across a lot of git repositories had to be moved to a different git host.
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
> Different projects depended on different versions of the same internal libraries, we had to...
This is one of the main motivations for using a monorepo with all third party dependencies imported all the way in, and only one version for everything.
Of course it causes a whole bunch of other problems and is kinda expensive to scale, so most places won't do it.
Isn’t this exactly why the Go team recommended vendoring deps for years and years before go modules came up? Even now it takes one simple command to vendor them.
Google runs a monorepo which makes that kind of organization possible. Most companies are going to have a mismatch of teams using incompatible library versions.
Pick your poison. Each approach has some seriously negative trade offs in the extremes.
Well vendoring is a technical solution of the same caliber as adding module aliases to the go.mod. By that I mean, sure, it keeps the code compiling right now, but all your source files still contain references to abandoned infrastructure. It changes when you have to do the work, but unless you want your code to reference abandoned infrastructure forever, you still need to do the work.
> And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
That makes no sense? As long as you’re using go modules, you can just host an internal go proxy for internal modules similar to proxy.golang.org to archive the old versions, and they won’t depend on the old git host.
I think is usually a symptom of how hard it is for the organization to do things, the technical side of this is not actually that hard.
I say this because I’ve gone through a few of these, some with monorepos (the easiest, search and replace and you’re done), some not (yes the hardest but you can just vendor).
At least in the situations I’ve faced this was genuinely not horrible beyond just the organization itself being complicated about the solution.
How do you propose it's solved in an easier way, given the constraints that 1) you don't want references to old infrastructure in your code and 2) you want to do the move without changing how the code works? Is there some secret trick we missed which would've made this much easier?
Clone Server A -> Server B. All code still refers to A.
Update code to refer to B
You can now EOL Server A, and B becomes the new canonical reference.
I think the key is simply that you can keep both A + B running during the migration, so you just need to be able to do a code freeze for the duration of the migration. And a single person can easily do this migration with a few python scripts and an hour.
Bonus points: code freeze guarantees your #2 - no changes to how the code works.
Of course, I'd assume that most complaints come from situations where this "secret trick" isn't viable
I don't understand. If B already exists and is working, isn't this just Ctrl+F, compile, push? I'm utterly unfamiliar with the "Go" language
No, it's not. Here's a comment where I described a concrete hypothetical scenario: <a href="https://news.ycombinator.com/item?id=49871032">https://news.ycombinator.com/item?id=49871032
I don't know enough about git but shouldn't you be able to take all your git data (refs, blobs etc) and import it to your destination host and have this work without issues? It might be a lot of data sure but the infra is surely cheaper than multiple weeks of engineering effort
Moving the repositories over was no problem at all! But once all the data is on the new hosts, all the Go source files still reference the old git host, since imports in Go are URLs to a git host.
Similar pain as upgrading a module to a new major version (eg: v1 -> v2) [1], but at scale, because of modules dependencies: every module in the domain are affected at once as they must change namespace, like if everyone changed its major version.
Even easier than that, you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from; you can point to a folder on disk or to another forge, and if you want total control you can indirect everything to your own artifact cache via GOPROXY (which can be something as simple as a static folder of source code).
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
com.sun is an identifier, nothing more. No tooling sees com.sun and assumes, "oh that means I can make an HTTP request to the IP address which sun.com resolves to".
In Go, the package identifiers are literally URLs. Go's own tooling assumes that it can make an HTTP request to the URL and that the response will be HTML with particular tags which Go's tooling will parse and use to resolve a git repository which can be 'git clone'd. Leaving them as-is when you abandon the infrastructure they reference literally means leaving dead links in your source code.
That's just the default, which you can easily override if you need to. IMO there is nothing wrong with treating Go package names as arbitrary identifiers like com.sun.* If you're spending a lot of time updating imports because the underlying repo has moved from its original URL, then just don't do that. It's unnecessary.
The Go tooling won't do that if you configure it properly (e.g. via a 'replace' directive in your go.mod or <a href="https://go.dev/ref/mod#goproxy-protocol" rel="nofollow">https://go.dev/ref/mod#goproxy-protocol)
Any tool that automatically assumes that a URL in a Go module import is 'trusted' is just a broken tool.
Any tooling that looks at Go module imports should respect the contents of go.mod.
If you haven't added a replace directive to your go.mod, then you certainly haven't replaced all instances of the old URLs with updated ones.
> you can use the 'replace' statement in your go mod to change where the Go build system will try to pull the dependencies from
Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].
So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.
[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.
[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.
> Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).
What about anyone else using your code? You can find and replace your own code, but do you have access to all of the code relying on your go library? Is it all yours? Customers? Other developers?
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
I mean yeah, but other languages don't make it quite so easy to make this mistake. Once your Go project gets to a size where it makes sense for different files to live in different folders (which, given Go's folder-based module system, often happens at just a couple hundred lines of code), the most straightforward way which the tooling nudges you towards means putting GitHub URLs (or URLs to whatever git host website you happen to use) in your source code.
I do something like this in all my projects now. Except that I don't use example.com, I just use the name of the project. So if I'm working on a program called Frobnicator, I'll just have `module Frobnicator;` in my go.mod and do things like `import "Frobnicator/lib/blah"` in my source files. It works really well to be honest for non-library software.
But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.
yourname.internal/mymodule is even better. But don't expect to Go toolchain (especially "go get" or "go mod download") to work.
I think this is one of the biggest missteps made in Go and in all honesty I don't think it was any more intuitive than going through a package manager. Probably less, looking back.
Everything else is fine. Generics, errors, whatever...
Yeah, the only problem is when your whole company decides to move to a different custom domain entirely, e.g. from contoso.com to xtools.cn (which doesn't even resolves if you're not on the corporate VPN), and half of your friendly teams, whose libraries you depend on, drag out their own migration by fiddling with their local /etc/hosts and .gitconfig and .ssh/config (yes, they also wrote bespoke scripts that do those changes to their CI machines; no, trying to run those same scripts on your CI machines breaks your own shims) instead of updating their source code.
I don't love that either, having all of your stack point to your own domains for how it can be accessed is nice. But in my opinion, it's desirable to be able to build your whole stack from source code. Of course you download external dependencies, but for code you own you should be able to build it from source.
I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.
I also think that vendoring or at least caching your dependency tree somewhere local, so you can build everything from ground up to a known state.
It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.
I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.
In the new world of AI and all the supply chain attacks on centralised package managers, I think circling back to vendoring probably makes more sense than depending on the network in your build step. Which in some ways is always great, as you mention, in terms of working offline. Using your own domain doesn't help there either, except exposing the package if the domain expires.
Even if you don't want to go as far as vendoring (which can get a bit out of hand with, say, NPM), a middle ground is using Artifactory or whatever as a proxy.
Then be more deliberate about when you update. AI may even help here where static analysis doesn't, because you might be able to use inference to see if a dep upgrade is even needed. Unless it has a severe vuln you probably don't need to track latest.
thih9 · · focus · HN ↗
I’d remove “go” from the above, i.e. I think same applies to other stacks.
Even using GitHub domain links in code comments gets problematic long term. Ie when a migration happens and those links start pointing nowhere.
handoflixue · · focus · HN ↗
Why can't you just search-and-replace? Presumably all of them refer to "GitHub.com" and not much else code will, so I'd think this was an exceptionally easy case.
Even easier for comments, since them being obsolete for a few hours during a migration doesn't exactly break anything.
gravypod · · focus · HN ↗
mort96 · · focus · HN ↗
It was horrible. A team of people spent weeks. Different projects depended on different versions of the same internal libraries, we had to create a branch for each depended-on commit and make a version of that commit with the new URLs. The expressed goal was to end up with a system that's "exactly the same" as the old, just on a new host. Changing which version of libraries projects depends on would introduce unnecessary risk.
And the result is a code base where bisects are broken and where it's impossible to build an old version of any of the code without a ton of work.
dmoy · · focus · HN ↗
This is one of the main motivations for using a monorepo with all third party dependencies imported all the way in, and only one version for everything.
Of course it causes a whole bunch of other problems and is kinda expensive to scale, so most places won't do it.
dlisboa · · focus · HN ↗
0cf8612b2e1e · · focus · HN ↗
Pick your poison. Each approach has some seriously negative trade offs in the extremes.
mort96 · · focus · HN ↗
oefrha · · focus · HN ↗
That makes no sense? As long as you’re using go modules, you can just host an internal go proxy for internal modules similar to proxy.golang.org to archive the old versions, and they won’t depend on the old git host.
sjbzbeiks · · focus · HN ↗
I say this because I’ve gone through a few of these, some with monorepos (the easiest, search and replace and you’re done), some not (yes the hardest but you can just vendor).
At least in the situations I’ve faced this was genuinely not horrible beyond just the organization itself being complicated about the solution.
mort96 · · focus · HN ↗
handoflixue · · focus · HN ↗
Clone Server A -> Server B. All code still refers to A.
Update code to refer to B
You can now EOL Server A, and B becomes the new canonical reference.
I think the key is simply that you can keep both A + B running during the migration, so you just need to be able to do a code freeze for the duration of the migration. And a single person can easily do this migration with a few python scripts and an hour.
Bonus points: code freeze guarantees your #2 - no changes to how the code works.
Of course, I'd assume that most complaints come from situations where this "secret trick" isn't viable
mort96 · · focus · HN ↗
handoflixue · · focus · HN ↗
[deleted] · · focus · HN ↗
[deleted]
mort96 · · focus · HN ↗
sjbzbeiks · · focus · HN ↗
gnaman · · focus · HN ↗
mort96 · · focus · HN ↗
dolmen · · focus · HN ↗
<a href="https://go.dev/doc/modules/major-version" rel="nofollow">https://go.dev/doc/modules/major-version
lelandbatey · · focus · HN ↗
The "module name is network path" is a convenient convention but not at all some "limitation" of the tooling.
mort96 · · focus · HN ↗
someonebaggy · · focus · HN ↗
mort96 · · focus · HN ↗
In Go, the package identifiers are literally URLs. Go's own tooling assumes that it can make an HTTP request to the URL and that the response will be HTML with particular tags which Go's tooling will parse and use to resolve a git repository which can be 'git clone'd. Leaving them as-is when you abandon the infrastructure they reference literally means leaving dead links in your source code.
someonebaggy · · focus · HN ↗
mort96 · · focus · HN ↗
someonebaggy · · focus · HN ↗
foldr · · focus · HN ↗
mort96 · · focus · HN ↗
foldr · · focus · HN ↗
mort96 · · focus · HN ↗
foldr · · focus · HN ↗
Any tool that automatically assumes that a URL in a Go module import is 'trusted' is just a broken tool.
mort96 · · focus · HN ↗
foldr · · focus · HN ↗
If you haven't added a replace directive to your go.mod, then you certainly haven't replaced all instances of the old URLs with updated ones.
someonebaggy · · focus · HN ↗
sleepybrett · · focus · HN ↗
mort96 · · focus · HN ↗
sleepybrett · · focus · HN ↗
quectophoton · · focus · HN ↗
Heads up for anyone who doesn't know: this only works at the "top level". Any replace directives in your dependencies will be ignored[1].
So for example if you have a dependency tree like [main -> thirdpartyframework -> golang.org/x/net/http2], and thirdpartyframework uses a vulnerable version of `golang.org/x/net/http2`, you can't just fix it by patching the thirdpartyframework repository with a replace directive; no, because that would be too convenient. Instead, the replace directive needs to be at the main module, where it doesn't make sense and is inconvenient.
Even though I like Go, it really seems like they don't care about anything other than monorepos. As soon as you need to work with forks, mirrors, or even just private modules[2][3], the tooling actively works against you. Also using your custom module proxy is a pain.
[1] See: <a href="https://go.dev/ref/mod#go-mod-file-replace:~:text=replace%20directives%20only" rel="nofollow">https://go.dev/ref/mod#go-mod-file-replace:~:text=replace%20...
[2]: If you've only used private modules hosted on GitHub you might not have noticed too much pain because the Go tooling has hardcoded behavior specifically for GitHub and a few mainstream forges. You don't find out about this until you try to self-host something like Forgejo on your own domain thinking it would Just Work(tm), but it doesn't, and now you're left wondering why tf it works with GitHub but not with your own forge instance.
[3]: I think there's no hardcoded code for SourceHut, so you might be able to experience the inconvenience by hosting private modules in there.
Joker_vD · · focus · HN ↗
Wait, what? If you decide, in your own application, to force all your dependencies to use a specific version of golang.org/x/net/http2, then obviously you'd want to be able to put this directive into your own application's source instead of going around patching 3rd-party repositories (that's just rude).
mbreese · · focus · HN ↗
Even in the case where it’s an internal only Lu array, it can be complicated to refactor a library name.
handoflixue · · focus · HN ↗
I did not realize it ran that deep - I'm used to distributing EXE files, not anything that would need to know internal dependencies like that.
mort96 · · focus · HN ↗
someonebaggy · · focus · HN ↗
someonebaggy · · focus · HN ↗
mort96 · · focus · HN ↗
But it's something you have to do as an active choice; the tooling, and your colleagues, will nudge you towards using URLs to your primary git host's web front-end. And it naturally doesn't work for libraries.
dolmen · · focus · HN ↗
Many tools will fail if you break this contract.
mort96 · · focus · HN ↗
dolmen · · focus · HN ↗
<a href="https://datatracker.ietf.org/doc/draft-davies-internal-tld/" rel="nofollow">https://datatracker.ietf.org/doc/draft-davies-internal-tld/
ljm · · focus · HN ↗
Everything else is fine. Generics, errors, whatever...
zahlman · · focus · HN ↗
Joker_vD · · focus · HN ↗
sleepybrett · · focus · HN ↗
throwawayffffas · · focus · HN ↗
I have seen too many cases of requirements on internal projects that prohibit devs from working because oops the vpn is down now or oops gitlab is under load and the pipeline for your dependency won't finish for the next hour.
bayindirh · · focus · HN ↗
It's not only an issue of availability or depending on circumstances you can't control, but a matter of reproducibility hence dependability.
I want to be able to compile critical software when a zombie apocalypse hits, and I have a growing discomfort about trusting outside parties for their availability and benevolence.
ljm · · focus · HN ↗
Even if you don't want to go as far as vendoring (which can get a bit out of hand with, say, NPM), a middle ground is using Artifactory or whatever as a proxy.
Then be more deliberate about when you update. AI may even help here where static analysis doesn't, because you might be able to use inference to see if a dep upgrade is even needed. Unless it has a severe vuln you probably don't need to track latest.