index.golang.org: support filtering by module name

[amita-seal]

The index.golang.org service is a valuable resource for querying information about Go modules. Currently, it provides a comprehensive index of all modules, but lacks the ability to filter results based on module names.

I propose enhancing index.golang.org by adding support for filtering the index based on specific module names. This feature would greatly improve the usability of the index by allowing users to narrow down search results to relevant modules, making it easier to find specific packages and their versions.

Consider a scenario where a developer wants to find all versions (including unstable ones) of a specific module, e.g., github.com/example/module. With the proposed feature, the developer could directly query index.golang.org for this module, bypassing unrelated entries and saving time.

Some benefits include:
Efficiency: Speeds up the search process by directly fetching relevant module information.
User Experience: Enhances the usability of index.golang.org for developers by providing more precise search results.
Scalability: Helps in managing and navigating the growing number of Go modules by making the index more searchable.

Adding support for filtering by module name in index.golang.org will significantly improve its functionality and usability, providing a more efficient and user-friendly experience for the Go community.

[gabyhelp]

Similar Issues

• index.golang.org: exclude unavailable module versions from default results #59320

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[hyangah]

Can you use proxy.golang.org/<module>/@v/list end point or, use go list -m --versions <module> (higher level interface)?

[amita-seal]

These endpoints do not show unstable module versions.
If unstable versions weren't so common that could have been enough, but as you can see here, they are very common.

[seankhliao]

I question the need to find "all versions". Every commit can be a version and pulled in to code. You'd be better off looking at the source repository which will have additional context on which commit/version is appropriate.

If it were purely for discovery, I believe pkg.go.dev should be sufficient?

[amita-seal]

It looks like pkg.go.dev only shows stable versions, too.

From what I understand, searching through all commits on the source repository includes versions that were never pulled from proxy.golang.org. Also, specific commits can come from non-default branches, which means it's much heavier to find relevant versions: You'd need to query for all commits of all branches, then cross-reference with proxy.golang.org. For a popular package like github.com/docker/docker, that means going over ~150,000 commits.

Also, in the more general case, I find that the querying possibilities of the index are very limited. Getting 2000 records in each request, with 27,722,863 records, means it takes ~13500 requests to go over the entire index.
What usages was this designed for originally? How is one expected to get information from the index without better querying options?

[seankhliao]

what's the use case for needing every pulled version of module that doesn't involve the rest of the ecosystem?
it'd be more efficient for you to pull it once then keep it up to date with since, and run any analysis locally

[amita-seal]

My specific use case is running some business logic on each pulled version of a module. A few examples I can think of: static analysis of code in search for vulnerabilities, building an SBOM, etc.

I agree that it's more efficient to store a continuously updating mirror of the index locally, but that means each user needs to do so fo themselves. If the index had better querying options, it would have saved that work being done multiple times.

[hyangah]

What usages was this designed for originally?

This is designed for services that constantly watch to discover new modules. For example, pkg.go.dev discovers new modules by querying it using updated since. Search or any advanced query support is beyond the scope of this service. Anyone can build a service on top of index.golang.org if want. Closing since we don't have a plan to expand the scope of the service.