Fetch module source files using Git or HTTP

[casey]

A few people would like to be able to load module source files from git or using HTTP.

I think that this probably isn't a good idea, but I wanted to make this issue for discussion.

The feature would work something like this, with some made up syntax:

mod foo http('https://foo.com/foo.just')

Just would then fetch the file https://foo.com/foo.just and use it as the source for module foo.

The main reason that I think this would be a bad idea is the complexity of the feature, the difficulty of getting it right, and the ease of workarounds.

Some hard questions regarding how the feature would work:

• Would just automatically fetch files, or would you have to tell it to explicitly? The latter is probably better, since you might not want just to fetch and execute files remotely.
• Which protocols would just support? HTTP is pretty easy, but a common ask is for git support too.
• How frequently would just update module source files? We'd probably need a subcommand for re-fetching new source files.

And the workaround looks something like this:

mod? foo

update:
  curl -O https://foo.com/foo.just

Then, you could run just update to fetch the source file, which would then be loaded the next time you ran just. Using git would looks something like:

mod? foo

update:
  #!/usr/bin/env bash

  if [[ -d foo ]]; then
    cd foo
    git pull
  else
    git clone [email protected]/foo.git
  fi

Then, you could run just update to fetch the git repo foo, or update it if it already exists.

This allows the user to get whatever behavior they want, so I'm not sure a dedicated feature would be an improvement.

[laniakea64]

I agree with @casey - the workaround is simpler implementation, more versatile for justfile authors, and easier for end-users to read and understand.

Also, adding any network-using feature to just would impact just's existing compatibility with offline dev environments. It would mean offline dev environments start needing special consideration. Usually, when a networking-enabled tool accommodates offline dev environments, the environments need to make a change to use the accommodation (e.g. setting an env var, passing additional command-line switch, and/or finding alternative ways to run the online part separately and transfer/save the result). And in the case of just, that could potentially get hairier when considering recipes that invoke another just instance. An individual justfile doesn't need to think about any of this.

[WladyX]

Ok, agree with your points, one question comes to mind.
Let's say multiple users use this system. It's not clear to me, i have not played with modules just yet tbh, how they would import the main just modules, because they have different PATHs.
eg:

• one user may keep the modules in ~/just_modules
• another user may keep the modules in ~/gitrepos/just_modules

but this line in the justfile will be common for both

mod foo 'PATH'

[casey]

You could have both users import a shared justfile, instead of a shared justfile importing the users modules.

[WladyX]

Let me present a use case:
You have 10 microservices, you have some a justfile for each with docker build, docker run, etc.
It is hard to update all if you want to make a change, eg add an argument to docker build, it's easier to update a module that they all depend on, you update in only one place.
You are not the only one working with these microservices, so the modules can reside in different local PATHs.

[laniakea64]

• one user may keep the modules in ~/just_modules

• another user may keep the modules in ~/gitrepos/just_modules

but this line in the justfile will be common for both

mod foo 'PATH'

That PATH can be a relative path, relative to the location of the justfile containing the mod statement.

[fdmysterious]

Agree that adding that kind of network-related stuff could add some uneeded complexity.

Of the few people that are asking for this feature, what are the identified use cases?

For instance, if the relevant use-case is recipe re-use, the file could be fetched using some simple recipe dependency:

ensure_env:
    #!/bin/bash
     if [[ -d foo ]]; then
        cd foo
        git pull
      else
        git clone [email protected]/foo.git
      fi

reciepe_using_dependency *args: ensure_env
     just -f .deps/foo.just foobar_recipe {{args}}

Let's take @WladyX case. Let's say that the docker_build.just file contains this rule:

build target_name path:
    docker buildx -t "{{target_name}}" {{path}}

Then this file could be called in the specific micro-service's Justfile:

cwd := `pwd`
image_name := "my_super_image"

ensure_env:
    # Do whatever is necessary to fetch submodule files

build: ensure_env
    just -f  <path_to_docker_just.build>/docker_build.just build {{image_name}} {{cwd}}

Overall I believe that this kind of need could be filled by some cookbook snippets ?

[WladyX]

Had to adapt it a bit because it said:

error: Path-prefixed recipes may not be used with `--working-directory` or `--justfile`.

so my working example is:

in the main repo i have:

juster % cat infisical-test.just
# test infisical env
_infisical-test path env_var:
    echo "Testing Infisical env"
    infisical run -e prod --path="{{ path }}" -- printenv {{ env_var }}

and in a child repo i have:

argo % cat justfile

# download master just repo
_ensure_env:
    #!/bin/bash
     if [[ -d juster ]]; then
        cd juster
        git pull
      else
        git clone [email protected]:repo/juster.git
      fi

# Show this help
help:
    just -l

import? 'juster/infisical-test.just'
infisical-test: _ensure_env
    just _infisical-test "/K8S-CC" "INFTOK_K8S_CC_PROD"

seems to work as expected.
Thank you!

[casey]

I'm going to close this for now, since I think that the workarounds are reasonable enough.

[gsemet]

Would it be possible to have your "update-mod" recipe "required" to use mod, to avoid the optional mod?

I mean, something like:

• mod are declared as non-optional, but one of the recipe is declared as dependency for the retrieval
• if the module is not here (1) not up to date (2), fetch it using the "update-mod" recipe.

(1) and (2) are much more trickier than most people think. And I would like this implemented somehow in its own module !

(1): the location can be a subfolder from the current project, but on some shared runner we might want to to place it in a shared location.

(2): Also, avoid conflict would require to set the sha1 or the ref of the original location in the path somehow. So different implementation might be possible, so I would let that implemented in a module

If you can provide some kind of basic "fetch" capability (not all systems have "curl" for instance), from http and git (with ref support and of course creds), and allow to make this shared/reuse module in its own dedicated module, this would make "just" so powerful.