[feature] Can revision calculation for SCM mode be changed to allow the user to identify URL pattern?

[RochaStratovan]

What is your suggestion?

We are using the Capturing Git scm information pattern. Specifically for one of the reasons identified in the documentation. We work with code where usage of the header files and libraries is open to everyone, but for security reasons the code is limited to people with the appropriate permissions.

However, we are hitting the issue outlined in the Credentials management section.

We use Gitlab with two factor authentiation.

For development work, we use SSH. However the CI/CD pipelines we have use HTTP combined with their unique job tokens.

In gitlab, every job generates a token that is good for the duration of that job and then becomes invalid. It's unique for every job.

We have multiple linux and windows compiles and they are creating different revisions for the same SCM SHA because of the origin URL.

The origin URL is in the format

https://gitlab-ci-token:<CI_JOB_TOKEN>@git.stratovan.com/Stratovan/foo/bar/bar.git

This is of course problematic because the generated package has a unique value,
the CI_JOB_TOKEN for every job compile. Moreover, the gitlab ci/cd pipeline uses https, while all devs use ssh. Meaning the ci/cd packages wouldn't be useable by devs.

I know conan recoimmends using SSH and provids a workflow for github to securely use SSH. But that method is not available in GitLab. The GitLab CI/CD SSH method is for the runners to have an appropriate /.ssh folder with the keys, etc. That makes SSH extremely insecure in GitLab because any repository yaml could dump the contents of those folders, or save the folder as an aritfact, etc. Long story short, using SSH in Gitlab CI/CD is not a secure or acceptable solution.

My team and I have batted around some ideas for this and we're currently using a workaround of

git remote set-url origin <SSH-PATH>
git conan export-pkg .
git remote set-url origin <original-path>

In our replacement scenario, we only do replacement when the origin has git.stratovan.com as that is our code manager.

However, we were wondering about a feature that might allow one to create a pattern for the origin URL for situations like this.

For example we noticed the the two origins that might be used are

                          ssh:https://[email protected]:2222/Stratovan/Encircle/Encircle.git
https://gitlab-ci-token:${CI_JOB_TOKEN}@git.stratovan.com/Stratovan/Encircle/Encircle.git
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It seems like a filter pattern could be used to remove the prefix using a regular expression such as ^.*?git.stratovan.com\(:2222\)* from the origin and use just the remainder of the string Stratovan/Encircle/Encircle.git for the revision calculation.

Could a feature like this be added? What would be the reason to not do something like this?

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @RochaStratovan

Thanks for your request.

Yes, this is something that we also had some awareness of and we wanted to try to improve it, lets try to have a look.

[shoeffner]

You should be able to configure your setup as follows:

Your GitLab instance is: git.stratovan.com
Your repository is: Stratovan/Encircle/Encircle.git

First, specify your URL as https. This makes it also possible for your IDEs to make it clickable if supported etc.:

class Recipe(ConanFile):
    url = "https://git.stratovan.com/Stratovan/Encircle/Encircle.git"

Second, configure your CI to use a credential helper (note that the quoting and escaping might be off):

git config credential.helper '!f() { test "$1" = "get" && printf "username=gitlab-ci-token\npassword=%s" "${CI_JOB_TOKEN}"; }; f'

You can make it more specific with git config credential."https://git.stratovan.com".helper ... if you want/need to.
We actually modify our CI-Docker image's ~/.gitconfig to contain the following (you can also use this to verify if the command line call worked; it should result in this entry; note that this is adapted and I might again have some quoting errors):

[credential "https://git.stratovan.com"]
        helper = "!f() { test \"$1\" = \"get\" && printf \"username=gitlab-ci-token\\npassword=%s\" \"${CI_JOB_TOKEN}\"; }; f"

Third, tell your ssh-key-using users to setup their ~/.gitconfig files to include (note the trailing slashes and colons):

[url "[email protected]:"]
	insteadOf = "https://git.stratovan.com/"

Their ~/.ssh/config should include your non-standard ssh-port (it might be possible to include that in the .gitconfig as well, but I prefer to keep ssh and git configurations more or less separate):

Host git.stratovan.com
    Port 2222

For your non-ssh users you do not need to change anything, they should be prompted by their OS for their credentials and perform any MFA-flows within the browser.

The result should be:

• A readable URL which can be clicked, copied, etc.
• A credential helper for proper authentication inside the CI pipeline without having to deal with auth inside the conan recipe
• A choice for your developers which protocol they want to use (ssh or https) with very little overhead (they can configure ssh once and then forget about it as it will be configured for all of your internal repositories).

You can find some additional background and discussion in conan-io/conan#15684 (comment).

[memsharded]

Hi @RochaStratovan

Did you manage to test the suggestions from @shoeffner (thanks very much, great idea and great explanation!)

I think that approach sounds a solid one, we might add it to the docs even.

[RochaStratovan]

@memsharded,

Apologies. We've recently had some company restructuring and my conan work/investigations has been interrupted. I have not had a moment to test the proposed method, but it seemed very sound and good.

[memsharded]

No prob, thanks for the feedback!

I am moving this to the docs for adding the above hints there, no change necessary at the client at the moment, but please don't hesitate to open new tickets if there is any gap when you manage to test it, thanks.