-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tag to digest resolving is not working in Knative 1.3.0 #12761
Comments
How are the pull credentials configured for the private image? |
/triage needs-user-input |
The credentials are defined in a secret of type |
Can you paste your |
Reason why I'm asking is because I fixed this case for gitlab before where fetching the creds from the k8s secret wasn't working google/go-containerregistry#1299 |
The other option is listing the steps you took to create your image pull secret |
The problem isn't just with gitlab, it's also with azurecr. I created the imagepullsecret using this secret definition:
And the dockerconfigjson for gitlab and azurecr: |
Can you give exact steps when creating the registry credential and also the exact image prefix ie. not The Gitlab fix was normalizing the URL schemes in the dockerconfigjson - so knowing whether it shows |
I pasted the dockerconfigjson in my earlier comment to clearly show there was no To create the secret I ran the following command for gitlab:
I also tried providing a scheme like http in the docker-server arg, but it also didn't work. As for ACR, I ran a similar command with the docker-server as |
I have the same issue upgrading from knative serving 1.1.4 to 1.3.0. I created the dockerconfigjson secret for gcr registry in my namespace and added as imagePullSecrets in the default service account. On 1.1.4 It works fine. Reverting to 1.1.4 started working again. I follow the knative serving setup by yaml |
I've also tested knative serving 1.2.2 and it works well. Reading the knative-serving Release I've noticed that with knative-serving 1.2.3 the github.com/google/go-containerregistry/pkg/authn/k8schain was refactored. I think that's what creates the issue. Although the service account is created correctly, the knative controller cannot schedule the pods because it cannot contact the registry. |
The k8schain refactor pre-1.3 definitely sounds like the culprit. I'm not able to reproduce in the minimal unit test added in google/go-containerregistry#1335 -- this doesn't necessarily mean it isn't an issue (it seems fairly obvious that it is) just that I haven't been able to nail down where in k8schain's secret handling code it's going wrong. |
Actually on 1.3.0 still needs this workaroud #12642 to download the image from the private gcr. |
/assign @dprotaso |
Confirmed the bug - I made a simple mistake of not bumping the sub package ( Lines 12 to 13 in f4ea3ac
I'll bump the dependency and get patch releases out tomorrow. |
Patch releases are out: v1.3 - https://github.com/knative/serving/releases/tag/knative-v1.3.1 |
I seem to be facing the same issue in the patch as well. @thepiger do you mind confirming this? |
I tested this by creating three docker cred k8s secrets with three different variations on the registry url
kubectl create secret docker-registry gitlab-auth-with-no-scheme \
--docker-server=registry.gitlab.com \
--docker-email= [email protected] \
--docker-username= someemail \
--docker-password=$GITLAB_TOKEN
kubectl create secret docker-registry gitlab-auth-with-http \
--docker-server=https://registry.gitlab.com \
[email protected] \
--docker-username= someemail \
--docker-password=$GITLAB_TOKEN
kubectl create secret docker-registry gitlab-auth-with-https \
--docker-server=https://registry.gitlab.com \
--docker-email= [email protected] \
--docker-username= someemail \
--docker-password=$GITLAB_TOKEN Then I created Knative Services using the imagePullSecret ie. apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: test-https-scheme
spec:
template:
spec:
imagePullSecrets:
- name: gitlab-auth-with-https
containers:
- image: registry.gitlab.com/someemail/test/golang:latest Let me know if your scenario is different and I can confirm |
From the first tests I did, it seems that the patch has solved the problem. |
@dprotaso it's working with gitlab but not azure container registry. Using the same secret in the azure container registry works when providing a digest but not when providing a tag. |
Also I'm not sure if this behavior is intentional or not, but using a gitlab registry secret in this format works fine:
However using this secret format (not saying this is correct) doesn't work unless I add
|
Thanks for the feedback - was finally able to get an azure account and confirmed that it's just broken. Here's the breakdown so far: K8s Deployment Image PullFor comparison with a vanilla K8s deployment
Oddly - K8s works even with a partial match - ie. Knative Service
|
Ok new releases are out with the fixes https://github.com/knative/serving/releases/tag/knative-v1.3.2 Please let me know if there are any issues |
For future reference this is the test I used to confirm the controllers behaved properly - This was placed under the path package revision
import (
"context"
"fmt"
"net/http"
"os/exec"
"testing"
"github.com/google/go-containerregistry/pkg/authn/k8schain"
"github.com/google/go-containerregistry/pkg/name"
"k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/tools/clientcmd"
)
func TestRealResolveGitlab(t *testing.T) {
img := "registry.gitlab.com/dprotaso/test/nginx:latest"
// Create a tag pointing to an image on our fake registry.
tag, err := name.NewTag(img, name.WeakValidation)
if err != nil {
t.Fatal("NewTag() =", err)
}
kubeconfig := "/Users/dprotasowski/.kube/config"
// use the current context in kubeconfig
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
t.Fatal(err)
}
// create the clientset
client, err := kubernetes.NewForConfig(config)
cases := []string{
"registry.gitlab.com",
"registry.gitlab.com/dpro",
"registry.gitlab.com/dprotaso",
"registry.gitlab.com/dprotaso/test",
"registry.gitlab.com/dprotaso/test/nginx",
}
// Resolve our tag on the fake registry to the digest of the random.Image().
dr := &digestResolver{client: client, transport: http.DefaultTransport, userAgent: "test-agent"}
for _, c := range cases {
t.Run(c, func(t *testing.T) {
err := client.CoreV1().Secrets("default").Delete(context.TODO(), "gitlab-container-secret", metav1.DeleteOptions{})
if err != nil && !errors.IsNotFound(err) {
t.Log("failed to delete secret")
t.Fatal(err)
}
cmd := exec.Command("kubectl", "create", "secret", "docker-registry", "gitlab-container-secret",
fmt.Sprintf("--docker-server=%s", c),
"--docker-email=EMAIL",
"--docker-username=UESRNAME",
"--docker-password=SECRET")
output, err := cmd.Output()
if err != nil {
t.Log(string(output))
t.Fatal(err)
}
opt := k8schain.Options{
Namespace: "default",
ServiceAccountName: "default",
ImagePullSecrets: []string{"gitlab-container-secret"},
}
resolvedDigest, err := dr.Resolve(context.Background(), tag.String(), opt, emptyRegistrySet)
if err != nil {
t.Fatal("Resolve() =", err)
}
// Make sure that we get back the appropriate digest.
digest, err := name.NewDigest(resolvedDigest, name.WeakValidation)
if err != nil {
t.Fatal("NewDigest() =", err)
}
t.Log(digest)
})
}
}
func TestRealResolveAzure(t *testing.T) {
img := "dtestcontainer.azurecr.io/dave/nginx:latest"
// Create a tag pointing to an image on our fake registry.
tag, err := name.NewTag(img, name.WeakValidation)
if err != nil {
t.Fatal("NewTag() =", err)
}
kubeconfig := "/Users/dprotasowski/.kube/config"
// use the current context in kubeconfig
config, err := clientcmd.BuildConfigFromFlags("", kubeconfig)
if err != nil {
t.Fatal(err)
}
// create the clientset
client, err := kubernetes.NewForConfig(config)
cases := []string{
"dtestcontainer.azurecr.io",
"dtestcontainer.azurecr.io/dav",
"dtestcontainer.azurecr.io/dave",
"dtestcontainer.azurecr.io/dave/nginx",
}
// Resolve our tag on the fake registry to the digest of the random.Image().
dr := &digestResolver{client: client, transport: http.DefaultTransport, userAgent: "test-agent"}
for _, c := range cases {
t.Run(c, func(t *testing.T) {
t.Run("setup", func(t *testing.T) {
err := client.CoreV1().Secrets("default").Delete(context.TODO(), "azure-container-secret", metav1.DeleteOptions{})
if err != nil && !errors.IsNotFound(err) {
t.Log("failed to delete secret")
t.Fatal(err)
}
cmd := exec.Command("kubectl", "create", "secret", "docker-registry", "azure-container-secret",
fmt.Sprintf("--docker-server=%s", c),
"--docker-username=EMAIL",
"--docker-password=PASSWORD")
output, err := cmd.Output()
if err != nil {
t.Log(string(output))
t.Fatal(err)
}
})
t.Run("resolve", func(t *testing.T) {
opt := k8schain.Options{
Namespace: "default",
ServiceAccountName: "default",
ImagePullSecrets: []string{"azure-container-secret"},
}
resolvedDigest, err := dr.Resolve(context.Background(), tag.String(), opt, emptyRegistrySet)
if err != nil {
t.Fatal("Resolve() =", err)
}
// Make sure that we get back the appropriate digest.
digest, err := name.NewDigest(resolvedDigest, name.WeakValidation)
if err != nil {
t.Fatal("NewDigest() =", err)
}
t.Log(digest)
})
})
}
} |
same above issue facing with docker hub registry ! any solution ! @dprotasomy image tag is like username/imagename:$CI_COMMIT_SHA |
@dprotaso Does knative load an image from local docker registry? I am using KinD.
Error in logs: |
What version of Knative?
Expected Behavior
I should be able to create a knative service that pulls an image with a defined tag from a private registry, the exact same service definition file was working with knative
0.25.x
Actual Behavior
I get
failed to resolve image to digest
when deploying an image with a defined tag. I tried it with azurecr as well as registry.gitlab.com, same error. However, when I added the image digest explicitly it worked.I'm aware that adding the
registriesSkippingTagResolving
is one workaround, but it won't be useful for me since users might deploy images in my app from registries I'm not aware of.Steps to Reproduce the Problem
Using knative v1.3.0, attempt deploying a service with an image from a private registry with a defined tag.
The text was updated successfully, but these errors were encountered: