kyaml panics on non-ASCII character comments #3417

ordovicia · 2021-01-05T02:41:50Z

Describe the bug

kustomize v3.9.1 (kyaml enabled by default) panics when a YAML includes non-ASCII character comments.
With --enable_kyaml=false, kustomize build succeeds without panic.

Files that can reproduce the issue

kustomization.yaml:

resources:
- a.yaml

a.yaml:

# こんにちは
# 你好
# 안녕하세요
apiVersion: v1
kind: Namespace
metadata:
  name: a

Any of these non-ASCII lines causes the panic.

Expected output

kustomize build does not panic and emits a YAML.

Actual output

$ kustomize build
panic: runtime error: index out of range [20] with length 20 [recovered]
        panic: runtime error: index out of range [20] with length 20

goroutine 1 [running]:
gopkg.in/yaml%2ev3.handleErr(0xc00086d4a0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/yaml.go:276 +0x85
panic(0x525a460, 0xc000680660)
        /usr/local/Cellar/go/1.15.6/libexec/src/runtime/panic.go:969 +0x1b9
gopkg.in/yaml%2ev3.write(0xc0004c8000, 0xc000680640, 0x14, 0x20, 0xc00086b8e8, 0x1)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:90 +0x245
gopkg.in/yaml%2ev3.yaml_emitter_write_comment(0xc0004c8000, 0xc000680640, 0x14, 0x20, 0xc000496420)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:1978 +0x1ae
gopkg.in/yaml%2ev3.yaml_emitter_process_head_comment(0xc0004c8000, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:1111 +0x88
gopkg.in/yaml%2ev3.yaml_emitter_emit_block_mapping_key(0xc0004c8000, 0xc000509630, 0x1, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:775 +0x90
gopkg.in/yaml%2ev3.yaml_emitter_state_machine(0xc0004c8000, 0xc000509630, 0x1)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:297 +0x365
gopkg.in/yaml%2ev3.yaml_emitter_emit(0xc0004c8000, 0xc0004c8240, 0xc000496418)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/emitterc.go:153 +0x14f
gopkg.in/yaml%2ev3.(*encoder).emit(0xc0004c8000)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:81 +0x39
gopkg.in/yaml%2ev3.(*encoder).emitScalar(0xc0004c8000, 0xc00026ef18, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc000696f02, 0x5fbe800, 0x0, ...)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:417 +0x31a
gopkg.in/yaml%2ev3.(*encoder).node(0xc0004c8000, 0xc0004ab540, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:559 +0x136f
gopkg.in/yaml%2ev3.(*encoder).node(0xc0004c8000, 0xc0004ab360, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:508 +0xb25
gopkg.in/yaml%2ev3.(*encoder).node(0xc0004c8000, 0xc0004ab220, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:463 +0x245
gopkg.in/yaml%2ev3.(*encoder).nodev(0xc0004c8000, 0x51ed980, 0xc0004ab220, 0x16)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:421 +0x79
gopkg.in/yaml%2ev3.(*encoder).marshalDoc(0xc0004c8000, 0x0, 0x0, 0x51ed980, 0xc0004ab220, 0x16)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/encode.go:101 +0x23a
gopkg.in/yaml%2ev3.(*Encoder).Encode(0xc00059c078, 0x51ed980, 0xc0004ab220, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/gopkg.in/[email protected]/yaml.go:251 +0xea
sigs.k8s.io/kustomize/kyaml/yaml.String(0xc0004ab220, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/yaml/types.go:214 +0x1b3
sigs.k8s.io/kustomize/kyaml/yaml.(*RNode).String(...)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/yaml/rnode.go:465
sigs.k8s.io/kustomize/kyaml/yaml.(*RNode).MarshalJSON(0xc000324d80, 0xfbe0c28, 0x8, 0x8, 0x0, 0x203000)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/yaml/rnode.go:671 +0x37a
sigs.k8s.io/kustomize/api/internal/wrappy.(*WNode).MarshalJSON(0xc00059c128, 0x203000, 0x203000, 0x203000, 0x8, 0x400bc1f)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/wrappy/wnode.go:172 +0x2e
sigs.k8s.io/kustomize/api/resource.(*Resource).MarshalJSON(0xc0004ab900, 0x5502400, 0x400f2b0, 0xc00040ea60, 0x20, 0x20)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/resource/resource.go:86 +0x33
sigs.k8s.io/kustomize/kyaml/filtersutil.GetRNode(0x5502400, 0xc0004ab900, 0xc0004ab900, 0x5502400, 0xc0004ab900)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/filtersutil/filtersutil.go:60 +0x35
sigs.k8s.io/kustomize/kyaml/filtersutil.ApplyToJSON(0x5504300, 0xc00040ea60, 0xc00086d888, 0x1, 0x1, 0xc00086d890, 0x44d39d7)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/filtersutil/filtersutil.go:26 +0xa9
sigs.k8s.io/kustomize/api/builtins.(*LabelTransformerPlugin).Transform(0xc0002d47e0, 0x555f500, 0xc000370b80, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/builtins/LabelTransformer.go:29 +0x12e
sigs.k8s.io/kustomize/api/internal/target.(*multiTransformer).transform(0xc00040ea40, 0x555f500, 0xc000370b80, 0xc0001c8201, 0x4)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/multitransformer.go:40 +0x79
sigs.k8s.io/kustomize/api/internal/target.(*multiTransformer).Transform(0xc00040ea40, 0x555f500, 0xc000370b80, 0xc0001c82c0, 0x4)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/multitransformer.go:35 +0x85
sigs.k8s.io/kustomize/api/internal/accumulator.(*ResAccumulator).Transform(...)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/accumulator/resaccumulator.go:136
sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).runTransformers(0xc000324400, 0xc000370b40, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/kusttarget.go:269 +0x24a
sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).accumulateTarget(0xc000324400, 0xc000370b40, 0x0, 0x4, 0x1)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/kusttarget.go:194 +0x28d
sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).AccumulateTarget(0xc000324400, 0x40, 0xc000324b80, 0xc000370a01)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/kusttarget.go:155 +0xce
sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).makeCustomizedResMap(0xc000324400, 0x0, 0x0, 0x55575a0, 0x5fbe800)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/kusttarget.go:110 +0x2f
sigs.k8s.io/kustomize/api/internal/target.(*KustTarget).MakeCustomizedResMap(...)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/internal/target/kusttarget.go:106
sigs.k8s.io/kustomize/api/krusty.(*Kustomizer).Run(0xc00086dd20, 0x53283d5, 0x1, 0x0, 0x0, 0x0, 0x0)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/sigs.k8s.io/kustomize/[email protected]/krusty/kustomizer.go:77 +0x29b
sigs.k8s.io/kustomize/kustomize/v3/internal/commands/build.(*Options).RunBuild(0xc0003f0bd0, 0x55021a0, 0xc00018a008, 0x0, 0x0)
        /private/tmp/kustomize-20201230-914-h6stee/kustomize/internal/commands/build/build.go:148 +0xb5
sigs.k8s.io/kustomize/kustomize/v3/internal/commands/build.NewCmdBuild.func1(0xc000273b80, 0x5fbe800, 0x0, 0x0, 0x0, 0x0)
        /private/tmp/kustomize-20201230-914-h6stee/kustomize/internal/commands/build/build.go:70 +0xad
github.com/spf13/cobra.(*Command).execute(0xc000273b80, 0x5fbe800, 0x0, 0x0, 0xc000273b80, 0x5fbe800)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0xc000273600, 0x0, 0xffffffff, 0xc00010a058)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /Users/brew/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
        /private/tmp/kustomize-20201230-914-h6stee/kustomize/main.go:19 +0x2a

Kustomize version

$ kustomize version
{Version:kustomize/v3.9.1 GitCommit:7439f1809e5ccd4677ed52be7f98f2ad75122a93 BuildDate:2020-12-30T00:43:15+00:00 GoOs:darwin GoArch:amd64}

Platform

macOS 10.15.7

Additional context

The text was updated successfully, but these errors were encountered:

Shell32-Natsu · 2021-01-05T18:40:01Z

Thanks for the report. Will take a look.

Shell32-Natsu · 2021-01-05T18:45:54Z

This bug is caused by go-yaml/yaml#538. Looks like yaml library has updated and fixed it. kyaml needs to use a later version of yaml.

ordovicia · 2021-01-06T01:24:17Z

Thank you for investigation!

ordovicia · 2021-02-04T03:40:47Z

Seems that this issue is resolved in kustomize v3.9.2.
Thank you!

$ kustomize version
{Version:kustomize/v3.9.2 GitCommit:e98eada7365fc564c9aba392e954f306a9cbf1dd BuildDate:2021-01-21T14:07:31+00:00 GoOs:darwin GoArch:amd64}

$ cat kustomization.yaml
resources:
- a.yaml

$ cat a.yaml
# こんにちは
# 你好
# 안녕하세요
apiVersion: v1
kind: Namespace
metadata:
  name: a

$ kustomize build
apiVersion: v1
kind: Namespace
metadata:
  name: a

iamnogada · 2021-05-08T13:05:33Z

With version v4.1.2
I got the same error with this version.

# 한글(korean)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: employees
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /

kustomize build . > temp.yaml
Error: map[string]interface {}{"apiVersion":"networking.k8s.io/v1", "kind":"Ingress", "metadata":map[string]interface {}{"annotations":map[string]interface {}{"nginx.ingress.kubernetes.io/rewrite-target":"/"}, "name":"employees", "namespace":"sam"}, "spec":map[string]interface {}{"rules":[]interface {}{map[string]interface {}{"host":"sam.htdp1.kubepia.net", "http":map[string]interface {}{"paths":[]interface {}{map[string]interface {}{"backend":map[string]interface {}{"service":map[string]interface {}{"name":"employees", "port":map[string]interface {}{"name":"http"}}}, "path":"/", "pathType":"Prefix"}}}}}}}: yaml: invalid trailing UTF-8 octet

ordovicia · 2021-05-15T14:14:09Z

I confirmed that, as @iamnogada reported, kustomize v4.1.2 has the same issue.

brianpursley · 2021-06-23T02:20:36Z

I just tried in 4.1.3 and don't get the error. I do get the error in 4.1.2 though, so it looks like it might be fixed (?)

@ordovicia Can you give it a try with 4.1.3?

natasha41575 · 2021-06-23T02:33:44Z

Adding a note that this is a duplicate of #3358, where they reported that the issue still appears with the following kind of comment:

apiVersion: networking.k8s.io/v1
kind: Ingress # あ
metadata:
  name: blog

ordovicia · 2021-06-23T02:48:31Z

The bug seems to have been fixed in kustomize 4.1.3. Thank you.

kustomization.yaml:

resources:
- a.yaml
- b.yaml

a.yaml:

# こんにちは
# 你好
# 안녕하세요
apiVersion: v1
kind: Namespace
metadata:
  name: a

b.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress # あ
metadata:
  name: blog

Output:

$ kustomize version
{Version:kustomize/v4.1.3 GitCommit:0f614e92f72f1b938a9171b964d90b197ca8fb68 BuildDate:2021-05-21T17:30:04+01:00 GoOs:darwin GoArch:amd64}

$ kustomize build
apiVersion: v1
kind: Namespace
metadata:
  name: a
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: blog

Shell32-Natsu self-assigned this Jan 5, 2021

Shell32-Natsu added area/kyaml issues for kyaml kind/bug Categorizes issue or PR as related to a bug. triage/accepted Indicates an issue or PR is ready to be actively worked on. labels Jan 5, 2021

Shell32-Natsu mentioned this issue Jan 20, 2021

Strategic Merge Patch doesn't seem to handle character other than basic ascii #3486

Closed

ordovicia closed this as completed Feb 4, 2021

110y mentioned this issue Feb 15, 2021

kustomize build cause the panic with non-ASCII characters #3605

Closed

Shell32-Natsu mentioned this issue Apr 16, 2021

Upgrade go-yaml #3789

Closed

ordovicia reopened this May 15, 2021

ordovicia closed this as completed Jun 23, 2021

natasha41575 mentioned this issue Jun 23, 2021

kustomize build panics if the comment contains multibyte characters #3358

Closed

mgoltzsche mentioned this issue Oct 18, 2021

feat: update to kyaml 0.12.0 mgoltzsche/khelm#26

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kyaml panics on non-ASCII character comments #3417

kyaml panics on non-ASCII character comments #3417

ordovicia commented Jan 5, 2021 •

edited

Loading

Shell32-Natsu commented Jan 5, 2021

Shell32-Natsu commented Jan 5, 2021

ordovicia commented Jan 6, 2021

ordovicia commented Feb 4, 2021

iamnogada commented May 8, 2021 •

edited

Loading

ordovicia commented May 15, 2021

brianpursley commented Jun 23, 2021

natasha41575 commented Jun 23, 2021 •

edited

Loading

ordovicia commented Jun 23, 2021

kyaml panics on non-ASCII character comments #3417

kyaml panics on non-ASCII character comments #3417

Comments

ordovicia commented Jan 5, 2021 • edited Loading

Shell32-Natsu commented Jan 5, 2021

Shell32-Natsu commented Jan 5, 2021

ordovicia commented Jan 6, 2021

ordovicia commented Feb 4, 2021

iamnogada commented May 8, 2021 • edited Loading

ordovicia commented May 15, 2021

brianpursley commented Jun 23, 2021

natasha41575 commented Jun 23, 2021 • edited Loading

ordovicia commented Jun 23, 2021

ordovicia commented Jan 5, 2021 •

edited

Loading

iamnogada commented May 8, 2021 •

edited

Loading

natasha41575 commented Jun 23, 2021 •

edited

Loading