support quota set on overlay snapshot #5859

yylt · 2021-08-11T03:22:59Z

support set quota on overlay snapshot

as discussed in issue #3329

enable quota set by follows:

Make sure the root directory of containerd is mounted with 'pquota'
Set 'overlay' as the default snapshot and enable quota in 'overlay' config
Set the default quota size in CRI

config.toml like this

  [plugins.'io.containerd.cri.v1.runtime']
    enable_selinux = false
    default_snapshot_quota_size = '2M'   #+
...
  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = ''
    upperdir_label = false
    sync_remove = false
    slow_chown = false
    mount_options = []
    enable_quota = true  #+

check containerd root mount info

# /etc/fstab
...
/dev/mapper/containerd_root  /var/lib/containerd  xfs     defaults,pquota        0 0

# mount ...
/dev/mapper/containerd_root on /var/lib/containerd type xfs (rw,relatime,prjquota)

check container had been set quota

xfs_quota -x -c "report -h" /var/lib/containerd/

Signed-off-by: Yang Yang [email protected]

k8s-ci-robot · 2021-08-11T03:23:08Z

Hi @yylt. Thanks for your PR.

I'm waiting for a containerd member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

theopenlab-ci · 2021-08-11T04:04:56Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 4m 52s (non-voting)
containerd-test-arm64 : SUCCESS in 6m 12s (non-voting)
containerd-integration-test-arm64 : SUCCESS in 24m 31s (non-voting)

AkihiroSuda · 2021-08-11T13:01:26Z

snapshots/overlay/overlayutils/fs_linux.go

+ See the License for the specific language governing permissions and
+ limitations under the License.
+*/
+


This file seems copied from Moby. Please clarify the origin of the code and the license in code comments.

AkihiroSuda · 2021-08-11T13:01:53Z

snapshots/quota/projectquota.go

+#define Q_XGETPQUOTA QCMD(Q_XGETQUOTA, PRJQUOTA)
+#endif
+*/
+import "C"


Can we avoid cgo

Can we avoid cgo

Done!

dmcgowan · 2021-08-11T18:34:02Z

Quota handling is a topic we should discuss at a high level first and possibly a good topic for a future community meeting. However, getting library support for setting project quotas across different backing FS makes sense to do now.

As an end feature, we need to discuss how it will be integrated with Kubernetes and how we can use it from our own tooling and go libraries. It seems backwards to have the cri layer here enable the quota then the snapshot set a static quota. I would think the quota would be set by the client adding a quota and the snapshotter enabling it via configuration and erroring out if the backing FS couldn't support it when enabled. The difficult part is figuring out from the client perspective whether quota is enabled so it knows whether it can avoid expensive ephemeral storage accounting (such as in the Kubelet).

lining2020x · 2021-11-11T02:48:47Z

pkg/cri/config/config.go

@@ -266,6 +266,9 @@ type PluginConfig struct {
 // of being placed under the hardcoded directory /var/run/netns. Changing this setting requires
 // that all containers are deleted.
 NetNSMountsUnderStateDir bool `toml:"netns_mounts_under_state_dir" json:"netnsMountsUnderStateDir"`
+ // SupportSetQuota indicate to set quota on container read-write snapshot, when snapshot do not
+ // support quota set, it will do nothing.
+ SupportSetQuota bool `toml:"support_set_quota" json:"supportSetQuota"`


Maybe EnableRootfsQuota or EnableSetQuotasounds better.

pacoxu · 2022-06-09T05:35:03Z

The difficult part is figuring out from the client perspective whether quota is enabled so it knows whether it can avoid expensive ephemeral storage accounting (such as in the Kubelet).

In Kubernetes, there is a feature gate and a check method SupportsQuotas, here is the KEP "Quotas for Ephemeral Storage" kubernetes/enhancements#1029 from Kubernetes.
https://github.com/kubernetes/kubernetes/blob/226323178e23b4c476001266beab1e2f116b3879/pkg/volume/util/fsquota/common/quota_linux_common_impl.go#L180-L190

// SupportsQuotas determines whether the filesystem supports quotas.
func SupportsQuotas(mountpoint string, qType QuotaType) (bool, error) {
	data, err := runXFSQuotaCommand(mountpoint, "state -p")
	if err != nil {
		return false, err
	}
	if qType == FSQuotaEnforcing {
		return strings.Contains(data, "Enforcement: ON"), nil
	}
	return strings.Contains(data, "Accounting: ON"), nil
}

yylt · 2022-06-16T07:23:22Z

sorry for so long time later.

As an end feature, we need to discuss how it will be integrated with Kubernetes and how we can use it from our own tooling and go libraries. It seems backwards to have the cri layer here enable the quota then the snapshot set a static quota. I would think the quota would be set by the client adding a quota and the snapshotter enabling it via configuration and erroring out if the backing FS couldn't support it when enabled. The difficult part is figuring out from the client perspective whether quota is enabled so it knows whether it can avoid expensive ephemeral storage accounting (such as in the Kubelet).

it is good suggestion. now quota setter focus on overlay, and with pure go.

and update snapshots litter, add new option WithQuotaSize. mostly if the option not define, will use default size in overlayfs configurtion

such as:

  [plugins."io.containerd.snapshotter.v1.overlayfs"]
    root_path = ""
    quota_size = "128MB"

pacoxu

I tried to build a containerd to do some test. Too many quota is created in my env.

[root@daocloud ~]# xfs_quota -xc "report -a -p -h"  | head -n 100
Project quota on /var/lib/kubelet (/dev/sdc)
                        Blocks
Project ID   Used   Soft   Hard Warn/Grace
---------- ---------------------------------
#0           820K      0      0  00 [------]
volume1048590      0     8E     8E  00 [------]
volume1048605      0     8E     8E  00 [------]
volume1048607      0     8E     8E  00 [------]
volume1048610      0     8E     8E  00 [------]

Project quota on /var/lib/containerd (/dev/sdb)
                        Blocks
Project ID   Used   Soft   Hard Warn/Grace
---------- ---------------------------------
#0          10.2G      0      0  00 [------]
#2              0      0      0  00 [------]
#3              0    15G    15G  00 [------]
#4            12K    15G    15G  00 [------]
#5              0    15G    15G  00 [------]
#6              0    15G    15G  00 [------]
#7              0    15G    15G  00 [------]
#8              0    15G    15G  00 [------]
#9              0    15G    15G  00 [------]
#10             0    15G    15G  00 [------]
#11             0    15G    15G  00 [------]
#12             0    15G    15G  00 [------]
#13             0    15G    15G  00 [------]
#14             0    15G    15G  00 [------]
#15             0    15G    15G  00 [------]
#16             0    15G    15G  00 [------]
#17             0    15G    15G  00 [------]
#18             0    15G    15G  00 [------]
#19             0    15G    15G  00 [------]
#20             0    15G    15G  00 [------]
#21             0    15G    15G  00 [------]
#22             0    15G    15G  00 [------]
#23             0    15G    15G  00 [------]
#24            4K    15G    15G  00 [------]
#25             0    15G    15G  00 [------]
#26             0    15G    15G  00 [------]
#27             0    15G    15G  00 [------]
#28             0    15G    15G  00 [------]
#29            4K    15G    15G  00 [------]
#30             0    15G    15G  00 [------]
#31             0    15G    15G  00 [------]
#32             0    15G    15G  00 [------]
#33             0    15G    15G  00 [------]
#34             0    15G    15G  00 [------]
#35            4K    15G    15G  00 [------]
#36            4K    15G    15G  00 [------]
#37             0    15G    15G  00 [------]
#38          540K    15G    15G  00 [------]
#39             0    15G    15G  00 [------]
#40             0    15G    15G  00 [------]
#41             0    15G    15G  00 [------]
#42             0    15G    15G  00 [------]
#43             0    10G    10G  00 [------]
#44             0    10G    10G  00 [------]
#45             0    10G    10G  00 [------]
#46             0    10G    10G  00 [------]
#47             0    10G    10G  00 [------]
#48             0    10G    10G  00 [------]
#49             0    10G    10G  00 [------]
#50             0    10G    10G  00 [------]
#51             0    10G    10G  00 [------]
#52             0    10G    10G  00 [------]
#53             0    10G    10G  00 [------]
#54             0    10G    10G  00 [------]
#55             0    10G    10G  00 [------]
#56             0    10G    10G  00 [------]
#57             0    10G    10G  00 [------]
#58             0    10G    10G  00 [------]
#59             0    10G    10G  00 [------]
#60             0    10G    10G  00 [------]
#61             0    10G    10G  00 [------]
#62             0    10G    10G  00 [------]
#63             0    10G    10G  00 [------]

I set the size to 10G at first and 15G later.

[root@daocloud ~]# xfs_quota -xc "report -a -p -h"  | wc -l
1956507

pacoxu · 2022-06-23T08:02:42Z

snapshots/overlay/plugin/plugin.go

@@ -57,6 +62,15 @@ func init() {
 oOpts = append(oOpts, overlay.WithUpperdirLabel)
 }

+ ic.Meta.Exports["root"] = root


dup with line 74.

pacoxu · 2022-06-23T08:03:39Z

snapshots/overlay/overlay.go

+ fsMagic, err := overlayutils.GetFSMagic(root)
+ if err != nil {
+ return nil, err
+ }
+ if config.quotaSize > 0 {


Suggested change

fsMagic, err := overlayutils.GetFSMagic(root)

if err != nil {

return nil, err

}

if config.quotaSize > 0 {

if config.quotaSize > 0 {

fsMagic, err := overlayutils.GetFSMagic(root)

if err != nil {

return nil, err

}

pacoxu · 2022-06-23T10:02:49Z

snapshots/overlay/overlay.go

+ } else {
+ size = config.quotaSize
+ }
+ return quotaCtl.SetAllQuota(size, targets...)


setting quota may need some logs. Probably, some debug log.

yylt · 2022-06-27T09:49:51Z

I tried to build a containerd to do some test. Too many quota is created in my env.

[root@daocloud ~]# xfs_quota -xc "report -a -p -h"  | head -n 100
Project quota on /var/lib/kubelet (/dev/sdc)
                        Blocks
Project ID   Used   Soft   Hard Warn/Grace
---------- ---------------------------------
#0           820K      0      0  00 [------]
volume1048590      0     8E     8E  00 [------]
volume1048605      0     8E     8E  00 [------]
volume1048607      0     8E     8E  00 [------]
volume1048610      0     8E     8E  00 [------]

Project quota on /var/lib/containerd (/dev/sdb)
                        Blocks
Project ID   Used   Soft   Hard Warn/Grace
---------- ---------------------------------
#0          10.2G      0      0  00 [------]
#2              0      0      0  00 [------]
#3              0    15G    15G  00 [------]
#4            12K    15G    15G  00 [------]
#5              0    15G    15G  00 [------]
#6              0    15G    15G  00 [------]
#7              0    15G    15G  00 [------]
#8              0    15G    15G  00 [------]
#9              0    15G    15G  00 [------]
#10             0    15G    15G  00 [------]
#11             0    15G    15G  00 [------]
#12             0    15G    15G  00 [------]
#13             0    15G    15G  00 [------]
#14             0    15G    15G  00 [------]
#15             0    15G    15G  00 [------]
#16             0    15G    15G  00 [------]
#17             0    15G    15G  00 [------]
#18             0    15G    15G  00 [------]
#19             0    15G    15G  00 [------]
#20             0    15G    15G  00 [------]
#21             0    15G    15G  00 [------]
#22             0    15G    15G  00 [------]
#23             0    15G    15G  00 [------]
#24            4K    15G    15G  00 [------]
#25             0    15G    15G  00 [------]
#26             0    15G    15G  00 [------]
#27             0    15G    15G  00 [------]
#28             0    15G    15G  00 [------]
#29            4K    15G    15G  00 [------]
#30             0    15G    15G  00 [------]
#31             0    15G    15G  00 [------]
#32             0    15G    15G  00 [------]
#33             0    15G    15G  00 [------]
#34             0    15G    15G  00 [------]
#35            4K    15G    15G  00 [------]
#36            4K    15G    15G  00 [------]
#37             0    15G    15G  00 [------]
#38          540K    15G    15G  00 [------]
#39             0    15G    15G  00 [------]
#40             0    15G    15G  00 [------]
#41             0    15G    15G  00 [------]
#42             0    15G    15G  00 [------]
#43             0    10G    10G  00 [------]
#44             0    10G    10G  00 [------]
#45             0    10G    10G  00 [------]
#46             0    10G    10G  00 [------]
#47             0    10G    10G  00 [------]
#48             0    10G    10G  00 [------]
#49             0    10G    10G  00 [------]
#50             0    10G    10G  00 [------]
#51             0    10G    10G  00 [------]
#52             0    10G    10G  00 [------]
#53             0    10G    10G  00 [------]
#54             0    10G    10G  00 [------]
#55             0    10G    10G  00 [------]
#56             0    10G    10G  00 [------]
#57             0    10G    10G  00 [------]
#58             0    10G    10G  00 [------]
#59             0    10G    10G  00 [------]
#60             0    10G    10G  00 [------]
#61             0    10G    10G  00 [------]
#62             0    10G    10G  00 [------]
#63             0    10G    10G  00 [------]

I set the size to 10G at first and 15G later.

[root@daocloud ~]# xfs_quota -xc "report -a -p -h"  | wc -l
1956507

yes, it becauce set quota committed layer too. but it is not expected

pacoxu · 2022-07-06T06:58:46Z

I got some new errors with latest code.

7月 06 14:57:03 paco containerd[10220]: time="2022-07-06T14:57:03.299383030+08:00" level=info msg="loading plugin "io.containerd.snapshotter.v1.overlayfs"..." type=io.containerd.snapshotter.v1
7月 06 14:57:03 paco containerd[10220]: time="2022-07-06T14:57:03.299968488+08:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.overlayfs" error="Failed to set quota limit for projid 1 on /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/backingFsBlockDev: function not implemented"
7月 06 14:57:03 paco containerd[10220]: time="2022-07-06T14:57:03.300169800+08:00" level=warning msg="could not use snapshotter overlayfs in metadata plugin" error="Failed to set quota limit for projid 1 on /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/backingFsBlockDev: function not implemented"

And the quota is not set as expected.

yylt · 2022-07-29T08:00:05Z

I try command GOGC=off containerd -c myconfig.toml, and it always successed, and use code below, but it does not work.

	_, _, errno := syscall.Syscall6(syscall.SYS_QUOTACTL, uintptr(qcmd(Q_XSETQLIM, XFS_PROJ_QUOTA)),
		(*[2]uintptr)(unsafe.Pointer(&backingFsBlockDev))[0], uintptr(projectID),
		uintptr(unsafe.Pointer(&fd)), 0, 0)

and use bpftrace to debug, and the syscall _sys_quotactl retval could be 0, -2 , -20.

bpftrace -e 'kretprobe:__x64_sys_quotactl {printf(%s, retval %d\n",kstack,retval)}'

finally, use below to translate string to uintptr could be work fine.

	devbyte := append([]byte(backingFsBlockDev), 0)

	_, _, errno := syscall.Syscall6(syscall.SYS_QUOTACTL, uintptr(qcmd(Q_XSETQLIM, XFS_PROJ_QUOTA)),
		uintptr(unsafe.Pointer(&devbyte[0])), uintptr(fd.id),
		uintptr(unsafe.Pointer(&fd)), 0, 0)

yylt · 2024-04-03T12:14:01Z

/retest

k8s-ci-robot · 2024-04-03T12:14:15Z

@yylt: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

yylt · 2024-04-08T02:00:30Z

Is it necessary to use the new version of ackagep in this way to use fs magic? This looks quite ugly.

require github.com/containerd/continuity v0.4.4-0.20240406122502-56a67e3ba427

pacoxu · 2024-04-11T03:05:36Z

Is it necessary to use the new version of ackagep in this way to use fs magic? This looks quite ugly.

require github.com/containerd/continuity v0.4.4-0.20240406122502-56a67e3ba427

Can this be done in a seperate PR?

yylt · 2024-04-11T03:31:55Z

Can this be done in a seperate PR?

It doesn't seem appropriate, because the overlay judgement of whether the file system supports setting quotas requires using magic fs.

Signed-off-by: Yang Yang <[email protected]>

k8s-ci-robot · 2024-07-01T21:28:02Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot added the needs-ok-to-test label Aug 11, 2021

yylt force-pushed the quota branch from e27997e to a90db5f Compare August 11, 2021 03:38

AkihiroSuda reviewed Aug 11, 2021

View reviewed changes

AkihiroSuda added the status/needs-discussion Needs discussion and decision from maintainers label Aug 11, 2021

AkihiroSuda added this to the 1.6 milestone Aug 11, 2021

AkihiroSuda added kind/enhancement kind/feature labels Aug 11, 2021

dmcgowan added this to New in Code Review via automation Aug 11, 2021

dmcgowan moved this from New to Needs Discussion in Code Review Aug 11, 2021

This was referenced Aug 16, 2021

Filesystem quotas #759

Open

Support disk usage stats for containerd #5882

Closed

lining2020x reviewed Nov 11, 2021

View reviewed changes

dmcgowan modified the milestones: 1.6, 1.7 Dec 9, 2021

yylt force-pushed the quota branch from a90db5f to 72e9222 Compare June 16, 2022 07:23

yylt force-pushed the quota branch from 72e9222 to d165339 Compare June 16, 2022 08:01

pacoxu reviewed Jun 23, 2022

View reviewed changes

yylt force-pushed the quota branch from d165339 to 07398d4 Compare June 27, 2022 09:45

yylt force-pushed the quota branch 2 times, most recently from f72d83c to ed16c4b Compare July 29, 2022 07:42

dmcgowan marked this pull request as draft November 30, 2022 01:53

yylt force-pushed the quota branch 4 times, most recently from fcf0f99 to bdf63b4 Compare April 3, 2024 10:00

yylt force-pushed the quota branch from bdf63b4 to c778fe4 Compare April 15, 2024 09:56

k8s-ci-robot added size/XXL and removed size/XL labels Apr 15, 2024

yylt force-pushed the quota branch from c778fe4 to af6c8c7 Compare April 15, 2024 13:02

yylt marked this pull request as ready for review April 16, 2024 01:36

k8s-ci-robot removed the do-not-merge/work-in-progress label Apr 16, 2024

dmcgowan modified the milestones: 2.0, 2.1 Apr 25, 2024

k8s-ci-robot added the needs-rebase label May 3, 2024

yylt force-pushed the quota branch from af6c8c7 to 7d34225 Compare May 4, 2024 14:40

k8s-ci-robot added needs-rebase and removed needs-rebase labels May 4, 2024

yylt force-pushed the quota branch from 7d34225 to 68262eb Compare May 8, 2024 09:26

k8s-ci-robot removed the needs-rebase label May 9, 2024

k8s-ci-robot added the needs-rebase label Jun 6, 2024

yylt force-pushed the quota branch from 68262eb to 3a6f524 Compare June 11, 2024 07:23

k8s-ci-robot removed the needs-rebase label Jun 11, 2024

support quota set

1b3905f

Signed-off-by: Yang Yang <[email protected]>

yylt force-pushed the quota branch from 3a6f524 to 1b3905f Compare June 24, 2024 03:03

k8s-ci-robot added the needs-rebase label Jul 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support quota set on overlay snapshot #5859

support quota set on overlay snapshot #5859

yylt commented Aug 11, 2021 •

edited

Loading

k8s-ci-robot commented Aug 11, 2021

theopenlab-ci bot commented Aug 11, 2021

AkihiroSuda Aug 11, 2021

AkihiroSuda Aug 11, 2021

yylt Apr 2, 2024

dmcgowan commented Aug 11, 2021

lining2020x Nov 11, 2021 •

edited

Loading

pacoxu commented Jun 9, 2022

yylt commented Jun 16, 2022

pacoxu left a comment

pacoxu Jun 23, 2022

yylt Jun 27, 2022

pacoxu Jun 23, 2022

pacoxu Jun 23, 2022

yylt Jul 29, 2022

yylt commented Jun 27, 2022

pacoxu commented Jul 6, 2022 •

edited

Loading

yylt commented Jul 29, 2022

yylt commented Apr 3, 2024

k8s-ci-robot commented Apr 3, 2024

yylt commented Apr 8, 2024

pacoxu commented Apr 11, 2024

yylt commented Apr 11, 2024

k8s-ci-robot commented Jul 1, 2024

support quota set on overlay snapshot #5859

Are you sure you want to change the base?

support quota set on overlay snapshot #5859

Conversation

yylt commented Aug 11, 2021 • edited Loading

k8s-ci-robot commented Aug 11, 2021

theopenlab-ci bot commented Aug 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmcgowan commented Aug 11, 2021

lining2020x Nov 11, 2021 • edited Loading

Choose a reason for hiding this comment

pacoxu commented Jun 9, 2022

yylt commented Jun 16, 2022

pacoxu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yylt commented Jun 27, 2022

pacoxu commented Jul 6, 2022 • edited Loading

yylt commented Jul 29, 2022

yylt commented Apr 3, 2024

k8s-ci-robot commented Apr 3, 2024

yylt commented Apr 8, 2024

pacoxu commented Apr 11, 2024

yylt commented Apr 11, 2024

k8s-ci-robot commented Jul 1, 2024

yylt commented Aug 11, 2021 •

edited

Loading

lining2020x Nov 11, 2021 •

edited

Loading

pacoxu commented Jul 6, 2022 •

edited

Loading