Spinning container lab in gitlab runner not working

[priyacshah]

Hi, I am trying to deploy clab in Kubernetes gitlab runner. I am finding this issue when I run clab dep -t command

INFO[0000] Containerlab v0.52.0 started                 
INFO[0000] Parsing & checking topology file: agent.clab.yml 
WARN[0000] failed to enable LLDP on docker bridge: open /sys/class/net/br-e3d71209374a/bridge/group_fwd_mask: read-only file system 
WARN[0000] errors during iptables rules install: not available 
WARN[0000] Unable to init module loader: stat /lib/modules/6.1.54-talos/modules.dep: no such file or directory. Skipping... 
INFO[0000] Creating lab directory: /builds/clab/agent/clab-agent 
INFO[0000] Creating container: "agent"          
INFO[0000] Running postdeploy actions for Nokia SR Linux 'agent' node 
WARN[0000] Unable to open /etc/hosts file for srl node agent: open /var/lib/docker/containers/ead22274eddd65444221b89a43fead993d89d8733cc8f2f6a2907ce83136ed4/hosts: no such file or directory 
WARN[0000] Unable to populate hosts for node "agent": open /var/lib/docker/containers/ead22274eddd65444221b89a43fead993d89d8733cc8f2f6a2907ce83136ed4/hosts: no such file or directory 
ERRO[0003] failed to create exec in container /agent: Error response from daemon: Container ead22274eddd65444221b89a43fead993d89d8733cc8f2f6a2907ce83136ed4 is not running 
ERRO[0003] agent: failed to execute cmd: "/opt/srlinux/bin/sr_cli -d info from state system app-management application mgmt_server state | grep running" with error Error response from daemon: Container ead22274eddd65444221b89a43fead993d89d8733cc8f2f6a2907ce83136ed4 is not running 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x25f7afb]

goroutine 16 [running]:
github.com/srl-labs/containerlab/clab/exec.(*ExecResult).GetReturnCode(...)
        github.com/srl-labs/containerlab/clab/exec/exec.go:141
github.com/srl-labs/containerlab/nodes/srl.(*srl).Ready(0xc00087a0e0, {0x36d7d08?, 0xc000a07950?})
        github.com/srl-labs/containerlab/nodes/srl/srl.go:333 +0x1db
github.com/srl-labs/containerlab/nodes/srl.(*srl).PostDeploy(0xc00087a0e0, {0x36d7d08, 0xc000a07950}, 0xc000012750)
        github.com/srl-labs/containerlab/nodes/srl/srl.go:262 +0x25a
github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes.func1(0x0?, 0xc0008602a0, 0x0?)
        github.com/srl-labs/containerlab/clab/clab.go:573 +0x7c2
created by github.com/srl-labs/containerlab/clab.(*CLab).scheduleNodes
        github.com/srl-labs/containerlab/clab/clab.go:638 +0x15e ```

[steiler]

Hi @priyacshah,
thanks for reporting this issue. I have fixed the nil pointer issue in #1952. However, this will just make clab not crash.
But the underlaying problem is a different.
As stated in the first ERROR log, the container was no longer running at that point in time. You should take a look at the docker logs to see if you can find the root cause for why the container itself stopped / crashed.

ERRO[0003] failed to create exec in container /agent: Error response from daemon: Container ead22274eddd65444221b89a43fead993d89d8733cc8f2f6a2907ce83136ed4 is not running 

[priyacshah]

Thanks for the response. Attaching the snapshot of docker logs.

[hellt]

@priyacshah this is related to the way your runner is set up and it seems containerlab can't do much...

Given you have permission issues as per your screenshot I suggest you try to figure out how to run privileged containers in your runner

[priyacshah]

Made some progress but now it showing me this error

+ sudo -E clab dep -c -t ./lab
INFO[0000] Containerlab v0.53.0 started                 
INFO[0000] Parsing & checking topology file: securityagent.clab.yml 
INFO[0000] Removing /builds/a-team/securityagent/clab-securityagent directory... 
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU=1500 
WARN[0000] failed gleaning v4 and/or v6 addresses from bridge via netlink, falling back to docker network inspect data 
Error: failed to lookup link "br-f8343219f9fe": Link not found

Here is my config.toml file

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test runner"
  url = "<url>"
  id = 504759
  token = "<token>"
  token_obtained_at = 2024-03-27T18:55:57Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = ["FF_NETWORK_PER_BUILD=1"]
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "linux"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
    shm_size = 0
    network_mtu = 0

[hellt]

I don't think clab will work with a gitlab's docker runner with that config.

This is because you mount the docker socket to the runner' container, but you don't mount the network namespace, hence the container gets created outside of the runner's netns but tries to lookup the docker bridge in the runner's namespace.

What could work is using a docker image that has docker installed inside.
For example, you can build a container similar to that one https://github.com/srl-labs/clabernetes/blob/main/build/launcher.Dockerfile

[priyacshah]

Thanks. I installed the docker inside the docker image. Finally able to deploy clab in gitlab runner docker executor with the below config. There was a mounting issue for which I added volume & network_mode = "host" to resolve above issue. But I am getting this warning for which I am not sure how to resolve yet:

Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-fendbp3-project-66846-concurrent-0-b5fd77b78e08c810-docker-0 probably didn't start properly.
Health check error:
create service container: Error response from daemon: conflicting options: host type networking can't be used with links. This would result in undefined behavior (services.go:208:0s)
Service container logs:

config.toml:

concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "test docker runner"
  url = "<URL>"
  id = 504888
  token = "<TOKEN>"
  token_obtained_at = 2024-04-01T20:06:22Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  FF_NETWORK_PER_BUILD = true
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    tls_verify = false
    image = "<artifactory_image>"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache","/var/run/docker.sock:/var/run/docker.sock","/var/run/netns:/var/run/netns","/var/lib/docker/containers:/var/lib/docker/containers","/etc/hosts:/etc/hosts","<project_build_path>:<project_build_path>"]
    shm_size = 0
    network_mtu = 1500 
    network_mode = "host"