Mounting fargate volume using EFS doesn't create sub directories #79

Kidist-Abraham · 2020-09-23T18:25:51Z

We are planing to use AWS EFS as persistent storage for our container on fargate service. The problem we are encountering is, all the directories the container uses needs to be ready prior to running the task. Or else, there will be a file not found error while the task runs. There was no sunch problem when we run the task using the non-persistent storage. Is there any work around for this?

The text was updated successfully, but these errors were encountered:

Cappuccinuo · 2020-09-24T12:48:48Z

Hey @Kidist-Abraham ,

Thanks for reaching out to us. I will assume you are using ECS Fargate since you are talking about task.

Can you share the general idea of you task definition? Is the root cause of the issue due to the PV is not fully mounted before some directories operation?

Thanks.

Kidist-Abraham · 2020-09-28T16:31:25Z

Hey @Cappuccinuo, thanks for the response.
Yes I am running fargate version 1.4.0 with AWS EFS for mounting the volume.
The task Is running the image inputoutput/cardano-node:1.18.0
This image has its entry point executable in /nix/store/..../bin/entry-point. The error I am getting is (it is built with nix)

18:22:02container_linux.go:349: starting container process caused "exec: \"/nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point\": stat /nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point: no such file or directory"
container_linux.go:349: starting container process caused "exec: \"/nix/store/w8cwn94bw2x86rzk0qralfdl6rsnwjdw-entry-point/bin/entry-point\": no such file or directory

This error comes after the container is already started. This is the first and only log
The weird thing about this is it works when I am not using AWS EFS and mount the volumes to non-persistence storage ECS provides. So I thought it has something to do with EFS

Cappuccinuo · 2020-09-29T03:29:48Z

Hey @Kidist-Abraham ,

Thanks for the explanation.

Can you provide the general steps/workflows for us to reproduce this issue? It will be great if you can share the sample yaml file.
Is the cardano-node this tool on github?

Thanks.

Kidist-Abraham · 2020-09-29T08:48:17Z

Yes, that is the repo.

Here is the most general step (Please let me know if something is not clear and I will provide the detail). I created the services with GUI.

Create a vpc, attach subnets to it, attach network ACL to the subnets with allowing everything from everything.
Create a security group ,in my case it is called cardano-wallet with the following rules on the inbound rules

and allowing all traffic to everwhere on the outbound
3. Create another security group to be used by EFS, in my case EFS-cardano-wallet with the following rules on the in bound rules

and allowing all traffic to everwhere on the outbound

Create an EFS files system with
Performance mode - General Purpose
Throughput mode - Bursting
Lifecycle policy - None
Automatic backups - Disabled
Encrypted - No
File system policy -

{
    "Version": "2012-10-17",
    "Id": "ExamplePolicy01",
    "Statement": [
        {
            "Sid": "ExampleSatement01",
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "elasticfilesystem:ClientRootAccess",
                "elasticfilesystem:ClientMount",
                "elasticfilesystem:ClientWrite"
            ],
            "Resource": "<FS ARN>"
        }
    ]
}

Add mount target - (IP address created automatically )

Create a task definition using the following JSON file (replace with your fs id and with your AWS account ID). I used the default ecsTaskExecutionRole and ecsTaskRole provided by AWS ECS.

{
  "ipcMode": null,
  "executionRoleArn": "arn:aws:iam::<AWS ID>:role/ecsTaskExecutionRole",
  "containerDefinitions": [
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/wallet-node",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [],
      "command": [
        "mkdir -p /nix/store"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "NETWORK",
          "value": "testnet"
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/ipc",
          "sourceVolume": "node-ipc"
        },
        {
          "readOnly": null,
          "containerPath": "/nix/store",
          "sourceVolume": "node-config"
        },
        {
          "readOnly": null,
          "containerPath": "/data",
          "sourceVolume": "node-testnet-db"
        }
      ],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "inputoutput/cardano-node:1.18.0",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": null,
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": false,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "node"
    },
    {
      "dnsSearchDomains": null,
      "environmentFiles": null,
      "logConfiguration": {
        "logDriver": "awslogs",
        "secretOptions": null,
        "options": {
          "awslogs-group": "/ecs/wallet-node",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      },
      "entryPoint": null,
      "portMappings": [
        {
          "hostPort": 8090,
          "protocol": "tcp",
          "containerPort": 8090
        }
      ],
      "command": [
        "serve",
        "--node-socket",
        "/ipc/node.socket",
        "--database",
        "/wallet-db",
        "--listen-address",
        "0.0.0.0",
        "--testnet",
        "/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json"
      ],
      "linuxParameters": null,
      "cpu": 0,
      "environment": [
        {
          "name": "NETWORK",
          "value": "testnet"
        }
      ],
      "resourceRequirements": null,
      "ulimits": null,
      "dnsServers": null,
      "mountPoints": [
        {
          "readOnly": null,
          "containerPath": "/ipc",
          "sourceVolume": "node-ipc"
        },
        {
          "readOnly": null,
          "containerPath": "/config",
          "sourceVolume": "node-config"
        },
        {
          "readOnly": null,
          "containerPath": "/wallet-db",
          "sourceVolume": "wallet-testnet-db"
        }
      ],
      "workingDirectory": null,
      "secrets": null,
      "dockerSecurityOptions": null,
      "memory": null,
      "memoryReservation": null,
      "volumesFrom": [],
      "stopTimeout": null,
      "image": "inputoutput/cardano-wallet:latest",
      "startTimeout": null,
      "firelensConfiguration": null,
      "dependsOn": [
        {
          "containerName": "node",
          "condition": "START"
        }
      ],
      "disableNetworking": null,
      "interactive": null,
      "healthCheck": null,
      "essential": true,
      "links": null,
      "hostname": null,
      "extraHosts": null,
      "pseudoTerminal": null,
      "user": null,
      "readonlyRootFilesystem": null,
      "dockerLabels": null,
      "systemControls": null,
      "privileged": null,
      "name": "wallet"
    }
  ],
  "placementConstraints": [],
  "memory": "4096",
  "taskRoleArn": "arn:aws:I am::<AWS ID>:role/ecsTaskRole",
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:<AWS ID>:task-definition/wallet-node-testnet-with-efs",
  "family": "wallet-node-testnet-with-efs",
  "requiresAttributes": [
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.efsAuth"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.efs"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.task-iam-role"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.container-ordering"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.25"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "targetId": null,
      "targetType": null,
      "value": null,
      "name": "ecs.capability.task-eni"
    }
  ],
  "pidMode": null,
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "networkMode": "awsvpc",
  "cpu": "2048",
  "revision": 12,
  "status": "ACTIVE",
  "inferenceAccelerators": null,
  "proxyConfiguration": null,
  "volumes": [
    {
      "efsVolumeConfiguration": {
        "transitEncryptionPort": null,
        "fileSystemId": "<FS ID>",
        "authorizationConfig": {
          "iam": "DISABLED",
          "accessPointId": null
        },
        "transitEncryption": "DISABLED",
        "rootDirectory": "/"
      },
      "name": "node-ipc",
      "host": null,
      "dockerVolumeConfiguration": null
    },
    {
      "efsVolumeConfiguration": {
        "transitEncryptionPort": null,
        "fileSystemId": "<FS ID>",
        "authorizationConfig": {
          "iam": "DISABLED",
          "accessPointId": null
        },
        "transitEncryption": "DISABLED",
        "rootDirectory": "/"
      },
      "name": "node-config",
      "host": null,
      "dockerVolumeConfiguration": null
    },
    {
      "efsVolumeConfiguration": {
        "transitEncryptionPort": null,
        "fileSystemId": "<FS ID>",
        "authorizationConfig": {
          "iam": "DISABLED",
          "accessPointId": null
        },
        "transitEncryption": "DISABLED",
        "rootDirectory": "/"
      },
      "name": "node-testnet-db",
      "host": null,
      "dockerVolumeConfiguration": null
    },
    {
      "efsVolumeConfiguration": {
        "transitEncryptionPort": null,
        "fileSystemId": "<FS ID>",
        "authorizationConfig": {
          "iam": "DISABLED",
          "accessPointId": null
        },
        "transitEncryption": "DISABLED",
        "rootDirectory": "/"
      },
      "name": "wallet-testnet-db",
      "host": null,
      "dockerVolumeConfiguration": null
    }
  ]
}

Create the AWS ECS service with the VPC created, cardano-wallet security group , the tax definition above. Fargate 1.4.0 version.

Cappuccinuo · 2020-10-09T07:57:54Z

Hey @Kidist-Abraham ,

I can reproduce the issue. Do you have specific reason to mount fs to /nix/store?

{
      "readOnly": null,
      "containerPath": "/nix/store",
      "sourceVolume": "node-config"
}

Since when I test the cardano-node docker, I found that the config file is under that path, if you mount fs to that path, the docker cannot find the config file.

[ec2-user@ip-172-31-43-114 ~]$ docker run -v /data -e NETWORK=testnet inputoutput/cardano-node:1.18.0
Starting cardano-node run: /nix/store/442brx5kpqdx9z9ccdd5z3zi5km56i05-cardano-node-exe-cardano-node-1.18.0/bin/cardano-node run
--config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json
--database-path /data/db
--topology /nix/store/dpajyi2vaychwps1x7d20c2ddls4kf62-topology.yaml
--host-addr 127.0.0.1
--port 3001
--socket-path /ipc/node.socket





..or, once again, in a single line:
/nix/store/442brx5kpqdx9z9ccdd5z3zi5km56i05-cardano-node-exe-cardano-node-1.18.0/bin/cardano-node run --config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json --database-path /data/db --topology /nix/store/dpajyi2vaychwps1x7d20c2ddls4kf62-topology.yaml --host-addr 127.0.0.1 --port 3001 --socket-path /ipc/node.socket
Listening on http:https://127.0.0.1:12798
[a0356538:cardano.node.release:Notice:5] [2020-10-09 07:35:00.37 UTC] CardanoProtocol
[a0356538:cardano.node.networkMagic:Notice:5] [2020-10-09 07:35:00.37 UTC] NetworkMagic 1097911063
[a0356538:cardano.node.version:Notice:5] [2020-10-09 07:35:00.37 UTC] 1.18.0
[a0356538:cardano.node.commit:Notice:5] [2020-10-09 07:35:00.37 UTC] e8fb486e44aceb4b9962d7b3b76849becd510820
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened imm db with immutable tip at Origin and epoch 0
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened vol db
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Replaying ledger from genesis
[a0356538:cardano.node.ChainDB:Info:5] [2020-10-09 07:35:00.38 UTC] Opened lgr db

We can see that the docker default config file is --config /nix/store/dfidzx8ggnj6sgzkz8ay7n225sy9nin3-config-0.json.

I changed the task definition part to

"containerDefinitions": [ 
    {
      ...
     // Remove the Command to create the /nix/store
      "mountPoints": [
        ...
        {
          "readOnly": null,
          "containerPath": "/test/store",
          "sourceVolume": "node-config"
        },
        ...
      ],
      ...
      "name": "node"
    },

And now the node can be launched, while the wallet cannot because

Failed to parse genesis configuration. You may want to check the filepath given via --genesis and make sure it points to a valid JSON genesis file. Here's (perhaps) some helpful hint: GenesisDataIOError /config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json: openBinaryFile: does not exist (No such file or directory)

I think this is due to my fs does not have "/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json".

Kidist-Abraham · 2020-10-12T11:36:23Z

@Cappuccinuo
I just tried this and the node is running 👍
But I was mounting to /nix/store so that the wallet can find the path *testnet-byron-genesis.json in /nix/store/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json.
You are getting this error because /config in the wallet should have been pointing to /nix/store in the node.
I just tried to mount node-config to / and change the command in wallet from/config/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json to /config/nix/store/0gdqdymj06c31f35xshw1g85bgrbh3wl-testnet-byron-genesis.json but I am having an error container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:70: creating device nodes caused \\\"errno 524\\\"\""

Kidist-Abraham · 2020-10-13T21:15:00Z

if you mount fs to that path, the docker cannot find the config file.

Why? The container is mounting /nix/store to node-config, but it should still be able to access /nix/store

Cappuccinuo · 2020-10-14T01:46:45Z

In the definition,

if you mount fs to that path, the docker cannot find the config file.

Why? The container is mounting /nix/store to node-config, but it should still be able to access /nix/store

In my understanding for the container definition,

sourceVolume (node-config) is the name of the volume to mount, in your case, it is the efsVolume.

containerPath is the path on the container to mount the volume at, in your case, it is the /nix/store. If you mount the fs to the path /nix/store, the folder content will be override by your fs content, that is why I say the docker cannot find the config file.

Kidist-Abraham · 2020-10-14T11:16:08Z

Shouldn't the fs content be overridden by the content in the container instead?
Curios on why this docker-compose file runs fine https://github.com/input-output-hk/cardano-wallet/blob/master/docker-compose.yml
Or why this works docker run -v node-config:/nix/store -e NETWORK=testnet inputoutput/cardano-node:1.18.0

anshul0915zinnia · 2021-09-24T10:06:50Z

we are also facing same issue we are mounting efs to all our task def and use efs as central logging directory so we want every application log there logs in separate application . like we are trying to mount like this

container path	efs path
/var/log/abc-app/	/application-logs/abc-app/logs/
/var/log/xyz-app/	/application-logs/xyz-app/logs/

problem is here that we need to create the subdirectories in efs manually

iekdosha · 2024-01-07T11:31:18Z

we are also facing same issue we are mounting efs to all our task def and use efs as central logging directory so we want every application log there logs in separate application . like we are trying to mount like this

container path efs path
/var/log/abc-app/ /application-logs/abc-app/logs/
/var/log/xyz-app/ /application-logs/xyz-app/logs/
problem is here that we need to create the subdirectories in efs manually

I have the same issue as you do

RyanStan added the enhancement label May 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mounting fargate volume using EFS doesn't create sub directories #79

Mounting fargate volume using EFS doesn't create sub directories #79

Kidist-Abraham commented Sep 23, 2020

Cappuccinuo commented Sep 24, 2020

Kidist-Abraham commented Sep 28, 2020 •

edited

Loading

Cappuccinuo commented Sep 29, 2020

Kidist-Abraham commented Sep 29, 2020 •

edited

Loading

Cappuccinuo commented Oct 9, 2020 •

edited

Loading

Kidist-Abraham commented Oct 12, 2020

Kidist-Abraham commented Oct 13, 2020 •

edited

Loading

Cappuccinuo commented Oct 14, 2020

Kidist-Abraham commented Oct 14, 2020 •

edited

Loading

anshul0915zinnia commented Sep 24, 2021 •

edited

Loading

iekdosha commented Jan 7, 2024

Mounting fargate volume using EFS doesn't create sub directories #79

Mounting fargate volume using EFS doesn't create sub directories #79

Comments

Kidist-Abraham commented Sep 23, 2020

Cappuccinuo commented Sep 24, 2020

Kidist-Abraham commented Sep 28, 2020 • edited Loading

Cappuccinuo commented Sep 29, 2020

Kidist-Abraham commented Sep 29, 2020 • edited Loading

Cappuccinuo commented Oct 9, 2020 • edited Loading

Kidist-Abraham commented Oct 12, 2020

Kidist-Abraham commented Oct 13, 2020 • edited Loading

Cappuccinuo commented Oct 14, 2020

Kidist-Abraham commented Oct 14, 2020 • edited Loading

anshul0915zinnia commented Sep 24, 2021 • edited Loading

iekdosha commented Jan 7, 2024

Kidist-Abraham commented Sep 28, 2020 •

edited

Loading

Kidist-Abraham commented Sep 29, 2020 •

edited

Loading

Cappuccinuo commented Oct 9, 2020 •

edited

Loading

Kidist-Abraham commented Oct 13, 2020 •

edited

Loading

Kidist-Abraham commented Oct 14, 2020 •

edited

Loading

anshul0915zinnia commented Sep 24, 2021 •

edited

Loading