Issues with FreeNAS NFS Permissions using Docker

[charles]

I’ve been having permissions issues on my Ubuntu server when using Docker.

Storage configuration

I use an Ubuntu Server to run Docker containers. This Ubuntu server is only used to run Docker containers.

I’ve created some datasets on my TrueNAS server to serve as storage for the Docker containers.

On the Ubuntu server nfs/docker/ is a local directory where all the NFS shares are mounted (via /etc/fstab)

When I run df -h I can confirm that the NFS shares are mounting as the free space showing on them far exceeds the storage on my Ubuntu server.

The Ubuntu server user account is simply charles which has a UID of 1000 and a GID of 1000. The datasets on TrueNAS are set to owner and group matching the UID and GID of 1000. (In addition to this they are set such that @everyone has full control in the ACL manager.

Additionally in TrueNAS, the NFS sharing options are set such that the mapall user is the user account on TrueNAS with UID of 1000 and mapall group is the group on TrueNAS with a GID of 1000.

Docker issues

I am trying to setup a Nginx Proxy Manager and am getting erorrs that I believe are permissions errors, but not necessarily sure of that. I think my issue is actually not specific to setting up this specific container- rather with the storage configuration I’m using to run my containers.

Regardless- here is the output that is leading me to believe the problems are permissions / storage related.

charles@VMdocker:/nfs/docker/nginxreverseproxy$ docker-compose up -d
Creating nginxreverseproxy_db_1 ... error

ERROR: for nginxreverseproxy_db_1  Cannot start service db: error while creating mount source path '/nfs/docker/nginxreverseproxy/data/mysql': chown /nfs/docker/nginxreverseproxy/data/mysql: operation not permitted

ERROR: for db  Cannot start service db: error while creating mount source path '/nfs/docker/nginxreverseproxy/data/mysql': chown /nfs/docker/nginxreverseproxy/data/mysql: operation not permitted
ERROR: Encountered errors while bringing up the project.

I have tried to create directories and have no issues doing so.

charles@VMdocker:/nfs/docker/nginxreverseproxy$ mkdir test1
charles@VMdocker:/nfs/docker/nginxreverseproxy$ ls -l
total 6
drwxrwxrwx 3 charles charles    3 Mar 10 20:28 data
-rwxrwxrwx 1 charles charles 1066 Mar 10 20:28 docker-compose.yml
drwxrwxrwx 2 charles charles    2 Mar 10 20:40 test1
charles@VMdocker:/nfs/docker/nginxreverseproxy$ rmdir test1
charles@VMdocker:/nfs/docker/nginxreverseproxy$ ls -l
total 5
drwxrwxrwx 3 charles charles    3 Mar 10 20:28 data
-rwxrwxrwx 1 charles charles 1066 Mar 10 20:28 docker-compose.yml

Here is the docker-compose file I am using for the Nginx Proxy Manager which is taken directly from the Nginx Proxy Manager setup instructions.

version: "3"
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: always
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
    environment:
      # These are the settings to access your db
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "*********"
      DB_MYSQL_PASSWORD: "*********"
      DB_MYSQL_NAME: "npm"
      # If you would rather use Sqlite uncomment this
      # and remove all DB_MYSQL_* lines above
      # DB_SQLITE_FILE: "/data/database.sqlite"
      # Uncomment this if IPv6 is not enabled on your host
      DISABLE_IPV6: 'true'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: always
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: '*********'
      MYSQL_PASSWORD: '********'
    volumes:
      - ./data/mysql:/var/lib/mysql

Conclusion / TLDR

If my issues are with how I’m setting up the Nginx Proxy manager then that is fine - I will create a thread in their Github / fourm but I’m suspicious that my issues are actually due to using Docker on an Ubuntu server without using any local storage. That is I’m trying to use the TrueNAS server for all storage aspects of my docker containers.

Any feedback on the understanding of NFS permissions I’ve laid out in the earlier part of my post? I’m definitley new to NFS but I read through the TrueNAS documentation relating to NFS and thought I understood it clearly enough. My understanding is that NFS does not care about the names of any users or groups, rather NFS only considers the UID and GID, so as long as your client UID/GID matches the dataset permissions as well as the NFS mapall user/ mapall group than I should be good to go.

Any help is much appreciated!

[jay]

I’m not sure I completely understand why it’s failing. I also use TrueNAS on my end for container storage, and it seems to work fine. However, I’m using Kubernetes for mine and you’re using Docker Compose for yours, and I’ve not tried Docker Compose with TrueNAS-backed storage yet.

Usually, the problem comes down to username. I had to use the “mapall” option in the share, and map it to the username I set aside for it. Then, in the data store, I set the permissions of it to be owned by that user. But if I’m understanding your post correctly, it sounds like you’ve already done that.

To best understand the issue, try to manually “walk through” the directories and create test files. As a user on the Ubuntu system, mount the NFS share, and via the commandline just try to do something like “touch testfile.txt” to see if you can write, and traverse the directories down to where the container should be writing.

I know that isn’t much, but I’d probably be more helpful if I was doing it the same way as you. Since I have my containers set up in kubernetes, I’m creating a physical volume call, and a physical volume claim call, and Kubernetes is managing that. But at the same time, if TrueNAS has the datastore set to the proper user ID for the owner, and you recursively set that, and you set the mapall option to that same user, there really isn’t anything else I can think of that would be required.

Also, ensure that the NFS share in TrueNAS has the IP address of your server whitelisted.

[charles]

Thanks Jay.

I had to use the “mapall” option in the share, and map it to the username I set aside for it. Then, in the data store, I set the permissions of it to be owned by that user. But if I’m understanding your post correctly, it sounds like you’ve already done that.

Correct - i’ve already set the mapall option in TrueNAS to be the user that owns the dataset.

just try to do something like “touch testfile.txt” to see if you can write, and traverse the directories down to where the container should be writing.

I have the shares mounted to the ubuntu system automatically by way of the /etc/fstab method of mounting on boot. I had been doing this by doing mkdir test and then ls -l to view it. I had no issues creating a directory. Just tried the touch testfile.txt as you suggested and same result - no issues creating the file, no issues opening it with nano on the cli. No issues then deleting it.

But at the same time, if TrueNAS has the datastore set to the proper user ID for the owner, and you recursively set that, and you set the mapall option to that same user, there really isn’t anything else I can think of that would be required.

Well at least we are on the same page! I thought I might be missing something obvious! I will post something in the IX Systems community to see if anyone there maybe knows what’s going on.

[charles]

I’ve made a bit of progress.

If anyone finds themselves in a similar situation, look into the NFS “feature” (in this case, a problem) called root-squash.

The idea is that the TrueNAS server prevents the NFS client from performing root operations.

I think disabling “root squash” on this particular TrueNAS dataset will hopefully fix my issues. That being said, I’m not sure I feel comfortable giving the Ubuntu server root access to my TrueNAS server so I’m going to investigate this a little further before I do so.

Hopefully this is helpful to someone else in the future if they find themselves troubleshooting the same problem as I am.

Edit:
Related links:

NFS share

Edit2:

SOLVED!

The NFS setting maproot user and maproot group is the key here. Here is a screenshot of how they should be set. This is essentially how someone disables root squash via TrueNAS webUI. Configuring it in this manner allows the linux client acting as root to act as root on the TrueNAS server and thus permits the docker container to perform chown operations as needed. Fantastic!