Proxmox and TrueNAS Scale issue

1

So this is giving me headaches for a few days. I have TrueNAS Scale running in a VM with a dedicated nic and drives. My Proxmox host has a separate dedicated nic and drives. I am up to date on everything. I have set up datasets and shares in TrueNAS and can access the SMB shares with out issue from my Windows clients. The NFS shares are another issue.

When in a vm on the same host or another host, I can mount the nfs shares and have RW capabilities to the share. These are tested with vanilla Debian v11.3 vm’s with nothing but the nfs client added. Simple “mount -vvv xxx.xxx.xxx.xxx:/mnt/data-01/ds-proxmox /opt/truenas” and no issues coping, editing, or deleting files/directories.

When I am on the Proxmox host or any other Proxmox host and try to the same thing, I can mount the share and list the files but cannot edit them and attempting to edit anything locks up the system where I have to reboot the host to clear it. Probably a NFS busy state and the session is dead.

I have tried many different permissions and NFS3/4 on the TrueNAS scale side but nothing works from any Proxmox host. I even have a share set with POSIX OPEN and running into the exact same issue from the Proxmox host. This prevents me from created NFS storage, shared or not, on any Proxmox host where the NFS share is running on TrueNAS scale.

At this point I have to believe there is something going on with Proxmox related to accessing TrueNAS Scale NFS shares, as it is the only common denominator that does not work properly. I posted this on the Proxmox forum as well as any help would be appreciated. This sounds like a video opportunity to me and have posted it on the Lawrence Systems forum as well to see if anyone can tackle it and figure out what the disconnect can be. If it is a Proxmox issue then I cannot and probably will not be the only one running into this.

2

Welcome to the forum!

How did you mount the NFS? Using Proxmox built-in NFS mount feature, or in the CLI in a certain path?

3

Both. I tested by creating a mount point under opt and mounted the TrueNAS share there. That is where I did all my testing. In the Proxmox GUI I can also create the shared storage but as soon as I try to upload anything to it, like an ISO, I get the upload to the tmp directory but it hangs when it tries to cp the iso to the proper directory on the TrueNAS mounted share. It just hangs there and all the storage locations eventually show “?” question marks. At that point a reboot is needed to clear everything.

It clearly is acting like a permissions issue but I have tried many options in TrueNAS with out any luck. I am not sure what Proxmox is doing under the covers as I can mount these shares inside a Debian VM and not have the same issue. Just on the Proxmox hosts.

4

It sounds like a bug to me, maybe something with Proxmox kernel, which is not the default Debian kernel. Can you try creating a proxmox VM and try to debug it inside a VM that is not part of a cluster?

5

I posted this over in the Proxmox forums as well:

So here is my setup

I have a single host v7.2-4. On it I have a single VM running TrueNAS Scale v22.02.2.

The VM has a dedicated NIC passed in via PCI Passthrough and all drives passed via iscsi plus no config on the host for the nic or ports. The host has a seperate NIC config’d with a bridge. The TrueNAS VM does not use the host NIC or drives.

TrueNAS is set up properly, pools, datasets, users, and shares all good. I have a single share to a dataset and the export file on TrueNAS shows RW and no_root_squash.

I created a vanilla Debian 11 VM and installed the NFS common package on the same host. As root, I was able to mount the share to the ds_proxmox dataset/share. In that VM running on the same host using the only bridge on the host, I have full access to the share and can cp, write, delete to my hearts content.

On the Proxmox host, from the commandline or GUI, I can mount the same ds_proxmox dataset/share from the TrueNAS VM. When using the GUI, it creates all the directories and sub-directories. At the commandline, I can create a file on the share by using touch. If I try to VI the file, it hangs up the entire mount point. In the GUI, if I try to upload an ISO, it uploads to the tmp directory and hangs when attempting to cp the file to the directory on the share. It does create a 0-byte file but that is it.

Why can I have full access to the share between VM’s on the same host as the TrueNAS VM but I cannot fully access the share from the host it self? It shouldn’t be a routing issue as the NIC in the TrueNAS VM is removed from the host OS via iommu and pci passthrough. The client VM’s on the host, except for the TrueNAS VM, share the same bridge as the host.

I get if I was all using the same bridge that NFS could get lost but not with PCI passthrough of the TrueNAS nic. So how does someone mount an NFS share inside a VM back to the proxmox host file system. I cannot mount it locally then add it to Proxmox as a Directory and I cannot mount it as a NFS share.

The only thing I know that works is if I map a host directory into an LXC container running an NFS server. Then I can mount that share from the container back to the host as shared storage. I have that working but that is no substitute for something like TrueNAS.

6

Again, the situation sounds weird and I am expecting a kernel bug in the Proxmox kernel. If you can create a VM running Proxmox inside your Proxmox host and manage to replicate the issue with the same 7.2.4, but this time as a VM, then it’s likely an issue with Proxmox itself. It shouldn’t take you more than 10 minutes to download a Proxmox iso and set a VM with it, then SSH into it and try to mount that same NFS export from TrueNAS Scale.

I’m curious why you gave your VM iscsi drives instead of passing the SATA ports or the entire SATA controller to it, but that’s besides the point. And since mounting that NFS share on other OS, like Debian, works, then TrueNAS Scale is out of question.

Other than that, try running a tail -f on /var/log/messages on Proxmox, then try mounting the path and creating stuff, then see what message gets displayed there before freezing. Also worth having a watch every second on dmesg and output it to a file when you mount the path and try writing stuff to it, maybe it would give an indication of what may be going on to make Proxmox hang.

7

Thanks for the response! I will try the Proxmox VM tomorrow. As for the iSCSI, it was a quick fix as PCI passthrough didn’t seem to be working properly but I figured our that it was just a me issue after I already had it all set up. I will go back through and rebuild it after I figure out why the NFS share is hanging.

1 Like
8

Ok, so I created a nested Proxmox VM and used host cpu’s per the wiki and had no issue just like in my test debian VM. I did get the following in journalctl:

Jul 04 12:05:35 r820-01 systemd[501499]: mnt-pve-nfs820\x2d01\x2dproxmox.mount: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support:
░░
░░ The unit UNIT has successfully entered the ‘dead’ state.
Jul 04 12:05:35 r820-01 systemd[1]: mnt-pve-nfs820\x2d01\x2dproxmox.mount: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: Debian -- User Support
░░
░░ The unit mnt-pve-nfs820\x2d01\x2dproxmox.mount has successfully entered the ‘dead’ state.
Jul 04 12:08:08 r820-01 pvedaemon[2323]: root@pam successful auth for user ‘root@pam’
Jul 04 12:09:56 r820-01 pvestatd[2303]: got timeout

and

root@r820-01:~# journalctl -xe
Jul 04 12:28:29 r820-01 kernel: ? nfs_check_verifier+0x51/0xa0 [nfs]
Jul 04 12:28:29 r820-01 kernel: ? nfs_do_lookup_revalidate+0x167/0x260 [nfs]
Jul 04 12:28:29 r820-01 kernel: filemap_write_and_wait_range+0x88/0xd0
Jul 04 12:28:29 r820-01 kernel: nfs_wb_all+0x27/0xf0 [nfs]
Jul 04 12:28:29 r820-01 kernel: nfs_setattr+0x1d7/0x1f0 [nfs]
Jul 04 12:28:29 r820-01 kernel: notify_change+0x347/0x4d0
Jul 04 12:28:29 r820-01 kernel: chmod_common+0xc4/0x180
Jul 04 12:28:29 r820-01 kernel: ? chmod_common+0xc4/0x180
Jul 04 12:28:29 r820-01 kernel: do_fchmodat+0x62/0xb0
Jul 04 12:28:29 r820-01 kernel: __x64_sys_chmod+0x1b/0x20
Jul 04 12:28:29 r820-01 kernel: do_syscall_64+0x5c/0xc0
Jul 04 12:28:29 r820-01 kernel: ? kern_select+0xf1/0x180
Jul 04 12:28:29 r820-01 kernel: ? handle_mm_fault+0xd8/0x2c0
Jul 04 12:28:29 r820-01 kernel: ? exit_to_user_mode_prepare+0x37/0x1b0
Jul 04 12:28:29 r820-01 kernel: ? syscall_exit_to_user_mode+0x27/0x50
Jul 04 12:28:29 r820-01 kernel: ? __x64_sys_select+0x25/0x30
Jul 04 12:28:29 r820-01 kernel: ? do_syscall_64+0x69/0xc0
Jul 04 12:28:29 r820-01 kernel: ? irqentry_exit+0x19/0x30
Jul 04 12:28:29 r820-01 kernel: ? exc_page_fault+0x89/0x160
Jul 04 12:28:29 r820-01 kernel: ? asm_exc_page_fault+0x8/0x30
Jul 04 12:28:29 r820-01 kernel: entry_SYSCALL_64_after_hwframe+0x44/0xae
Jul 04 12:28:29 r820-01 kernel: RIP: 0033:0x7f6aa7452a17
Jul 04 12:28:29 r820-01 kernel: RSP: 002b:00007fff68d26d68 EFLAGS: 00000202 ORIG_RAX: 000000000000005a
Jul 04 12:28:29 r820-01 kernel: RAX: ffffffffffffffda RBX: 00000000000001a4 RCX: 00007f6aa7452a17
Jul 04 12:28:29 r820-01 kernel: RDX: 000055fcbdd87e60 RSI: 00000000000001a4 RDI: 000055fcbe4ef1c0
Jul 04 12:28:29 r820-01 kernel: RBP: 0000000000000001 R08: 00007fff68d26c70 R09: 00007f6aa7522be0
Jul 04 12:28:29 r820-01 kernel: R10: 000055fcbdd87de0 R11: 0000000000000202 R12: 0000000000000001
Jul 04 12:28:29 r820-01 kernel: R13: 000055fcbe4be460 R14: 000055fcbe4c3880 R15: 000055fcbe4ef1c0
Jul 04 12:28:29 r820-01 kernel:
Jul 04 12:30:30 r820-01 kernel: INFO: task vi:505749 blocked for more than 1208 seconds.
Jul 04 12:30:30 r820-01 kernel: Tainted: P O 5.15.35-2-pve #1
Jul 04 12:30:30 r820-01 kernel: “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
Jul 04 12:30:30 r820-01 kernel: task:vi state:D stack: 0 pid:505749 ppid:504811 flags:0x00000000
Jul 04 12:30:30 r820-01 kernel: Call Trace:
Jul 04 12:30:30 r820-01 kernel:

My host system was installed using Deb 11 and updated to Proxmox per their wiki. I needed to go this route as I have a custom partition layout utilizing a clover usb boot drive with Proxmox installed on a PCI adapter with an NVMe drive. The installer doesn’t support that install but the proxmox install config is supported. I do not believe that the infrastructure is the issue at all as I can access the shares from nested VM’s just not from the host.

9

To me, it does look like a kernel issue, especially with the kernel complaining about the vi task being blocked for 20 minutes.

Try something else. Since it appears you are using vi when the issue is happening, try doing a

strace -o /tmp/trace.out vi /whatever/you/were/editing.txt

I assume you are trying to save / write something and that fails. We may be able to figure out what is happening by going towards the end of the trace.out log.

In all honesty, I don’t recommend the debian to proxmox conversion. It’s better to just install proxmox on anything, even a VM, power it off, boot into the installer again, but this time go into a console (ctrl+alt+f2), and recreate the file system however you desire.

This is how I got my OS rootfs on a NVME M.2 USB enclosure on, but booting from an SD card. I’m not using Proxmox or Debian, but the idea is the same: install the OS anywhere, create a new FS on another disk, mount all partitions or logical volumes accordingly, just rsync the data, recreate fstab and if needed lvm.conf, reinstall grub and make the config and you’re set. It is a bit involved, but nothing too difficult.

10

Solved. I posted this over on the Proxmox forums as well at the Lawrence Systems community.

Since I have a full 10G network, I took my MTU up to 9000 in order to utilize jumbo frames. This seemed to work for pretty much everything but NFS. And it only hangs up NFS when it actually is trying to affect data by editing a file with VI or copying a file across the mount point. Why. I don’t know but I suspect someone with deep networking knowledge could figure it out. For me, I guess I will live with the performance hit and throttle mtu back to 1500 in order to at least have NFS work.

I certainly would welcome an explanation as to why NFS can’t move the jumbo packets or how to figure out where in the network the chock point actually resides. I certainly would welcome a video from Tom, Jay, or both on how to diagnose and remediate this bottleneck.