NFS Secondary Storage Does not want to mount in SSVM.

[whitetiger264]

ISSUE TYPE

• Bug Report

COMPONENT NAME

NFS & SSVM

CLOUDSTACK VERSION

4.17.2

CONFIGURATION

KVM Host, using the advanced network.

OS / ENVIRONMENT

KVM Host & CS Host Both AlmaLinux 8

SUMMARY

NFS Secondary Storage Fails to mount in SSVM upon SSVM or CS restart.

STEPS TO REPRODUCE

Most like will not be able to reproduce it may only be in my environment this is caused.

I have two NFS Secondary storage, and they run on a completely external network entirely remotely from my CloudStack network. These are their public IP addresses:

1. NFS One: 102.165.XXX.YYY
2. NFS Two: 102.165.XXX.ZZZ

I have a private network for my cloud stack environment, which is 192.168.50.0/24 and both my management and storage fall within this network using the same 192.168.50.1 gateway. But keep in mind, my actual NFS storage is remote via internet public IP.

For months I have been connecting to my NFS secondary storage via public IP with zero issues until one morning it just failed. The failed mount error in the logs shows:

```
2023-02-16 10:24:22,423 ERROR [storage.resource.NfsSecondaryStorageResource] (agentRequest-Handler-2:null) GetRootDir for nfs://nfsip/data/secondary failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to create local folder for: /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 in order to mount nfs://102.165.XXX.ZZZ/data/secondary
com.cloud.utils.exception.CloudRuntimeException: Unable to create local folder for: /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 in order to mount nfs://102.165.XXX.ZZZ/data/secondary
```

Upon further investigation, I found that when SSVM or CS is rebooted, the SSVM makes the following entry in the IP Route Table:

- 102.165.XXX.ZZZ via 192.168.50.1 dev eth1

This is correct according to Cloudstack because that's the default gateway for the storage network. However, if I remove this entry from the IP route table in the SSVM, the NFS mount is successful, because now it connects via the default public route.

So here we can see why the mount fails:

```
root@s-145-VM:~# mount -t nfs 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary
root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.ZZZ:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:11 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.ZZZ,clientaddr=192.168.50.53'
mount.nfs: mount(2): Operation not permitted
mount.nfs: trying text-based options 'addr=102.165.XXX.ZZZ'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 102.165.XXX.ZZZ prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: trying 102.165.XXX.ZZZ prog 100005 vers 3 prot UDP port 892
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 102.165.XXX.ZZZ:/data/secondary
```

As you can see, it's trying to mount from a private IP address `192.168.50.53`, and because NFS is not in the same network, it will fail as it is not permitted. 

Now, here's the weird part, like I said this has worked for months. My second NFS secondary storage is also on the same remote network: 102.165.XXX.YYY when I mount to this NFS storage from SSVM it mounts perfectly fine without any issues:

```
root@s-145-VM:~# mount -t nfs -vvv 102.165.XXX.YYY:/data/secondary /mnt/SecStorage/test
mount.nfs: timeout set for Thu Feb 16 14:07:58 2023
mount.nfs: trying text-based options 'vers=4.2,addr=102.165.XXX.YYY,clientaddr=197.189.XXX.YYY'
root@s-145-VM:~#
```

The reason is that SSVM does not end up creating a route in the routing table such as 102.165.XXX.YYY via 192.168.50.1, no. It remains to use the default route (public), which is why we can see the SSVM public IP 197.189.XXX.YYY.

This now leaves me with a few questions:

1. Can I not make use of remote/external NFS storage?
2. Why does SSVM not create a routing path for the second NFS and force it to only create for the 102.165.XXX.ZZZ NFS?
3. Why has this been working for months if the answer to question 1 is no?
4. Why does SSVM mount the second remote NFS server perfectly fine and not just do the same for my one NFS, which I really need.

EXPECTED RESULTS

NFS should mount accordingly regardless of private or public NFS server network as secondary storage. 

My Goal

If using external NFS servers is not recommended, then I will happily configure new NFS servers via a private network to work accordingly. However, I am left with a loop problem. Because my NFS with my actual data cannot automatically mount upon CS or SSVM restart, the verification checks on my templates fail. They are 100% downloaded, but their ready state shows as "No". When checking the DB, the `template_view` table, their state shows as "Migrating". Now because NFS does not mount, checks cannot complete, and it no prevent me from moving my data from this existing NFS storage to a new NFS storage. 

So if we cannot fix the bug above, is there a way I can set the state of these templates that show as "Migrating" to "Ready" so that I can just move my data accordingly? I have tried to update the table, but I am returned with an SQL error that this table is not updatable. 

And if this is not possible, is there any way I can move my data from this NFS storage to a new NFS storage such as using RSYNC or so?

I am REALLY stuck with a looping problem and would appreciate all the efforts possible. 

[whitetiger264]

Hi @weizhouapache as discussed via email, here's my issue. Hopefully, someone you know can assist with this here. As you're aware, I only want to find a way to get my templates to a "Ready" state so that I can move my data to a new NFS storage, as explained at the end of the bug report.

[DaanHoogland]

@whitetiger264
As for the route problem; have you tried adding two separate physical networks for the different storage networks? I have not tried this but this seems at first glance the way to implement this.
Any storage should be reachable on the physical network of type storage network. I'm not sure if this works or if the solution should be searched in that direction at all. But if your issue needs implementation I would say it is in that direction.

And thanks for the "My Goal" chapter, we have been thinking of simplifying the issue template a bit, but your addition is very useful at times. 👍

[whitetiger264]

@DaanHoogland Do I understand correctly that you want me to add a network range for my two NFS storage servers in the existing storage type in the zone's physical interface? i.e here:

Or where exactly are you referring to I should add them? Sorry, I am fairly new to Cloudstack.

As for the "My Goal" part, I am just looking for a method where I can do the template verification check without needing to restart the CS or SSVM for this specific case. Is this possible?

If not, is there a way I can edit the template_view table without getting the "unupdatable" error? PS I actually need a solution, because this is my production environment. This is not testing purposes.

[DaanHoogland]

@whitetiger264 , I didn't but that is a viable option as well, I was thinking you would add two physical networks with traffic type storage. I think we should explore your option first.

If the ssvm has the storage mounted it will verify the templates if thay are not already verified. (if all works as expected) So that goal should be implicitely solved once the mounting part is fixed.

I missed the "unupdateable" error, did you describe that in the mail thread?

[weizhouapache]

see my answers inline

1. Can I not make use of remote/external NFS storage?

yes, if NFS storage is reachable via gateway of storage network

2. Why does SSVM not create a routing path for the second NFS and force it to only create for the 102.165.XXX.ZZZ NFS?

When secondary storage VM starts, it should create routes for all NFS secondary storages via gateway of storage network.

3. Why has this been working for months if the answer to question 1 is no?

As you told me yesterday, this issue began when your router is rebooted. Maybe some rules/routes are not persistent on your router, which disappeared afer reboot.

4. Why does SSVM mount the second remote NFS server perfectly fine and not just do the same for my one NFS, which I really need.

see 2. The SSVM should have a route to secondary remote NFS server as well. But it is true, your secondary NFS server might be unreachable too.

@whitetiger264 can you please destroy or restart SSVM, and then

• upload the management-server.log ,
• cloud.log in SSVM
• results of "ip route", "ip a" in SSVM

[whitetiger264]

@weizhouapache

The SSVM should have a route to the secondary remote NFS server as well. But it is true, your secondary NFS server might be unreachable too.

Both NFS servers are on the same network space 102.165.XXX.ZZZ & 102.165.XXX.YYY. Routing works for one NFS but not the other. I have checked the firewall and it's FULLY open to redirect all traffic from LAN to WAN, there are no specific port settings or host settings in place, it's fully open. And as tests have shown, I can telnet port 2049 from my local gateway 192.168.50.1 to the NFS server that's failing to mount. This is why this problem is so weird to debug.

In any case, I have destroyed the SSVM once more, please find attached logs, as requested.

• cloud.log
• management.log

And here is the ip route and ip a commands. Take note, in IP route it does not add the second NFS server, which is successful as asked in question 2 above:

root@s-146-VM:~# ip route
default via 197.189.xxx.xxx dev eth2 
1.1.1.1 via 192.168.50.1 dev eth1 
8.8.8.8 via 192.168.50.1 dev eth1 
10.0.0.0/8 via 192.168.50.1 dev eth1 
102.165.xxx.yyy via 192.168.50.1 dev eth1 
156.38.xxx.xxx/29 via 192.168.50.1 dev eth1 -> CS Host Subnet
156.38.xxx.xxx via 192.168.50.1 dev eth1 -> CS Host Public IP
169.254.0.0/16 dev eth0 proto kernel scope link src 169.254.110.50 
172.16.0.0/12 via 192.168.50.1 dev eth1 
192.168.0.0/16 via 192.168.50.1 dev eth1 
192.168.50.0/24 dev eth1 proto kernel scope link src 192.168.50.74 
192.168.50.0/24 dev eth3 proto kernel scope link src 192.168.50.106 
197.189.xxx.xxx/29 dev eth2 proto kernel scope link src 197.189.xxx.xxx

root@s-146-VM:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 0e:00:a9:fe:6e:32 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    altname ens3
    inet 169.254.110.50/16 brd 169.254.255.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:94:00:00:19 brd ff:ff:ff:ff:ff:ff
    altname enp0s4
    altname ens4
    inet 192.168.50.74/24 brd 192.168.50.255 scope global eth1
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:06:00:00:1f brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    altname ens5
    inet 197.189.xxx.xxx/29 brd 197.189.211.7 scope global eth2
       valid_lft forever preferred_lft forever
5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 1e:00:fc:00:00:3d brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 192.168.50.106/24 brd 192.168.50.255 scope global eth3
       valid_lft forever preferred_lft forever
root@s-146-VM:~# 

[whitetiger264]

@DaanHoogland

I think we should explore your option first.

I will try this a bit later or as a last resort. Let's see if the logs attached show any errors we can explore.

I missed the "unupdateable" error, did you describe that in the mail thread?

Yes, I have explained this in the email thread as well, here's one I mentioned it again today and I have discussed it with Wei too:

The setting you're referring to is secstorage.nfs.version I have configured it to "4" and when manually mounting I also specified version 4. Still refuses to connect. At this point I do not care about fixing this specific issue because its clear that Cloudstack does not seems to like external/remote public IP NFS secondary storage.

For now, is there ANY way I can trigger the template verification process without needing to restart CS or SSVM? If I restart any of them, then SSVM cannot automatically mount, therefore, failing the template verificaiton process and therefore my templates shows as "No" or "Migrating" state in the DB.

Or is there anyway I can just edit the table from "Migration" to "Ready" I just want to move my data away from this problematic NFS.

[weizhouapache]

@whitetiger264
I noticed the following logs.
Is 192.168.50.2 the IP of secondary NFS server ?
This might explain why there is no route to 102.165.XXX.ZZZ in SSVM.

2023-02-17 16:38:21,974 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-5:ctx-efa0fdcb) (logid:37ea06d6) Seq 51-7684829814154330113: Sending  { Cmd , MgmtId: 66988330791812, via: 51(s-146-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.50.2/mnt/datasecond","_role":"Image","nfsVersion":"4"}},"secUrl":"nfs://192.168.50.2/mnt/datasecond","certs":{},"postUploadKey":"fneB0iq9ETvOUHKxqHQBai1GeeKaBjvUxSQXHuZlv2mTW13wO2OOWH0OgrAIRH2eMMvkFORM9jwez1LPGnhXAg","nfsVersion":"4","wait":"0","bypassHostMaintenance":"false"}}] }
2023-02-17 16:38:25,266 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5:ctx-efa0fdcb) (logid:37ea06d6) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: success
2023-02-17 16:38:25,272 DEBUG [c.c.a.t.Request] (AgentConnectTaskPool-5:ctx-efa0fdcb) (logid:37ea06d6) Seq 51-7684829814154330117: Sending  { Cmd , MgmtId: 66988330791812, via: 51(s-146-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.SecStorageSetupCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://NFSServer1IP/data/secondary","_role":"Image","nfsVersion":"4"}},"secUrl":"nfs://NFSServer1IP/data/secondary","certs":{},"postUploadKey":"fneB0iq9ETvOUHKxqHQBai1GeeKaBjvUxSQXHuZlv2mTW13wO2OOWH0OgrAIRH2eMMvkFORM9jwez1LPGnhXAg","nfsVersion":"4","wait":"0","bypassHostMaintenance":"false"}}] }
2023-02-17 16:39:56,723 DEBUG [c.c.a.m.AgentManagerImpl] (AgentConnectTaskPool-5:ctx-efa0fdcb) (logid:37ea06d6) Details from executing class com.cloud.agent.api.SecStorageSetupCommand: GetRootDir for nfs://NFSServer1IP/data/secondary failed due to com.cloud.utils.exception.CloudRuntimeException: Unable to create local folder for: /mnt/SecStorage/91de6d1c-4c04-359c-82b5-fdcfe4a83da7 in order to mount nfs://NFSServer1IP/data/secondary

[whitetiger264]

@weizhouapache

Yes, 192.168.50.2 is my management server, which I use as my primary storage (currently) and used to use as my secondary storage but I have since done a "complete" data migration from this secondary NFS to now the NFS that fails to mount so it's currently in "ReadOnly" state. See here:

Can I proceed to remove it? Since you mentioned it affects the routing?

[weizhouapache]

ok @whitetiger264
I think there should be 3 com.cloud.agent.api.SecStorageSetupCommand in log, but there are only 2. Looks something wrong

[whitetiger264]

Is there really no other way to trigger the template verification checks without rebooting CS or SSVM? Or is there really no way for me to update the table template_view in the DB?

[weizhouapache]

Is there really no other way to trigger the template verification checks without rebooting CS or SSVM? Or is there really no way for me to update the table template_view in the DB?

template_view is a view which you cannot update.
You can update template_store_ref table

[whitetiger264]

@weizhouapache Thank you, updating the state to Ready for my two "Migrating" templates allows me to move my data from the "broken" NFS to the working one. So this should allow me to set up a new NFS server, one fully reachable by the local network.

In the meantime, should this remain open for the actual bug that's happening?

[weizhouapache]

@whitetiger264
from my point of view, there is a bug that the SSVM should have routes to all secondary storages, even if one of them is unreachable.