SR_BACKEND_FAILURE_453(, tapdisk experienced an error [opterr=Read-only file system], ) after forced TureNAS Core 12-U1 VM termination

Hi XCP-ng experts! (:

After a forceful VM termination via Xen Orchestra of a VM running TrueNAS Core 12-U1 (1 VirtualDisk & 22 HDDs passed-through)… the VM does not want to restart anymore.

The meat of the problem might be in the SMlog produced trying to start the VM (what I believe is the interesting part…follows):

Jan 10 16:42:49 flash8 SM: [4171] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:49 flash8 SM: [4171] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|c7c527c1-c20d-4878-84fc-2fabf7681d2f|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:c97150b1-65bc-40b2-b9e1-a00cdba2d98a', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdr', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:6c30742d-8acc-4679-8d13-98cea57b2ba6', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '6bfba503-ac79-42b1-911f-748d7157a5dd'}
[...]
Jan 10 16:42:49 flash8 SM: [4190] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:49 flash8 SM: [4190] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|108920c5-fcbc-45cc-ba5d-8f7c14050009|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:d38c7795-edd0-48e9-b22a-d11263f9ad0a', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdd', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:ba3d4b5e-05c9-4638-b3cf-6ce141dc5bf9', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '89e734bb-497e-418e-9a43-4abc72415e49'}
Jan 10 16:42:49 flash8 SM: [4193] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:49 flash8 SM: [4193] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|4efaac24-4ae4-4a01-93c4-35417662b1bd|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:77d74849-c689-44c6-a97e-8ef19cf7a724', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdh', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:d33ac24e-d253-4364-9a83-c891f95cc3d9', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '25d7240f-acf8-40e7-b5eb-7a591d52f679'}
Jan 10 16:42:49 flash8 SM: [4196] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:49 flash8 SM: [4196] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|4b2a3702-beb9-4865-8b18-9cde488e274d|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:7141e62c-3734-4e21-8ea6-78d79ca2be88', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdk', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:ad71be37-8666-4a2b-89a6-eed19c7657ef', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': 'eaf6de52-dad9-473d-9d9a-1428f495932a'}
Jan 10 16:42:49 flash8 SM: [4198] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:49 flash8 SM: [4198] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|33b712c4-f4a0-46ad-a4ae-6caf55c089a9|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:1553a4fe-1523-4e61-8bd8-98d47ed7a1a3', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdi', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:5c361417-1829-4a4c-95fd-825f09c981e9', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': 'f42f9a79-bb14-4c6d-846e-a693946f4d20'}
Jan 10 16:42:50 flash8 SM: [4314] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:50 flash8 SM: [4314] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|9c2915cb-73dd-4fe5-b125-7b39281f3a34|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:ae0fdfa2-b501-496a-bd9b-d4b99c3eb6a2', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdj', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:d269debe-88d5-4a8b-84e7-8ae06e5d7194', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': 'cbdf4132-cc83-4a7c-bc40-57d7ff3e04a5'}
Jan 10 16:42:50 flash8 SM: [4319] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
[...]
Jan 10 16:42:50 flash8 SM: [4343] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|a60a84d8-3045-43b2-af9b-961f5a0db9ab|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:d0b8e25a-b930-4797-aa6f-04586f826e41', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sda', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:91ae400b-4abc-4881-b164-5c82a42eb0f1', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '977116ae-8942-4b0a-b91a-78030c9ba6fb'}
Jan 10 16:42:50 flash8 SM: [4443] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:50 flash8 SM: [4443] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|11a77d2c-1330-4776-9eec-d4b4e7fa530f|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:5644bd27-bdaf-4453-84f9-2ff505a676fd', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdv', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:ebc6eb23-d5f7-404b-8c4b-a466748b7105', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '232c77c1-5b1d-4917-b78d-118ff4e05404'}
Jan 10 16:42:50 flash8 SM: [4448] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:50 flash8 SM: [4448] vdi_epoch_begin {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|58e3d701-b9e9-4abf-a142-37ebfe1c86c2|VDI.epoch_begin', 'vdi_ref': 'OpaqueRef:e61f0bc3-37d0-4ca2-b50c-feb0750a9dac', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '/srv/hddpassthrough/sdf', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:337fe384-6bbd-4b17-bdd1-f067d4bca9a6', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_epoch_begin', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': 'ef1c268c-7634-4e22-b3f7-25fc0bdf54bb'}
Jan 10 16:42:50 flash8 SM: [4446] lock: opening lock file /var/lock/sm/f7485280-25a3-fb1d-0468-18cb3edb5d43/sr
Jan 10 16:42:50 flash8 SM: [4446] LVMCache created for VG_XenStorage-f7485280-25a3-fb1d-0468-18cb3edb5d43
Jan 10 16:42:50 flash8 SM: [4446] lock: opening lock file /var/lock/sm/.nil/lvm
Jan 10 16:42:50 flash8 SM: [4446] lock: acquired /var/lock/sm/.nil/lvm
Jan 10 16:42:50 flash8 SM: [4446] ['/sbin/vgs', '--readonly', 'VG_XenStorage-f7485280-25a3-fb1d-0468-18cb3edb5d43']
Jan 10 16:42:50 flash8 SM: [4446]   pread SUCCESS
[...]
Jan 10 16:42:51 flash8 SM: [4545] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/89e734bb-497e-418e-9a43-4abc72415e49', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/89e734bb-497e-418e-9a43-4abc72415e49', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEjg5ZTczNGJiLTQ5N2UtNDEgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIDg5ZTczNGJiLTQ5N2UtNDE4ZS05YTQzLTRhYmM3MjQxNWU0OSA=', 'vdi-uuid': '89e734bb-497e-418e-9a43-4abc72415e49', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4548] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/6bfba503-ac79-42b1-911f-748d7157a5dd', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/6bfba503-ac79-42b1-911f-748d7157a5dd', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEjZiZmJhNTAzLWFjNzktNDIgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIDZiZmJhNTAzLWFjNzktNDJiMS05MTFmLTc0OGQ3MTU3YTVkZCA=', 'vdi-uuid': '6bfba503-ac79-42b1-911f-748d7157a5dd', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4555] Set scheduler to [noop] on [/sys/dev/block/254:9]
Jan 10 16:42:51 flash8 SM: [4558] Set scheduler to [noop] on [/sys/dev/block/254:11]
Jan 10 16:42:51 flash8 SM: [4551] Set scheduler to [noop] on [/sys/dev/block/254:8]
Jan 10 16:42:51 flash8 SM: [4558] tap.activate: Launched Tapdisk(aio:/srv/hddpassthrough/sdi, pid=4956, minor=11, state=R)
Jan 10 16:42:51 flash8 SM: [4555] tap.activate: Launched Tapdisk(aio:/srv/hddpassthrough/sdl, pid=4943, minor=9, state=R)
Jan 10 16:42:51 flash8 SM: [4551] tap.activate: Launched Tapdisk(aio:/srv/hddpassthrough/sdc, pid=4932, minor=8, state=R)
Jan 10 16:42:51 flash8 SM: [4558] DeviceNode(/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/f42f9a79-bb14-4c6d-846e-a693946f4d20) -> /dev/xen/blktap-2/tapdev11
Jan 10 16:42:51 flash8 SM: [4551] DeviceNode(/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/73f1b9f0-dded-48de-85d0-db2f632cb2f5) -> /dev/xen/blktap-2/tapdev8
Jan 10 16:42:51 flash8 SM: [4555] DeviceNode(/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/e80647cb-e52d-4147-b90e-fc17334a885f) -> /dev/xen/blktap-2/tapdev9
Jan 10 16:42:51 flash8 SM: [4558] NBDLink(/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/f42f9a79-bb14-4c6d-846e-a693946f4d20) -> /run/blktap-control/nbd4956.11
Jan 10 16:42:51 flash8 SM: [4551] NBDLink(/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/73f1b9f0-dded-48de-85d0-db2f632cb2f5) -> /run/blktap-control/nbd4932.8
Jan 10 16:42:51 flash8 SM: [4555] NBDLink(/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/e80647cb-e52d-4147-b90e-fc17334a885f) -> /run/blktap-control/nbd4943.9
Jan 10 16:42:51 flash8 SM: [4558] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/f42f9a79-bb14-4c6d-846e-a693946f4d20', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/f42f9a79-bb14-4c6d-846e-a693946f4d20', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEmY0MmY5YTc5LWJiMTQtNGMgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIGY0MmY5YTc5LWJiMTQtNGM2ZC04NDZlLWE2OTM5NDZmNGQyMCA=', 'vdi-uuid': 'f42f9a79-bb14-4c6d-846e-a693946f4d20', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4551] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/73f1b9f0-dded-48de-85d0-db2f632cb2f5', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/73f1b9f0-dded-48de-85d0-db2f632cb2f5', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEjczZjFiOWYwLWRkZWQtNDggIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIDczZjFiOWYwLWRkZWQtNDhkZS04NWQwLWRiMmY2MzJjYjJmNSA=', 'vdi-uuid': '73f1b9f0-dded-48de-85d0-db2f632cb2f5', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4555] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/e80647cb-e52d-4147-b90e-fc17334a885f', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/e80647cb-e52d-4147-b90e-fc17334a885f', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEmU4MDY0N2NiLWU1MmQtNDEgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIGU4MDY0N2NiLWU1MmQtNDE0Ny1iOTBlLWZjMTczMzRhODg1ZiA=', 'vdi-uuid': 'e80647cb-e52d-4147-b90e-fc17334a885f', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4564]  = 0
Jan 10 16:42:51 flash8 SM: [4564] ***** ['/usr/sbin/tap-ctl', 'open', '-p', '4957', '-m', '10', '-a', 'aio:/srv/hddpassthrough/sdh'] failed: status=30, pid=4973, errmsg=Read-only file system: EXCEPTION <class 'blktap2.CommandFailure'>, ['/usr/sbin/tap-ctl', 'open', '-p', '4957', '-m', '10', '-a', 'aio:/srv/hddpassthrough/sdh'] failed: status=30, pid=4973, errmsg=Read-only file system
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 820, in launch_on_tap
Jan 10 16:42:51 flash8 SM: [4564]     TapCtl.open(pid, minor, _type, path, options)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 426, in open
Jan 10 16:42:51 flash8 SM: [4564]     cls._pread(args=args, input=input)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 295, in _pread
Jan 10 16:42:51 flash8 SM: [4564]     tapctl._wait(quiet)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 284, in _wait
Jan 10 16:42:51 flash8 SM: [4564]     raise self.CommandFailure(self.cmd, **info)
Jan 10 16:42:51 flash8 SM: [4564]
Jan 10 16:42:51 flash8 SM: [4564] ['/usr/sbin/tap-ctl', 'free', '-m', '10']
Jan 10 16:42:51 flash8 SM: [4564]  = 0
Jan 10 16:42:51 flash8 SM: [4564] ***** tapdisk failed exception: Tapdisk(aio:/srv/hddpassthrough/sdh): ['/usr/sbin/tap-ctl', 'open', '-p', '4957', '-m', '10', '-a', 'aio:/srv/hddpassthrough/sdh'] failed: status=30, pid=4973, errmsg=Read-only file system: EXCEPTION <class 'blktap2.TapdiskFailed'>, Tapdisk(aio:/srv/hddpassthrough/sdh): ['/usr/sbin/tap-ctl', 'open', '-p', '4957', '-m', '10', '-a', 'aio:/srv/hddpassthrough/sdh'] failed: status=30, pid=4973, errmsg=Read-only file system
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Jan 10 16:42:51 flash8 SM: [4564]     return self._run_locked(sr)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Jan 10 16:42:51 flash8 SM: [4564]     rv = self._run(sr, target)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/SRCommand.py", line 247, in _run
Jan 10 16:42:51 flash8 SM: [4564]     return target.attach(self.params['sr_uuid'], self.vdi_uuid, writable, caching_params = caching_params)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 1541, in attach
Jan 10 16:42:51 flash8 SM: [4564]     {"rdonly": not writable})
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 1694, in _activate
Jan 10 16:42:51 flash8 SM: [4564]     self._get_pool_config(sr_uuid).get("mem-pool-size"))
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 1330, in _tap_activate
Jan 10 16:42:51 flash8 SM: [4564]     options)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/blktap2.py", line 851, in launch_on_tap
Jan 10 16:42:51 flash8 SM: [4564]     raise TapdiskFailed(cls.Arg(_type, path), ctl)
Jan 10 16:42:51 flash8 SM: [4564]
Jan 10 16:42:51 flash8 SM: [4561] Set scheduler to [noop] on [/sys/dev/block/254:12]
Jan 10 16:42:51 flash8 SM: [4561] tap.activate: Launched Tapdisk(aio:/srv/hddpassthrough/sdu, pid=4954, minor=12, state=R)
Jan 10 16:42:51 flash8 SM: [4561] DeviceNode(/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/defd7bbc-e6a2-4015-a78b-dbc35cc87133) -> /dev/xen/blktap-2/tapdev12
Jan 10 16:42:51 flash8 SM: [4561] NBDLink(/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/defd7bbc-e6a2-4015-a78b-dbc35cc87133) -> /run/blktap-control/nbd4954.12
Jan 10 16:42:51 flash8 SM: [4561] result: {'params_nbd': 'nbd:unix:/run/blktap-control/nbd/1657b252-b70e-1fba-86d5-fb1a4e313496/defd7bbc-e6a2-4015-a78b-dbc35cc87133', 'o_direct_reason': 'SR_NOT_SUPPORTED', 'params': '/dev/sm/backend/1657b252-b70e-1fba-86d5-fb1a4e313496/defd7bbc-e6a2-4015-a78b-dbc35cc87133', 'o_direct': True, 'xenstore_data': {'scsi/0x12/0x80': 'AIAAEmRlZmQ3YmJjLWU2YTItNDAgIA==', 'scsi/0x12/0x83': 'AIMAMQIBAC1YRU5TUkMgIGRlZmQ3YmJjLWU2YTItNDAxNS1hNzhiLWRiYzM1Y2M4NzEzMyA=', 'vdi-uuid': 'defd7bbc-e6a2-4015-a78b-dbc35cc87133', 'mem-pool': '1657b252-b70e-1fba-86d5-fb1a4e313496'}}
Jan 10 16:42:51 flash8 SM: [4564] Raising exception [453, tapdisk experienced an error [opterr=Read-only file system]]
Jan 10 16:42:51 flash8 SM: [4564] ***** udev: EXCEPTION <class 'SR.SROSError'>, tapdisk experienced an error [opterr=Read-only file system]
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Jan 10 16:42:51 flash8 SM: [4564]     ret = cmd.run(sr)
Jan 10 16:42:51 flash8 SM: [4564]   File "/opt/xensource/sm/SRCommand.py", line 125, in run
Jan 10 16:42:51 flash8 SM: [4564]     os.strerror(e.get_error().get_error_code()))
Jan 10 16:42:51 flash8 SM: [4564]
Jan 10 16:42:51 flash8 SM: [5315] params = ['sr_uuid', 'subtask_of', 'vdi_ref', 'vdi_on_boot', 'args', 'vdi_location', 'host_ref', 'session_ref', 'device_config', 'command', 'vdi_allow_caching', 'sr_ref', 'vdi_uuid']
Jan 10 16:42:51 flash8 SM: [5315] vdi_attach {'sr_uuid': '1657b252-b70e-1fba-86d5-fb1a4e313496', 'subtask_of': 'DummyRef:|39157136-9881-4f9f-ab65-657aabeff702|VDI.attach2', 'vdi_ref': 'OpaqueRef:faadfeb4-d3a6-45a8-b336-8f53df9f3b4a', 'vdi_on_boot': 'persist', 'args': ['true'], 'vdi_location': '/srv/hddpassthrough/sdg', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:3807bab5-5967-43e7-b317-cc5d15cfbae3', 'device_config': {'SRmaster': 'true', 'location': '/srv/hddpassthrough'}, 'command': 'vdi_attach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:2a90eece-9b9a-4994-93f1-c772d67194b0', 'vdi_uuid': '7fd3ec67-7ed1-4574-9295-7995cadaa523'}
Jan 10 16:42:51 flash8 SM: [5315] lock: opening lock file /var/lock/sm/7fd3ec67-7ed1-4574-9295-7995cadaa523/vdi
[...]
Jan 10 16:42:57 flash8 SM: [10344] lock: released /var/lock/sm/.nil/lvm
Jan 10 16:42:57 flash8 SM: [10344] Entering _checkMetadataVolume
Jan 10 16:42:57 flash8 SM: [10344] LVMCache: will initialize now
Jan 10 16:42:57 flash8 SM: [10344] LVMCache: refreshing
Jan 10 16:42:57 flash8 SM: [10344] lock: acquired /var/lock/sm/.nil/lvm
Jan 10 16:42:57 flash8 SM: [10344] ['/sbin/lvs', '--noheadings', '--units', 'b', '-o', '+lv_tags', '/dev/VG_XenStorage-f7485280-25a3-fb1d-0468-18cb3edb5d43']
Jan 10 16:42:57 flash8 SM: [10344]   pread SUCCESS
Jan 10 16:42:57 flash8 SM: [10344] lock: released /var/lock/sm/.nil/lvm
Jan 10 16:42:57 flash8 SM: [10344] vdi_detach {'sr_uuid': 'f7485280-25a3-fb1d-0468-18cb3edb5d43', 'subtask_of': 'DummyRef:|9c961a98-a71f-49f6-a0bc-1de30f137a38|VDI.detach', 'vdi_ref': 'OpaqueRef:776d4877-b4eb-47f5-84df-b2549040c3cf', 'vdi_on_boot': 'persist', 'args': [], 'vdi_location': '68192b55-0700-4462-b7d5-86ed057ecc8d', 'host_ref': 'OpaqueRef:7056a1c6-2060-471f-aed1-c665e7e4e6e8', 'session_ref': 'OpaqueRef:5e117b55-12f7-494b-991c-123539d236e9', 'device_config': {'device': '/dev/disk/by-id/md-name-localhost:127-part3', 'SRmaster': 'true'}, 'command': 'vdi_detach', 'vdi_allow_caching': 'false', 'sr_ref': 'OpaqueRef:6f8032d8-226e-4f05-80e5-3da4e412c1a3', 'vdi_uuid': '68192b55-0700-4462-b7d5-86ed057ecc8d'}
Jan 10 16:42:57 flash8 SM: [10344] lock: opening lock file /var/lock/sm/68192b55-0700-4462-b7d5-86ed057ecc8d/vdi

Googled around and it looks like it might be a problem related to a CD-rom not ejected, but there are no ISO images mounted in the virtual CD-rom reader.
Tried to migrate the VDI to a different server. TrueNAS boots up correctly… of course the real disks are not there.

To me it looks like the sdh drive has some issues, but not really sure what is the best course of action to deal with it…

Does anyone have the slightest idea of what I can do to have the VM starting again and safely mount back the 22 disks ZFS pool?

Any help would be greatly appreciated!

Thanks!