Stale devices or devices not recognized on illumos#

I had to replace a server hardware, since the systemboard of my old X4270M2 decided to die. So I bought a X8-2L and migrated all disks and PCIe cards to it and booted. I will spare you all the hassle with old broken Firmware on the X8-2L which I solved by updating.

But in the end I got stuck in a scenario where I couldn't see all devices. Two NVMes and the Dual 10G Ethernet card where unable to access. The drivers wouldn't attach with update_drv/add_drv - even though I found the cards in prtconf -dD and with /usr/lib/pci/pcieadm show-devs .

This was very confusing and even devfsadm -Cv was unable to solve it. So I went the rough way and deleted /etc/path_to_inst which maps the pci paths to drivers. After rebooting with boot -avrw the situation was exactly the same.

I did boot also from an OpenIndiana Text Install USB stick, dropped into a shell and was able to see all devices, so I knew that this is not a hardware issue, but something with illumos device management. I was a bit lost, because up to my knowledge (until yesterday ;)) there was no other location for persistent device information.

And then I stumbled upon /etc/devices/ ...

localadm@spin:~$ ls -l /etc/devices
total 2066
-r--r--r--   1 root     root        2264 Apr  3 22:15 devid_cache
-r--r--r--   1 root     root         324 Apr  3 22:15 devname_cache
drwxr-xr-x   2 root     root           2 Apr  3 13:04 dli
-r--r--r--   1 root     root         388 Apr  3 22:25 mdi_scsi_vhci_cache
-r--r--r--   1 root     root         452 Apr  3 22:15 pci_unitaddr_persistent
-r--r--r--   1 root     root      920108 Apr  3 22:25 snapshot_cache

So I created a tar from it and then deleted the files (and kept /etc/devices/dli).

Then I re-created the boot-archive with bootadm update-archive -v and rebooted.

After that reboot all devices became accessible again.