!!Stale devices or devices not recognized on illumos

I had to replace a server hardware, since the systemboard of my old X4270M2 decided to die. So I bought a X8-2L and migrated all disks and PCIe cards to it and booted.
I will spare you all the hassle with old broken Firmware on the X8-2L which I solved by updating.

But in the end I got stuck in a scenario where I couldn't see all devices. Two NVMes and the Dual 10G Ethernet card where unable to access. The drivers wouldn't attach with {{{update_drv}}}/{{{add_drv}}} - even though I found the cards in {{{prtconf -dD}}} and with {{{/usr/lib/pci/pcieadm show-devs}}} .

This was very confusing and even {{{devfsadm -Cv}}} was unable to solve it.
So I went the rough way and deleted {{{/etc/path_to_inst}}} which maps the pci paths to drivers. After rebooting with {{{boot -avrw}}} the situation was exactly the same.

I did boot also from an OpenIndiana Text Install USB stick, dropped into a shell and was able to see all devices, so I knew that this is not a hardware issue, but something with illumos device management.
I was a bit lost, because up to my knowledge (until yesterday ;)) there was no other location for persistent device information. 

And then I stumbled upon {{{/etc/devices/}}} ...

{{{localadm@spin:~$ ls -l /etc/devices
total 2066
-r--r--r--   1 root     root        2264 Apr  3 22:15 devid_cache
-r--r--r--   1 root     root         324 Apr  3 22:15 devname_cache
drwxr-xr-x   2 root     root           2 Apr  3 13:04 dli
-r--r--r--   1 root     root         388 Apr  3 22:25 mdi_scsi_vhci_cache
-r--r--r--   1 root     root         452 Apr  3 22:15 pci_unitaddr_persistent
-r--r--r--   1 root     root      920108 Apr  3 22:25 snapshot_cache}}}

So I created a tar from it and then deleted the files (and kept {{{/etc/devices/dli}}}).

Then I re-created the boot-archive with {{{bootadm update-archive -v}}} and rebooted.

After that reboot all devices became accessible again.