I tested your theory on my own Ubuntu 18.04 desktop, PopOS, and Ubuntu Server (headless) all running in Parallel virtual machines and I was able to successfully copy my public/private key pairs from Ubuntu Desktop over to PopOS and later log into my Ubuntu headless server without needing a password. So, you are correct, this should work.
I will admit I needed to install the ssh service first on my ubuntu desktop, and popos first and then manually copy over the id_rsa (private), id_rsa.pub (public) and known_hosts files over to get this to work since the ssh service was not installed and listening on these newly installed default desktop systems.
Make sure you have installed ssh on both the old and new systems and try to scp (secure copy) the files over to PopOS and try again. Good luck.