Or are you talking about RDP? If your machine is vulnerable to RDP hacking, you've done something horribly wrong in your network setup. They're (presumably) all going to be connected to the internet through a NAT anyway, so opening a firewall exception to RDP LAN connections is really not any additional risk.
My particular setup has them all on the same 1GB wired ethernet switch; for most uses it's the same responsiveness as directly using the machine on its own physical hardware.
I've had good luck with the TESmart 4-port 4k HDMI KVM switch. There are also 2-port options, and 2-port-dual-monitor options.
I've mostly stopped using the actual monitor switching though; I just use remote desktop.
... a KVM is hardware.
Or are you talking about RDP? If your machine is vulnerable to RDP hacking, you've done something horribly wrong in your network setup. They're (presumably) all going to be connected to the internet through a NAT anyway, so opening a firewall exception to RDP LAN connections is really not any additional risk.
My particular setup has them all on the same 1GB wired ethernet switch; for most uses it's the same responsiveness as directly using the machine on its own physical hardware.