AutoIT, Sikuli or Kantu Automation all work fine in a virtual machine (e. g. Virtual Box). If you have a problem there, please provide details.
On the other hand, these tools need a desktop - so they can not run headless. This is by design. They can not be compared with tools like PhantomJS that just emulate a web browser.
The prerequisites for starting the object mapping mode are:
project is created and opened
a connection has been made to the AUT Agent (if this is successful, the "Stop AUT Agent" button on the toolbar will be enabled)
an AUT has been configured via the project properties
the AUT has been started. Your AUT is successfully started if the AUT can be seen to be ...