UI Automation (often refered to as UIA) relates to a set of interfaces designed to allow one programs to interact with or to control another via the screen, keyboard and mouse. The intention is to allow programs such as screen readers and alternatives to keyboard and mouse to be used with (almost) any application. Hence most applications can be used by people with impaired vision or dexterity problems (or more crudely put: people with disabilities).
Microsoft also support the MSAA interface (Microsoft Active Accessibility) which is an older interface with the same goals. MSAA was released with Windows 95 and UIA with Vindows Vista. See here for more about both.
Having got the UIA or MSAA interfaces to an application, other programs can interact with it. This includes automated testing. Test systems such as Microsoft's Coded UI use the UIA and MSAA interfaces of an application.
Most of the Microsoft application developement frameworks (such as Windows Forms, WPF and MFC) provide proper support for either MSAA or UIA. The problem is with custom controls. They need to include support for UIA or MSAA otherwise they will not work with the screen readers etc. I have not found any good documentation on how to write the required support code.