I think it's difficult to make through-the-UI tests reliable. The challenge comes down to the difficulty of reliably controlling and observing the variables that matter to your tests. Whether this is worthwhile depends on your ability to make your test code more resilient, and on the value of being able to run the tests automatically.
Asynchrony. For web applications, a common source of trouble is asynchrony. You take an action, and at some later time the app produces a result. And the delay can vary. How do you account for the varying delay?
Fixed waits. Some people do this by naively adding fixed waits into their test code. Click the button, wait a fixed amount of time, and check whether the result is displayed. This is troublesome because one day you'll run the test and the fixed wait won't be long enough. So you increase the duration of the fixed wait. Over time, this makes your tests more reliable, but very slow. Don't use fixed waits unless you absolutely can't use either of the following two techniques.
Latch. A better technique is to have the application use a latch, a variable that indicates whether some asynchronous operation has completed. When the operation starts, the application sets the latch variable to false. When the operation completes, the application sets the latch variable to true. Sometimes your tests can inject code into the application to implement the latches. Your injected code registers to be notified when the operation starts and stops, and sets the value of the latch accordingly. Then write your test code to watch for the latch variable to indicate that the operation is complete.
Polling. Another technique is for your test code to poll for the desired result. You click the button, then repeatedly check whether the result is displayed, pausing briefly in between checks. You put a maximum polling duration on your polling code, so that if the item isn't displayed within, say, 30 seconds, you fail the test. I usually end up creating a DSL to hide the details of the polling code and make it expressive, like this:
assertThat(submitButton, eventually(), is(present()));
Identifying elements. A second common challenge is how to identify the HTML/XML elements you want to observe and manipulate.
Meaningful ids and classes. It's best if the application is designed to make it easy to identify elements, by adding well-named
id attributes to unique elements, and well-named
class attributes for repeated structures (like lists or tables). These attributes make it easier to refer directly to the elements you're interested in. If your application doesn't have these, befriend a developer and beg for them. Otherwise, you'll have to identify elements by their relationships to other elements, which makes your tests depend on the current structure of the markup code. When the structure changes, your locators become invalid.
Simplify locators. A related challenge is to write locators that are amenable to changes in the markup. I often see people using inspection tools (Selenium IDE, Firebug, etc) to generate locators. Often the monstrous locators created by these tools include the complete path to the element, including indexes. If you use these locators, your tests become dependent on that full path. If the structure changes, your locators become invalid. It's better to examine each tool-generated locator to see if you can pick out the essential parts, then create a locator that relies on only those essential parts. (Then get a developer to add meaningful