In my personal opinion and observations, a lot of this comes down to cost and knowledge of the product. Many companies have tried solutions like this and QTP (hp Quick Test Pro) thinking that it was going to be a silver bullet for their testing resources but it ended up not being able to deliver what they expected it would (it takes time and knowledge no matter what solution you're using). Those same companies have then tried another, open source or free solution (or abandoned test automation all together), and with what their testers knew after trying to commercial product, were able to make it work somewhat better.
There are also the companies who take a look at the dollar value, and say flat out no. Even if not this, then there is the bundling factor when it comes to cost. Many companies already use HP's products like Quality Centre and Load Runner, and I would assume that there would be a certain discount around putting their own automation framework on top of this. The same is certainly true of Microsoft products. If a company is already purchasing a shwack load of Visual Studio licenses for the developers, why not spend a little extra to also get it for the testers.
To come to your question about 'newer' frameworks, I think that this comes down partially to cost, but also to usability for the testers who have to use the product. Personally, I want to use the combination of products to build my automation suite that I want. In addition to this, the user community for these 'newer' products is quite more diverse than what you'd get with commercial products like SilkTest or QTP. If a user from the community is not able to help you, one of the contributers, with a vested interest in the continuity of the project (even if just out of pride) is probably able to help you out.