Not cleaning up on tear down works as long as you are 100% sure that every test cleans up before setup correctly and always will (e.g., its setup won't get outdated). If you are the only tester and the only person using your environment - or, if you have a good 'clean the environment & restore to defaults' script / function that all testers use before test or which the harness runs automatically - then I think it can work.
However, most software does not want to stop and sit after a failure to maintain a pristine failure state with all of the data from the error still in its original form. Most software projects need to use test resources efficiently, and that means cleaning up and moving on to the next test. To have a debug-able failure, that means you need to save state anyways before moving on. Copy the error files and config files, back up the DB, dump machine information on CPU / RAM / etc. into log files, and so on, and put it all in one place so its easy for the tester investigating the issue to find everything s/he needs. Since most testers will want to save state before the test ends anyways, there's not a lot of advantages to avoiding a full cleanup during tear down.*
Also, if you don't clean up during tear down, you will need to maintain old tests when more configuration options are made available, so they set those new options correctly. This can be avoided by setting the machine to a 'default state' before running tests using a single function that is always called before tests, so you just update that one function - but if you are choosing not to cleanup after tests, test setup maintainability is something you likely want to plan for. If you don't have a process for ensuring test setup remains maintainable (and the feasibility of this kind of planning varies between companies), then cleaning up is a good idea.
I know in my company, I can't rely on other test developers to clean up everything before starting tests - largely because my company is planning to make huge changes to QA in the future, and I can't know for sure that the other testers using my harness will be skilled, or will understand the system well enough to think of all the things they will need to clean up. I don't get to develop policy, I can only try and influence it. I could let other test developers suffer for their bad habits, but I know that could rebound on me politically. So, I always clean up to the best of my ability.
There's also the case where the failure disrupts things so much that the cleanup code can't run. In this case, I personally like to have the failure to cleanup reported with the tear down for the actual test that caused the problem, rather than reported as a failure of the setup of the next test to run. If nothing else, it's easier to show to devs and management and explain as one failed test, not two.
*Not a lot of advantages to avoiding cleanup during teardown, but there are some: Maybe cleanup is expensive, and doing it during both setup and teardown is too time-consuming - cleanup during setup is definitely more important. Maybe you haven't had time to develop the 'save everything' infrastructure so you do want things to stop and sit, rather than having the run continue. Maybe you think the cases where a tester on your product doesn't clean up before testing are so rare that you would rather save the development time needed for cleaning up and trust that this piece of QA culture doesn't change over time in your company.