I'd disagree with the premise, or at least, disagree with breaking things into those two pieces. Because there are at least three types of software, if you want to look at a business software stack.
- Presentation Layer
- Operating System
And if you really want to get complicated, the Operating System isn't at the bottom. There's the hardware, which probably has several layers to it (BIOS, millicode, etc). And everything might be virtualized.
So, with that said, testing any of those isn't too different. At a base level, software is expected to take some input, and return some output. What's different is the thing that they're expected to take input from, and return output to.
Operating Systems exist to make life easier for other software running on the computer, by providing an interface between the hardware and programs running on top. So when testing OSes, you focus on the APIs that the OS provides, and the interface with the hardware.
Middleware, similarly, exists to make things easier for the display layer. It could be a database, a queue manager, or a host of other things. So, again, you focus on the APIs it provides, and the service it's supposed to provide.
And the presentation layer exists to present data to the end user, so you focus on the UI and data entry/retrieval. That's assuming that there is a UI; some business software talks only to other business software, in which case you focus on the data transformation that's happening.
The only thing that may be different is system testing; with Middleware and the Presentation Layer, you're probably also going to do some testing with a specific software stack, to in addition to testing the interfaces the software provides. But doing Operating System testing, you're probably going to have a suite of software you've got installed to make sure that real software doesn't break somehow.
Things like security flaws, run conditions, and so on can exist at any layer of software; all that really matters is how the flaw is exposed.