What you are looking for is called [*static code analysis*](https://en.wikipedia.org/wiki/Static_program_analysis). (Check [this wiki page](https://en.wikipedia.org/wiki/List_of_tools_for_static_code_analysis) for a list of tools.)

>What type of testing tool can automatically test this?

One of the most famous tools is [SonarQube](https://www.sonarqube.org/), which supports Java, JavaScript, C#, and more. It also offers integrations for various build tools and CI servers (including Jenkins). Plus: It is open source.

>And is it better to have it as a Jenkins hook or move it down and into the IDE the developer is using?

You probably want both. That is, some sort of code formatter within your IDE which is compliant with your tool's setting *and* the tool itself that checks everything that is pushed to the repository.

If, for instance, you use [trunk-based development](https://trunkbaseddevelopment.com/), you would break the build if you commit a style violation you didn't catch locally, since static code analysis tools typically mark builds unstable. And if you only check locally, you may integrate style violations when a developer has a misconfigured IDE. Therefore, you want to have both.