I am new to testing the security of applications.
I know basic security like invalid credentials but I don't know about mobile app security testing
Please let me know how to check?
There are some nice points that you can cover with a cheat-sheet from the guys over at OWASP.
My suggestion is that you should try to go point-by-point in the list here
Long story short for android applications is the following:
Decompile the APK file
Install the APK.
Intercept the requests.
Check for local storage
There are already a lot of tools in this App-Use, namely apktool, smali, baksmali, manifest explorer, jdgui, dex2jar, Android SDK, adb, burp suite, etc
To get started, may be you should refer the OWASP Mobile Security Testing Guide repository on Github.
It has tons of information, including what you should test, and how you should test, including information of tools which are used to test that particular scenario. It is a very active repository, with lots of activity happening everyday (Please star so that you can receive notifications).
The best part is that they start with the very basics - the architecture of both IOS and Android operating system and then build it from there.
Combined with the cheat sheet given in the other answer, you have a goldmine of scenarios for mobile security testing.