and then compare both videos
Simple comparison wouldn't work, the video is being compressed and decompressed and delay is being added in the middle so even in the best case you will need to align the streams.
Let's take it step by step-
You want to use some algorithm to compare a reference video stream to the received stream at the remote side.
Simply using the camera would make comparison problematic since you don't have the reference stream, also conditions might change from test to test making the results inconsistent.
Usually the best idea would be injecting a premade video stream, performance wise you would want to store it on the device on an SD card.
There are multiple options here-
You can use adb's record tool to simply record the screen
On some phones you can output the stream to hdmi and record it from there using traditional tools
If possible your software can output the decoded video stream to file instead of the stream, the main drawback here is that you don't test the software end to end.
You will first need to align your video streams in time, one way to achieve that is to insert some sycn pattern into the video stream itself for example 3D bar codes that can be analysed by software. Other options could be having something identifiable in the stream like a moving element of somekind.
The simplest and most reliable way is probably suing MOS and especially using PSNR since it is computationally supported on many platforms and environments
Video consumes a lot of space, you want your original and results to reside on an SD card or internal memory during the test while still keeping high quality of the video itself.
If your storage is limited you can compromise on synthetic or repetitive video streams.
As explained above you want a lot to be done on the mobile itself (PSNR can be done offline) and this affects the performance of the software itself and the mobile's OS.
There is not a lot that can be done, but it's always a good idea to keep that in mind while testing.
Tips and Tricks
While not perfect I've solutions using still images out of a video stream being used to assess video quality.
The simplest way is using adb to get screen captures, a somewhat better way is using high speed camera to capture the screen.
Both solutions requires time synching, the external camera could be mounted with the mobile inside a box (a big cardboard box works just fine).
Both would compare to a reference image captured before with some margin for errors, for example using ImageMagick
My experience is not totally postie on this, it is doable but the tests tend to be be unstable.
This is especially true on Android and when using high performance video encoding/decoding since it is hardware dependent and changes from model to model and with changes of OS versions.