I have an Android app that allows to make video calls and I need to test the quality of the video and if both video streamings match correctly.
How would you test it, in an automatic proccess?
I thought about putting 2 devices, one in front eachother, and in the middle a tablet that records with one camera one device, and with the front camera the other device and then compare both videos, but this solution seems too much complicated.
and then compare both videos
Simple comparison wouldn't work, the video is being compressed and decompressed and delay is being added in the middle so even in the best case you will need to align the streams.
Let's take it step by step-
You want to use some algorithm to compare a reference video stream to the received stream at the remote side.
Simply using the camera would make comparison problematic since you don't have the reference stream, also conditions might change from test to test making the results inconsistent.
Usually the best idea would be injecting a premade video stream, performance wise you would want to store it on the device on an SD card.
There are multiple options here- You can use adb's record tool to simply record the screen
On some phones you can output the stream to hdmi and record it from there using traditional tools
If possible your software can output the decoded video stream to file instead of the stream, the main drawback here is that you don't test the software end to end.
You will first need to align your video streams in time, one way to achieve that is to insert some sycn pattern into the video stream itself for example 3D bar codes that can be analysed by software. Other options could be having something identifiable in the stream like a moving element of somekind.
The simplest and most reliable way is probably suing MOS and especially using PSNR since it is computationally supported on many platforms and environments
Video consumes a lot of space, you want your original and results to reside on an SD card or internal memory during the test while still keeping high quality of the video itself.
If your storage is limited you can compromise on synthetic or repetitive video streams.
As explained above you want a lot to be done on the mobile itself (PSNR can be done offline) and this affects the performance of the software itself and the mobile's OS.
There is not a lot that can be done, but it's always a good idea to keep that in mind while testing.
While not perfect I've solutions using still images out of a video stream being used to assess video quality.
The simplest way is using adb to get screen captures, a somewhat better way is using high speed camera to capture the screen.
Both solutions requires time synching, the external camera could be mounted with the mobile inside a box (a big cardboard box works just fine).
Both would compare to a reference image captured before with some margin for errors, for example using ImageMagick
My experience is not totally postie on this, it is doable but the tests tend to be be unstable.
This is especially true on Android and when using high performance video encoding/decoding since it is hardware dependent and changes from model to model and with changes of OS versions.
There can be multiple test case of the application based on context. Some important Test case scenarios as follows
- Video calling icon should display when another users are online with webcam.
- Clicking on video calling icon should connect to the particular person with calling ring tone and callers name.
- Video resolution should good in all browsers and resolutions.
- video calls should start with sound enabled automatically.
- Video can be mute for temporary but Sound can be played. (And vice versa)
- Reconnect to the user is possible in the low network or quick disconnect/ connect of wifi connection.
- Chat/ text can be done along with the video calling.
- Joining/removing more people into same chat is possible.
- If one person is disconnecting the chart then the connection should not get fail with other persons.
- When the main caller will close the chat, then it should disconnect from all the persons.
- Video recording/sound recording functionality is working fine during video chat.
- Mute/ un-mute to sound functionality is working fine.
- Video calling window should be maximized/ minimized.
- The user should use other functionality of PC/ laptop/mobile, during the video call.
- The screen should divide and display all the person pics when the video call made for multiple persons. And disconnect with the person should remove the screen allocation for him/ her.
- If the person is not accepting the call request, then the call log should generate and should appear to the called person.
- Functionality should work fine with diff OS/devices & Networks.
