Mobile apps have become a necessity for every business, whether a startup or a well-recognized brand.
The Cashfree Payments mobile app lets our merchants easily access their Cashfree Payments account from anywhere, anytime, and track payments on the go!
Writing and maintaining the mobile automation code is not always a piece of cake. As a matter of fact, we frequently face many scenarios where automated test cases don’t work as expected and might lead to false positive or false negative results. In such cases, debugging and fixing the tests is the only way out.
Debugging issues in the mobile automation framework becomes more complicated when the mobile automation framework has a huge number of test cases. With the expansion of features in the application, the number of test cases gradually increases. In such a scenario, fixing complex issues of frameworks might require enhanced techniques of debugging.
In this post, we will deep dive into such essential debugging techniques that we have implemented in our mobile automation framework – that will not only help us fix the test automation issues easily but also save a good amount of debugging time.
Before getting into debugging, let us give you a quick overview of the basic elements of the mobile automation framework built using Pytest and Appium.
Introduction to framework
Mobile automation framework at Cashfree Payments follows Page Object Model design pattern with Python as the programming language. We use Appium as the automation tool, Pytest as the testing framework, and Allure is used for reporting.
Appium is an open source, cross platform test automation framework for automated testing of native, hybrid and mobile web apps on Android, iOS and Windows platforms. At the core, Appium is nothing but a HTTP server written in Nodejs that exposes the Rest API which can be used to automate the mobile apps.
Page Object Model
Page Object Model is the design pattern commonly used in web/mobile test automation that creates an object repository for web/mobile elements.
Every screen will have a corresponding screen class. Mobile elements will be defined as variables, which will be uniquely identified by locators and all the possible user interactions like click or send text or scroll can be implemented as methods of the class.
The tests can use the methods of this class whenever they need to interact with the UI of the screen.
Pytest is a testing framework that allows users to write test codes using python. It provides multiple features like command line support to run single test, all tests in a package, fixtures that can run before the test, Hooks can be used to log events of tests, like attach screenshot in case of test failure, Markers can be used for grouping tests like sanity,regression and plugins for reporting and parallel testing.
Allure Framework is a flexible lightweight multi-language test report tool that not only shows a very concise representation of what has been tested in a neat web report form, but allows everyone participating in the development process to extract maximum of useful information from everyday execution of tests. Allure provides a clear ‘big picture’ of what features have been covered, where defects are clustered, how the timeline of execution looks like and many other convenient things.
Debug Support for Mobile Automation
We have implemented multiple capabilities in our framework to help quickly debug and fix bugs discovered through automated tests.
- Video recording and device logs
Video recordings of mobile test case execution can play a critical role. Not only would these help in debugging issues more efficiently, we can use recorded videos to show test execution activities to internal stakeholders. Additionally, these recorded videos can be added to the project management tool while creating bugs, which helps in understanding the bug clearly.
Appium Dashboard plugin provides video recording of test and device logs(logcat and syslogs) for easy debugging of tests.
Some of the benefits of using appium dashboard plugins are as follows :
- Monitor all webdriver session api calls made by the test with full request and response details.
- Watch the live video of the test execution
- View device logs from Android(logcat) and iOS(syslogs) devices/emulators.
- Get insights on the performance of the application by analyzing the CPU and Memory usage during the test execution.
Installation of the plugin can be done by the below command (Note : plugin requires appium version 2.0)
appium plugin install –source=npm appium-dashboard
Once the installation is done, the plugin needs to be activated each time the appium server is started using below command
Now navigate to http://localhost:<appium-server-port>/dashboard to open the web app which will show the complete list of tests and its details that are being executed.
You can also analyze the CPU and Memory usage during the test execution by selecting app profiling
- Screenshot for failed tests
The easiest and quickest way to debug your automated testing script is using screenshots. The idea is to capture a screenshot of the mobile screen at the moment where the test script failed. And later, as a part of debugging, looking at the screenshots we can easily identify where and what was the reason for our test case failure.
Pytest supports the creation of hooks in a file called conftest.py that can be placed in the root folder where we wish to run the tests. This allows us to define, in Python, code to find test files, run them, and collect the results of the test execution.
def pytest_runtest_makereport(item, call):
outcome = yield
rep = outcome.get_result()
setattr(item, “rep_” + rep.when, rep)
Screenshot can be attached to the allure report with below command for failed tests. This method will be called post test execution and if the test has failed
allure.attach(driver.get_screenshot_as_png(), name=”screenshot”, attachment_type=AttachmentType.PNG)
- Live logging on CLI/Console Logs
Pytest 3.5 has introduced log_cli configuration. By setting the log_cli configuration option to true, pytest will output logging records as they are emitted directly into the console.
You can specify the logging level for which log records with equal or higher level are printed to the console by passing log_level. In order to enable this, all you have to do is create a configuration INI file in the root directory and add the below CLI options
- Text Logs
Python provides an in-built logging module which is part of the python standard library. So you don’t need to install anything.
To emit a log message, You can create a new logger using the ‘logger.getLogger(__name__)‘ method.__name__ variable will hold the name of the module (python file) that called the code.
Now, once you’ve created a new logger, you can set the logging at different log levels (DEBUG, INFO, ERROR, etc.) using setLevel(level) method.
logger = logging.getLogger(__name__)
The log formatter basically enriches a log message by adding context information to it. It can be useful to know when the log is sent and where (Python file, line number, method, etc.)
formatter = Formatter(fmt=’%(asctime)s – %(levelname)s – %(message)s’)
The TimedRotatingFileHandler class, located in the logging.handlers module, supports rotation of disk log files at certain timed intervals. In our mobile automation framework, the log files are rotated everyday with a backup count of 3 days
handler = TimedRotatingFileHandler(filename=log_filename, when=’midnight’, interval=1, backupCount=3)
Maintaining the automation framework becomes hard as the product grows and test cases increase. If we fall behind in fixing the automation failures, we will have many false positives (i.e., cases where the test fails because of a problem with the test, not a problem with the application). The above mentioned debug support essentials will help the team cut down on debugging and maintenance time, allowing the team to keep pace with software development teams.
Along with above capabilities, we can also add slack integration and email notifications for alerting purposes when a failure occurs.
If this sounds like something that hits all the right spots, cerebrally speaking, then we have some exciting opportunities lined up for engineers just like you!