Can You Build Automation as a Service?

Software systems are growing larger and ever more complex. At Cashfree Payments, a typical application or service is made up of thousands of lines of code. It uses many libraries or frameworks and is accessed by an increasing number of merchant platforms running with numerous configurations and users. To make matters interesting, new versions are pushed to users frequently, sometimes multiple times each day.

APIs form the backbone of web applications and they are treated more like the product than code these days. They are designed for direct consumption for external customers, well documented and are managed and monitored for performance and scalability.

Attempting to assess the quality of each product/feature by asking humans to manually interact with every feature just doesn’t scale. This is where automation comes into the picture. 

In this blog, we will run you through a few basic concepts of API and how the API test automation service is built at Cashfree Payments.

API Automation and Its Importance

API is the acronym for Application Programming Interface, which is a software intermediary that allows two applications to talk to each other. Each time you use an app to view the current stock price of a company or check the weather on your phone, you’re using an API behind the scenes.

By testing APIs, you will be able to validate the core functionality of the application in a short amount of time. You can also validate non-functional requirements like performance and security aspects of the application. Automating APIs will enable you to change the systems at a faster pace without fear. This will reduce your time to market for the new features built.

What if…?

The most common approach for automating REST APIs in java using open source tech is: 

1. Automate the tests using testNG/Junit framework and http-client/rest-assured library
2. Once automated, trigger the test cases from local via maven/testNG.xml file
3. Configure a job on Jenkins which will run the tests based on cron expression configured in build triggers.

Instead of following the above-mentioned flow, what if you can expose an API endpoint to trigger the tests? This will give you the advantage of:

1. Running tests from anywhere without checking out the code
2. Integrating the automation as part of the CI/CD process easily without any hassles like installing the dependencies for automation, configuration issues and versions mismatch
3. Testing new environments without thinking of seed data setup and tear down setup once the activity is complete
4. Analysing test results and comparing it with other runs/environments 

The Bigger Picture

Before I run you through the actual implementation of the test automation service, take a look at the following diagram. This gives you a fair idea on how the workflow looks and where exactly the test automation service fits in. 

Test automation service is a stand-alone service deployed on kubernetes (k8s) pod using Jenkins Pipeline script. This pipeline script code defines your entire build process, which includes stages for building an application, testing it and then deploying it. The test automation service is deployed on k8s because k8s is open source, highly portable, easily scalable and ops friendly.

For every request, the test automation service endpoint expects module name and URL as params. Based on the params, test cases are triggered against the given URL and the reports are stored in the AWS S3 bucket. Reports are stored in S3 bucket because AWS S3 provides industry-leading scalability, data availability, security, and performance, and you can retrieve any report at any time, from anywhere.

Inside Automation as a Service

Let’s take a look at how exactly the test automation is built in simple steps:

1. Automation service is a Java service built using spring boot framework
2. An API endpoint is exposed from test automation service that expects the module name and URL as params
3. API Test cases are automated using testNG and rest assured
4. Based on the params passed, the suite.xml gets created. With this suite.xml and configured listeners, the tests are triggered. You can refer to this link for more details – https://testng.org/doc/documentation-main.html#running-testng-programmatically
5. Once the test run is complete, the report gets generated and is uploaded to AWS S3
6. The response will have details like total_tests, tests_passed, tests_failed etc along with S3 report url. Anyone with permissions can access the reports and see it.

Sample request

curl –location –request POST ‘https://host-name/v1/automation/business-unit/trigger –header ‘Content-Type: application/json’
–data-raw ‘{
    “module_name”: “test-service”,
    “target_url”: “https://test.cashfree.com"
}’

Sample response

{
   “totalTestsCount”: 10,
   “testsPassedCount”: 6,
   “testsFailedCount”: 2,
   “testsSkippedCount”: 2,
   “testsFailed”: [“FailedTest-1″,”FailedTest-2”],
   “testsSkipped”: [“SkippedTest-1″,”SkippedTest-2”],
   “reportUrl”:”https://report-s3.zone.amazonaws.com/moduleName/report-timestamp.html"
}

Challenges Faced

The following are the top challenges faced while implementing test automation service, which got fixed in successive iterations:

1. Handling multiple requests at the same time

2. Handling concurrent requests for the same module

3. Show test reports in HTML format on Jenkins

4. Time out issues when the test run is taking longer than expected

5. Pretty print request and response in report file

Challenges related to multiple/concurrent requests and timeouts were solved using asynchronous execution support provided in spring. Async processes each request in a separate thread. Issues related to reports in Jenkins were resolved using the HTML publisher plugin.

Into The Future

Testing and automation culture is core to Cashfree Payments’ quality-first approach. 

I am very much excited about this journey. There are still lots of things that can be built as part of the test automation service like publishing the reports on internal communication channels(like slack), mocking/faking the APIs, generating test coverage metrics, etc.  

If you want to be part of this journey, please do visit our careers page!