Customizing soapUI reports

Note: For overall context for this blog please read integration testing with soapUI.

SoapUI Pro has extensive reporting functionality: creating reports on different levels (project, testsuite, testcase, etc.), different formats (pdf, cvs, xls, xml, etc.), easy customization and more. The feature provides three basic types of reports:

  1. Printable Reports : Reports are based on the JasperReports reporting engine, and a good understanding of how Jasper works is necessary to customize the reports created by soapUI.
  2. Data Export: lets you export the same underlying data that is used for printable reports to xml or csv files.
  3. JUnit-Style HTML Reports: to generate HTML reports very reminiscent of the “classic” ant/junitreport reports

Our requirement was to show functional test results with complete details in XML/HTML format. Some of the details we want to show as part of XML/HTML reports were:

Description of test case:

  • List of passed/failed test steps
  • Passed/failed assertions in each step
  • Capturing screen shots where failed test step invoked selenium (please read our blog about selenium integration with soapUI)
  • Fetch Selenium logs
  • Providing attachments like screen shots, request/response details in case of failed test step

Others

  • It should show results for different browsers
  • Test results can be publishable on Hudson or on specified web server
  • If published on Hudson result should be consistent for different browser and can be categorized by browser.

We decided that JUnit-Style report is right candidate to meet the above requirement as it provides functional test results in XML/HTML format as well as Hudson understands the JUnit test report XML format.

Now the challenge was to generate reports with complete details as per requirement above because default reporting functionality gives a simplified overview of functional test results. Refer screenshot.

To fulfill all this requirement we need to customize the JUnit-Style reports created by soapUI and that is not supported by soapUI!…

So what did we do?

We did some analysis and found that we can achieve what we need by extending com.eviware.soapui.report.JUnitReportCollector class and implementing custom behavior as per our requirements.

In our custom JUnitReportCollector, we are adding custom behavior in the afterStep method. As we required to show all steps with assertions regardless of they got success or failed, we added one more map to record steps status that are got success

HashMap<TestCase, String> status;

public IsplJunitReportCollector() {

reports = new HashMap<String, JUnitReport>();

failures = new HashMap<TestCase, String>();

status = new HashMap<TestCase, String>();

}

In beforeRun method we added common properties to system out so it can be shown by each test case. At this point we sets name of test case with prefix project.browser so that Hudson can separate test cases accordingly. Check code and Screen Shot below.

public void beforeRun(TestCaseRunner testRunner, TestCaseRunContext runContext) {

TestCase testCase = testRunner.getTestCase();

TestSuite testSuite = testCase.getTestSuite();

if (!reports.containsKey(testSuite.getName())) {

JUnitReport report = new JUnitReport();

String browser = PropertyExpansionUtils.getGlobalProperty(“Browser”);

String project = PropertyExpansionUtils.getGlobalProperty(“Project”);

report.setTestSuiteName(project + “.” + browser + “.” + testSuite.getName());

StringBuffer buf = new StringBuffer(“Date: “

+ new Date(System.currentTimeMillis()));

buf.append(“\nUIURL: “ + PropertyExpansionUtils.getGlobalProperty(“UIURL”));

buf.append(“\nStoreCode: “

+ PropertyExpansionUtils.getGlobalProperty(“StoreCode”));

buf.append(“\nVersion: “ + PropertyExpansionUtils.getGlobalProperty(“Version”));

report.setSystemOut(buf.toString());

reports.put(testSuite.getName(), report);

}

}

hudson_grooup_tc

Next customization point is afterStep method, which takes care of following points.

Proper formatting result to make readable even if deployed by hudson

Shows selenium log in case of failed step invoked browser

Provides screenshot link if available

Provides failure step’s details file link.

Shows all steps results irrespective of it is failed or passed

Show all assertions details if available

public void afterStep(TestCaseRunner testRunner, TestCaseRunContext runContext,

TestStepResult result) {

TestStep currentStep = result.getTestStep();

TestCase testCase = currentStep.getTestCase();

if (result.getStatus() == TestStepStatus.FAILED) {

StringBuffer buf = new StringBuffer();

if (status.containsKey(testCase)) {

buf.append((String) status.get(testCase));

}

if (failures.containsKey(testCase)) {

buf.append((String) failures.get(testCase));

}

Get report file and screenshots if available to attach. Hudson doesn’t support html formatted Stack trace but provide link for URL so we had provided plane text with URL of attachments. Here to check whether screen shot is available or not we are checking property of test case which will programmatically created by our custom groovy script which Invoking a seleniumRC test from soapUI (Please read our blog about Invoking a seleniumRC test from soapUI)

String screenshot = testCase.getPropertyValue(currentStep.getLabel() + “screenshot”);

String reportFile = getErrorReport(currentStep, runContext, result);

buf.append((new StringBuilder()).append(“\n” + result.getTestStep().getName()).append(

” Failed”).toString());

buf.append(lineBreak());

buf.append(showFile(reportFile, “View Details”));

buf.append(lineBreak());

//buf.append(“Absolute path: [“+reportFile+”]”);

buf.append(lineBreak());

buf.append(showFile(screenshot, “View Screenshot”));

buf.append(lineBreak());

Show Complete Message Logs:

buf.append((new StringBuilder()).append(

XmlUtils.entitize(Arrays.toString(result.getMessages())))

.append(lineBreak()).toString());

if (testRunner.getTestCase().getSettings().getBoolean(“Complete Message Logs”)) {

StringWriter stringWriter = new StringWriter();

PrintWriter writer = new PrintWriter(stringWriter);

result.writeTo(writer);

buf.append(XmlUtils.entitize(stringWriter.toString()));

}

failures.put(testCase, buf.toString());

}

Here is the code to show success steps information as well as all assertions

else {

StringBuffer buf = new StringBuffer();

if (status.containsKey(testCase)) {

buf.append((String) status.get(testCase));

}

buf.append(“\nTest Step: “ + currentStep.getName());

buf.append(“\tResult: “ + result.getStatus() + “”);

if (currentStep instanceof Assertable ) {

Assertable requestStep = (Assertable) currentStep;

if(requestStep.getAssertionCount()>0){

}

}

// any prior test-step failed!…

if (failures.containsKey(testCase)) {

String str = (String) failures.get(testCase) + “\n” + buf.toString();

failures.put(testCase, str);

} else {

status.put(testCase, buf.toString());

}

}

}

Screenshots:

— failed test case (failed UI test step)

seleniumts_before

After Customization

seleniumts_after_n
After Customization it shows failed test step with complete details, link to screen shot if it had invoked selenium as well as test steps that got success with assertions details. Next screenshot shows how it reports if deployed in Hudson.

seleniumts_after

Interested in our Testing Services?

+1 408-727-1100

By submitting this form, you agree that you have read and understand Infostretch’s Terms and Conditions. You can opt-out of communications at any time. We respect your privacy.

By submitting this form, you agree that you have read and understand Infostretch’s Terms and Conditions. You can opt-out of communications at any time. We respect your privacy.

Other stories you may enjoy...

Healthcare Apps and the Need for Security

One of the most exciting areas in tech right now promises to be “the most personal” ever. A key aspect of making wearable devices like the Apple Watch personal is through...

Developing an App for the 2020 General Election?

Here is a thought: With the UK General Election having just finished, could the next one in 2020 be the first to use a mobile app to allow people to vote? The polling...

Be honest. Describe the state of your test cases.

“There’s some dead wood in there.” “Hmmm…. Someone really needs to clean them up.” “A little outdated.” For those reading this in the northern hemisphere,...