Note: For overall context for this blog please read integration testing with soapUI.
SoapUI Pro has extensive reporting functionality: creating reports on different levels (project, testsuite, testcase, etc.), different formats (pdf, cvs, xls, xml, etc.), easy customization and more. The feature provides three basic types of reports:
- Printable Reports : Reports are based on the JasperReports reporting engine, and a good understanding of how Jasper works is necessary to customize the reports created by soapUI.
- Data Export: lets you export the same underlying data that is used for printable reports to xml or csv files.
- JUnit-Style HTML Reports: to generate HTML reports very reminiscent of the “classic” ant/junitreport reports
Our requirement was to show functional test results with complete details in XML/HTML format. Some of the details we want to show as part of XML/HTML reports were:
Description of test case:
- List of passed/failed test steps
- Passed/failed assertions in each step
- Capturing screen shots where failed test step invoked selenium (please read our blog about selenium integration with soapUI)
- Fetch Selenium logs
- Providing attachments like screen shots, request/response details in case of failed test step
Others
- It should show results for different browsers
- Test results can be publishable on Hudson or on specified web server
- If published on Hudson result should be consistent for different browser and can be categorized by browser.
We decided that JUnit-Style report is right candidate to meet the above requirement as it provides functional test results in XML/HTML format as well as Hudson understands the JUnit test report XML format.
Now the challenge was to generate reports with complete details as per requirement above because default reporting functionality gives a simplified overview of functional test results. Refer screenshot.
To fulfill all this requirement we need to customize the JUnit-Style reports created by soapUI and that is not supported by soapUI!…
So what did we do?
We did some analysis and found that we can achieve what we need by extending com.eviware.soapui.report.JUnitReportCollector class and implementing custom behavior as per our requirements.
In our custom JUnitReportCollector, we are adding custom behavior in the afterStep method. As we required to show all steps with assertions regardless of they got success or failed, we added one more map to record steps status that are got success
HashMap<TestCase, String> status;
public IsplJunitReportCollector() {
reports = new HashMap<String, JUnitReport>();
failures = new HashMap<TestCase, String>();
status = new HashMap<TestCase, String>();
…
}
In beforeRun method we added common properties to system out so it can be shown by each test case. At this point we sets name of test case with prefix project.browser so that Hudson can separate test cases accordingly. Check code and Screen Shot below.
public void beforeRun(TestCaseRunner testRunner, TestCaseRunContext runContext) {
TestCase testCase = testRunner.getTestCase();
TestSuite testSuite = testCase.getTestSuite();
if (!reports.containsKey(testSuite.getName())) {
JUnitReport report = new JUnitReport();
String browser = PropertyExpansionUtils.getGlobalProperty(“Browser”);
String project = PropertyExpansionUtils.getGlobalProperty(“Project”);
report.setTestSuiteName(project + “.” + browser + “.” + testSuite.getName());
StringBuffer buf = new StringBuffer(“Date: “
+ new Date(System.currentTimeMillis()));
buf.append(“\nUIURL: “ + PropertyExpansionUtils.getGlobalProperty(“UIURL”));
buf.append(“\nStoreCode: “
+ PropertyExpansionUtils.getGlobalProperty(“StoreCode”));
buf.append(“\nVersion: “ + PropertyExpansionUtils.getGlobalProperty(“Version”));
report.setSystemOut(buf.toString());
reports.put(testSuite.getName(), report);
}
}
Next customization point is afterStep method, which takes care of following points.
– Proper formatting result to make readable even if deployed by hudson
– Shows selenium log in case of failed step invoked browser
– Provides screenshot link if available
– Provides failure step’s details file link.
– Shows all steps results irrespective of it is failed or passed
– Show all assertions details if available
public void afterStep(TestCaseRunner testRunner, TestCaseRunContext runContext,
TestStepResult result) {
TestStep currentStep = result.getTestStep();
TestCase testCase = currentStep.getTestCase();
if (result.getStatus() == TestStepStatus.FAILED) {
StringBuffer buf = new StringBuffer();
if (status.containsKey(testCase)) {
buf.append((String) status.get(testCase));
}
if (failures.containsKey(testCase)) {
buf.append((String) failures.get(testCase));
}
Get report file and screenshots if available to attach. Hudson doesn’t support html formatted Stack trace but provide link for URL so we had provided plane text with URL of attachments. Here to check whether screen shot is available or not we are checking property of test case which will programmatically created by our custom groovy script which Invoking a seleniumRC test from soapUI (Please read our blog about Invoking a seleniumRC test from soapUI)
String screenshot = testCase.getPropertyValue(currentStep.getLabel() + “screenshot”);
String reportFile = getErrorReport(currentStep, runContext, result);
buf.append((new StringBuilder()).append(“\n” + result.getTestStep().getName()).append(
” Failed”).toString());
buf.append(lineBreak());
buf.append(showFile(reportFile, “View Details”));
buf.append(lineBreak());
//buf.append(“Absolute path: [“+reportFile+”]”);
buf.append(lineBreak());
buf.append(showFile(screenshot, “View Screenshot”));
buf.append(lineBreak());
Show Complete Message Logs:
buf.append((new StringBuilder()).append(
XmlUtils.entitize(Arrays.toString(result.getMessages())))
.append(lineBreak()).toString());
if (testRunner.getTestCase().getSettings().getBoolean(“Complete Message Logs”)) {
StringWriter stringWriter = new StringWriter();
PrintWriter writer = new PrintWriter(stringWriter);
result.writeTo(writer);
buf.append(XmlUtils.entitize(stringWriter.toString()));
}
failures.put(testCase, buf.toString());
}
Here is the code to show success steps information as well as all assertions
else {
StringBuffer buf = new StringBuffer();
if (status.containsKey(testCase)) {
buf.append((String) status.get(testCase));
}
buf.append(“\nTest Step: “ + currentStep.getName());
buf.append(“\tResult: “ + result.getStatus() + “”);
if (currentStep instanceof Assertable ) {
Assertable requestStep = (Assertable) currentStep;
if(requestStep.getAssertionCount()>0){
…
}
}
// any prior test-step failed!…
if (failures.containsKey(testCase)) {
String str = (String) failures.get(testCase) + “\n” + buf.toString();
failures.put(testCase, str);
} else {
status.put(testCase, buf.toString());
}
}
}
Screenshots:
— failed test case (failed UI test step)
After Customization
After Customization it shows failed test step with complete details, link to screen shot if it had invoked selenium as well as test steps that got success with assertions details. Next screenshot shows how it reports if deployed in Hudson.
