The Ultimate Guide to Agile Testing for Modern Software Teams: Part 2

In The Ultimate Guide to Agile Testing for  Modern Software Teams: Part 1, we looked at the agile testing life cycle, agile testing quadrants, and agile methodology in testing. In Part 2, we’ll go more into the agile testing process and check out agile testing methods, agile testing tools, agile test automation, and more!

Let’s look at other agile testing methods in modern software development.

Agile Testing Methods

As mentioned earlier, test-driven development (TDD) is an agile programming technique that requires developers, before they write any unit of code, to write an automated test for that code. Writing the automated tests is important because it forces the developer to take into account all possible inputs, errors, and outputs. 

• TDD allows an agile team to make changes to a project codebase and then quickly and efficiently test the new changes by running automated tests. 
• The result of using TDD is that agile teams will accumulate a comprehensive suite of unit tests that can be run at any time to provide feedback that their software is still working. 
• If the new code breaks something and causes a test to fail, TDD also makes it easier to pinpoint the problem and fix the bug.

Behavior-Driven Development (BDD)

Behavior-driven development testing (BDD) is an extension of test-driven development (TDD) that encourages collaboration between developers, QA, and non-technical or business participants on a software project. It extends TDD by writing test cases in a natural language called Gherkin that non-programmers and domain experts can read. 

BDD features are usually defined in a GIVEN WHEN and THEN (GWT) format, which is a semi-structured way of writing down test cases. A BDD feature or user story needs to follow the following structure:

• Describe who is the primary stakeholder of the feature
• What effect the stakeholder wants the feature to have?
• What business value the stakeholder will derive from this effect?
• Acceptance criteria or scenarios:
    Feature: Items on abandoned shopping carts should be returned to inventory
    In order to keep track of inventory
    As an on-line store owner
    I want to add items back into inventory when an on-line shopping cart is abandoned.
    Scenario 1: On-line shopping cart items not purchased within 30 minutes go back into inventory
    Given that a customer puts a black sweater into his shopping cart
    And I have three black sweaters in inventory.
    When he does not complete the purchase with 30 minutes (i.e. abandons the shopping cart)
    Then I should have four black sweaters in inventory

In TDD, the developers write the tests while in BDD the automated specifications are created by users or testers (with developers writing the underlying code that implements the test.) A key BDD best practice is to use, a collaborative approach to define the requirements and business-oriented functional tests for software products based on capturing and illustrating requirements using realistic examples instead of abstract statements.

User stories, which we described earlier, are used in agile development to help shift the focus on software projects from writing about software requirements to talking about them. User Stories aren't formal documents in the way traditional requirements are. They’re intended to be placeholders for conversations among the stakeholders on a project to get agreement on acceptance criteria for a particular piece of functionality. 

The Three Amigos

Another best practice to encourage this kind of conversation about functionality is to use "Three Amigos" meetings that involve a product owner (or a business analyst), a developer, and a QA tester, who get together (either face-to-face or online) to review the requirements, tests, and dependencies of a feature request on the backlog.

Acceptance Test Driven Development

Acceptance test driven development (ATDD) is another enhancement of test-driven development that promotes collaboration between business users, testers, and developers to define automated acceptance criteria before coding has begun. 

ATDD and TDD are complementary techniques: 

• ATDD helps describe the high-level business objectives, while TDD helps developers implement them as requirements. 
• ATDD helps ensure that all project members understand what is being implemented since failing ATDD tests provides quick feedback that requirements are not being met.

Although they don’t have to be written in the Gherkin language used on BDD projects, User Stories on ATDD projects should be well-defined. A best practice for doing this in a "Three Amigos" meeting is to follow the "3 C's" formula, devised by Ron Jeffries, that captures the components of a User Story:

1. Card – stories are traditionally written on notecards, and these cards can be annotated with extra details
2. Conversation – details behind the story come out through conversations with the Product Owner
3. Confirmation – acceptance tests confirm the story is finished and working as intended

A key part of ATDD tests is that they are run automatically whenever a change is made to the source code. In addition to testing the application, automated acceptance tests are useful in measuring the progress your project team is making since, on an agile project, working software is considered to be the only objective measure of progress.

Exploratory Testing

There are many exploratory testing methods. Exploratory tests are tests where the tester actively controls the design of the tests as those tests are performed and uses information gained while testing to design new and better tests. 

• Exploratory testing is done in a more freestyle fashion than scripted testing, where test cases are designed in advance. 
• Exploratory testing is important on Agile projects since developers and other team members often get used to following a definite process (such as those dictated by TDD, BDD, or ATDD testing approaches) and can stop thinking outside the box. 
• Because of the desire for fast consensus among self-organizing Agile teams (even globally distributed ones), collaboration can devolve into groupthink. 
• Exploratory testing combats this tendency by allowing a team member to play the "devil's advocate" role and ask tough, ‘what if’-type testing questions.

Example of Exploratory Testing

An example of exploratory testing on an online shopping website would test the following functionality from the perspective of an end-user (in any order the tester chooses): Login, List of Products, Loading and Unloading of a Shopping Cart, Payment Processing, Order History, and Product Shipment.

When is Exploratory Testing Useful?

Exploratory testing is especially useful in complex testing situations when little is known about the product, or as part of preparing a set of scripted tests. The complexity of modern applications makes it difficult for prescribed manual or automated tests to find all of the edge cases needed to fully capture your end user’s experience. Exploratory testing excels in allowing non-testers to go through an application and provide clear, captured, annotated data for your team to replicate any issues that are found.

Because exploratory testing is a time-consuming process, a best practice requires the Project Owner to budget enough time so that the tester can understand the business requirements and the purpose of the app being tested. Testers should also use testing tools and other resources that can help document any bug found clearly and concisely so that developers can recreate and fix the bug. Exploratory testing is best done in a collaborative environment where testers work closely with developers.

Session-Based Testing

Session-based testing is a type of structured exploratory testing that requires testers to identify test objectives and focus their testing efforts on fulfilling them. Test sessions differ from test cases in two ways: First, more than one test can be carried out in a single session. Second, like test cases, test sessions can tell you who tested what, but they can also tell you how, when, and why the tests were done.

Pre-written test scripts can be run during a session, but because test sessions emphasize test objectives over specific test cases, testers are encouraged to create and execute more tests based on what they've discovered and learned. This type of exploratory testing is an extremely powerful way of optimizing test coverage without incurring the costs associated with writing and maintaining test cases.

Formal Session-Based Testing

Formal Session-Based Testing uses a charter, which is a goal or agenda for a test session that is created by the test team before the start of testing from a specification, test plan, or by looking at the results from previous sessions. An example of a charter for session-based testing of the shopping cart application (the example used in the Exploratory Testing section above) would be: “Test adding items to the shopping cart.” Since a session is an uninterrupted period spent testing, typically lasting one to two hours. session-based testing is an ideal way to test the performance and functionality of your apps, i.e. how long it takes to add or remove items from the shopping cart or the speed and accuracy of calculations performed.

Best practices for Session-Based Test Management include setting aside uninterrupted time for a session and limiting the session to just one charter (charters should be prioritized and can be broken down into smaller logical units that can be tested in separate sessions). 

• It’s also important to do a session debriefing, which is a meeting between test leads and the testers who completed the session. 
• Debriefing topics should include what happened during the session, what was achieved, challenges faced, pending issues, and feedback from the tester to improve future sessions.

Agile Testing Tools

(Agile Test Automation Pyramid)

Outside-in versus Inside-out Testing

When following agile testing best practices, agile teams generally follow one of two approaches when it comes to testing their applications, either outside-in or inside-out. In the outside-in approach, teams start by focusing on the end user's perspective and attempt to describe high-level desired functionality and goals for the software under test in the form of user stories. In every iteration of Sprint, user stories are refined until the agile team and the Product Owner/Customer Representative can agree on the acceptance criteria, which determine that a User Story works as planned. Testing then goes 'inward' and code is written to test smaller and smaller components until you reach the unit-test level.

In the inside-out or bottom-up approach, agile teams start with unit tests using JUnit or other unit testing frameworks (collectively known as xUnit) at the lowest level of the Test Automation Pyramid (see figure above). As the code evolves due to refactoring, testing efforts evolve as well as the team moves upward to acceptance level testing, which tests business logic at the API or service level using tools like SoapUI, Fitnesse, or Cucumber [Cucumber runs automated acceptance tests written in the behavior-driven development (BDD) style described above.] The top of the pyramid and the last thing tested is the user interface (UI), which can be tested using tools like TestComplete or Selenium.

Inside-out and outside-in are different but complementary approaches to testing. Software quality control relies on the related notions of verification and validation (V&V) that check to see that a software system meets specifications and that it fulfills its intended purpose. The terms verification and validation are often used interchangeably but have different meanings:

• Verification: Checks the software concerning specifications ("Is our team building the code right?")

• Validation: Checks the software concerning customer's expectations ("Are we building the right code?")

On modern agile testing projects, validation and verification steps overlap and take place continuously since agile team members must engage with customers and other stakeholders throughout the project-- to do things like prioritizing bug fixes and enhancements on the team's project backlog --and not just after a separate test phase at the end of the project. This requires effective communication at all levels of the business since team members need to understand what features need to be built and who needs each feature.

Limitations of Spreadsheets

In addition to specialized tools for agile testing, agile teams can benefit from test management or test case management tools that can help manage their testing efforts, record test results, and generate reports that provide useful feedback to testers, developers, and project leads. Many agile teams often rely on a spreadsheet application like Microsoft Excel as a test case management, documentation, and reporting tool. There are significant risks to using spreadsheets to store and process test cases, however, especially on multi-team projects where individual teams often adapt spreadsheets to their specific needs, which can cause problems when it comes to getting uniform reports. If two or more people are working at the same time on a spreadsheet file, there's also the danger of corrupting the file or creating other security risks. This is one of the many reasons that, as agile teams and projects get larger, they’re likely to switch to a test management tool that provides a central repository for all their testing efforts.

Test Management Tools

There are a variety of test case management tools available for different needs, with both cloud-based and on-premise installation options. Some tools like Zephyr (Standalone Edition) and HipTest come with integrated issue and requirements tracking, while other test case management tools are integrated with popular issue tracking tools such as Atlassian’s Jira or Bugzilla (both Zephyr and HipTest provide Jira plug-ins).

Agile Test Automation

Agile automation testing works on agile projects by running a large number of tests repeatedly to make sure an application doesn’t break whenever new changes are introduced—at the Unit-. API- and GUI-level. For many agile development teams, these automated tests are executed as part of a Continuous Integration (CI) build process, where developers check code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect errors and conflicts as soon as possible. CI tools such as Jenkins, Bamboo, and Selenium are also used to build, test, and deploy applications automatically when requirements change to speed up the release process.

Test automation allows agile teams to execute more tests in less time, increasing coverage and freeing human testers to do more high-level, exploratory testing. Since automation test scripts are reusable, they can be used to do more comprehensive testing by testing repetitive steps with different data sets, such as those for cross-browser or cross-device compatibility.

Among the risks of automation are those related to the need for version control and maintainability of test scripts and test results. Choosing the right automation testing tool is critically important since you want to avoid ones that are incompatible with other software testing tools in your test environment. Once you’ve got a test automation tool that works well with your other testing tools, automated tests should also be run regularly to provide continuous feedback about the health of your entire system, preferably through a continuous integration approach described above, as opposed to manually.

The Best Agile Test Management Tool

There are a few best agile tools. However, the best agile test management tool is one that enables agile teams to work collaboratively in the different areas described above--Continuous Build, Continuous Integration, Continuous Delivery, and Continuous Deployment-- to speed up the release of high-quality software. Doing that effectively means it should be able to integrate with other project management, issue tracking, and automation tools in your agile development and testing toolchain. It also should have live reporting features since you need to maintain real-time visibility into the products in your software delivery pipeline. This is important so that information about bugs, inefficiencies, or other issues can be shared and acted on in real-time. If you're a small or medium-sized business, it also helps to choose a tool that can scale to fit the needs of an enterprise or very large enterprise team.

As agile methodologies evolve, testing becomes even more integral to the software development lifecycle. Embrace these agile testing practices to ensure your software consistently meets the highest standards of quality and reliability. Don't let quality be an afterthought—engage with our QA experts today to streamline your Agile testing strategy