Iteration is one of the cornerstones of agile development, and of any agile team.

There are some key benefits to using an agile methodology to deliver a digital product; chief among them is that agile prioritizes client and user involvement. When you’re trying to introduce new ways of working, (which is one of the main tenets of the Code for Canada fellowship) client and user involvement is hugely important.

How We Work

We work in two week sprint cycles. Every two weeks, we deliver a piece of usable software that can be tested with real users. We run product testing during each sprint. This helps us check our work, and make changes based on real feedback.

One of the biggest challenges we’ve faced during this project is having to push back against traditional waterfall business requirements. Working agile means working in small, fast iterations. This is a scary adjustment, but it’s also a huge mechanism for risk assessment and reduction. Even though it often feels like we don’t have a “plan” or answers to tough questions, it’s because we’re trying to be flexible enough to pivot our build if we need to. You can’t do that in a waterfall environment.

One method we use to get closer to knowing what to build, is product testing.

Product Testing 101

When people ask us why we do product testing, we tell them that it’s the best chance we have to make sure we’re building something usable, and in-line with user needs. In the case of the eMIB, while there are a number of important specifications on the content side, because it’s an interactive simulation, it’s critical that the testing platform itself be easy to use.

Product testing helps us:

• Validate design choices with real users
• Get feedback on the product while it’s being built, not after it’s finished
• It’s a reminder that you are not your user
• You can test assumptions and different designs, to see what works best

We want to create an experience where users are able to focus their energy on succeeding in the test, which means reducing the barriers to onboarding as much as possible.

In some of our testing sessions to date, we’ve learned that our notepad feature was too narrow; that the navigation wasn’t compatible with a screenreader; that we needed to provide a way for people to zoom in on the org charts; and that people weren’t clear on where they were supposed to go once they finished reading the instructions. These valuable insights have helped us build better iterations of the eMIB that show measurable improvements over early versions.

For example, look at how much our notepad has changed since our first iteration:

There are a few simple steps to follow to test a product.

1. Get a prototype ready. You can test on something as lo-fi as a piece of paper, or in an application like Figma, or live in code, which is primarily what we’ve been doing.
2. Write a script. Have an idea of what you’re testing before someone comes in. For example, our early tests for the eMIB focused only on how fast people were reading the test instructions and background information. Currently, we’re testing subtle feature changes to the inbox functionality. In each case, we ask the user to take on the persona of a test taker. We time them, and put them in a quiet room with the test full-screen, to best simulate a testing environment.
3. Metrics. It’s important to know how you’re going to assess the results of your test. Are you looking to reduce clicks? Improve time per page? For a handful of our testing sessions, we measured how long it was taking people to read the background information, and whether improvements to the onboarding was helping decrease the time. That gave us a clear idea of if those changes were working, because we wanted the interface to be as easy as possible for a new user.
4. Analyze. The most important and most fun part of product testing is learning something new from an actual user. If you’re going to solicit feedback, it’s important to put that feedback together, find the common issues users were experiencing, and make a plan to improve on the product.
5. Iterate. Now that you know what needs fixing, get to it! You might want to run a product testing again on the same features with your new changes, or do some A/B testing to see which change performs best.

There are lots of great resources out there on testing your product. We think it’s a really important practice that should be introduced in digital government, and hope the PSC continues to after our fellowship term ends.