Reflections on AEP Testing Process

Hello everyone, my name is Ron Evans, and I’m a technology consultant who has worked on various levels of the design of AEP in the past, and recently I was brought back on board to lend a hand in the software testing process. My recent role included leading the interview process of early users of the software, which is always a fascinating experience.

The initial experience that a first-time user has with new software is an incredibly useful event to witness and record. And it only happens once for each person — after that, they are already somewhat aware of the software and how it works. So, user experience testing at this stage is designed to gather the most amount of feedback from users, on all aspects of their experience — things like their thoughts on the look and feel of the software, their expectations on what it should “do,” and how effectively it “does” that at this point in development. Even people’s facial expressions are important if you can get them — they can tell you when someone “doesn’t get it” right away, and how long it takes them to figure “it” out. It’s incredibly interesting from a software design perspective, but even more interesting from a psychological perspective.

For AEP, a number of individual artists and small organizations were invited to participate in this first round of testing. They were given the url and password to the AEP demo site, and asked to create an account, login, play around, create some content, and generally kick the software around a bit and see how it worked.

The testers were split into two groups — one group was given detailed instructions on how to login and do all of these things, and the other was given almost no information. This A/B test was designed to show how easily understood the functionality of this early stage of development was without training for the “B” group, vs. the “hand holding” of the “A” group.

After a set testing period, I contacted the testers individually, and scheduled interview times to meet with them and discuss their experience in AEP. The AEP staff and I had designed a number of questions to ask the testers to help them describe their experience in a way that would create useful feedback for the AEP developers. There were 15 questions, including:

* What do you think AEP is now? What is it being designed to accomplish?
* How do you hope those goals and functionality will fit into your organization?
* What did you think about the look and design of the site? How does it compare to other sites you frequent?

The feedback from the testers was very interesting — it’s remarkable how certain people are very focused on one aspect — say, look & feel — while others are totally focused on some other specific aspect. With the variety of testers we were able to interview, we received a healthy view of how these early users view AEP, and that info will be used to improve the product into the next testing stage.

Sharing a product at an early a stage is always somewhat anxiety-inducing for any developer. But the feedback you receive can offer the potential for detailed levels of course correction that would be too costly to discover later on, as well as validation that the choices you’ve made up to this point in development were ultimately accepted and understood by your users. I’d like to thank the initial testers for their dedication and great feedback, and I look forward to working with them in future testing stages!


Ron Evans, Principal
Groupofminds.com Arts Marketing Consulting

Oct. 30, 2010, Filed under category Program info


Comments are closed.