Alright. So it’s been a week since I was given the chance to be “test manager” for our small project. I spent some time cataloging my reactions, and figured it would be interesting to others. I’ve written them as a play-by-play, to best describe my reactions as I had them. This is a long blog post — almost too long — but I think the detail is worth it, both for me and for anyone who’s interested.

Click “Read More” below to view the full play-by-play.

Sunday, Day 0

1. I started out using SBTM as a model with hopes that we could be the first open-source company — or company in general — to release a large set of SBTM session notes.
2. I wrote a proposal, which detailed a scaled-down version of SBTM with less bureaucracy (ie., metrics) and less note-taking. As I was writing, I realized that the hopes in #1 would not come to fruition — at least, initially — as those hopes were secondary to the main goals of the test team. [Arguably, these were not goals at all, but held the possibility of becoming goals later].
3. In the proposal, I emphasized debriefing as a way to get the information I needed to give to stakeholders. Note taking was de-emphasized — though not totally wiped out — as I was pretty sure intensive note-taking would not go over well in my more laissez-faire environment.

Monday, Day 1

1. Development wasn’t ready for us to test what we needed to. Although my initial reaction was to say “Give us anything you can; early and bad is better than late and slightly less-bad,” I held back the urge as it seemed pessimistic (though likely true). Also, I was new.
2. I spent some time talking with our de-facto release manager about our weekly test strategy leading up to our big release. I also read up on some of the Bach brothers’ SBTM papers, as well as some from James Lyndsey, and others, to make sure I was on top of my game.
3. I spent time doing other work that didn’t apply to this position.

Tuesday, Day 2

1. I was given my testers. I had initial talks with them about the role of the test group, and from what perspective we were doing our work. To my surprise, they were very accommodating. [I've been told that developers often see testing as a side-task, and so I was reassured in the goodness of my organization when I saw the enthusiasm the testers brought to table].
2. To give the testers a feel for the process, I let them roam around on the fun tasks before getting down to business. This was, say, breaking into things hacker-style as opposed to exploring the more mundane parts of the system. [How do you teach someone -- or at least, communicate -- the importance and usefulness of exploration without giving them something fun to explore?]
3. At the same time as #2 above, I talked with stakeholders both formally and informally to go over their thoughts on the process. There was little expectation of the “how” of our testing — they were leaving that up to me — and more of an expectation on the result. As a testing group, we were supposed to find functional bugs (and comment on any “higher level” bugs as we saw fit), reporting our findings through a bug tracking system. There was no talk of metrics. [Many of these meetings -- even some of the formal ones -- were over smoke breaks. I don't smoke.]
4. The testers finished their first session, and we went into debriefing. I got two separate reactions, one from each tester. One tester took overwhelmingly long notes, giving me everything from the things that passed to the things that failed, in chronological order, with personal reactions every step of the way. [I was amazed. These notes were awesome, but I wasn't sure what to do with them.]. The other tester gave me very direct notes, focusing specifically on bugs. There were no reactions, and no hint of happy paths tried and succeeded. [I was still amazed. These were long notes. This tester totally beat up a problem area in our software, and had many bugs to show for it.]
5. After the debriefing was over — and after they went over the problems that needed reporting — I tried telling them about “proper bug reporting” practices [This, in some respects, made me feel like a process-weenie in a management position. Do I need to convince my organization of the value of these practices?]. These were the practices I learned in school, which I personally strive for when writing bug reports — I figured they applied here. I did my best to communicate how I do things, though I stumbled a bit as I tried expressing the things that have become rote and habit. [Yes, I do think they apply. How far should I push this, and to what extent am I only pushing formality?]
6. I let them explore more of the software, pulling in a new charter. This time, I tried communicating how important this charter was to stakeholders. I got kickback from the testers as they didn’t see how, for instance, joining the site was more important then preventing hackers from executing denial of service attacks, or somehow letting malicious users take over our software. [Uh oh. I've let them have too much fun. How can I communicate non-obvious priorities?] We talked about this for a while, and after all the other events of the day, they eventually didn’t get around to starting a second charter.
7. Right before it was quittin’ time, I was told that stakeholders wanted a recap of the bugs we found over the course of each day. I complied, and sent them an email with bug ID numbers and some short text regarding severity.

Wednesday, Day 3

1. I decided to start this day off with a quick priorities meeting to try and reiterate the needs of the stakeholders. I didn’t feel like our previous talks communicated that joining the site — all X number of ways to do it — was the main priority for the first week. [Stakeholders wanted this solid. I had to keep harping on it.] I was afraid my testers would think of me as a dictator, as if I was simply “telling them what to do.” In some respects, I was, but they understood and were very accommodating. In still not trying to be the dictator, I told them that while in-session, keep a look out for new charters that could be interesting. [I wanted them to feel involved; it's their job too, ya know.]
2. I sent them off on some joining charters and they did really well. They found bugs that I totally didn’t think of [The beauty of multiple testers.], and were modeling out the system without me suggesting modelling as a heuristic. Although they did a great job, they did something I didn’t expect: They skipped debriefing. Instead of waiting until debriefing to go over what bugs were found, how to report them, and when to explore in more depth, they simply just started reporting them. [They circumvented me as the manager, the information-gatherer, the advisor. This is the second day of real testing. Where else could I inject areas for review?] I took it in stride, and listened to their comments. They thought it was odd to take notes — especially long notes — and it felt more natural to simply report them. I’ve heard that one of the Bachs [Or maybe it was Michael Bolton.] was able to coach their testers in how to take notes that don’t “get in the way.” I wasn’t sure how to do this. As well, I had no business-case for why we should take these notes; I knew the SBTM model said so, but why should we take them? I decided to scrap formal note-taking in favor of more detailed debriefings. If they weren’t going take notes, then I was going to record what was important to me. After all, I was the one directly reporting the information to stakeholders. Who else were these notes for?
3. I got wind that one of the stakeholders was actually reading our notes! [They were public, after all.] This wasn’t really a problem — at least, in a large sense of the word — but none of these notes were formatted for the stakeholders’ eyes. From the outside, they were gibberish. They were long reactions. Hints at bugs. Chronological actions that may or may not be relevant. They were nothing final, and were nothing — on their own — important. Formatting these notes would take too much time for me [I had no idea how they should be formatted, and I wasn't going to reformat my testers' notes]; they’d also take too much time for my testers [Why should they be focusing their time on formatting, when they could be focusing it on testing?]. Since there was no business case for these notes — none visible, at least — and there was no reason to create extra work for ourselves, I decided to tighten the channel in which stakeholders would receive information from me.
4. I ended up talking to a stakeholder about my email reports, and thought that the information I was giving them — the tightened version — would be suitable as a blog post, possibly informative to the rest of the organization. This idea turned out to be a hit, and was the impetus for my blog posts on our “internal” operations blog. [This blog's not really internal; you can keep track of our posts at your leisure.]
5. The testers finished their second session, and we went into debriefing. Since they weren’t taking detailed notes, I had them CC me on all their bug reports. [I had to stay updated somehow.] I had them go over their bug reports with me, and I found there was still some coaching to do regarding “How to write a good bug report.” I decided to elicit the advice of Elisabeth Hendrickson, as she had a great article that was right at the top of Google. [They really liked this, and it came across as skill-building.]
6. Since some things I learned in school have become habit for me — and, since I can’t possibly argue the many sides of testing for which I’ve had little experience — I decided that, in order to channel some of the enthusiasm of my testers, I’d create a resources page they could look at during their own time. This worked like a charm. One tester, during lunch at their workstation [It happens sometimes, though mostly voluntarily], decided to read from the resources page. They loved what they read, and wondered where they could get more. The software-testing list was brought up, and the tester sent me an email as a reminder to get them signed up.
7. After debriefing and some coaching on bug reports, my testers started contemplating which charters to do next. Since I had mentioned earlier that they should be part of the charter-creation-process, one tester happily jumped on a “Pretend you’re a hacker” charter the other tester had created. Although the charter was a good one, and good for when security was our highest priority, it wasn’t something the stakeholders wanted at that very moment. [Oh no! He volunteered for the fun one. "Dictator, dictator!" What do I do now???]. I decided to double check the importance of security-related charters with the closest stakeholder I had on hand. As I expected, they weren’t as important as our joining charters. I asked the tester if he could instead pull a joining charter — not to ruin the fun, and all — and I reiterated that the joining charters were the most important. The tester calmly chose a joining one [I love my organization.] and tackled it much like the one before.
8. I spent the rest of the day writing up a blog post to communicate the events of the day. I focused mostly on bugs, but thought it was important to communicate the number of duplicates that were found and re-ticketed [Bug tracking systems aren't the best at recognizing duplicates; the search functionality of the one we use doesn't help much either.].
   

Thursday, Day 4

1. I started out this day with another meeting, but this time to re-emphasize priorities, and go over how we, as a team, want to create charters. Previously, we were using a wiki to manage our charters and our sessions [Horrible choice, but we were dog-fooding our software; it seemed a good idea at the time.]. Aside from hitting the points that I wanted to hit, we decided to use our bug tracking system as our charter management system. We came out with something like this [Voluntarily written by one of my testers, without any hint of expectation from me.].
   
2. This day, weirdly, was uneventful. Other than getting good bugs from my testers, good debriefings, and some awesome test ideas, things went pretty smoothly. I did ask my supervisor if she’d order some testing books for my group [Where did my college textbooks go? I checked once I got home, and couldn't find a single one.]. Hopefully those will come in before the testing group disbands in March.

Friday, Day 5

1. This too, was an uneventful day. I finally got around to getting my tester signed up on the software-testing list; aside from about one to one-and-a-half sessions today [It was Friday], there was nothing too exciting.

Reactions

• I am totally amazed at how well my testers took on their new role. I expected developers to be stereotypical-developers, and to resist testing either consciously, or subconsciously. No resistance was apparent here; they ate it up, and from my new-to-management experience/expectations, they did a great job.
• Although I like to think I’m in a test-management job — which, to some degree, I am — I’m mostly in a position of working with my testers, and not simply telling them what to do. I do more managerial tasks, say, communicating with stakeholders, but we’re all working together. This is a good perspective, as I would hate becoming — and working for — a pointy haired boss with his own office and little involvement.
• SBTM was a good model to start, but it’s too much bureaucracy for our small project, and in our short time frame. Instead of going full on SBTM, we’ve evolved into semi-structured SBTM, holding onto sessions, charters, and debriefing, and getting rid of detailed notes and possibly-informative metrics. Formal reporting is focused completely on the bugs and duplicates that were found — as that’s most important to developers — and happens daily through internal blog posts. Informal reporting happens at irregular times (such as during smoking breaks), is less structured, and is focussed more on the qualitative aspects of our process.
• The “charter” idea has really hit home with stakeholders. Through their own doing, they’ve come to view our process as “making strategic passes at charters.” For instance, we attack the charters that are more important first, and our first attempt at attacking each charter they’ve called our “first pass.” This gives the impression that we’re covering the important areas (which we are) and covering these areas multiple times, without following strict test-cases and test-scripts that remove the tester from the act of testing. [My testers really liked that they could use their brains at will; they wouldn't react well to me telling them exactly what to do, in a micro-managerial way.]

Note that this is only the first week, and things are bound to change; even so, it was useful to re-cover my decisions, explain them coherently, and reassess the decisions that I made. This was a good play-by-play, and if time permits, there may be more next week.

UPDATE: Continue on to week two. >>