“There are more things in heaven and earth, Horatio, than are dreamt of in your philosophy.” — Hamlet, Act 1, scene 5.

Oh the eternal words of Shakespeare. His works still apply today, and his words span the world over. You’ve heard him quoted before. If only he knew how I’ve quoted him today.

I’ve been doing some thinking. Not just the everyday thinking — not about what to eat tonight, or what I’m going to wear tomorrow (see, I rarely even do that). But the kind of thinking you usually do when you’re stuck fifty feet in a well, and all you can do to entertain yourself is knit. Except you don’t like knitting. So you let your imagination run wild.

My imagination lately has been running around Agile culture. The good, the bad, the not so good and the not so bad. The main takeaway, in the whirlwind of possibilities and dot-matrix printer-like results, is a question: Is Agile a culture of confirmation? And is this okay?

Micheal Bolton expresses his views on confirmation in his blog post here. In general, his view is that confirmation is the simple act of checking the correctness of known assumptions about a piece of software, while testing is the act of garnering new and useful information. I’ll admit, when I first wrote this blog post I hadn’t read Micheal’s, and I had my own ideas, but he expresses those ideas much better than I could. This post, then, is a bit of spin.

I was calling confirmation verification. I was redefining the term widely used in testing, and describing it colloquially as making sure stuff works and does what’s expected. The inverse then, and the million dollar question and what I care most about, is, what — and who — is making sure that same stuff doesn’t do what it’s not expected?

Now, this is a bit of a simple description, but let’s get specific, and bring this to Agile. Agile has standups. It has TDD, it has retrospectives. It has championed “the iteration”, continuous integration, and the utility of scripting languages. It’s also reduced the need for extensive design documentation, which is my personal favorite, and it promotes, at least in its marketing, very close interaction with the customer.

And let’s be clear: All of this is good stuff, and it’s very valuable. But there’s something missing. A hole. A way of thinking that isn’t quite included in the hype, and in the massive jigsaw puzzle of software development philosophy, there’s a piece yet to be filled.

The piece, of course, is the unknown. Continuous integration, TDD, unit tests, large suites of functional tests, etc., can only verify that the software is working. It can tell us when things stop working — which is perhaps its greatest strength — but it can’t tell us anything about the problems we don’t write tests for; the problems we aren’t aware could be problems. Sometimes these are user interface issues. Sometimes these are product oversights. Sometimes these are problems we think our automated tests cover, but don’t actually test. There could be any number of reasons. But if these problems are left unfound, they make it to the customer’s screen. And this, I think, is where much of the trouble lies.

The Agile philosophy and marketing material I’ve seen says nothing about detailed, investigative testing. Instead, it promotes what I see as rigorous confirmation using primarily automated testing practices. These practices verify the happy paths and the not-so-happy paths that we can think of at the time of writing, but they miss the things we don’t know. As well, when it’s time for the customer — the product manager or the client, not the end user — to review the software for correctness, they generally focus on their positive assumptions of functionality rather than focusing on the unknowns. And I think this is important. Because it might be my Kaner and context-driven upbringing, but the only techniques I can think of to help us discover what we don’t know revolve around exploration and manual testing. And an internal sense of curiosity. I just can’t think of any other way to learn.

In that respect — and a tip of the hat to those who gave me much appreciated feedback — I’ve learned from a previous post about dating that I need to state my conclusions more clearly. Here, I intend to be clear. Resisting another quote and some iambic pentameter, or some vague call for idealism, the moral of this story is: Agile practices are useful, have value, and I appreciate them. But in many cases, they can be made better when bolstered by a team of interested, investigative testers. It’s a matter of managing our unknowns, and I think the culture should embrace that.

Now — to end this soliloquy. Because the dot-matrix printer just died.

Fin. Exuent left.

Kurt Schrader, otherwise known as “The Shrade”, “Shradester”, and my boss, recently wrote a post about how we roll at Intent Media. And by how we roll, I mean how we use Cucumber to power automated functional tests of a large scale ad platform. Oh ya, that.

When I first mentioned Cucumber to a close member of the context-driven testing community, the reaction was “Oh, so you’re just entering manual scripted tests into a computer.” She’s South African, so it was more like, “You’re ent-ring scripted tastes into the computah!” (Ya, she knows who she is.)  Though I felt her pings of discontent quickly, the answer was — “Well ya, I guess…”

In the tools world, Cucumber is the blood relative of both your normal manually-scripted tests and, say, FIT or FitNesse, with a bent toward integration tests (if that’s what you use it for). It’s manual scripted testing because you type your manual scripts into the computer; it’s automated testing because you also define pieces of code that match up with each line of each test. All Cucumber does is find the right match — and then happily goes chug, chug, chug.

It’s been a fun ride so far, and I don’t have many complaints. There is a question about whether to write imperative or declarative steps throughout your tests, as well as where to put your implicit state (we leave it in the browser). But as Kurt mentioned, we’ve been highly successful so far. I’d be interested to see if we have the same troubles that plague manual scripted tests over the long term, or if automating them keeps them in the public eye (i.e., continuous integration = continuous test fixing). We shall see.

In any case, it should be noted that Cucumber isn’t our only line of defense, and part of the reason Cucumber works is because of everything else going on around it. We have PMs doing acceptance testing, devs doing TDD and unit testing, I’m leading exploratory testing (which arguably the PMs are doing as well), and Cucumber is backing us up with integration tests that both the devs and I write. And that’s only functional: We have performance testing, user testing, smoke testing, and the whole nine-yards going on elsewhere throughout the company.

So ya, I’m proud to say that’s how we roll. If you’re interested, I’d encourage you to follow Kurt’s blog for more updates about the technical aspects of Intent Media, and to see how we’re building “the next great online advertising startup” (TM).

I’d like to express this articulately some day, but for now I’ll just stand on the shoulders of greatness — and then correct him:

Given enough eyeballs, all shallow bugs are shallow. — Linus Torvalds Tim Coulter

Maybe it’s an economy of scale thing. e.g., When is there ever enough competent, manageable, able-to-fix-your-problem eyeballs?

I saw Star Trek today and was given a little treat as I walked into the theater.

I used Fandango. I bought the tickets online, and then printed them using Fandango’s machine stationed in the movie theater lobby. Instead of getting the usual ticket and receipt, I was given this little extra:

It was a ticket for Watchmen! Not only is Watchmen out of theaters, but the date the movie was intended to be shown was March 9th. If you look closely, on the bottom of each stub, you’ll see a printing date of May 29th — way past the showing date.

So the question is: Did Fandango’s machine erroneously print me an extra ticket, leading to this little tidbit? Or did I pay for a ticket I never picked up?

The world may never know.

Not only does the Google Earth plugin let me know it’s not supported in my environment, but it provides me with a link to more information. All in all, I think it’s beautiful.

My coworker said it’s like applauding the chef for washing his hands, as if there’s an expectation that all software should tell you such information. To me, it feels more like the chef personally apologizing for not having the linguine — and then telling me when to come back later!

More importantly, it’s almost as if I have a relationship with the company.

When I see error messages like these, it presents a level of polish that even on my first impression suggests a quality product. As a user, I appreciate it.

It’s beautiful. Take a look.

I got to it using Fandango’s search box in the top right hand corner twice in a row. The first time I searched for something that returned no results (try “Court St.”, as if you want to search for a theater on Court St.). The second time I used the box to search for something else, like “Star Trek”.

Not a huge error, but I’m always a fan of professional error messages.

Saw this once and had to reproduce it. It’s completely random, but it showed up after viewing a couple tags in Shazam:

On the heels of James Bach’s Buccaneer Scholar, a new idea, book and blog where he details his experience with exploratory thinking and learning, comes my own wrestling, not only with how I learn, but what I should learn. Due to recent events I’ll make clear in a few weeks, this question has begun to capture all of my spare CPU cycles — and it won’t quit.

The crux of the question, I think, is this: When I was thirteen, I set a goal to go to college, get a degree, and work for Microsoft (then the most popular software company in existence). While in college, I latched onto open-source software and the free software movement — happily ditching Windows for Gentoo, Mandrake, then finally Ubuntu — and gleefully traded Microsoft for a slightly newer and more exciting company called Google. Google had free food, funky office chairs, and they fit the goal I previously made when I was thirteen. My sense of self was intact, no harm done.

When it came time to interview, though, things were messier than I expected. I was still trying to finish school — to graduate, move out of the dorm and wrap up all the extracurriculars that consumed most of my spare time; and I was, on the more personal side, simply trying to grasp what was happening as I made the transition from college life to real life.

It was all too fast. Not only was I not accepted by Google (that’s a story for another time), but I didn’t know where else to go from there. In magicians terms, I made all but the prestige of my childhood goals, and I was left like a confused audience member contemplating whether I should stand up and applaud.

But, prestige aside, I realized my longstanding goals ended after college, and whether or not I was employed by Big Popular Company mattered little to anything but my ego. Even if I did achieve my goal, I’d still be left with the same question: What happens next?

When I learn, I need to know the big picture. I like to know where I’m going, why I’m going there, and in general, how to get there — but not always. I know the devil’s in the details, but I like figuring out the details as I go along.

Looking back, I had more or less achieved my goal — college, degree, employment — but I didn’t have the big picture telling me where to go next. I still don’t. I’d assume even James’ Buccaneer Scholar, with his exploratory state of mind, would have some type of charter, though I can’t believe he’d always know where he is going. James, I’d like to hear you expound upon that if you haven’t already.

This leads to the more practical dilemma: Of my interests, which path do I choose? The following is a list of career paths I could be interested in taking. Though I run the risk of taking the totally incorrect approach to, well, life, of these career paths, I can’t help but be interested in their prestige. Here goes:

• Independent test consultant.
• Product Manager/Test Manager for large software engineering projects.
• Ph.D in Software Engineering or Cognitive Science, researching and teaching software engineering and/or software testing.
• Software Developer for cutting-edge, Web 2.0 technologies (I am that now).
• Entrepreneur for my own software development company. (I’ll probably need an MBA).
• Entrepreneur for my own software testing firm. (Again, an MBA).
• Tech Journalist for popular blogs and magazines. (I’ll probably need to study journalism).
• Politician. (I’ll probably need to study political science, though this brings me to my next interest…)
• Something in the arts, say, acting.

And as an aside, this whole process feels like one big game of Twister.

James Bach just started a new blog called How I Learn Stuff where he depicts the life of a Buccaneer Scholar. I’ve yet to fully understand his use of the term, but the posts and comments there are quite moving. This is likely a blog to watch.

A recently-made mockup of The Open Planning Project’s website, released only within the company, shows an about page with a sub-heading and description that says we as a company are “driving best practices.” I haven’t made my distaste for this phrase known yet — and by no means is this post an announcement, though I wouldn’t mind if they read it; but I am a bit surprised we use the term. Though some have said it better (well, one at least), here’s my impression of the phrase and how it appears to be used in our context:

1. First, I agree fully with James Bach. Bias aside, best practices are like best friends: You never know when you’ll have a falling out. That’s sad, I know, but since I was little, I never wanted to choose a “best” friend because I thought it devalued the qualities of the other friends I had. Perhaps I’ll choose one when I’m old?
2. Through conversations with coworkers, most on the programming side, there seemed to be this implicit understanding that “best practice” really meant “the best practice we know of right now.” This still doesn’t pass the context argument, but there seems to be an understanding that “best,” here, is not absolute over time… though it sometimes seems to be touted as so. There also seems to be this notion of, “Most bridges are built in way X, and therefore, way X is the best practice.” This might actually hold some weight in the development side of software engineering just as I assume the phrase holds in some contexts within structural engineering — I mean, let’s not forget Tacoma Narrows. But in testing, I’d assume stakeholders’ interests (among other variables) are too fickle and disparate to choose one “best” way of managing them. (I’d love to hear more from structural engineering folks to see how this phrase is used, if at all.)
3. In my company, “best practice” feels like a marketing term. This is one of James’ arguments. If the quality of the software produced is a measure of the practice (this isn’t always the case), I wouldn’t say our practices are “the best.” That said, if there’s truth in advertising, maybe there’s a context — perhaps a geospatial one, an area where TOPP excels — where we really are leading the practice. Does that mean we’re the “best”? I don’t know.

I talked to the CEO of an optics-based software company recently, and he described his product not as the “best” product on the market, but as the best for a certain context — in his case, the high-end precision optics market. Not the low-to-mid end, where software is of lower quality but is much cheaper. I felt his description and use of the term “best” worked because he gave a context, though he was sampling over a single variable, quality, which to each person is fairly subjective — that is, if you subscribe to Jerry’s view, described here.

As humans, though, we seem to place the label “best” by ignoring variables we think are uninteresting, all with the intent of rallying behind a common leader. At one point, the software industry was led by IBM. After that it was Microsoft. Now it’s probably Google. Which is the best, you ask? That’s like asking which is the best car company. Maybe what you’re really asking is, “Which company hasn’t turned into an 800 lb. gorilla?”

I’ve probably belabored the point already, but the word “best,” in my opinion, is simply one person’s perception of the environment in which things exist, sampled at a certain point in time. They choose variables that they’re interested in, explicitly or implicitly, then find a leader amongst those available. What they don’t take into account when assigning their label is the concept of time, or that over time perceptions change. Or maybe they take time into account, but the “best” label simply doesn’t hold as things change. And we all know how fast the software world changes…

But again: The variables they sample over may not have value to others, even if the label holds, for them, over time.

My last analogy, simply because I had to hit home while making a cheap shot at Britney Spears, is that maybe the concept of “best” in “best practice” is just as fickle as “’til death do us part” in 50% of American marriages. Perhaps we assign the term too quickly. Or, maybe, we’re just looking for social status in a world that rewards that sort of thing.