Tuesday, March 10, 2015

Mobile Testing - Why is my hair on Fire!

This is a presentation I gave at my local software quality users association, SQuAD in Denver Colorado.  A link to the PDF is Mobile Testing -Why is my Hair on Fire.

Thursday, May 8, 2014

My StarEast Presentation

Just finished up giving my presentation at StarEast testing conference today.
I must say, best testing conference I have been to, enjoyed the conference and the people.

Had standing room only!

Check it out at DevOps:Where in the World is Test?

I welcome stories and comments.

Thursday, September 26, 2013

DevOps: Where in the world is Test?

Last week we had our SQuAD 2103 software testing conference in Denver, CO.  Our line up included Key notes from Julian Harty (Master Mobile Guru) and Theresa Lanowitz (Software quality industry Guru) plus talks from Lee Copeland and Lisa Crispin.  It was a star studded line up for only $400.

I presented on the topic of DevOps and how a test/quality organization fits into that model.  The presentation starts off with some definitions of what DevOps is and how it is really the "practice" of the "principles" of the Lean Software movement. 

"DevOps: Where in the world is Test?"
My first thought when I heard DevOps, was that of 2 "back yard brawlers" battling it out in a no holds barred fight.  Later I heard terms like collaboration, discipline, and agile with Ops.  But I still hadn’t heard the word I was looking for <test>.  It started to feel like two sumo wrestlers with no referrer watching over them.  Finally,  it wasn’t until after I did my research and implemented a successful DevOps QA team that I understood,  how the “ritual” would happen and how test would continue to be an extremely important part of the overall process, (i.e. the Referee ensuring that Dev was still playing on a level field with Ops).

So what is DevOps:  It refers to the emerging professional movement that advocates a collaborative working relationship between Development, QA and IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience and security of the production environment.

It became common placed around 2009, as the convergence of numerous adjacent and mutually reinforcing movements:
  • The Velocity Conference movement, especially the seminal “10 Deploys A Day” presentation given by John Allspaw and Paul Hammond
  • The “infrastructure as code” movement (Mark Burgess and Luke Kanies),
  • the “Agile infrastructure” movement (Andrew Shafer) and the Agile system administration movement (Patrick DeBois)
  • The Lean Startup movement by Eric Ries
  • The continuous integration and release movement by Jez Humble
  • The widespread availability of cloud and PaaS (platform as a service) technologies (e.g., Amazon Web Services).
Well you might be asking yourself, Why not Lean instead.  Lean comes from Lean Manufacturing and is a set of principles for achieving quality, speed & customer alignment.  Lean development can be summarized by seven principles:
  • Eliminate waste
  • Amplify learning
  • Decide as late as possible
  • Deliver as fast as possible
  • Empower the team
  • Build integrity in
  • See the whole

LEAN is the Principles and DevOps is the Practice.
  • DevOps is especially complementary to the Agile software development process, as it extends and completes the continuous integration and release process by ensuring the code is production ready and providing value to the customer
  • DevOps patterns can be derived from as “The Three Ways.” They describe the values and philosophies that frame the processes, procedures, practices, as well as the prescriptive steps.
  • The only difference is the narrower focus of DevOps. With the loop as tight as possible, knowledge gained from the actual system is fed back into the next iteration of the product so the system becomes better in small, but tight, increments.
Gene Kim, author of "The Phenoix Project" describes DevOps in "3 ways"
The "First Way" is:
  • The focus is on all business value streams, in other words it begins when requirements are identified, are built in Development, tested in Test and then transitioned into Operations, where the value is then delivered to the customer as a form of a feature or service.
  • The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system.
The "Second Way" is:
  • Creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
  • The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The "Third Way" is:
  • Creating a culture that fosters at two things: continual experimentation,  which requires taking risks and learning from success and failure; and understanding that repetition and practice is the prerequisite to mastery. We need both of these equally.
  • The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
You can imagine the problem that a Test/QA organization might have.  A high deployment rate typically associated with DevOps work streams will often put enormous pressure on Test/QA organizations.  So how what needs to change to help solve this issue:
  • The job of Testing is no longer to find defects, necessarily.
  • The job of Testing changes to prevent defects.
  • We must push testing up early
    – Tests improve the conversation between customers and developers
    – Tests become executable specifications
Some of the possible solutions to this issue are:
  • QA must be testing early and often.
  • Integrate QA into the flow through automated tests.
  • Whenever code is checked in, automated tests are automatically run, and issues must be fixed right away, just as if a developer checked in code that didn’t compile.
  • By integrating functional, integration and information security testing into the daily operations of Development, defects are found and fixed more quickly than ever.
  • *******Never Pass issues Downstream.
In the end, the benefits of Test/QA being involved in the DevOps model are:
  • Faster time to market (reduced cycle times and higher deploy rates)
  • Increased Quality (i.e., increased availability, increased change success rate, fewer failures)
  • Increased organizational effectiveness (increased time spent on value adding activities vs. waste, increased amount of value being delivered to the customer).
Good luck in implementing your own ideas into the DevOps model and I look forward to hearing your feedback.

Tuesday, June 25, 2013

Interviewing Potential Software test engineers

I typically I look for key words in resumes as a decision if I am going to do a phone interview.  Things like REST, Selenium, web services, Firebug are some of the "buzz words" I look for in my current position.
Once I find a "good" resume, I divide up my questions into technical and non-technical.

My first question always is:
Describe to me in detail the most challenging defect/issue you have personally discovered, how you discovered it and the way you debugged it including the tools you used.

The reason I ask this question first is, it immediately sets the tone for the experience level of the person I am interviewing.  If they "look" highly experienced on their resume and they come back with a basic example, I know most of what is on their resume is "buzz words".  However if they come back with a really solid answer, I know that most of what their resume tells me is good.

Next I ask a few technical questions based on their experience with specific tools or concepts.

Then I always ask this question:
How do you stay technical?
There is no correct answer to this question, rather I am looking for them to give me an answer instead of not having one.  Answers I think would come from someone quickly would be:  I read blogs, I read books, I participate in OpenSource projects, I play around with new technology or software, etc....  The worse answer someone can give me is, "well I really don't know".  Interview over!!!

Then I will ask some more technical questions, regarding their specific product experience.

Finally if they make it through all of that with acceptable answers, my final question is:
Do you feel Software Quality / Test is more of an engineering practice or artistic practice?

There is NO right answer to this in my mind.  I have my own opinions (discussion for a different day) but what I am looking for is, have they really ever thought about this topic.  Every highly experienced quality/test engineer I have ever run into, always has an opinion on this topic.  ALWAYS!!!

Also a key thing for me is, I am not afraid to end an interview 5 minutes into it or 30 minutes passed the time it was scheduled to be over.  If I feel you are not telling me the truth or trying to snow me, I end the interview immediately, I know that sounds harsh but really do any of us doing interviews have time to waste when we know we won't be offering them the job. 
I am also not afraid to run the interview long if I am on the fence and I feel like I need to dig further into their answers to help them articulate what they are trying to say.  One of the best engineers I ever hired, had a really hard time interviewing but because I could see flashes of what his abilities were, I spent a lot of extra time helping him answer his questions by continuing to tweak my questions to help pull out his answers.

Tuesday, June 18, 2013

SQuAD presentation on SQE Architecture

I recently presented at SQuAD (Software Quality users Association of Denver) on the SQE Architecture topic.

Please find the PDF version of the presentation at

Any feedback is welcomed.

Monday, June 3, 2013

SQuAD Testing conference in Denver

SQuAD Announcement pending!!!
I am so excited about an upcoming announcement about the SQuAD Testing conference in Denver on Sept 16 and 17, 2013.

I can't make the announcement just yet but all I can say is hardwork does pay off.  After working on this for almost 2 months, I received great news over the weekend.

Watch the blog in the coming days as we will be making this big announcement as well as the Registration website announcement.

Where else can you go to a testing conference with some big names in a setting like Denver Colorado for under $400?

Registration is limited so once the announcement is made, don't wait.

We will also be making the announcement on my Twitter feed at #erikstensland60 and at our Meetup.com site, look for SQUADCO.

Thursday, May 16, 2013

DevOps - Where in the world is Test?

In the past year, I have had the opportunity to be instrumental in building a DevOps team which included a set of highly skilled test engineers. So I thought I would share some of my thoughts and experiences during this opportunity.

So what DevOps is:
WikiPedia - DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.  DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

So the after reading that definition of what DevOps is, I was very disappointed in the fact that there was no mention of Test professionals.  Now maybe the author felt that Test is apart of the Information Technology professionals but that is a stretch in my mind.  So I decided that I would take the opportunity to write my own definition of what DevOps should be in this new world.

So the story starts with a senior Software Architect at my current company who is a bit of a cowboy (which matches my style) and I having a discussion about how were were going to build a team that would implement DevOps.  After many days of discussions on how a traditional Test team just would not fit into this new world, we agreed that we needed to architect this team with software engineers that focused on code development and software engineers that focused on test development. 

Now maybe this isn't new to some companies, i.e. Google or MicroSoft, but for this company it was a "revolutionary" idea and caused a major stir in the current Test organization.  With the help of the Sr Software Architect, we convinced the Test management to allow us to run with this idea.

We appropriately named the project team, Manhattan Project, and our philosophy was "We are either going to revolutionize the world or get us all fired!"  Thankfully we revolutionized the internal culture and now have a major effort underway to move all internal teams into a DevOps model.

Well that sounds easy right?  The answer was a resounding NO..... First we had to find Software engineers that were prepared to treat Test engineers as equals and not a sub-species of the engineering world.  Second we had to find Test engineers that were skilled enough to build test tools, review code, fix bugs they found and were skilled at looking outside the box for new automation approaches.  Finally we had to find Ops engineers that were willing to focus on release automation versus doing manual releases.  All of this was a "huge" challenge and required a bit of a tyrant mentality when it came to people trying to tell us how things should work, which meant how things had been done in the past.

After 2 months of hiring the proper team members, pulling from places like Yahoo, Microsoft and other high end internet companies, we had built a team of 10 software engineers, 5 test engineers and 3 operations engineers. 

As the team started to gel and produce code, we had several challenges to over come.  One was none of the existing internal testing tools would work in a continuous integration model.  The test engineers started to build a tool that could handle both UI and web service automated testing using the same tech stack that the software engineers were using to build the software.  This allowed for easy debugging efforts by either the Software or Test engineers.  The largest issue this team faced was environments.  See in this company's traditional software engineering approach, we had 5 different environments that code had to propagate through in order to get to the Production data center.  The reason this was a major challenge for the DevOps team was, not only did they change tech stacks but they decided that the code would live in the Cloud.  So connecting the existing environments up to this new Cloud environment regarded a lot of bullying the traditional network teams.

After several months, lots of late nights and many decisions made with the idea of "asking for forgiveness later", we successfully released our Pre-Prod environment and Production environments to the Cloud 3 days before hurricane Sandy hit the East coast.  That is important because Amazon's East coast region data centers had a chance of being taken out.  In those 3 days, our Ops automation and Test automation allowed us to setup our Production site into the West coast region and be ready to switch over if the worse case happened.  Today if we had to move to any of the other Amazon regions, from start to finish would take half a day.

The success of having Test engineers that had the respect of the Software engineers as well as the Ops engineers allowed this DevOps team to have 52 software releases in 20 working days with NO customer impacts reported.  You do the math, but with out the high level of test automation and the ability of the test engineers to identify the offending code, this would have never been achievable.

So even though the name DevOps doesn't include the word Test, the concept really doesn't work successfully unless you ensure that Test is apart of the overall culture of the teams.

The new DevOps definition should read:
DevOps is a software development method that stresses communication, collaboration and integration between software engineers, test engineers and information technology (IT) engineers.  DevOps is a response to the interdependence of software engineering, test engineering and IT operations. It aims to help an engineering organization rapidly produce software products and services.

Feel free to share your experiences with DevOps, good or bad!!!