Thursday, May 8, 2014

My StarEast Presentation

Just finished up giving my presentation at StarEast testing conference today.
I must say, best testing conference I have been to, enjoyed the conference and the people.

Had standing room only!

Check it out at DevOps:Where in the World is Test?

I welcome stories and comments.

Thursday, September 26, 2013

DevOps: Where in the world is Test?

Last week we had our SQuAD 2103 software testing conference in Denver, CO.  Our line up included Key notes from Julian Harty (Master Mobile Guru) and Theresa Lanowitz (Software quality industry Guru) plus talks from Lee Copeland and Lisa Crispin.  It was a star studded line up for only $400.

I presented on the topic of DevOps and how a test/quality organization fits into that model.  The presentation starts off with some definitions of what DevOps is and how it is really the "practice" of the "principles" of the Lean Software movement. 

"DevOps: Where in the world is Test?"
My first thought when I heard DevOps, was that of 2 "back yard brawlers" battling it out in a no holds barred fight.  Later I heard terms like collaboration, discipline, and agile with Ops.  But I still hadn’t heard the word I was looking for <test>.  It started to feel like two sumo wrestlers with no referrer watching over them.  Finally,  it wasn’t until after I did my research and implemented a successful DevOps QA team that I understood,  how the “ritual” would happen and how test would continue to be an extremely important part of the overall process, (i.e. the Referee ensuring that Dev was still playing on a level field with Ops).

So what is DevOps:  It refers to the emerging professional movement that advocates a collaborative working relationship between Development, QA and IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience and security of the production environment.

It became common placed around 2009, as the convergence of numerous adjacent and mutually reinforcing movements:
  • The Velocity Conference movement, especially the seminal “10 Deploys A Day” presentation given by John Allspaw and Paul Hammond
  • The “infrastructure as code” movement (Mark Burgess and Luke Kanies),
  • the “Agile infrastructure” movement (Andrew Shafer) and the Agile system administration movement (Patrick DeBois)
  • The Lean Startup movement by Eric Ries
  • The continuous integration and release movement by Jez Humble
  • The widespread availability of cloud and PaaS (platform as a service) technologies (e.g., Amazon Web Services).
Well you might be asking yourself, Why not Lean instead.  Lean comes from Lean Manufacturing and is a set of principles for achieving quality, speed & customer alignment.  Lean development can be summarized by seven principles:
  • Eliminate waste
  • Amplify learning
  • Decide as late as possible
  • Deliver as fast as possible
  • Empower the team
  • Build integrity in
  • See the whole

LEAN is the Principles and DevOps is the Practice.
  • DevOps is especially complementary to the Agile software development process, as it extends and completes the continuous integration and release process by ensuring the code is production ready and providing value to the customer
  • DevOps patterns can be derived from as “The Three Ways.” They describe the values and philosophies that frame the processes, procedures, practices, as well as the prescriptive steps.
  • The only difference is the narrower focus of DevOps. With the loop as tight as possible, knowledge gained from the actual system is fed back into the next iteration of the product so the system becomes better in small, but tight, increments.
Gene Kim, author of "The Phenoix Project" describes DevOps in "3 ways"
The "First Way" is:
  • The focus is on all business value streams, in other words it begins when requirements are identified, are built in Development, tested in Test and then transitioned into Operations, where the value is then delivered to the customer as a form of a feature or service.
  • The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system.
The "Second Way" is:
  • Creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
  • The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The "Third Way" is:
  • Creating a culture that fosters at two things: continual experimentation,  which requires taking risks and learning from success and failure; and understanding that repetition and practice is the prerequisite to mastery. We need both of these equally.
  • The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
You can imagine the problem that a Test/QA organization might have.  A high deployment rate typically associated with DevOps work streams will often put enormous pressure on Test/QA organizations.  So how what needs to change to help solve this issue:
  • The job of Testing is no longer to find defects, necessarily.
  • The job of Testing changes to prevent defects.
  • We must push testing up early
    – Tests improve the conversation between customers and developers
    – Tests become executable specifications
Some of the possible solutions to this issue are:
  • QA must be testing early and often.
  • Integrate QA into the flow through automated tests.
  • Whenever code is checked in, automated tests are automatically run, and issues must be fixed right away, just as if a developer checked in code that didn’t compile.
  • By integrating functional, integration and information security testing into the daily operations of Development, defects are found and fixed more quickly than ever.
  • *******Never Pass issues Downstream.
In the end, the benefits of Test/QA being involved in the DevOps model are:
  • Faster time to market (reduced cycle times and higher deploy rates)
  • Increased Quality (i.e., increased availability, increased change success rate, fewer failures)
  • Increased organizational effectiveness (increased time spent on value adding activities vs. waste, increased amount of value being delivered to the customer).
Good luck in implementing your own ideas into the DevOps model and I look forward to hearing your feedback.

Tuesday, June 25, 2013

Interviewing Potential Software test engineers

I typically I look for key words in resumes as a decision if I am going to do a phone interview.  Things like REST, Selenium, web services, Firebug are some of the "buzz words" I look for in my current position.
Once I find a "good" resume, I divide up my questions into technical and non-technical.

My first question always is:
Describe to me in detail the most challenging defect/issue you have personally discovered, how you discovered it and the way you debugged it including the tools you used.

The reason I ask this question first is, it immediately sets the tone for the experience level of the person I am interviewing.  If they "look" highly experienced on their resume and they come back with a basic example, I know most of what is on their resume is "buzz words".  However if they come back with a really solid answer, I know that most of what their resume tells me is good.

Next I ask a few technical questions based on their experience with specific tools or concepts.

Then I always ask this question:
How do you stay technical?
There is no correct answer to this question, rather I am looking for them to give me an answer instead of not having one.  Answers I think would come from someone quickly would be:  I read blogs, I read books, I participate in OpenSource projects, I play around with new technology or software, etc....  The worse answer someone can give me is, "well I really don't know".  Interview over!!!

Then I will ask some more technical questions, regarding their specific product experience.

Finally if they make it through all of that with acceptable answers, my final question is:
Do you feel Software Quality / Test is more of an engineering practice or artistic practice?

There is NO right answer to this in my mind.  I have my own opinions (discussion for a different day) but what I am looking for is, have they really ever thought about this topic.  Every highly experienced quality/test engineer I have ever run into, always has an opinion on this topic.  ALWAYS!!!

Also a key thing for me is, I am not afraid to end an interview 5 minutes into it or 30 minutes passed the time it was scheduled to be over.  If I feel you are not telling me the truth or trying to snow me, I end the interview immediately, I know that sounds harsh but really do any of us doing interviews have time to waste when we know we won't be offering them the job. 
I am also not afraid to run the interview long if I am on the fence and I feel like I need to dig further into their answers to help them articulate what they are trying to say.  One of the best engineers I ever hired, had a really hard time interviewing but because I could see flashes of what his abilities were, I spent a lot of extra time helping him answer his questions by continuing to tweak my questions to help pull out his answers.

Tuesday, June 18, 2013

SQuAD presentation on SQE Architecture

I recently presented at SQuAD (Software Quality users Association of Denver) on the SQE Architecture topic.

Please find the PDF version of the presentation at

Any feedback is welcomed.

Monday, June 3, 2013

SQuAD Testing conference in Denver

SQuAD Announcement pending!!!
I am so excited about an upcoming announcement about the SQuAD Testing conference in Denver on Sept 16 and 17, 2013.

I can't make the announcement just yet but all I can say is hardwork does pay off.  After working on this for almost 2 months, I received great news over the weekend.

Watch the blog in the coming days as we will be making this big announcement as well as the Registration website announcement.

Where else can you go to a testing conference with some big names in a setting like Denver Colorado for under $400?

Registration is limited so once the announcement is made, don't wait.

We will also be making the announcement on my Twitter feed at #erikstensland60 and at our site, look for SQUADCO.

Thursday, May 16, 2013

DevOps - Where in the world is Test?

In the past year, I have had the opportunity to be instrumental in building a DevOps team which included a set of highly skilled test engineers. So I thought I would share some of my thoughts and experiences during this opportunity.

So what DevOps is:
WikiPedia - DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.  DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

So the after reading that definition of what DevOps is, I was very disappointed in the fact that there was no mention of Test professionals.  Now maybe the author felt that Test is apart of the Information Technology professionals but that is a stretch in my mind.  So I decided that I would take the opportunity to write my own definition of what DevOps should be in this new world.

So the story starts with a senior Software Architect at my current company who is a bit of a cowboy (which matches my style) and I having a discussion about how were were going to build a team that would implement DevOps.  After many days of discussions on how a traditional Test team just would not fit into this new world, we agreed that we needed to architect this team with software engineers that focused on code development and software engineers that focused on test development. 

Now maybe this isn't new to some companies, i.e. Google or MicroSoft, but for this company it was a "revolutionary" idea and caused a major stir in the current Test organization.  With the help of the Sr Software Architect, we convinced the Test management to allow us to run with this idea.

We appropriately named the project team, Manhattan Project, and our philosophy was "We are either going to revolutionize the world or get us all fired!"  Thankfully we revolutionized the internal culture and now have a major effort underway to move all internal teams into a DevOps model.

Well that sounds easy right?  The answer was a resounding NO..... First we had to find Software engineers that were prepared to treat Test engineers as equals and not a sub-species of the engineering world.  Second we had to find Test engineers that were skilled enough to build test tools, review code, fix bugs they found and were skilled at looking outside the box for new automation approaches.  Finally we had to find Ops engineers that were willing to focus on release automation versus doing manual releases.  All of this was a "huge" challenge and required a bit of a tyrant mentality when it came to people trying to tell us how things should work, which meant how things had been done in the past.

After 2 months of hiring the proper team members, pulling from places like Yahoo, Microsoft and other high end internet companies, we had built a team of 10 software engineers, 5 test engineers and 3 operations engineers. 

As the team started to gel and produce code, we had several challenges to over come.  One was none of the existing internal testing tools would work in a continuous integration model.  The test engineers started to build a tool that could handle both UI and web service automated testing using the same tech stack that the software engineers were using to build the software.  This allowed for easy debugging efforts by either the Software or Test engineers.  The largest issue this team faced was environments.  See in this company's traditional software engineering approach, we had 5 different environments that code had to propagate through in order to get to the Production data center.  The reason this was a major challenge for the DevOps team was, not only did they change tech stacks but they decided that the code would live in the Cloud.  So connecting the existing environments up to this new Cloud environment regarded a lot of bullying the traditional network teams.

After several months, lots of late nights and many decisions made with the idea of "asking for forgiveness later", we successfully released our Pre-Prod environment and Production environments to the Cloud 3 days before hurricane Sandy hit the East coast.  That is important because Amazon's East coast region data centers had a chance of being taken out.  In those 3 days, our Ops automation and Test automation allowed us to setup our Production site into the West coast region and be ready to switch over if the worse case happened.  Today if we had to move to any of the other Amazon regions, from start to finish would take half a day.

The success of having Test engineers that had the respect of the Software engineers as well as the Ops engineers allowed this DevOps team to have 52 software releases in 20 working days with NO customer impacts reported.  You do the math, but with out the high level of test automation and the ability of the test engineers to identify the offending code, this would have never been achievable.

So even though the name DevOps doesn't include the word Test, the concept really doesn't work successfully unless you ensure that Test is apart of the overall culture of the teams.

The new DevOps definition should read:
DevOps is a software development method that stresses communication, collaboration and integration between software engineers, test engineers and information technology (IT) engineers.  DevOps is a response to the interdependence of software engineering, test engineering and IT operations. It aims to help an engineering organization rapidly produce software products and services.

Feel free to share your experiences with DevOps, good or bad!!!


Wednesday, May 8, 2013

The Value of a Software Quality Engineering Architect


Software Quality Engineering (SQE) Architects have broad roles inside a software quality organization.  Their main focus is to understand the software from an architecture level by being able to link integration points, or as many points as possible from with-in their domain, so they can guide testing strategies across boundaries.  By being involved in the early software architecture phases, SQE Architects ensure that testability is a consideration during design and help drive down the overall cost of testing. 
Other areas of focus include developing test infrastructure, authoring automation-testing frameworks, evaluating new testing methodologies, improve performance or integration testing efficiencies, implementing new testing tools and spearhead new testing technologies.
The common thread across all SQE architect roles is to provide technical leadership and strategic direction for testing across the organization as a whole.
In Pearson it is also expected that in addition to accountability to their current product lines such as LearningStudio, MMND or OpenClass, that these senior engineers will consistently look beyond the current release for integration issues and may have several deliverables not tied to a particular product release.

SQE Arch responsibilities

A SQE architect has in-depth knowledge of a variety of technology domains, testing techniques and methodologies used both inside and outside of Pearson. They provide technical assistance and/or advice to both Software Development and Software Quality Engineering on how best to ensure software is testable and how to test software.
The question often arises - “Shouldn’t the SQE manager be doing all of this anyway?”  In the past, the SQE Manager was the individual providing technical leadership and formulating team strategy. However, as the organization continued to grow and the SQE manager took on additional responsibilities, it made sense for a SQE architect to step in and assist with or deliver on these responsibilities and strategies. Now the SQE architect leads the technical strategy while the SQE manager can focus on team building and resource management.
A SQE architect is also expected to be able to affect change not only across the testing teams, but between other engineering disciplines as well. SQE architects must drive testability across all disciplines, providing guidance, feedback, and suggestions to improve testing practices across an entire engineering team.  A great example of that is how the test team fits into the “DevOps” model that was built around the Manhattan project.  The SQE Architect collaborated closely with Software Development management and Software Architects to come up with an approach that allows the team to release software quickly but most importantly it has a high level of overall quality.
While a few SQE architects may be focused on a specific problem or improvement, such as integrated automation suites or performance/scalability challenges or automation strategies, the goal for the SQE architect investment should be long-term improvement of the organizations testability, improved testing processes and technical growth.

Senior test engineers, when faced with an urgent problem or situation in need of quick improvement can typically find a fast or easy solution. Broad or recurring issues, such as unified UI/Service level testing frameworks or how to best utilize the Cloud for generating load on a product, however require a SQE architect to research and design a long-term solution that drives down the cost of testing.

The SQE architects think long-term and lay out paths for solving challenging issues over a long period and not necessarily focused on fighting daily fires.

SQE Competency model

The competency of a SQE Architect is not sufficient to be a specialist in any one domain or technology and requires a wide and fairly deep understanding of a gamut of technologies and tools.  They are very similar to a Software Architect, in such that they have a broad range of experience in a large area of technology.

SQE Architects must have:
    Extensive Technical skills covering Product, Technologies and Sound knowledge of domain / areas being handled.
    Knowledge of current industry wide Quality & Test processes and practices, Tools and techniques
    Ability to work with teams.  The “ability to influence” despite not having direct reporting relationships is very key. The ability to collaborate and co-operate is important
    Excellent communication skills – within and outside of the SQE organization, across teams, with customers – both horizontally and vertically, is important. Effective negotiation skills are very important too.
    Ability to focus and prioritize is important. Understanding the distinction between the urgent and the important and effectively prioritizing tasks is key
    Must understand the customer / user needs
    Self-management is a key attribute expected of a SQE Architect. Being able to work without the need for follow-up or “too much” management is important. The SQE Architect should be self-motivated and a self-starter.
    Ability to motivate their self and others is important as well as being able to set a good example for the other team members to follow
    Ability to set goals is also key. In many instances, the SQE architect will need to define and set goals including stretch-goals as appropriate
    Patience and a touch of humility are valuable, especially in all dealings with team members. This is especially true when trying to mentor or guide other team members; the ability to articulate in ways that are understood by the listener at their level is necessary while also possessing good listening skills. The humility to acknowledge need for continuous learning and to undertake a program of learning to constantly update skills and keep abreast of current developments in the industry is vital
    Ability to strategize and look ahead and at the big picture
    A great deal of maturity, accountability, high degree of integrity, highest levels of pro-active behavior, ability to take initiative and professional behavior is naturally expected
    Sound Project Management abilities are important
    Software Analysis & Design knowledge/experience is needed while also having a solid background in Software Quality & Testing. Must have hands-on experience having performed both functional and non-functional testing and be able to review requirements, design and even code as needed.


In summary, I hope the above whitepaper is useful in gaining a general understanding of the value of a SQE Architect role and some of the expectations surrounding this position. This is in no way complete or a full representation of the responsibilities / requirements of the SQE Architect role. Each group within the larger SQE organization has its own additional expectations that form the entire SQE Architect's role.  However the key take-a-ways for the value that a SQE architect adds to the overall engineering discipline can be summed up in the below bullet points:

·      SQE Architects drive down testing costs by being involved early in software architecture to ensure testability is considered during design.
·      SQE Architects pull teams together and facilitate technical alignment.
·      SQE Architects are members of SQE leadership “close to the action”.
·      SQE Architects analyze product lines and recognize testing gaps across multiple teams.
·      SQE Architects know and understand the customer.
·      SQE Architects collaborate and help facilitate diverse teams including Dev, Sys Archs, Operations, PMO and Product Management.
·      SQE Architects ensure consistency of process and tools in a test organization.
·      SQE Architects bring about visibility into the product and help facilitate a complete high-quality solution.
·      SQE Architects provide technical leadership and strategic direction for SQE management.