Wednesday, January 22, 2014

Tablet Apps, should you be "Testing in the Wild"?

Today almost 34% of consumers in the US own a tablet of some kind and 43% of those users spend more time on their tablet then a desktop according to a study by Google Inc.


In my mind this data reinforces that fact that companies need to have an amazing mobile/tablet application to have any hope of surviving in the growing mobile market place.


Experts agree that the most important area to start when creating these apps is: Ease of Use.  So you ask yourself, how do we do that?


Make sure searching is very obvious and easy:  Just like with all good web applications, users need to be able to search and find things very easily.  The suggestions are to put the search box at the top of the screen with both a "magnifying glass" and the word "search" to let the users know they can type a query in there.


Be "Big" on browsing:  Tablets are not much heavier than most magazines, and many users use their tablets in the same way they would page through a magazine.   Focus should be on images and minimal navigation options so they don't take up much space.  Also allowing users to enlarge images by tapping or stretching their fingers and offering visual signs such as arrows to notify users that there is more content available.


Minimize the need for typing:  Although it is easier to type on a tablet then a mobile phone, it still is not the way most users interact with their tablets.   Focus on collecting information that is only "absolutely" necessary and consider using multiple choice questions to reduce the amount of typing required.  Another great point is to increase the entry field size to make the form field boxes large enough to tap.  The same should go for buttons, increase the size so they fit the larger tip of a finger.


Now that we know what areas of your tablet apps should be designed in, we can talk about hot spots for testing these types of apps.


Usability should be your top concern:  If users do not find your app intuitive, have trouble finding features or simply don't like the layout, it can drive them away very quickly or worse drive up costs by calling your call centers asking questions.  Colors, layouts, imagery, etc can have negative impacts on the overall user experience.  Of course this becomes a more of a exploratory testing exercise because no automation tool can provide this "usability" experience information.


Load Capacity is highly critical:  Many companies have learned hard lessons when an unexpected (or expected) level of load negatively impacts their apps.  With today's consumers ability to Blog/Facebook/Tweet about such negative experiences, a company can see a hugh inconvenience with an app that is sluggish or worse completely crashes.  There are studies that show once a user experiences such an issue, they will not come back to your app.  Using a "Hybrid" model for doing load testing gives you a clearer picture of how an app performs under pressure.  This "hybrid" model allows you to do manual functional testing while the app is under a heavy synthetic old.  A typical load test will not tell you if a user "feels" your app being slow.


In comes, "Testing in the Wild".  This is a concept that started with the DevOps movement but really has been expanded with 3rd party companies that provide "crowd source testing".  By providing an application that can be instrumented and monitored in production, I argue that monitoring this real data will provide teams with more insight and actionable information than traditional pass/fail test cases.  I have worked with teams that have huge automated and manual regression suites and they still release with defects found by customers.  I realize that tablet apps are arguably harder to do this type of testing in production because release cycles are dependent on Apple and Google.  However there are creative ways to install and release beta versions of your apps to specific users that you have worked out agreements to provide detailed information and not go to their social network and ruin your brand.


I consider "Testing in the Wild" an extremely viable option for both web and mobile/tablet testing and you should too.

Wednesday, December 18, 2013

Testing in Production

Testing in Production

I am sure you are saying "You are crazy, we would never do that".  Well I am sure you are thinking back to the old days when dev would by-pass QA and throw crappy code into Production, only to see customers filling the call centers with complaints.  However as Seth Eliot from Microsoft points out "Testing in Production isn't skipping QA".

I agree with James Whittaker when he talks about how software testing hasn't changed in 20+ years.  Testing teams need to change with the rapid change in software development processes, no longer should we be thinking that we need to run full regressions on every software release.  Our thinking should be "lets do enough testing to catch show stoppers and after the release monitor user activity to see if users are experiencing issues not previously caught" says Eliot.

The point here is that real time data collection is key to ensuring your new release is successful. Eliot says "Testing in Production techniques often leverage the massive flow of data constantly emitted from our services. Hotmail measures actual performance experienced by users (anonymously of course) across all operating systems, browsers, and locations around the world to pinpoint trouble spots and optimize performance.”

Even if you are not as large as Microsoft, if you can implement just 2 metrics to track real-time, you will gain a huge ROI.  I highly suggest monitoring "page views and page load times", if page views flatlines, the timing stats give an immediate clue to what might be going on and where.

Other items to monitor would be things like the rate and details of run-time JavaScript errors occurring on the site, CPU, API requests, system response time as well as anonymized user data and finally data emitted from synthetic transactions that you run constantly in production.

I argue that monitoring real data will provide teams with more insight and actionable information than traditional pass/fail test cases.  I have worked with teams that have huge automated and manual regression suites and they still release with defects found by customers yet because of their release process it takes them hrs, days or weeks to fix these issues.

I bet you are still saying "You are crazy to test in production", remember with this continuous deployment idea you have the ability to roll back a build if something goes wrong. Continuous deployment is done in small chunks (dev, test and release) and because of that, it should be simple to roll back a release to an old version if needed.

Yes there are challenges and risks associated with Testing in Production, but in combination with your teams ability to release software in small pieces quickly and "fail fast", you gain a huge amount of information about how your application will behave in the wild.  When market leaders like Microsoft, Google, Facebook and Twitter run experiments in production all the time, the data they garner from these experiments is invaluable to them.

Is "Testing in Production" right for you?  I suggest you start small with an area of your application where you can create small releases, have a large amount of quick running automated tests and can implement real time monitoring and then watch to see if the phone lines start ringing before you notice the issue.

Thursday, September 26, 2013

DevOps: Where in the world is Test?

Last week we had our SQuAD 2103 software testing conference in Denver, CO.  Our line up included Key notes from Julian Harty (Master Mobile Guru) and Theresa Lanowitz (Software quality industry Guru) plus talks from Lee Copeland and Lisa Crispin.  It was a star studded line up for only $400.

I presented on the topic of DevOps and how a test/quality organization fits into that model.  The presentation starts off with some definitions of what DevOps is and how it is really the "practice" of the "principles" of the Lean Software movement. 

"DevOps: Where in the world is Test?"
My first thought when I heard DevOps, was that of 2 "back yard brawlers" battling it out in a no holds barred fight.  Later I heard terms like collaboration, discipline, and agile with Ops.  But I still hadn’t heard the word I was looking for <test>.  It started to feel like two sumo wrestlers with no referrer watching over them.  Finally,  it wasn’t until after I did my research and implemented a successful DevOps QA team that I understood,  how the “ritual” would happen and how test would continue to be an extremely important part of the overall process, (i.e. the Referee ensuring that Dev was still playing on a level field with Ops).


So what is DevOps:  It refers to the emerging professional movement that advocates a collaborative working relationship between Development, QA and IT Operations, resulting in the fast flow of planned work (i.e., high deploy rates), while simultaneously increasing the reliability, stability, resilience and security of the production environment.

It became common placed around 2009, as the convergence of numerous adjacent and mutually reinforcing movements:
  • The Velocity Conference movement, especially the seminal “10 Deploys A Day” presentation given by John Allspaw and Paul Hammond
  • The “infrastructure as code” movement (Mark Burgess and Luke Kanies),
  • the “Agile infrastructure” movement (Andrew Shafer) and the Agile system administration movement (Patrick DeBois)
  • The Lean Startup movement by Eric Ries
  • The continuous integration and release movement by Jez Humble
  • The widespread availability of cloud and PaaS (platform as a service) technologies (e.g., Amazon Web Services).
Well you might be asking yourself, Why not Lean instead.  Lean comes from Lean Manufacturing and is a set of principles for achieving quality, speed & customer alignment.  Lean development can be summarized by seven principles:
  • Eliminate waste
  • Amplify learning
  • Decide as late as possible
  • Deliver as fast as possible
  • Empower the team
  • Build integrity in
  • See the whole

LEAN is the Principles and DevOps is the Practice.
  • DevOps is especially complementary to the Agile software development process, as it extends and completes the continuous integration and release process by ensuring the code is production ready and providing value to the customer
  • DevOps patterns can be derived from as “The Three Ways.” They describe the values and philosophies that frame the processes, procedures, practices, as well as the prescriptive steps.
  • The only difference is the narrower focus of DevOps. With the loop as tight as possible, knowledge gained from the actual system is fed back into the next iteration of the product so the system becomes better in small, but tight, increments.
Gene Kim, author of "The Phenoix Project" describes DevOps in "3 ways"
The "First Way" is:
  • The focus is on all business value streams, in other words it begins when requirements are identified, are built in Development, tested in Test and then transitioned into Operations, where the value is then delivered to the customer as a form of a feature or service.
  • The outcomes of putting the First Way into practice include never passing a known defect to downstream work centers, never allowing local optimization to create global degradation, always seeking to increase flow, and always seeking to achieve profound understanding of the system.
The "Second Way" is:
  • Creating the right to left feedback loops. The goal of almost any process improvement initiative is to shorten and amplify feedback loops so necessary corrections can be continually made.
  • The outcomes of the Second Way include understanding and responding to all customers, internal and external, shortening and amplifying all feedback loops, and embedding knowledge where we need it.
The "Third Way" is:
  • Creating a culture that fosters at two things: continual experimentation,  which requires taking risks and learning from success and failure; and understanding that repetition and practice is the prerequisite to mastery. We need both of these equally.
  • The outcomes of the Third Way include allocating time for the improvement of daily work, creating rituals that reward the team for taking risks, and introducing faults into the system to increase resilience.
You can imagine the problem that a Test/QA organization might have.  A high deployment rate typically associated with DevOps work streams will often put enormous pressure on Test/QA organizations.  So how what needs to change to help solve this issue:
  • The job of Testing is no longer to find defects, necessarily.
  • The job of Testing changes to prevent defects.
  • We must push testing up early
    – Tests improve the conversation between customers and developers
    – Tests become executable specifications
Some of the possible solutions to this issue are:
  • QA must be testing early and often.
  • Integrate QA into the flow through automated tests.
  • Whenever code is checked in, automated tests are automatically run, and issues must be fixed right away, just as if a developer checked in code that didn’t compile.
  • By integrating functional, integration and information security testing into the daily operations of Development, defects are found and fixed more quickly than ever.
  • *******Never Pass issues Downstream.
In the end, the benefits of Test/QA being involved in the DevOps model are:
  • Faster time to market (reduced cycle times and higher deploy rates)
  • Increased Quality (i.e., increased availability, increased change success rate, fewer failures)
  • Increased organizational effectiveness (increased time spent on value adding activities vs. waste, increased amount of value being delivered to the customer).
Good luck in implementing your own ideas into the DevOps model and I look forward to hearing your feedback.

Tuesday, June 25, 2013

Interviewing Potential Software test engineers

I typically I look for key words in resumes as a decision if I am going to do a phone interview.  Things like REST, Selenium, web services, Firebug are some of the "buzz words" I look for in my current position.
Once I find a "good" resume, I divide up my questions into technical and non-technical.

My first question always is:
Describe to me in detail the most challenging defect/issue you have personally discovered, how you discovered it and the way you debugged it including the tools you used.

The reason I ask this question first is, it immediately sets the tone for the experience level of the person I am interviewing.  If they "look" highly experienced on their resume and they come back with a basic example, I know most of what is on their resume is "buzz words".  However if they come back with a really solid answer, I know that most of what their resume tells me is good.

Next I ask a few technical questions based on their experience with specific tools or concepts.

Then I always ask this question:
How do you stay technical?
There is no correct answer to this question, rather I am looking for them to give me an answer instead of not having one.  Answers I think would come from someone quickly would be:  I read blogs, I read books, I participate in OpenSource projects, I play around with new technology or software, etc....  The worse answer someone can give me is, "well I really don't know".  Interview over!!!

Then I will ask some more technical questions, regarding their specific product experience.

Finally if they make it through all of that with acceptable answers, my final question is:
Do you feel Software Quality / Test is more of an engineering practice or artistic practice?

There is NO right answer to this in my mind.  I have my own opinions (discussion for a different day) but what I am looking for is, have they really ever thought about this topic.  Every highly experienced quality/test engineer I have ever run into, always has an opinion on this topic.  ALWAYS!!!

Also a key thing for me is, I am not afraid to end an interview 5 minutes into it or 30 minutes passed the time it was scheduled to be over.  If I feel you are not telling me the truth or trying to snow me, I end the interview immediately, I know that sounds harsh but really do any of us doing interviews have time to waste when we know we won't be offering them the job. 
I am also not afraid to run the interview long if I am on the fence and I feel like I need to dig further into their answers to help them articulate what they are trying to say.  One of the best engineers I ever hired, had a really hard time interviewing but because I could see flashes of what his abilities were, I spent a lot of extra time helping him answer his questions by continuing to tweak my questions to help pull out his answers.

Tuesday, June 18, 2013

SQuAD presentation on SQE Architecture

I recently presented at SQuAD (Software Quality users Association of Denver) on the SQE Architecture topic.

Please find the PDF version of the presentation at
http://files.meetup.com/2625872/SQE%20Architecture-SQUAD.pdf

Any feedback is welcomed.
-Erik

Monday, June 3, 2013

SQuAD Testing conference in Denver

SQuAD Announcement pending!!!
I am so excited about an upcoming announcement about the SQuAD Testing conference in Denver on Sept 16 and 17, 2013.

I can't make the announcement just yet but all I can say is hardwork does pay off.  After working on this for almost 2 months, I received great news over the weekend.

Watch the blog in the coming days as we will be making this big announcement as well as the Registration website announcement.

Where else can you go to a testing conference with some big names in a setting like Denver Colorado for under $400?

Registration is limited so once the announcement is made, don't wait.

We will also be making the announcement on my Twitter feed at #erikstensland60 and at our Meetup.com site, look for SQUADCO.


Thursday, May 16, 2013

DevOps - Where in the world is Test?

In the past year, I have had the opportunity to be instrumental in building a DevOps team which included a set of highly skilled test engineers. So I thought I would share some of my thoughts and experiences during this opportunity.

So what DevOps is:
WikiPedia - DevOps is a software development method that stresses communication, collaboration and integration between software developers and information technology (IT) professionals.  DevOps is a response to the interdependence of software development and IT operations. It aims to help an organization rapidly produce software products and services.

So the after reading that definition of what DevOps is, I was very disappointed in the fact that there was no mention of Test professionals.  Now maybe the author felt that Test is apart of the Information Technology professionals but that is a stretch in my mind.  So I decided that I would take the opportunity to write my own definition of what DevOps should be in this new world.

So the story starts with a senior Software Architect at my current company who is a bit of a cowboy (which matches my style) and I having a discussion about how were were going to build a team that would implement DevOps.  After many days of discussions on how a traditional Test team just would not fit into this new world, we agreed that we needed to architect this team with software engineers that focused on code development and software engineers that focused on test development. 

Now maybe this isn't new to some companies, i.e. Google or MicroSoft, but for this company it was a "revolutionary" idea and caused a major stir in the current Test organization.  With the help of the Sr Software Architect, we convinced the Test management to allow us to run with this idea.

We appropriately named the project team, Manhattan Project, and our philosophy was "We are either going to revolutionize the world or get us all fired!"  Thankfully we revolutionized the internal culture and now have a major effort underway to move all internal teams into a DevOps model.

Well that sounds easy right?  The answer was a resounding NO..... First we had to find Software engineers that were prepared to treat Test engineers as equals and not a sub-species of the engineering world.  Second we had to find Test engineers that were skilled enough to build test tools, review code, fix bugs they found and were skilled at looking outside the box for new automation approaches.  Finally we had to find Ops engineers that were willing to focus on release automation versus doing manual releases.  All of this was a "huge" challenge and required a bit of a tyrant mentality when it came to people trying to tell us how things should work, which meant how things had been done in the past.

After 2 months of hiring the proper team members, pulling from places like Yahoo, Microsoft and other high end internet companies, we had built a team of 10 software engineers, 5 test engineers and 3 operations engineers. 

As the team started to gel and produce code, we had several challenges to over come.  One was none of the existing internal testing tools would work in a continuous integration model.  The test engineers started to build a tool that could handle both UI and web service automated testing using the same tech stack that the software engineers were using to build the software.  This allowed for easy debugging efforts by either the Software or Test engineers.  The largest issue this team faced was environments.  See in this company's traditional software engineering approach, we had 5 different environments that code had to propagate through in order to get to the Production data center.  The reason this was a major challenge for the DevOps team was, not only did they change tech stacks but they decided that the code would live in the Cloud.  So connecting the existing environments up to this new Cloud environment regarded a lot of bullying the traditional network teams.

After several months, lots of late nights and many decisions made with the idea of "asking for forgiveness later", we successfully released our Pre-Prod environment and Production environments to the Cloud 3 days before hurricane Sandy hit the East coast.  That is important because Amazon's East coast region data centers had a chance of being taken out.  In those 3 days, our Ops automation and Test automation allowed us to setup our Production site into the West coast region and be ready to switch over if the worse case happened.  Today if we had to move to any of the other Amazon regions, from start to finish would take half a day.

The success of having Test engineers that had the respect of the Software engineers as well as the Ops engineers allowed this DevOps team to have 52 software releases in 20 working days with NO customer impacts reported.  You do the math, but with out the high level of test automation and the ability of the test engineers to identify the offending code, this would have never been achievable.

So even though the name DevOps doesn't include the word Test, the concept really doesn't work successfully unless you ensure that Test is apart of the overall culture of the teams.

The new DevOps definition should read:
DevOps is a software development method that stresses communication, collaboration and integration between software engineers, test engineers and information technology (IT) engineers.  DevOps is a response to the interdependence of software engineering, test engineering and IT operations. It aims to help an engineering organization rapidly produce software products and services.

Feel free to share your experiences with DevOps, good or bad!!!

Sources:
http://en.wikipedia.org/wiki/DevOps