The Curious Case of Mobile App Testing ;-)

Before starting with the main topic - we may have a quick look to a chart researched at Keane and published as a white paper with the title "Testing Mobile Business Applications". It is some sort of continuation of my last post (specially the last paragraph). This chart provides us with a recommendation about  testing strategy while testing mobile business applications.

Now moving to the original post i.e. the curious questions about mobile app testing we may have a look to the following Q and A. ;-)


1.) What is the difference between Mobile Testing and Mobile Application Testing?
:::::: Mobile testing refers to a complete testing of mobile phones which includes - Protocol testing, device testing and application testing. If we try to elaborate a bit more - testing the support of 4 bands of mobile phone and calling in each of the band, battery, connectivity (such as bluetooth, wlan, etc.),  network frequency, cell selections, handover, encryption, software compatibility, etc. along with multimedia and applications testing are referred to Mobile Testing.


Whereas, mobile application testing is the testing of softwares, games and others utilities built on the mobile platform by third parties or the same vendor for targeted device. Here the main concern is about the functionality of the application under different circumstances and environments.


2.) What is your approach while Testing Mobile Applications?
:::::: The most important thing while testing a mobile application is the approach towards the testing. To my understanding and experience the things we have to consider and prioritize while testing a mobile application can be listed as below:
-- Application development platform (such as J2ME, Windows Mobile, Blackberry, Simbian, iOS, Android, etc.). The reason behind this is the fact that nature and behavior of the applications are different and classed according to the platform they are built on.
-- Device screen size, memory allocation, navigation (cause application behavior depends largely on these things when portability of application on multiple devices is a factor.)
-- Device network or carrier (for some application which deal with data transfer - calls, data calls, frequency of network are crucial things).
-- Application Domain (such as gaming, location based system, imaging, video, etc.).
-- Test strategies (UI tetsing, functional tetsing, stress tetsing, load tetsing, network tetsing, memory tetsing, battery/charge tetsing, simultaneous run of application or calls or data calls while application is running)
-- Maintaining common standard checklist (UI and UX stability, readability of texts, status of call/data call while app is running, app behavior while flip/slider opens or closes, sms or data port blocking if app is using those ports, etc.).


3.) What are the things specific to Mobile Application would you emphasis on while writing test plan for Mobile Applications?
:::::: The things that we must give emphasis can be listed as below:
-- Testing Scope [Scope is very much important because there is a large amount of test items available for mobile app testing. So defining scope like - testing with/without carriers, calls, data calls, wap, variant of operating systems, etc. is explicitly important while writing a test plan.]
-- Testing Strategy [The strategy and methodologies for unit testing, system testing, regression testing and pass/fail check-listing - few things like application characteristics, stability, connectivity, launching, UI and application accessing internal resources or user data.]
-- Test Environment Setup [Environment is another bug thing for test plan for mobile app. Which parts of the app will be tested on emulator and which parts will be on real device, what will be the device versions, how many variant of the devices will be present while testing, what will be the possible carriers, what will be the test locations -- this questions is needed to be answered in the test plan explicitly.]
-- Control Procedure [How to handle the reports, documentations, change requests while testing is undergone -- should be confirmed from conversation/discussion with the PM and should be listed in the test plan.]


4.) What's your view on testing of Mobile Application on Emulators?
:::::: Testing mobile applications in emulators has some limitations of its own. Testing on emulator has some serious issues for which they differs a lot from testing on real devices. We may list some of the important difficulties as follows:

-- Making voice/data calls while running applications are different than the actual scenario in an emulator.

-- From an emulator it cannot be determined if the phone is connected to carrier or not.

-- While testing in an emulator there is no support for testing the real charging (battery) state while running the application.

-- Considering the memory usage, in emulator it is hard to calculate or test the memory usage or CPU performance.

-- Considering graphics elements, emulator performance can differ from real device depending on the platform where the emulator is installed.

-- Considering the input/output scenario, the tests that depends on accessing user data or internal devices resources like memory cards or something like that -- cannot be done in emulators. 

5.) What is the difference between Mobile Operating Systems and Mobile Platforms?
:::::: This was one of the initial questions came in my mind when I came to know about mobile platforms years back. While working with development of mobile application and while testing mobile apps I have built some idea on my own about it. Here I am listing the ideas mixed with some definitive theoretical answers:
-- Mobile OS is a system software which resides upon device hardware and it runs the device hardwares such as networking devices, graphics etc. In short it operates the mobile phone system.
-- Where as Mobile Platforms are another applications that runs on top of the OS layer yet exposes the hardware through a more focused set of interfaces and functionality. In short platforms are built on top of the mobile OS and uses the device connectivity options provided by the OS to interact with the device to execute/run applications.
-- To sum up the issue, we can just say - in terms of software, the Operating System runs the hardware, the platform runs on top of the OS to host applications.  

6.) What actually are the application certification programs like True Brew Testing (TBT), Symbian Signed Test Criteria, Java Verified Program?
:::::: Application certification programs are test criteria which must be passed by an application before it is made available to operators for distribution. So from the definitions it is clear that the criteria are defined by the platform authorities as a standard to make the application connectivity and accessibility almost uniform to end-users. Every platform has their defined sets of criteria.
-- True Brew Testing (TBT) is the Brew® platform compatibility criteria. Brew® defines some item specification document and some charging for submission and passing the application to make it ready for distribution from the operators.
-- Symbian Signed Test Criteria are defined by the Symbian Developers Network and used for testing application built for Symbian OS and Symbian platform. The guideline defines a great deal of information and criteria along with cases where exemptions are given. There are a great deal of detailing about the criteria and even test steps.
-- Similarly Java Verified is the industry-recognized Java testing and signing program for Java ME apps which has to go through a testing, signing and verification process through the Java Verified Submissions portal. If the apps meets the testing criteria, they'll gain the Java Verified seal of approval. And this applies to all apps, irrespective of their simplicity or complexity. So the signing implies a minimum quality of the application.

7.) What so special in testing a Location Based Mobile Application?
:::::: The little specialty of location based mobile applications are very much clear from the name itself. In general sense it requires testing the application on various location to make the application produce expected outputs for the location difference. Location-based applications often require field testing before launching the product. Setting up "the field" almost similar to real life usage scenario may produce various testing scenarios which can often be a logistics issue. Moreover, real life field environment is not very much suitable for controlled testing and there can be so much uncontrolled, unexpected and unwanted scenarios which can easily affect the test results. Mainly these are the complexities of testing Location Based applications.

I hope I will write more about mobile application testing in upcoming days. Those should be platform specific write ups. Lets see.

Read more »

My Introduction to Mobile App Testing

Mobile application platforms like Android, iOS, Blackberry etc. are now trendy things. The market for mobile application is growing everyday and the demand for this applications are rising as the technology is advancing. Most of the general software testing principles apply equally well for mobile application -- in a general overview -- but in my small experience I have found some cases and curious questions where it differs! There is a huge variety in mobile application technology, platform, devices and they have their own complexity while developing. So while testing - this "extra' complexities needs to be handled. We can spot out some specific areas of mobile application testing where extra care for testing is required.


1) Technology specific testing: All of the technologies has their own properties as well as own pros-cons. While testing mobile apps testers needs to be more careful and conscious about the technology pros-cons.


2) Platform specific testing: Each and every platform for mobile apps has its own properties. And the nature and behavior of the platforms and the apps at the top of it conveys them. Testers need to be concerned about these issues while testing.


3.) Device specific testing: My little experience on developing J2ME applications tells me that even if an app runs on emulator quite smoothly - unlike desktop apps, it may not suit all the J2ME supported devices -- even if the version and Api versions are all same. That enables quite a big consciousness while testing a mobile app.





Now if we like to create a list of things to be considered specially while testing mobile applications -- we can create a list which also covers up the basic things we do while testing a mobile app.


First, Functional Testing. As per definition - this testing will cover up testing the basic functionality of the app. In fact, the procedure starts right from the application store (like iTunes for iOS, android store, etc.) because the quality of the app will depend on if it comes to users with ease or downloads seamlessly. Now techniques for functional testing may vary in two ways: first is testing the app using a simulator (most of the case it is called emulators) and the second is testing the app where it will be used i.e. on device. In ideal case, both of the approach should be followed. Because, if some of the feature of the app like clicking on a button or link works on emulator but not on the device -- that should be a device specific fault. Additionally, the running environment (sdk version, api version, etc.) for the app in emulator and the device should be same or at least similar. Like in regular SQA process, this testing should be started at the very beginning of development cause changing in app design or architecture costs a lot.


Secondly, Usability Testing. To the best of my understanding it is the single most critical testing for mobile applications. Mobile devices are of a HUGE variety. Screen size, scrolling buttons, joy-stick, trackballs, touch screen inputs, folding, sliding and many more. Now for the same app, user experience and easiness differs hugely based on these things. Like in a form with 7/8 inputs can cost 4 clicks to a user of Nokia 2700 phone while it is just a smooth slide to a Nokia 5800 owner (Leaving alone the iPhones and androids :S). And first impression of user while using a mobile app creates the future market scope for the app. And who doesn't know that only word-of-mouth of bad experience can be broad-casted to the market and hence can destroy the business of the app. That is why usability testing of an app is the critical most step in testing.


Thirdly, Performance Testing. Here comes another point in related to app in addition with device specific test and that is performance of the network carrier. If the network carrier is stable enough and the device is a smooth performer on the carrier - risks of crashing the application minimizes to a great deal - specially for the apps which requires data or call transfers. Another issue is region or location. Performance of carriers may vary location to locations. If the performance of the app is poor to end-users they are meant to be diverted to other similar apps. In most of the cases, performance of the app is likely to be measured with the help of tools cause it is very hard to create a real life "load scenario' in front of the testers.


Finally, Selecting Testing Methodology. In our organization, mobile apps are being tested in-house with the development team in front of the PM with client's direct feedback. This method has both pros and cons - it is a controlled process i.e. it is not very much tough to create the diverse test environment with different types of devices with different carriers and under use-case scenarios of the dev team. But the fact is it is too much expensive (in accordance to time-money and resources). Among the other methods can be testing with simulators/emulators -- but as I mentioned earlier they doesn't give you the 100% real UX scenario. There can be community testing or beta testing, that is releasing the app within a limited community and they use it with limited features (or may be full) and report problems found. This is more effective than others but is time consuming and not all the areas can be covered (naturally) cause the users in a community is of more or less similar type. My observation here is -- the methodology should be chosen with respect to the nature, volume, budget and of course its device-carrier requirements. A hybrid of all the methodologies can be a good solution for medium level app.






Next I am willing to write something about interesting and curious questions I faced (and my team members faced) about mobile app testing. So long bye for now.

Read more »

Testing Lesson 1 : Information, Communication and Challenges

In my previous post, I expressed that I am a lazy  and irregular writer. This second post of the site proves it as it came after almost one month. Though I have studied a lot of things (and naturally forgot most of them :-P), I will note down gradually what seems to be important to me and what I have to remember. In this chronicle, today I will note something like -- Information system in testing, Bug significance and communication and key challenges one may face. The last point will be a bit theoretical as my testing career is only one month long, and for the rest two points theory along with some personal observation will be there. For learning purpose, I have followed testingeducation.org, Google and Selim vai's advice.



Information System:
     The major purpose of testing is to assure the quality of a product i.e. make sure a product does what it is intended to. Along with this, it is important to keep the status of the product viable to the product developer / owner / beneficiary or other stakeholders as each of the product has a business value. For this reason, information systems related to quality of a product plays the most vital role as the objective of testing.

Upon the received information from QA persons --

• The developer fixes the wrong in programming
• The system administrator can make the system running for expected condition
• The designer can fix the user experience bugs
• The product owner can know the status of development, fixes needed, redesigning criteria, etc.

The above list signifies the reasons of importance of a good information system. Now we can list the major information acquiring or investigation tasks of a tester which can be as follows:

1. Find important bugs (by comparing actual with expected output from a product).
2. Assess the quality-severity-importance of fixing-frequency-etc. of the found bug.
3. Stop release of premature products and suggest managers to take release decision.
4. Assess probable cost and control of product related support.
5. Find such scenarios / cases where using the product may not cause problems.
6. Mark the product as safe or standard to some predefined scale.
7. Pass the information gathered in process 1 to 6 to due authority.

The no 7 task leads us to next topic which is "communication'.

Communication:

     Communication in QA means to act as a information system and pass "relevant' information to stakeholders and make the faults (that are found during investigations) fixed. A successful communication has multiple dimensions of aspects like -- 

• make the developer successfully recreate the error, 
• make the project manager successfully reason the importance of an error,
• make the product owner successfully understand the current development status,
• reduce the complexity of development decision making process, etc.

If we consider the above points as "ends' the "means' towards communication lies in Bug reports. The content of bug reports may vary from one stakeholder to another depending on the information one wants, but the basic principle for a clear bug reports may be the same. If we attempt to write them they can be as follows:

--The bugs in the report has to be verifiable that is it has to be a real problem and the steps to reproduce the bugs should be mentioned clearly.

-- When a developer / Project manager looks at a report, he tends to make decision about the bugs - to fix it or not to fix it or fix it later or it's not a bug. A good bug report must clarify about the severity and cost of the bug fixing (and of course opportunity cost) so that they can make the decision.

-- A good bug report contains reference between the problem found and the stakeholders' concern about that issue.

-- If a fault or problem found is because of the technology but not the work, a bug report has to contain the statement with proper reference.

-- Last but not the least, the writing style and format of the report has to be motivating (so that stakeholders take that as interesting) not attacking.

Interestingly, the points of motivation stated in BBST looks simple yet powerful. I can't really help myself from sharing them -

-- it looks really bad

-- it will affect lots of people

-- it can be easily recreated

-- product owner can sue you as it breaks the contract

-- it will hamper the image of the company

-- top management wants it to be fixed

-- it's interesting ;-)

Motivating people to do something is always a challenge. In the next topic I will try to write the key challenges of testing as it is described in BBST.

Challenges:

(1) Complete testing

-- No testing process can be marked as a "Complete Testing'.

(2) Complete coverage

-- "Completeness' depends on specific population of test cases.

-- Even if complete coverage is done through hundreds of thousands of test cases, it never ensures complete testing.

-- It is based on percentage but there are still some points which deals with thoughtful judgement are not covered here.

(3) Negligence

-- If the cost for "preventing' a bug is greater than the loss done by the bug then solving the bug is unreasonable.

-- If the cost for "preventing' a bug is equal to or lower than the loss done by it then the Company is negligible.

(4) Bug Workflow 

-- Maintain proper products-specific workflow.

-- Design the workflow to deal with conflicts of interest between stakeholders.

(5) The tester's time-spent vs the developer's time-spent trade-off.

-- The more time spent investing a bug, the fewer bugs are found and reported.

-- If one hour of tester's work saves 10 minutes of developer's time, is it cost worthy?

(6) Experimenter Effects 

-- To report or not to report and to fix or not to fix dilemma.

-- At each level of bug workflow the results of that level needs to be verified and approved.

(7) Motivating Stakeholders

-- Stated at previous topic.

(8) Not Report most bugs but report right bugs and get most of them fixed.

-- A good tester is not someone who reports most bugs but gets the most bugs fixed.

-- A good tester is not someone who reports most bugs but reports the right bugs.

This write up is a novice approach towards learn testing. So many things may not be covered or many unnecessary things may have been added. I think as time goes by I will be experienced enough to find the wrongs.

Read more »

Hello to the testing world

This is my first day at testing department. I have been assigned here a week ago.


Though I have a career of almost 4 years in web application development (in LAMP environment), I said 'yes' to the management offer to join the QA team --- as I was always interested to work with issues like --- security, performance, information system management, etc (I have academic works in these fields too). I thought it to be a chance to change the way of my working as it is something of my interest. That, of course, does not mean that I don't have any interest in software development anymore. Rather, I think the experience will boost my personal development career (for Inversionz Garage mainly).

I have named this blog as "SubZeroTester' because I have almost no previous experience with QA team. I have only textbook knowledge about this very section. So I can better be labeled at "sub zero level'! 

With many questions and eagerness to learn something, I am starting the new horizon of my career. In this very blog, I will be writing my experience, findings and my way of learning -- how to QA! Mainly documentations, blogs, slides, videos and man-to-man communication will be my mean of learning. I agree I am a very much irregular writer (that can be seen from my other blogs)  --  this write-ups will be my own guide towards tracking things I have done.


see you there!!!!!!

Read more »