An improper warning message that might have resulted in product failure

In my team, there is a group that tests a file sharing application. Knowing the detailed functionality of this application is impertinent to what we are going to discuss in this post. But I must mention how a user may use this application as there hides the food for the post. This is a small application (but having a lot of combinations of tests) where user can share files up to certain GBs. User has to provide the email id(s) of the person(s) with whom he wants to share the files along with file(s) up to a maximum allowed size.

The application was stable and performing well. For some reference purpose, we shared the application with couple of other users. Within next few minutes, I got a ping from one of them and he told me that he was unable to share files with users. I was surprised. I inquired about what exactly went wrong. He said he entered two email ids and attached a couple of files that come within the maximum allowed size and then clicked on Send button. Then he got a warning message that read something like "Sorry! We found the following email Ids entered in sender or receiver fields, which are blocked by our system: "xxx@yahoo.com, xxx@gmail.com". Please remove them and try again". He then told me that he had already tried with several other email ids but to fail. He thought there was a serious problem and the application is not able to perform its intended task. He even said that it was a showstopper until I asked him to hold on.

I asked him what were the things that came to his mind when he saw this warning message. His reply was:

1. First attempt: The system has blocked only the emails that I had provided. I was clueless about why an application would block my email ids especially because it was the first time I was using the application. What algorithm it must be using to block email ids or it is just that the application didn't like my name in the email id :)

2. Second attempt (with another set of email ids): The same warning message.There has to be a problem with the system and not with the email ids I am providing.

3. Third attempt: I just wanted to confirm that the problem was with the system and provided another set of email ids and BANG! I was right. The system has a problem. It has blocked every other email ids. What are the ids then it has not blocked? Do I have any information on that? NO.

Apart from these three attempts, he said, I have marked another thing "Please remove them and try again". I simply followed the instruction pretending to be a dumb and removed the email ids I had provided and clicked on the Send button. Bollocks! Sender and Receiver fields are mandatory. I knew this had to happen. Otherwise whom do I share the files with? Then why did it ask me to remove the email ids and try. Oh, did it ask me to remove the email ids, provide new set of email ids (which are of course not blocked by the system) and retry? I don't know.

He was curious to know what exactly the problem was. He asked me why it behaved like this. He wanted to know what are the sets or rather types of email ids that are not blocked and are eligible.

My answer to all his questions was "Commercial email service providers like Yahoo, Gmail, Rediffmail etc. are not allowed. You have to use only official email ids".

Did we convey this message to the users who are mostly likely to use email ids provided by these so called commercial mail service providers(CMSP)and get to see this very warning message? How the hell on earth anyone could know that email ids provided by CMSP are not allowed by the system. It may happen and the probability is very high that a user, after having tried many such email ids with failure each time, in exasperation may give up and look for a better application. He may never come back again to use this application.

We lose a customer and then many customers and then the product fails. Just because we didn't put a proper message.

A test report

When I don’t have much project work to do, I do a lot of different things that add certain value to my skill as a tester. I read blogs, watch podcasts and videos. Sometime I pick up an open source project from sites like http://sourceforge.net and test it. I prepare test report and analyze it myself. In this process I learn many things. On one such occasion, I picked up a product, tested it and prepared a test report. You might like to see the test report. Below it is:

                Test Report: The Number Race 2.3.4
                                            (Author: Prabhat Nayak)

Test Mission:
The mission is to test “The Number Race” application which has been awarded as the project of the month for November by sourceforge.net and find out some important problems in a session of two hours. The quality criteria that are going to be covered in this session are “Usability” and “Capability”. I’ll not prepare any document before my actual testing. It will be more of exploratory testing and keeping note as I test.

Test Environment:
Operating System: Windows XP Professional, Service Pack 3
Application Version: NumberRace 2.3.4

Test Areas:
1. Installer
2. Select Player
3. Menu
4. Level 1 Game
5. Uninstaller

Test Approach:
I downloaded the application from http://sourceforge.net/projects/numberrace/. I divided my two hours of time into 3 phases as below:

Phase1 (30 mins): This phase was dedicated to gathering information about the application. This included who are the intended users, what is the purpose of building the application and how to use the application.

Phase 2 (60 mins): This phase involved actual learning/testing efforts of the application. I started my exploration from installing the application. Then I created/selected a player and played a level. I then uninstalled the application. During all these activities I took notes of my observations.

Phase 3 (30 mins): In this phase, I prepared the final test report on my findings.

Observations:
1. Double clicking the installer invokes the language selection window. In this window appears the “how to start” instruction which could hardly be read. Ironically this is the only place which shows this information. Refer to “Starting_Point.JPG”.

2. Installation-step-5: If "Create shortcuts in the Start-Menu" checkbox is unticked, "Create additional shortcuts on the desktop" option gets disabled. In this way one cannot create only the desktop shortcut without creating Start-Menu shortcut.

3. Installation-step-6: After installation gets completed, Instead of button name "Quit", "Finish" could have been more meaningful.

4. After launching the application, "No language pack installed" dialog box appears. It asks to download the needed language pack without giving any information on where to download it. Please refer to “Language pack.JPG”.

5. The start page doesn’t clearly display the starting point by clicking which the game could be started.

6. While creating a player, providing invalid information (other than leaving it blank) and clicking the “Enter” button doesn’t result in creating a player but it doesn’t show any error/warning message either. Message is shown only in case a field is left blank.

7. Age filed is validated only against non-numeric data. It accepts 0 and all negative numbers.

8. Pre specified “Level” for an existing player cannot be modified.

9. Very less information on “how to play a level” is displayed prior to start the level. It doesn’t suffice in understanding the level.

10. After a move is finished, it takes few seconds for the sound to play and hence delays the process of going to the next move.

11. After completion of a level, the player still remains in the same level. It should get alleviated to the next level.

12. On clicking question mark icon on Select Player window, a window is invoked. This window is blank and reveals no information. Either it should display some information or it should be removed. Please refer to “Questionmark.JPG”.

13. Clicking “Quit” button terminates the application without displaying a confirmation message.

14. Uninstalling from the folder hierarchy (for eg. C:\Program Files\NumberRace-2.3.4\Uninstaller) leaves some empty folders.

Issues:
Issue #1:
Summary: 000101: Unticking "Create shortcuts in the Start-Menu" checkbox disables "Create additional shortcuts on the desktop" option
Product: NumberRace 2.3.4
Description: In step 5 of installation, when "Create shortcuts in the Start-Menu" checkbox is ticked, "Create additional shortcuts on the desktop" checkbox gets disabled. The only way to get a desktop shortcut is to allow the installation to create Start menu shortcut.
Steps: 1. Double click the installer
2. Go to step 5 of installation
3. Untick the "Create shortcuts in the Start-Menu" checkbox
Impact: A user can never have a desktop shortcut without having a start menu short cut. Start menu shortcut is unwanted for those who like to launch the application from desktop shortcut. They might not like this feature.
Attachements: Please refer to image “Shortcut_Issue.JPG”
=============================================

Issue #2:
Summary: 000102: Age field accepts 0 and negative numbers
Product: NumberRace 2.3.4
Description: Age field in Select Player window takes 0 as well as all negative numbers which are invalid. It is only validated against non-numeric data
Steps: 1. Go to Select Player window
2. Fill all the fields excluding Age field with valid data. Provide 0 or any negative number on Age field.
3. Click on Enter button
Impact: Age of a player is very important when this application is used to track the real progress in terms of capability vs. age of a child. In that case allowing wrong data as age can mislead the assessment.
Attachements:
=============================================
Issue #3:
Summary: 000103: Pre-specified Initial Level for an existing player cannot be modified
Product: NumberRace 2.3.4
Description: Initial Level field in Select Player window shows which level a particular player is in. According to the level selected, the player gets a game of that level to play. An existing player cannot change his level even if he has finished the same level and eligible to graduate to the next level.
Steps: 1. Go to Select Player window
2. Fill all the fields with valid data.
3. Select any level (1-14) from the Initial Level list.
4. Click on Enter button
5. Click on Modify button
6. The Initial Level list is disabled
Impact: An existing player cannot graduate to the next level by selecting a higher level even if he has completed the specified level. In a way to do it, either he has to create another player or delete the existing one and recreate a player with the same details.
Attachements: