When developers finish a first draft of a new feature or bug fix, they move the corresponding JIRA issue into the "Needs 1st Level Review" column of the Kanban board and un-assign it. Someone else on the team then assigns the issue to themself and moves the issue to "Reviewing" column. Reviewing involves two types of reviewing: code review (discussed elsewhere) and functional testing.
When testing new feature, we always check that the new feature or bug fix works as described in the story. In addition, check subsystems in IGB that might be affected by the change.
If you find a problem, we make a note of it in the corresponding JIRA card, re-assign it to the developer, and move the store back to the To-Do lane on the JIRA board. Always include a note in the story describing what you did, even if there are no problems.
For testing, the "deliverable" is a report on what you did to check for errors.
Pre-merge functional testing means testing a developer's changes before it is merged into the main-JDK8 branch via a pull request. For functional testing, you should download the developer's branch-specific installer, install it on your system, and test it there. (The developer should have rebased their branch onto the latest main-JDK8 prior to testing.) Assign the corresponding JIRA card to yourself and move it to "First Level Reviewing" while working on it.
When performing post-merge testing, use the main-JDK8 branch installer to test. As with pre-merge functional testing, if you notice any problems, describe them in the corresponding JIRA card, move it back to the To-Do column, and re-assign it to the original developer.
Note: The main-JDK8 branch installer is a pre-release version of IGB.
Before making an official release of IGB, we test release candidate installers on up-to-date versions of the three major platforms - Mac, Windows, and Linux. This is important because operating systems differ in how they handle security, and this changes with new versions. For example, an IGB installer that worked great on older versions of the Apple OS may become un-installable on a newer version. (Here is an example from release 9.0.2.)
Another reason we test on all three platforms is that IGB is using native file chooser components that are native to each platform. Also, how graphics are handled differ between platforms. For changes to the IGB UI, it is important to check them on each platform.
For release testing, we may set up some computers side-by-side and go through each smoke testing and systems testing protocol as a group, testing each protocol on the three platforms simultaneously. This can save time because it allows comparing behavior across operating systems and uncovers OS-specific problems.
To check IGB functionality on a system, you should test in two ways:
Note that to test IGB and IGB installations, you need to understand how the IGB installation process works on the system you're testing.
IGB installers are using the Install4J tool to install the compiled IGB code together with a "sandboxed" Java virtual machine that is used only to run IGB. When a user installs IGB, a folder gets created in the part of the file system dedicated to individual users' applications. Administrator (root) privileges are not required to install IGB, so this location is nearly always located in the user's personal or home directory. In addition, be aware that the Install4j installer software runs every time IGB launches. When IGB launches, the Install4J runs, too. It sends a request to the IGB server to read a file called "updates.xml." When we release a new version of IGB, we edit this file residing on the server, incrementing the release number as required. Then, when the installer reads this file the next time a user launches IGB, it will invite the user to download and install a new version if available.
Also note that IGB saves information about user preferences to the local system. IGB is using the Java preferences API for this. This means that to mimic a completely naive system, you must clear the user preferences. The easiest way to do this is by launching IGB and running the Reset Preferences to Defaults command, available in the Other Options tab of the Preferences window.
The preferences can also be cleared by deleting the following file, depending on the operating system.
Windows:
Mac: /Users/<user>/Library/Preferences/com.affymetrix.igb.plist
Linux:
Last but not least, IGB stores IGB-specific data in a folder, which is different depending on the operating system.
Windows: C:\Users\<user>\AppData\Roaming\IGB
Mac: /Users/<user>/.igb
Linux: /home/<user>/.igb
To mimic an entirely naive system, you should delete this or move it a new location for safekeeping.
If using a GUI file browser, make sure that the settings allow you to see hidden files. |
To remove all traces of IGB and mimic a naive system:
To mimic an existing user of IGB:
The test target is the version of IGB that you are planning to test. This is either a branch-specific installer (in the case of agile testing) or a release candidate (for release testing.)
Download the test target installer and run it.
Note: Instead of installing it in the default location, install the test target version (branch build or release candidate) on your computer in a directory named for the release. In addition, answer "no" when asked if you want to create a shortcut. Following this convention will reduce the chance for conflicts that may appear to be errors but really are only artifacts of the testing process.
Congratulations! Finding errors is awesome!
It is far better for us to find errors - so that we can fix them before releasing the software to our user community.
If you find what appears to be an error, the first thing you should do it document how to reproduce it.
Create step-by-step instructions describing how to trigger the error. Make sure that the you can trigger the same error behavior on a completely naive system (see above.) Add these instructions to the existing JIRA card for the particular new feature or bug fix you are testing, or create a new JIRA card if the original card has already been closed or if you think the error you discovered should be documented in a new card. (Feel free to consult a colleague if you are not sure.)
Important: If you open a new JIRA card, give it a title that starts with "Investigate" and then describe the problem.
For an example, see Investigate why IGB hangs on Mac.
Next, find out: When did the error behavior first get introduced? For this, do two things: