The following factors are important during tool selection:
i. Assessment of the organization’s maturity (e.g. readiness for change);
ii. Identification of the areas within the organization where tool support will help to improve testing processes;
iii. Evaluation of tools against clear requirements and objective criteria;
iv. Proof-of-concept to see whether the product works as desired and meets the requirements and objectives defined for it;
v. Evaluation of the vendor (training, support and other commercial aspects) or open-source network of support;
vi. Identifying and planning internal implementation (including coaching and mentoring for those new to the use of the tool).
OR
The industry experts have suggested following four major criteria for selection of testing tools.
1) Meeting requirements.
2) Technology expectations.
3) Training / skills.
4) Management aspects.
1) Meeting Requirements:
a) There are many tools available in the market today but rarely do they meet all the requirements of given product or a given organization. Evaluating different tools for different requirements involves lot of effort, money and time. Huge delay is involved in selecting and implanting test tools.
b) Test tools may not provide backward or forward compatibility with the product-under-test (PUT).
c) Test tools may not go through the same amount of evaluation for new requirements. For example: some tools had Y2K-problem.
d) A number of test tools cannot distinguish between a product failure and a test failure. This increases analysis time and manual testing. The test tools may not provide the required amount of trouble-shooting/debug/error messages to help in analysis.
For example, in case of GUI testing, the test tools may determine the results based on messages and screen coordinates at run-time.
Hence, if the screen elements of the product are changed, it requires the test suite to be changed. The test tool must have some intelligence to proactively find out the changes that happened in the product and accordingly analyze the results.
2) Technology Expectations:
a) In general, test tools may not allow test developers to extend / modify the functionality of the framework. So, it involves going back to the tool vendor with additional cost and effort. Very few tools available in market provide source code for extending functionality or fixing some problems. Extensibility and customization are important expectations of a test tool.
b) A good number of test tools require their libraries to be linked with product binaries. When these libraries are linked with the source code of the product, it is called as the “instrumented code”.
This causes portion of testing be repeated after those libraries are removed, as the results of certain types of testing will be different and better when those libraries are removed. For example, the instrumented code has a major impact on the performance testing since the test tools introduce an additional code and there could be a delay in executing the additional code.
c) Finally, test tools are not 100% cross-platform. They are supported only on some O.S. platforms and the scripts generated from these tools may not be compatible on other platforms.
Moreover, many of the test tools are capable of testing only the product, not the impact of the product/test tool to the system or network. When there is an impact analysis of the product on the network or system, the first suspect is the test tool and it is uninstalled when such analysis starts.
3) Training Skills:
Test tools require plenty of training, but very few vendors provide the training to the required level. Organization-level training is needed to deploy the test tools, as the users of the test suite are not only the test team but also the development team and other areas like SCM (Software Configuration Management). Test tools expect the users to learn new language/scripts and may not use standard languages/scripts. This increases skill requirements for automation and increases the need for a learning curve inside the organization.
4) Management Aspects:
A test tool increases the system requirement and requires the hardware and software to be upgraded. This increases the cost of the already-expensive test tool. When selecting the test tool, it is important to note the system requirements and the cost involved in upgrading the software and hardware needs to be included with the cost of the tool. Migrating from one test tool to another may be difficult and requires a lot of effort. Not only is this difficult, as the test suite that is written cannot be used with other test tools but also because of the cost involved. As the tools are expensive and unless the management feels that the returns on investment (ROI) are justified, changing tools are generally not permitted. Deploying a test tool requires as much effort as deploying a product in a company. However, due to project pressures, test tools effort at deploying gets diluted, not spent. Thus, later it becomes one of the reasons for delay or for automation not meeting expectations. The support available on the tool is another important point to be considered while selecting and deploying the test tool.