Good Practices in Vendor Evaluation & Selection for Computerised Systems and Software

Good Practices in Vendor Evaluation & Selection for Computerised Systems and Software

In the ever-evolving landscape of Computer System Validation (CSV) vendor assessment and selection has become more crucial than ever before. In this article, we will explore the key considerations of vendor assessment, guiding towards making informed decisions that align with the organisation’s needs and regulatory requirements.
From background checks to evaluating system features, assessing vendor capabilities and ensuring ongoing support and maintenance, this article outlines steps to follow during the vendor assessment as well as post-selection supporting activities.

Background Checks and Business Viability
Before committing time and effort to the detailed assessment of the vendor and product (computerised system/software) it is important to establish the viability of both with some basic checks.

1. Due Diligence – Checking financial stability and length of time in the industry of the vendor and whether business conditions present any risk – for example, acquisition or merger status.
2. Referrals – A good approach would be to check similar projects, reviews or references from past clients.
3. Product demonstration – This should give a strong indication as to the product’s suitability (sometimes performed against a prospective/draft URS). The features and use cases of the software should be presented, as well as other services from the vendor (e.g. development, configuration, testing, post-implementation support). It is useful to have a cross-functional team to attend (IT, business users, engineering, finance) so the session can cover all aspects of requirements (the vendor may also need a similar cross-functional team).
4. Cost Evaluation and Licensing – While cost should not be the sole determining factor, it is important to evaluate the total cost of ownership (TCO). Consider upfront costs for software development, ongoing maintenance and validation fees, and any additional expenses (such as trips to vendor premises for detailed assessment or product testing. Assess the licensing model and consider factors such as scalability, number of users and any potential costs associated with future system development or upgrades.

Vendor Approach To Quality (Software Development, Maintenance and Support)

When basic checks have been completed (with satisfactory results), a formal assessment of the vendor is typically performed. The key objective is to establish the extent to which quality by design is embedded into the product (and associated services). This is the core aspect of Vendor Assessment. Note if the outcome of basic checks identify issues, it is sometimes possible to work collaboratively to resolve or mitigate these. There are various methods via which this can be achieved, from checklist questionnaires to formal audit at the vendor’s premises. The effort invested should be commensurate with the assessed impact of the product/service to operations (impact to business, regulatory compliance and patient safety/product quality) – greater effort will allow broader scope and more detailed insights into the vendor’s strengths and weaknesses.

1. As previously identified, the scope of products/services should be clearly established (for example, software product, installation/commissioning, configuration & testing, validation support (IQ & OQ), support & maintenance).
2. Processes and Procedures – A quality system should be in place, with well-defined processes for the development, testing, maintenance and ongoing support of products.
3. Staff Experience and Training – Checking the skills and number of employees and organisational structure (against relevant processes in the quality system and technical/vocational qualifications). For software, there should be segregation of duty between developers and testers (and/or quality assurance).
4. Product Documentation – Thorough documentation of the system/software should be available covering its design, architecture, functionality, configuration, testing and use (i.e. manuals). There should be strong processes for how these are maintained and kept current – staff should demonstrate in-depth knowledge of their areas of expertise.
5. Post-Implementation Support – The vendor should operate a robust support service, with well-defined processes and tools (e.g. case management) – a good indicator is whether metrics are available, showing response times and closure rates of various severities of issues. These may be covered by contractual arrangements, for example, Service Level Agreements.

Functions and Features of the System / Software
Before committing to significant investment, as well as assessing the vendor, a comprehensive understanding of the product should be built up. Note this is sometimes performed by Design Qualification – the points below may be carried out before this (and typically require less effort).

1. Regulatory or Quality Requirements – If the product must support regulatory or quality requirements it is essential to establish what the functionality solution looks like – good examples include 21 CFR part 11 (e.g. how are electronic signatures implemented and controlled by the system?), Audit trail and data archiving.
2. Must-Have Requirements (URS) – Similar to the previous point, high-priority requirements (which sometimes are established with a draft or high-level preliminary URS) should be checked. As outlined in the previous section functionality can be assessed by means of demonstration. It is also important to check how non-functional requirements (for example acceptable downtime for maintenance, hours of availability of support desk) will be met.

Supporting the Project

For some types of Systems and Software, the implementation effort may be significant and complex – this is particularly true for information systems with large numbers of users (e.g. ERP, Document Management) and any system requiring customisation (software code) or extensive configuration. In these cases it may be important to evaluate the vendor’s capability in some (or all) of the following areas:

1. Project Management – Does the vendor have dedicated project management? Do they have a pre-defined approach, tasks and methods and delivery experience? Larger projects typically require a dedicated project team on the client side – good vendors should be able to advise on roles, responsibilities and amount of effort typically required.
2. Development & Configuration – Where the vendor is responsible, as well as the assessment of their capabilities in these areas, the vendor should be able to provide the structure, processes tasks and schedule for this phase. Where required, alignment to client’s processes.
3. Documentation – Where specific documentation is created or modified for the product or service, there should be a clear understanding of what these will be; who will be responsible for generating, reviewing and approving; who will own the documents during the project phase and beyond (for example configuration records may need to be transferred to and maintained by the client). In some cases, alignment to the client’s processes may be required (for example – test defect management).
4. Testing and Validation – Similarly to documentation; the approach, who, what and how with regard to testing and validation must be assessed and agreed upon. For large test initiatives, a strategy, planning and specific tools (for status tracking and defect management) will be required. Sometimes the vendor may offer validation testing (typically IQ and OQ) – this can be very efficient, but it is essential to review these and supporting processes to determine suitability, and potential gaps to help define the client-led test effort (typically PQ).
5. Enhanced Support – It can be appropriate, particularly with large highly configured or customised systems, to have a period of enhanced support after the initial implementation of the system – sometimes referred to as hypercare. This may involve an onsite presence from the vendor and the ability to react to and resolve issues quickly (including fixing performance problems, and bugs and fine-tuning workflows). After Hypercare support is normally catered for by an approved Service Level Agreement.

Selecting the right vendor for CSV projects is essential to ensure compliance, efficiency, and successful system implementation. The effort employed should be commensurate with the size, importance and areas of risk that may arise regarding the system or service. By following the best practices outlined in this article, you can make informed decisions and achieve scalable approaches that align with your organisation’s needs and regulatory framework.
Our team at BPV, have the skills and expertise to assist you in vendor selection and assessment as well as providing services related to CSV and equipment validation.
To find out more about our validation services, please get in touch with us at [email protected]