Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
TECHNICAL EVALUATION GUIDELINES
FOR
SELECTION OF VENDOR
FOR
ANDHRA PRADESH TECHNOLOGY SERVICES
National Institute for Smart Government
August 2015
Technical Evaluation Guidelines for Selection of Vendor for APTS
2 | P a g e
Document Control
Document Title: Technical Evaluation Guidelines for selection of Vendors for e-Governance
Projects
Document Status: <Version Number>
Abstract: Example: <<This document details the Technical Evaluation Guidelines for selection of
Vendors for e-Governance Projects >>.
Document Publication History
(All revisions made to this document must be listed in chronological order, with the most recent revision
at the top.)
Date Author Version Remark
25.08.2015 NISG, Hyderabad 1.2
21.08.2015 NISG, Hyderabad 1.1
Reviewers
Date Reviewer Remarks
21.08.2015 APTS
Distribution
Version Name Location
Technical Evaluation Guidelines for Selection of Vendor for APTS
3 | P a g e
Contents
1 Introduction ................................................................................................................... 5
2 Designing Technical Evaluation ....................................................................................... 7
3 Technical Evaluation Criteria ........................................................................................... 8
3.1 Category One: Suggested Technical Evaluation ............................................................... 9
3.2 Category Two: Suggested Technical Evaluation ............................................................. 15
4 Detailed Methodology for Evaluation and Parameters .................................................. 18
4.1 System Functionality: 20 Points ..................................................................................... 18
4.2 Technology: 20 Points .................................................................................................... 20
4.3 India Specific Capabilities: 8 points ................................................................................ 26
4.4 Government Specific Deployment: 7 Points .................................................................. 26
4.5 Training: 10 Points .......................................................................................................... 27
4.6 Certification, Credentials: 7 points................................................................................. 29
4.7 Profile of Proposed Team Members: 20 Points ............................................................. 29
4.8 Project Methodology, Support & Documentation: 8 points .......................................... 31
Technical Evaluation Guidelines for Selection of Vendor for APTS
4 | P a g e
List of Abbreviation
Abbreviation Description
CV Curriculum Vitae
CMMI Capability Maturity Model Integration
COTS Commercial of the Shelf
ERP Enterprise Resource Planning
ISO International Organization for Standardization
IE Internet Explorer
LPC Lowest-Priced Conforming
MMP Mission Mode Project
MSME Micro, Small and Medium Enterprises
PoC Proof of Concept
OS Operating System
QCBS Quality and cost Based Selection
RFP Request for Proposal
WLAN Wireless Local Area Network
Technical Evaluation Guidelines for Selection of Vendor for APTS
5 | P a g e
1 Introduction
The Evaluation phase covers the procurement process from close of offers to the selection of the
preferred bidder and the selection of the successful bidder. The key components of an Evaluation
process could also be illustrated as:
Evaluation model
Evaluation criteria
Rating Scale
Panel decision making process
Due diligence requirements
This outcome is achieved by carefully considering each offer on an equal basis, against the evaluation
criteria. In providing a broad framework for the selection of the best value-for-money bidder, the
following have been covered.
The Evaluation Process (depending on whether a one- or two-stage procurement method is used) will
generally follow the following steps:
• Evaluation team members complete conflict-of-interest declarations
• Evaluation team members are briefed on the evaluation process and receive training if necessary.
• Copies of the bidders’ offers, scoring scale and evaluation forms are distributed to each evaluation
team member.
Note: If a two-envelope method is being used (i.e. price has been requested in separate, sealed
envelopes) only the non-price parts of the offers are distributed to the evaluation team members, with
the price envelopes remaining unopened and securely stored
• Evaluation team members complete individual assessments of the bidders’ offers. This ensures each
evaluation team member has adequately reviewed the offers prior to the team evaluation meeting and
that no single team member is able to exert inappropriate influence on the outcome by being better
prepared.
• The evaluation team members meet and complete a “team” evaluation. Team scores may be reached
by a number of methods, with averaging the individual scores and reaching a team consensus score
being the two most common. Of the two, the consensus scoring method is the most frequently used, as
it allows a score to be agreed based on consideration of all the evaluation team members’ opinions and
observations. The averaging method does ensure a persuasive or dominant individual is not able to
exercise undue influence. However, it also means an individual’s scores that, based on reconsideration
Technical Evaluation Guidelines for Selection of Vendor for APTS
6 | P a g e
of the supplier response in light of other team members’ comments would have been higher or lower,
still form part of the calculation for a final score
• If price has been requested in separate, sealed envelopes, these envelopes are opened after the team
scoring of the non-price criteria and a ranking agreed by the evaluation team. The ranking method will
be determined by the evaluation model used. For example:
- Lowest-priced conforming (LPC) offer is automatically is the most responsive bidder
- Highest-scoring bidder based on QCBS assessment is the most responsive bidder
• As a matter of “Good Practice” undue weightage should not be given to the presentation. The key
reason being that the presentations are quite often delinked from the proposal submitted and hence
may not be contractually binding on the bidder. The bidders may be invited to come and discuss the
areas requiring clarifications in the bids OR they can be asked to present the documents in the form
submitted to the Nodal Agency highlighting all the evaluation areas. In this manner the evaluation
committee can review the evaluation done by one member / support team. However this may require a
higher time to be budgeted for such discussions. It may be noted that the bidders should not be allowed
to change any documents (for e.g. new project citation and work order cannot be taken into cognizance
at this stage). However any document substantiating the clarifications asked by the evaluation team,
may be provided.
• Reference-checking or further due diligence is carried out if the bidder is not well known to the nodal
agency. The purpose of the reference-checking is to verify the assessed level of performance, capability
or expertise and to satisfy the agency generally that the supplier is likely to deliver what it has offered.
Further due diligence, such as financial viability checks, is generally only conducted where the
procurement is high risk, high profile or high value
A rightfully defined and objective technical evaluation criteria & methodology ensures that the most
competent bidder offering the right quality solution/service and price is selected - the Most Responsive
Bidder– best solution and not just best price.
Technical Evaluation Guidelines for Selection of Vendor for APTS
7 | P a g e
2 Designing Technical Evaluation
As a part of the design for Technical evaluation, the Nodal Agency has to make decisions/choices on the
following things, but not limited to:
- Evaluation model: lowest price/weighted attribute etc.
- Develop Rating Scale to guide evaluation panel scoring
- Determine panel decision making process: mathematical average/panel moderation
-Identify the information required from suppliers: supplier details/response to
requirements/pricing/format etc.
- Identify any required additional steps: interview/presentation/site visit(s) etc.
- Identify any optional additional steps: reserve the right to interview/presentation/site visit etc.
- The need for Government department, Ministry due diligence requirements (if any).
Technical Evaluation Guidelines for Selection of Vendor for APTS
8 | P a g e
3 Technical Evaluation Criteria
Technical Evaluation Criteria are the bid response parameters on which the evaluation is carried out to
arrive at a final (technical) score for each qualified agency.
Hence, the Technical Evaluation Criteria SHOULD:
Be as objective as possible, breaking the scoring down to individual identifiable components
Have direct and perceptible linkage to nature and scope of work
Use the most relevant scoring / weighting scheme to evaluate; weighting should be basis their
importance to the Government or project’s outcomes. The weightings must be disclosed in the
tender document.
Establish the scoring guidelines prior to contacting bidders creating the RFP. Then, when the
proposals are received, score them based on the criteria established in the RFP
Have scoring for each component of the solution rather than an overall score for the solution
Provide weights / maximum marks for each Technical evaluation criterion; weights should be as
per their importance to the project or project’s outcomes and must be disclosed in the tender
document.
In case of a software solution, evaluate the coverage of or degree of match to functional and
technical requirements by the solution
It may be a good practice that the evaluations should be done with the maximum score of 100 marks, so
that appropriate resolution could be provided to the criterions having less marks / sub-criterions.
Based on the above advisory, the evaluation criterion has to be chosen very carefully, as it defines the
filtering criterion on which the final Implementation Agency is to be shortlisted. Generally based on
qualitative assessment, one can categorize the projects in two categories:
Category ONE:
Under this category, the RFP is for projects where there is high level of clarity on the technology and the
solutions. These would be typical implementation of COTS/ERP projects OR any State MMPs. In e-
Governance space, these projects would be applications which have a simple citizen application
workflow. In such projects, the risk of technology feasibility is less.
In such RFP technical evaluation should provide clarity on solution till the Bill of Material stage in the
RFP document. It may be noted that in this case, the responsibility of technical feasibility of the
proposed solutions rests with the Nodal Agency.
Typically these should be on L1 basis.
Technical Evaluation Guidelines for Selection of Vendor for APTS
9 | P a g e
Category TWO:
Under this category, the RFP is for projects where there is inadequate clarity on the solution. For e.g.
these may be any large scale implementation of any Central MMP. These are risky projects and should
be on QCBS evaluation 70:30 (quality: Cost).
It may be noted that in this case, the responsibility of technical feasibility of the proposed solutions rests
with the Bidder. However the Proposal Evaluation Committee in this case should have expertise or
should have access to expertise to objectively evaluate & compare the various solutions components
proposed by the bidders.
In such projects the Nodal Agency should do a due diligence critical parameters of the project covering
System Functionality, Technology, current performance on key technologies proposed in isolation &
together as a stack, details on implementation experience of the bidder, Training methodology,
performance in Proof of concept (in case PoC is planned), Certifications, Past experience of the bidder in
executing similar assignments, size of those assignments, profile of team members and Project
Methodology.
3.1 Category One: Suggested Technical Evaluation
[The X1, X2 and X3 values in the table below has to be filled in such a manner that a) it is above the
value provided in the eligibility criterion b) it allows at least 8 agencies to score maximum marks for the
criterion]
S.No Criteria Basis for Valuation Max Marks Supporting
Company Profile 10
1. Average turnover from
System Integration/ICT
Systems Development
and Implementation
Work in last 3 years
(Turnover in Rs. Crores)
Greater than or equal to
<X1>: 10 marks
Between X2 and X1: 8
marks
Between X3 and X2 : 6
marks
Less than X3: 0 marks
[X1>X2>X3 and are
average Revenue Turnover
for last 3 financial years]
10 Extracts from the
audited Balance sheet
and Profit & Loss; OR
Certificate from the
statutory auditor
Relevant Strengths 60
2. Hardware and Network
Equipment Supply &
maintenance services to
When No. is :
equal to or more than 5
projects : 20 marks
20 Completion
Certificates from the
client; OR
Technical Evaluation Guidelines for Selection of Vendor for APTS
10 | P a g e
be demonstrated in a
maximum of <5 Nos.>
engagements of value
more than Rs. <50% of
the estimated value of
the hardware being
procured under this
project>.The work order
should have been issued
within the last 5 years,
as on <date>.
The projects should
have been either
completed or an on-
going project where
deliverable or milestone
has been successfully
met.
Weightages (W)
In case project is
completed and letter of
satisfaction available :
100% weightage
In case project in
progress and the Work
Order is more than 18
months old and letter of
satisfaction available:
80% weightage
In case project in
progress and the Work
Order is between 12- 18
months old and letter of
satisfaction available:
50% weightage
In case project in
progress and the Work
Order is less than 12
months old and letter of
satisfaction available:
25% weightage
equal to 4 projects : 16
marks
equal to 3 projects : 12
marks
equal to 2 projects : 8
marks
Less than 2 projects : 0
marks
The maximum marks for
each project is 4 marks.
These marks would be
multiplied by the
weightage as defined in
the previous column to
arrive at a cumulative
score.
Work Order + Self
Certificate of
Completion (Certified
by the Statutory
Auditor); OR
Work Order + Phase
Completion Certificate
(for on-going projects)
From the client
3. Software Development When No. is : 20 Completion
Technical Evaluation Guidelines for Selection of Vendor for APTS
11 | P a g e
& maintenance services
to be demonstrated in a
maximum of <Nos.>
engagements of value
more than <50% of the
estimated value of the
Software being
procured under this
project> that have
either been completed
or an on-going project
where deliverable or
milestone has been
successfully met
relevant to the
experience.
The work order should
have been issued within
the last 5 years, as on
<date>.
Weightages (W)
In case project
completed and letter of
satisfaction available:
100%
In case project in
progress and the Work
Order is more than 18
months old and letter of
satisfaction available:
80%
In case project in
progress and the Work
Order is between 12- 18
months old and letter of
satisfaction available:
50%
In case project in
progress and the Work
Order is less than 12
months old and letter of
satisfaction available:
equal to or more than 5
projects : 20 marks
equal to 4 projects : 16
marks
equal to 3 projects : 12
marks
equal to 2 projects : 8
marks
Less than 2 projects : 0
marks
The maximum marks for
each project is 4 marks.
These marks would be
multiplied by the
weightage as defined in
the previous column to
arrive at a cumulative
score.
Certificates from the
client; OR
Work Order + Self
Certificate of
Completion (Certified
by the Statutory
Auditor); OR
Work Order + Phase
Completion Certificate
(for on-going projects)
from the client
Technical Evaluation Guidelines for Selection of Vendor for APTS
12 | P a g e
25%
4. Turnkey services to be
demonstrated in a
maximum of <Nos.>
engagements of value
more than <50% of the
estimated value of the
turnkey services being
procured under this
project> that have
either been completed
or an on-going project
where deliverable or
milestone has been
successfully met
relevant to the
experience.
The work order should
have been issued within
the last 5 years, as on
<date>.
Weightages (W)
In case project
completed and letter of
satisfaction available:
100%
In case project in
progress and the Work
Order is more than 18
months old and letter of
satisfaction available:
80%
In case project in
progress and the Work
Order is between 12-18
months old and letter of
satisfaction available:
50%
In case project in
progress and the Work
Order is less than 12
months old and letter of
When No. is :
equal to or more than 5
projects : 20 marks
equal to 4 projects : 16
marks
equal to 3 projects : 12
marks
equal to 2 projects : 8
marks
Less than 2 projects : 0
marks
The maximum marks for
each project is 4 marks.
These marks would be
multiplied by the
weightage as defined in
the previous column to
arrive at a cumulative
score.
20 Completion
Certificates from the
client; OR
Work Order + Self
Certificate of
Completion (Certified
by the Statutory
Auditor); OR
Work Order + Phase
Completion Certificate
(for on-going projects)
from the client
Technical Evaluation Guidelines for Selection of Vendor for APTS
13 | P a g e
satisfaction available:
25%
Approach &
Methodology
20
5. Solution Proposed
Demonstration of
understanding of the
Department’s
requirements
Qualitative assessment
based on Demonstration
of understanding of the
Department’s
requirements through
providing:
− Solution proposed and
its components,
− Technologies used,
− Scale of implementation,
− Learning on Issues
− Challenges
− Challenges likely to be
encountered
− Mitigation proposed
− Client references:
15 A note
6. Approach and
Methodology to
perform the work in this
assignment
Qualitative assessment
based on
− Understanding of the
objectives of the
assignment: The extent to
which the Systems
Implementer’s approach
and work plan respond to
the objectives indicated in
the Statement/Scope of
Work
− Completeness and
responsiveness: The extent
to which the proposal
responds exhaustively to
all the requirements of all
the Terms of Reference
2.5 A note
7. Project work break
down structure
Qualitative assessment
based on timelines,
resource assignment,
dependencies and
milestones
2.5 A note
Technical Evaluation Guidelines for Selection of Vendor for APTS
14 | P a g e
Resource Profile 10
8. Resume of all key
technical resources
proposed for the
assignment
Qualitative Assessment 10
Inclusion of MSME Case to Case Basis
9. Bidder’s inclusion of
MSMEs in project
delivery through
allotment of at least
<10%> of contract value
to the project
As per Requirement As per
requirement.
[2 to 5 marks
to be allotted]
Letter of evidence and
commitment that
MSME will be
contracted for the
required value of
work.
Tools & Assets Case to Case Basis
10. Tools and Assets which
could be leveraged for
the assignment [for e.g.
Test Case Builders,
Effort Estimators, PMU
Tool, Load testing etc.,
depending on the
relevance to the Scope
of work]
As per requirement As per
requirement
A note and relevant
supporting
Technical Evaluation Guidelines for Selection of Vendor for APTS
15 | P a g e
3.2 Category Two: Suggested Technical Evaluation
S. No Criteria Basis for valuation Max Marks Supporting
1 System Functionality
Meeting the requirements of <department> in terms of how close the proposal is to the functional requirements for the solution as have been proposed for <department> (In case it is COTS, it should be measured by degree of customization required)
20% Compliance Note
2 Technology Demonstrated robustness of the technology deployed across other installations around the world, including – Scalability – Security – Ease of implementation
20% Note
3 India Specific Capabilities
Qualitative assessment based on the number of Projects of similar nature in India and size of those projects.
7% Note; and Completion Certificates from the client; OR Work Order + Self Certificate of Completion (Certified by the Statutory Auditor); OR Work Order + Phase Completion Certificate (for on-going projects) from the client
4 Industry Specific Capabilities
Qualitative assessment based on the Past experience of the bidder in executing similar assignments, size of those assignments. [The definition of “similar” should be such that it focuses on the areas which are “innovative” or where the technical feasibility is a
7% Note; and Completion Certificates from the client; OR Work Order + Self Certificate of Completion (Certified by the Statutory Auditor); OR Work Order +
Technical Evaluation Guidelines for Selection of Vendor for APTS
16 | P a g e
challenge in the context of the project]
Phase Completion Certificate (for on-going projects) from the client
5 Training Trainings proposed by the bidder and the amount of emphasis laid on Training the employees schedule details, locations, sessions and their description
7% Note
6 Certifications and Credentials
Relevant certifications (SEI-CMMI, ISO, etc.)
7% Copy of certificates
7 Profile of proposed team members
Relevant assignment experience / Years of experience / Number of Certifications in Technology specific to Solution proposed
20% CVs
8 Project Methodology, Support and Documentation
Qualitative assessment based on − Understanding of the objectives of the assignment: The extent to which the Systems Implementer’s approach and work plan respond to the objectives indicated in the Statement/Scope of Work − Completeness and responsiveness: The extent to which the proposal responds exhaustively to all the requirements of all the Terms of Reference
7% Note
Technical Evaluation Guidelines for Selection of Vendor for APTS
17 | P a g e
9 Inclusion of MSMEs in Project Delivery
As per requirement. 7% Letter of evidence and commitment that MSME will be contracted for the required value of work.
Each of these should be evaluated further in detail based on the following checklist. There would be
page, paragraph and line references for each criterion for easy reading and scoring by the evaluation
Committee.
Technical Evaluation Guidelines for Selection of Vendor for APTS
18 | P a g e
4 Detailed Methodology for Evaluation and Parameters
4.1 System Functionality: 20 Points
The Evaluation template should contain the Functional & Non Functional requirements.
<<A sample of the Evaluation checklist which should be made is as follows:>>
ID Title Description Effort Experience Sample Demo
Comments
1. Projected Demand for Service
Store data on the Projected Demand for Service over a planning horizon (say 3-5 years as decided by <DEPARTMENT>).
3 4 0
2. Information on Licensing Procedures
Store information on Licensing Procedures and related information that needs to be shared with the Applicant (This information could be displayed in the public website / printed in hard copies and dispatched to interested parties
3 4 0
……. ……
……. ……
In the comments column of the Evaluation Template, give a score from 0 to 10 for each of the
functionality.
For convenience divide the score as follows:
- Effort - 0-5 points
- Experience - 0-4 points
Technical Evaluation Guidelines for Selection of Vendor for APTS
19 | P a g e
- Sample Demo - 0-1 points
For each functionality give scores as follows:
- "Effort"- Lower the effort, more the score
- "Experience"- Higher the experience more the score
- “Sample Demo” - Points only if answer is "Yes".
E.g.
Bidder A: If Effort = 5 days, Experience = 3, Demo = N, then score = 7(4+3+0).
Bidder B: If Effort = 8 days, Experience = 1, Demo = Y, then score = 4(2+1+1).
Add scores given to each functionality (each row) for all the Modules to arrive at System Functionality
Score. Convert the each Total Score to a 20-Point Scale
Consider the following Example for explanation.
Assume that the entire Module has 160 functionalities. Each of the functionalities would carry 10 Points;
hence the Full Score would be 1600. Assume that a Bidder gets a Total Score of 1100 out of the Full
Score of 1600.
Then the Score on 20 Point Scale is (1100/1600) * 20 = 13.75
For large projects, even if it is a COTS solution, the evaluation can be carried out in this manner.
However the scoring needs to be carried out on the following basis.
- If functionality required is readily available: 5 Points (Maximum points)
- If functionality NOT readily available but can be customized through a work-around - 4 Points
- If functionality is NOT available and cannot be customized, but can be made available in a future
version within 3 months: 2 Points
- If NOT available at all: No Points – 0 (minimum).
Add score for each functionality for ALL modules to arrive at the System Functionality Score
Convert the each Bidder Total Score to a 20-Point Scale in the similar way as described above.
Technical Evaluation Guidelines for Selection of Vendor for APTS
20 | P a g e
4.2 Technology: 20 Points
The responses by Bidders would be reviewed for the following technology aspects of the proposed
solution:
- Specifications (platform, software, database design, etc.)
- Security and Scalability:
- Ease of Implementation
Also information would be collected from Bidder or references (other clients for which solutions were
implemented).Bidder furnishing comprehensive information closest to <DEPARTMENT>’s expectations
as listed in shall be awarded the maximum points.
Overall Scoring would be on a 20 – Point Scale.
Please refer the following table for Scoring Template.
Evaluation Criterion Benchmark / Preferences Rating Scale
1. Architecture
Application architecture Partitioned into three tiers (0-2)
Flexibility of these layers residing in different
hardware platform (0-1)
Justification on the suggested application
architecture with rationale and benefits (0-5)
0-2
0-1
0-5
Base foundation of major
sub-components.
Maintenance and
enhancements
Information provided on Major sub-
components and whether they are proprietary
OR Open (0-1)
Willingness to customize for <DEPARTMENT>
(0-2)
Ownership of source code for Customised
solution made available to <DEPARTMENT>
(0-1)
Description of process adopted by
bidders on maintenance and enhancements
(0-3)
Detailed description of Software development
Methodology (0-5)
0-1
0-2
0-1
0-3
0-5
Technical Evaluation Guidelines for Selection of Vendor for APTS
21 | P a g e
Distributed Component
Architecture
As a Best practice, and considering the
scalability of the solution, a Distributed
Component Architecture is preferred (0-4)
In case Bidder has suggested its own
architecture other than the above, based on
the reasoning provided assessment would be
done (0-3)
Clear description on customisation of
modifying existing business rules / introducing
new business rules (0-3)
0-4
0-3
0-3
2. Security
Identification
Data Integrity
Data Encryption
Data Confidentiality
Description of user authentication procedure
for the proposed application (0-3)
Details on Access Controls and Privileges
through application (0-3)
0-3
0-3
Data Availability
Auditability
Encrypted storage of Username and
Passwords (0-1)
Details on restriction of access to data content
in the database (0-2)
0-1
0-2
Firewalls and security
supported by the system.
Description of firewalls and security
components (such as intrusion detection
systems) in the proposed solution (0-5)
0-5
3. System Performance
Peak Volume and
associated average
response times,
Concurrency
Average volume handling capacity (0-4)
Database update response time (0-3)
Database retrieval response time (0-5)
Compliance with over and above the
minimum requirement specified for
concurrent users (0-2)
0-4
0-3
0-5
0-2
Benchmark studies on
technology components
Performance details for
previous projects in case
of developed solution
Availability of Graphs, Tables, Whitepapers
and related statistics of benchmarked studies
on performance (0-5)
0-5
0-3
Scalability Descriptions of procedure for scaling up the
application for additional users and additional
features furnished. (0-3)
Extent of dependency on the bidder for
inclusion of new features (0-2)
Compliance with increasing the user base to
the desired level, as specified by
0-3
0-2
0-5
Technical Evaluation Guidelines for Selection of Vendor for APTS
22 | P a g e
<DEPARTMENT> with least effort (time and
cost) would be given higher score (0-5)
Need for additional infrastructure such as
system software licenses / new hardware
requirements etc. Lower the need higher the
score (0-2)
0-2
4. Supported Platform O/S
Server platforms Detailed information on server platform with
adequate rationale would be assessed for.
More options for server platforms - higher the
score (0-2).
0-2
Operating System (OS) Details of OS the product would support.
More the options – higher the score (0-2)
0-2
Platform/OS combination Statistical information on the platform / OS
combination to support the rationale
furnished. (0-3)
0-3
5. Client Hardware/ OS
Hardware/OS
combinations for client
Details of hardware / OS combination
proposed for client systems with rationale.
(0-3)
0-3
User Interfaces Browser based user interfaces would be
preferred for suggested application,
considering the accepted standards in use
currently. (0-3)
In case browser based user interface not
suggested, the alternatives would be judged
based on the reasoning provided by bidder
(0-2)
0-3
0-2
6. Database/ Directory Support
Databases Relational databases which are being
supported adequately in present day context
would carry max score (0-3)
Hierarchical / flat files databases – low score
(0-1)
0-3
0-1
Product information (MS /
Oracle)
Information on the Database products with
recommendation and rationale (0-1)
0-1
Directory Services Information on system support to Directory
Services (0-1)
0-1
7. Data Architecture
Technical Evaluation Guidelines for Selection of Vendor for APTS
23 | P a g e
Logical and/or physical
data model.
Description of the proposed data models for
the application, supported by data models of
earlier developed applications (0-3)
0-3
Customisation Assess build-in flexibility for introducing
changes at field level
0-3
Control features for data
integrity
Details on Control features to ensure data
integrity such as updates, totals, cross-
checks, validations etc. (0-3)
0-3
Transaction management
system
Details of the transaction management
system (0-2)
0-2
Database layout Information on Database layout supported
with whitepaper(s) on database (0-2)
0-2
8. System management
Rollback, recovery, and
fault tolerance
Description of database failover, rollback,
recovery provisions and fault tolerance (0-3)
0-3
System management tools Details on provision of accepted system
management tools and utilities, with
appropriate notification features for effective
system administration. (0-2)
0-2
Accessibility of system
management interfaces.
Details on accessibility of system
management interfaces and provision for
remote administration. (0-2)
Facility from bidder on Web-based support
(on-line) would be preferred (0-2)
0-2
0-2
9. Web Server Support
Web servers compatibility Considering a future scenario of web-
enablement, integration with Web Server
would be assessed (0-5)
0-5
Functionality reside on the
web server.
Number of functionalities that can reside on
the web server and the level of difficulty
associated in making it possible (0-3)
0-3
10. Application Server Support
Application Server
integration
Proposed Solution running on Application
Server that is established and well known
would be preferred (0-3)
0-3
Interface between your
product and applications
servers
Details on the interface of the proposed
solution with application servers would be
assessed (0-2)
0-2
11. Single Sign-on
Integration with any Single
Sign-On packages
Provision to integrate with single-sign on
packages (0-1)
0-1
Technical Evaluation Guidelines for Selection of Vendor for APTS
24 | P a g e
12. Presentation requirements
Interfacing with a
presentation system
Information on presentation system would
be assessed for details (0-4)
0-4
Level of in-house
customisation possible
Ability to customise screens (0-2) 0-2
Browsers support Cross-browser support would be preferred
(eg. IE / Mozilla) (0-2)
0-2
Embedded browser
languages
Accepted scripting language within the
boundaries of sand-box. (0-2)
Client side Active-X controls would not be
preferred.
0-2
Interface design
compliance with standards
Usage of Standards such as optimised images
/ 256 colour palette would be preferred (0-2)
0-2
13. Session Management
Concurrent users, multiple
sessions
Description on handling of concurrent users,
multiple sessions, session cookies. (0-3)
0-3
System’s policy on session
time-outs.
Policy details on session time-outs (0-1) 0-1
14. Integration Capabilities
Integration capabilities
with external, third party
applications
Not a very significant point at this stage for
<DEPARTMENT>. However, if the bidder can
substantiate capability of the proposed
solution with commercially known third
party solution, it would be surely preferable.
(0-1)
0-1
15. Auditing / Reporting
Log files Complete audit trails and log filing features
of the proposed COTS as well as Custom
solution would be evaluated (0-5)
0-5
Log files be customization Ability of the log files to be customized and
scheduled; (0-2)
Access rights to view log files (0-1)
0-2
0-1
Querying capabilities Details of querying capabilities, and user-
friendliness of the user logs (0-2)
0-2
Procedure of audit trails Procedure of audit, reporting and review of
the proposed solution (0-2)
0-2
16. Disaster Recovery and Back-up
Disaster recovery
procedures
Description of the disaster recovery
procedures for data, application and client.
Details of the procedure address adequate
scenarios and the actions thereof (0-3)
0-3
Technical Evaluation Guidelines for Selection of Vendor for APTS
25 | P a g e
Archival policy Details of archival policy and procedure for
the proposed solutions. Higher the degree of
automatic features / scheduling in the
archival procedures, more would be score (0-
3)
0-3
17. General
Additional utility package
required
Completeness in the list of any additional
utilities (software / infrastructure) would be
assessed (0-2)
0-2
Any limitations in the
software/operating
system/file manager
Description of the limits in terms of number
of table entries, database size would be
assessed (0-2)
0-2
Interdependencies in the
modules
Details of the interdependencies across
modules and possible implementation
constraints would be assessed. (0-2)
0-2
Import / export facilities Provision of import / export utilities
especially to facilitate data entry / report
generation (0-2)
0-2
End-of-day processes Clear write-up on the end-of-day processes
and the associated dependencies (0-2)
0-2
Application support Compatibility with WLAN environment (0-3) 0-3
Total 183
Assign scores for each of the line items based on Bidder responses.
Arrive at sum totals for Bidder Total Scores.
Example:
Let’s say the Bidder A gets 120 out of 183.
Conversion of above score on a 20-point scale is as follows:
120 / 183 * 20 = 13.115
Technical Evaluation Guidelines for Selection of Vendor for APTS
26 | P a g e
4.3 India Specific Capabilities: 8 points
For assessing this criterion, the projects may not be limited to India, and can also include similar
countries having federal structure [or any other categorization as relevant] as India
Basis for awarding scores for this criterion is as follows:
- Number of clients using the product: 0-4 Points
If No. of clients using the product are:
More than 15: 4 Points
Between 10- 14: 3 Points
Between 5 – 9: 2 Points
Between 1- 4: 1 Point
- Maximum Size of the Installations by bidder: 0-4 Points
More than 150 Users: 4 Points
Between 50 – 150 Users: 3 Points
Between 25 - 50 users: 2 Points
Up to 25 Users: 1 Point
Example:
Number of Clients in India are 12; The Score is 3
Average No. of users = 100,
Then Bidder Total Score is 6 (i.e., Sum Total of 3 and 3).
Scoring would be on a 8-Point Scale
4.4 Government Specific Deployment: 7 Points
Number of Govt. clients using the product: 0-4 Points
More than 15: 4 Points
Between 10- 14: 3 Points
Between 5 – 9: 2 Points
Between 1- 4: 1 Point
Average Size of the Installations: 0-3 Points
More than 100 Users: 3 Points
Between 50 – 100 Users: 2 Points
Upto 50 Users: 1 Point
Scoring would be on 7-Point Scale
Technical Evaluation Guidelines for Selection of Vendor for APTS
27 | P a g e
Example:
Number of Clients in India are 8; The Score is 2
Average No. of users = 75,
Then bidder Total Score is 4 (i.e., Sum Total of 2 and 2).
Only in this case the projects will have to be specific for the Government / Public Sector.
Based on the information provided by Bidder, <Nodal Agency> would assess the client referenced in
relevant areas. The bidder must agree to allow <Nodal Agency> to check relevant details / feedback
from clients OR make site visits.
Bidder would provide information on each reference assignment for which the firm was legally
contracted. Based on an overall assessment, a score on a 7- point scale will be given.
4.5 Training: 10 Points
It will be based on the emphasis the bidder lays on providing training to the employees of
<DEPARTMENT>.
This can be measured primarily at the time of Bidder Presentation, and on a secondary basis through the
following:
- Training Methodology (including the handing over of the training material to the Nodal Agency, CBT or
web based, Toolkits etc.)
- Layers of Training
- Time devoted for Training
- Training Kit
- Assessment Process
Evaluation Criterion Benchmark / Preferences Rating Scale
Training model
Broad Course Content would be checked if it addressed all the relevant modules / functionalities specified by <DEPARTMENT> (0-4) Layers of Training proposed based on types of users: Top Management, Core Team and End User Group Phases of Training depending on the Project Time Lines – Basic Training during the beginning phase (0-2) – Module specific training as the system is configured and Pilot is being build (0-3) – Hands-on Training when the system is read for User Acceptance Tests (0-3)
0-4 0-3 0-10
Technical Evaluation Guidelines for Selection of Vendor for APTS
28 | P a g e
– Post implementation Training (0-2)
Training Approach
Train the Trainer As an accepted good practice, ‘Train the Trainer’ approach would be preferred (0-1) Number of Training Programs, Duration of each Program and Evaluation methods suggested for the Train the trainer would be assessed (0-3)
0-4
Alternative Approaches
In case the Bidder suggests any other alternative approaches, it would be evaluated suitable based on the rationale provided (0-3)
0-3
Training Deliverables
Training Aids In form of hand-outs detailing each module with suitable examples and assignments, as and when class-room trainings are organised Complete Training Manual on completion of all trainings, before the Go-live which would cover final configuration of the solution
0-5 0-5
Prior Training Experience
Prior training imparted during earlier project could be furnished for the following details:
Training Programme
Client
Duration
Number of Batches and Trainees Sample Training Documentations in earlier projects could be furnished by bidders
0-3 0-3
Total Score 40
Based on Bidder’s Responses, assign score to each line item detailed in the table above. Add up scores
to arrive at a Bidder Total Score and convert the total score to 10-Point Scale.
For Example: If the Bidder Total is 25 out of the Total Score 40
On a 10-Point Scale the Score would be 25/40 * 10 = 6.25
Technical Evaluation Guidelines for Selection of Vendor for APTS
29 | P a g e
4.6 Certification, Credentials: 7 points
This criterion would be used to assess the reputation of the company in the market that it is operating
in. Here companywide certifications should be given scores.
The scores for this criterion may also be supplemented by response received from the Bidder references
(prior clients). Responses from the Bidder References would be based on telephonic discussions /
personal visits to the referred personnel (as specified by the bidders) based on Credentials submitted by
Bidders in the format in the RFP document.
The following table provides the evaluation guidelines:
Evaluation Criteria Benchmark/ Preferences Rating Scale
Certification of Bidders
CMMi Level for the organisation (1 point for each level)
ISO Certification / Six Sigma Certification
Any other recognised certification from reputed authority for
the organisation
0-5
0-3
0-2
Feedback from Bidder Responses
Based on discussions with / visits to bidder’s referred clients scores would be awarded.
0-7
Total Score 17
Based on Bidder’s responses and Referred client feedback, assign score to each line item in above table
and arrive at the Bidder Total Score.
Convert the score to 10-Point Scale. (eg. If a bidder gets 14 out of 17, then a 7-Point Scale equivalent is
5.76 (14/17*10))
4.7 Profile of Proposed Team Members: 20 Points
A consolidated spread sheet for the sub-criteria would be tabulated in an excel sheet for easier
evaluation
The Scoring for this criterion will be simple yet significant.
The key parameters for evaluating the team members would be:
- Team Composition
- Years of Experience (out of which relevant experience would be considered for evaluation)
- Qualification
- Certifications
- Number and type of assignments handled
Technical Evaluation Guidelines for Selection of Vendor for APTS
30 | P a g e
- Proficiency in Local Language
Please refer following table for line-wise breakup
Evaluation Criterion Benchmark / Preferences Rating Scale
Overall Project Team
Structure
Composition of the Project Team Structure
proposed by Bidder
- Appropriate Number of Team members
with justification
- Roles and Responsibilities clearly defined
0-5
0-5
Total Years of Professional
Experience
Project Manager preferably to have 8 years
of experience
Functional Consultants & Technical Leads (to
preferably have 5 years of experience
Database Administrators to preferably have
5 years of experience
Consultants / Developers to have at least 3
years of experience
Process Consultant for Process
Documentation
0-16
0-5
0-5
0-3
0-5
Qualification Basic Qualification for each team member
should be either of the following:
Bachelor’s degree in Engineering
Master’s degree in Computer Science / Information Systems / MBA
Membership from reputed Chartered Accountancy institute
MCA / PGDCA
0-8
Certifications Project Management Certification for Project
Manager Technology Specific Certification
for Team members
0-2
Number of similar
assignments
Project Manager must have handled
preferably 8 assignments of similar nature
Other team members must have worked in
at least 2 complete implementation /
development
Experience in Government Sector / Utilities
0-4
0-4
0-2
Proficiency in Local
Language
At least 1 team member from each module
must be conversant in Local Language
Speaking and Writing
0-5
0-3
Technical Evaluation Guidelines for Selection of Vendor for APTS
31 | P a g e
More members in the team knowing Local
language would be seen as an advantage
Total 75
Based on Bidder’s responses, assign score to each line item detailed in the table above add up scores to
arrive at a Bidder total score convert the total score to 20-point scale
EXAMPLE:
If the Bidder’s Total Score is 45 out of the Total Score 75
On a 20-Point Scale the Score would be 45/75 * 20 = 12
4.8 Project Methodology, Support & Documentation: 8 points
Refer to main document & addendums
Evaluation Criterion Benchmark / Preferences Rating Scale
Project Methodology Software Development Methodology using the
accepted standards; evaluation would be based
on the rationale and clarity furnished in the
bidder’s response
Implementation methodology of solution would
be evaluated to assess bidder’s understanding
of <DEPARTMENT>’s requirements to comply
with the deliverables, training, handholding and
post-implementation support required to make
project successful.
0-8
0-8
Support Problem reporting and resolution
mechanism
Hotline Support
Simultaneous Support of various releases
Willingness to support customisation of
solutions
Handling Change Requests
Details of User Discussion Forum (s)
Escalation Mechanism
Future Upgrades
0 - 4
Documentation List the documents provided with your custom
solutions
User manuals
System Administration manual
0-3
Technical Evaluation Guidelines for Selection of Vendor for APTS
32 | P a g e
Security
System manuals – Architectures, Entity-
Relationship diagrams, Source code etc.
On-line Help
Documentation updates to correspond with
each software releases
Total Score 23
Based on Bidder’s Responses, assign score to each line item detailed in the table above Add up scores to
arrive at a Bidder Total Score Convert the total score to 8-Point Scale
Example:
If the Bidder Total is 15 out of the Total Score 23
On a 8-Point Scale the Score would be 15/23 * 8 = 5.22