[Federal Register Volume 64, Number 181 (Monday, September 20, 1999)]
[Notices]
[Pages 50854-50861]
From the Federal Register Online via the Government Publishing Office [www.gpo.gov]
[FR Doc No: 99-24363]
-----------------------------------------------------------------------
DEPARTMENT OF TRANSPORTATION
Federal Highway Administration
Federal Transit Administration
Transportation Equity Act for the 21st Century; Guidelines for
the Evaluation of Operational Tests and Deployment Projects for
Intelligent Transportation Systems (ITS)
AGENCIES: Federal Highway Administration (FHWA), Federal Transit
Administration (FTA), Department of Transportation (DOT).
ACTION: Notice.
-----------------------------------------------------------------------
SUMMARY: This document provides implementation guidance for section
5204(j)(1) of the Transportation Equity Act for the 21st Century (TEA-
21) 23 U.S.C 502 Note. Guidelines and requirements for the evaluation
of
[[Page 50855]]
operational tests and deployment projects for ITS ensure the
objectivity and independence of project evaluators to avoid any real or
apparent conflict of interest or potential influence on the outcome by
parties to such tests and projects. They also establish evaluation
funding levels based on the size and scope of each test or project to
ensure adequate evaluation. The ITS Joint Program Office (ITS JPO)
plans to disseminate the TEA-21 Evaluation Guidelines to affected
program offices within the DOT. Should it be deemed necessary to
establish any requirements for the evaluation of operational tests and/
or deployment projects, these would be established pursuant to
rulemaking to be issued in the future.
FOR FURTHER INFORMATION CONTACT: For technical information: Mr. Joe
Peters, (202) 366-2202, ITS Joint Program Office, (HOIT-1) FHWA. For
legal information: Mr. Wilbert Baccus, Office of the Chief Counsel,
FHWA (HCC-32), (202) 366-0780; Ms. Linda Sorkin, Office of the Chief
Counsel, FTA (TCC-24), (202) 366-1936. All are located at the U.S.
Department of Transportation, 400 Seventh Street, SW., Washington, DC
20590. Office hours are from 8 a.m. to 4:30 p.m., Monday through
Friday, except Federal holidays.
SUPPLEMENTARY INFORMATION:
Electronic Access
An electronic copy of the document may be downloaded using a modem
and suitable communications software from the Government Printing
Office's Electronic Bulletin Board Service at (202) 512-1661. Internet
users may reach the Federal Register's home page at: http://
www.nara.gov/fedreg and the Government Printing Office's database at:
http://www.access.gpo.gov/nara. The TEA-21 Evaluation Guidelines may
also be accessed at the U.S. DOT's ITS home page, through the
evaluation link, at http://www.its.dot.gov.
Background
Section 5204(j)(1) of TEA-21, Pub. L. 105-178, 112 Stat. 107, 455
(1998), states that the Secretary of Transportation (Secretary) shall
issue guidelines and requirements for the evaluation of operational
tests and deployment projects for the ITS. This document sets forth
TEA-21 Guidelines that are required to include provisions to ensure
objectivity and independence of evaluators so as to avoid real or
apparent conflicts of interest on the outcome of project evaluations.
The guidelines are further required to establish evaluation funding
levels based on the size and scope of each test or project to ensure
adequate project or operational test evaluations. Should it be deemed
necessary to establish any requirements for the evaluation of
operational tests and/or deployment projects, these would be
established pursuant to rulemaking to be issued in the future.
The objective of the TEA-21 Evaluation Guidelines is to fulfill the
requirements imposed on the Secretary in the referenced section by
accomplishing the following:
1. Defining the different categories of projects carried out under
subtitle C;
2. Defining, in general terms, different types of evaluations to be
conducted by projects in the categories defined;
3. Establishing criteria to guide the selection of evaluations to
be performed;
4. Defining, in general terms, procedures for ensuring objectivity
and independence of evaluating organizations;
5. Defining the funding mechanism to provide project evaluation
resources;
6. Providing a general description of the procedures [and
requirements] that project partnerships can expect when participating
in the types of evaluations defined;
7. Providing access through web site address to detailed evaluation
procedures to include examples of specific evaluation plans, test
plans, and reports.
The TEA-21 Evaluation Guidelines are published in the Federal
Register for informational purposes on our approach to satisfying the
requirements prescribed in section 5204(j)(1). Specific questions on
any of the material published in this notice should be directed to the
appropriate contact person named in the caption FOR FURTHER INFORMATION
CONTACT.
Authority: 23 U.S.C. 315; sec. 5204(j), Pub. L. 105-178, 112 Stat.
455 (1998) 23 U.S.C. 502 Note.
Issued on: September 13, 1999.
Kenneth R. Wykle,
Federal Highway Administrator.
Gordon J. Linton,
Federal Transit Administrator.
The text of the TEA-21 Evaluation Guidelines is presented as
follows:
United States Department of Transportation
Intelligent Transportation Systems
Evaluation Guidelines for the Transportation Equity Act for the 21st
Century
Operational Tests and Deployment Projects
I. Introduction
A. Background
The enactment of the Transportation Equity Act for the 21st Century
(TEA-21) [Pub. L. 105-178, 112 Stat. 107 (1998)] has expanded the focus
of the Intelligent Transportation Systems (ITS) program from one of
research and operational tests to one that now includes deployment of
ITS across the Nation. Subtitle C (Intelligent Transportation Systems
Act of 1998) under title V of TEA-21 authorizes investment to
accelerate the rate at which intelligent transportation systems are
incorporated in the Nation's surface transportation network to improve
transportation safety and efficiency, and to reduce costs and negative
impacts on communities and the environment. The effective
implementation of the deployment and integration activities specified
in subtitle C will rely heavily on active and rigorous evaluations of
those activities. Recognizing the evolving nature of the ITS program,
Congress included a mandate in TEA-21 which requires the Secretary of
Transportation (Secretary) to issue guidelines and requirements for the
evaluation of operational tests and deployment projects carried out
under the program. A prerequisite for the continuing support of
decision makers addressing policy and investment issues will be a clear
understanding of ITS system effectiveness. At the Federal level, the
Government Performance and Results Act [Pub. L. 103-62, 107 Stat. 285
(1993)] has established a formal process for program and budget
planning. ITS benefits information will be needed for assessing the
efficacy of Federal investment in ITS. At the project level, agencies
and partners in public-private project initiatives require the means to
monitor costs and system effectiveness on a continuing basis to support
management of systems and operations.
B. Legislative Requirements
The Congress has recognized the critical role of ITS evaluation in
section 5204(j)(1)(A) of subtitle C which prescribes that:
Subparagraph (A):
In General.--The Secretary shall issue guidelines and requirements
for the evaluation of operational tests and deployment projects carried
out under this subtitle.
Subparagraph (B):
Objectivity and Independence.--The guidelines and requirements
issued under Subparagraph (A) shall include provisions to ensure the
objectivity and
[[Page 50856]]
independence of the evaluator so as to avoid any real or apparent
conflict of interest or potential influence on the outcome by parties
to any such test or deployment project or by any other formal
evaluation carried out under this subtitle.
Subparagraph (C):
Funding.--The guidelines and requirements issued under Subparagraph
(A) shall establish evaluation funding levels based on the size and
scope of each test or project to ensure adequate evaluation of the
results of the test or project.
C. Objective
The objective of this document is to fulfill the ``guidelines''
requirement imposed on the Secretary by TEA-21, title V, subtitle C,
section 5204(j)(1)(A), (B), and (C) as referenced above.
D. Purposes of Evaluation Guidelines
In order to satisfy the applicable title V subtitle C criterion
imposed by TEA-21, guidelines must be issued which accomplish the
following purposes:
1. Define different categories of projects carried out under
subtitle C;
2. Define, in general terms, different types of evaluations to be
conducted by projects in the categories defined;
3. Establish criteria to guide the selection of evaluations to be
performed;
4. Define, in general terms, procedures for ensuring objectivity
and independence of evaluating organizations;
5. Define the funding mechanism to provide resources for
evaluations;
6. Provide a general description of the procedures [and
requirements] that project partnerships can expect when participating
in the types of evaluations defined;
7. Provide access through web site address to detailed evaluation
procedures including examples of specific evaluation plans and reports.
Guidelines in this document do not address detailed evaluation
procedures.
II. Terms of Reference
The establishment of clearly understood guidelines for evaluation
of projects funded with Federal ITS funding requires identification of
the categories of projects specified in the law. Similarly,
understanding of the Department's concept for conducting project
evaluations requires differentiation between types of evaluation.
A. Project Categorization
Projects to be carried out with Federal ITS funds are identified as
follows:
1. Operational Tests--Section 5207(c) specifies the conduct of
operational tests of intelligent vehicles and intelligent
infrastructure systems. Section 5207(c) further specifies that
operational tests conducted under this section shall be designed for
the collection of data to permit objective evaluation of the results of
the tests, derivation of cost-benefit information that is useful to
others contemplating deployment of similar systems, and the development
and implementation of standards.
2. ITS Deployment Program--The ITS Deployment Program provides
funding for two major project categories specified in sections 5208 and
5209. The two components are the ITS Integration Program and the
Commercial Vehicle Intelligent Transportation Infrastructure Deployment
Program, commonly known as the Commercial Vehicle Information Systems
and Networks (CVISN) Program.
a. Intelligent Transportation System Integration Program--Section
5208(a) specifies projects to be carried out under the Intelligent
Transportation System Integration Program in metropolitan and rural
areas. These projects comprise one major component of the ITS
Deployment Program defined above. The ITS Integration Program provides
Federal funding for the integration of multi-modal ITS components in a
variety of settings, including large regional areas (for example,
Statewide, multi-State, or multi-city), metropolitan areas, non-
metropolitan areas, and rural areas. ITS integration projects should
improve transportation efficiency; promote safety; enhance transit
integration; improve paratransit/demand-responsive transit operations,
including operations of health and human services providers; improve
traffic flow, including the flow of intermodal freight at ports of
entry; reduce emissions of air pollutants; improve traveler
information; promote tourism; enhance alternative transportation modes;
or support improved transportation systems operations, management and
maintenance.
As part of the Intelligent Transportation System Integration
Program, section 5208(g), Corridor Deployment and Coordination,
requires the Secretary to encourage multistate cooperative agreements,
coalitions, or other arrangements intended to promote regional
cooperation, planning, and shared project implementation for
intelligent transportation system projects. There are two areas of
implementation: Great Lakes ITS Implementation and Northeast ITS
Implementation.
(a) Great Lakes Implementation--Section 5208(g)(2)(A): The
Secretary is required to make grants under this subsection to the State
of Wisconsin to continue ITS activities in the corridor serving the
Greater Milwaukee, Wisconsin; Chicago, Illinois; and Gary, Indiana,
areas initiated under the Intermodal Surface Transportation Efficiency
Act of 1991 (ISTEA) [Pub. L. 102-240, 105 Stat. 1914 (1991)] and other
areas in the State.
(b) Northeast ITS Implementation--Section 5208(g)(3)(A): The
Secretary is required to make grants under this subsection to the
states to continue ITS activities in the Interstate Route I-95 Corridor
in the northeastern United States initiated under ISTEA.
b. Commercial Vehicle Intelligent Transportation System
Infrastructure Deployment--Section 5209 specifies that the Secretary
shall carry out a comprehensive program to deploy intelligent
transportation systems that improve safety and productivity of
commercial vehicles and drivers, while reducing costs of commercial
vehicle operations and regulatory requirements. The term ``Commercial
Vehicle Information Systems and Networks'' is defined (section 5211) as
the information systems and communications networks that support
commercial vehicle operations. Funding in section 5209 is used to
support CVISN projects which comprise the second major component of the
ITS Deployment Program defined above. CVISN objectives focus on
advancing technological capabilities and promoting the deployment of
intelligent transportation system applications to commercial vehicle
operations, including commercial vehicle, commercial driver, and
carrier-specific information systems and networks. Priority areas for
projects are those that encourage multi-state cooperation and corridor
development; improve the safety of commercial vehicle operations;
increase the efficiency of regulatory inspection processes to reduce
administrative burdens; improve the efficiency of enforcement efforts;
advance electronic processing of registration, driver licensing and
fuel tax information, inspection and crash data; promote communication
of information among the States; and, enhance safe passage of
commercial vehicles across international borders.
B. Evaluation Categorization
For the purpose of the guidelines, there are two categories of
project evaluations.
[[Page 50857]]
1. Self-Evaluations--In the interests of accepted sound management
practice, it is expected that ITS Deployment Program participants
should conduct locally executed and funded evaluations under the
auspices of the project partners. These self-evaluations, also
identified as local evaluations, incorporate certain minimum evaluation
and reporting requirements. Cross-cutting assessments of these local
evaluations will be conducted by the ITS Joint Program Office (ITS JPO)
at Headquarters (HQ), U.S. DOT, and will include gathering data and
dissemination of results.
2. National Evaluations--National evaluations are formal, in-depth,
independently conducted evaluations of operational tests of intelligent
infrastructure systems and selected projects carried out under the ITS
Deployment Program. In the case of selected ITS Deployment Program
projects, these evaluations will supplement and expand on the
activities of self-evaluations. The added resources available for
national evaluations will facilitate assessment of quantitative
performance measures and other aspects of project evaluation difficult
to pursue with the potentially limited resources assigned for self-
evaluations. National evaluations will be conducted under the auspices
of U.S. DOT, and will be closely monitored by a designated U.S. DOT
representative. The U.S. DOT has established the Intelligent Vehicle
Initiative (IVI) to improve the safety and efficiency of motor vehicle
operations by reducing the probability of motor vehicle crashes.
Evaluations of IVI field operational tests will be conducted under the
auspices of the IVI Program Manager in accordance with procedures
defined within the IVI Program and not associated with requirements for
national evaluations discussed in these guidelines.
Features and requirements of these categories are described below
in these guidelines.
III. Concept for Project Selection and Evaluation Funding
A. National Evaluations
1. Operational Tests--All operational tests of intelligent
infrastructure systems carried out under section 5207 shall be
administered formal, national evaluations under the auspices of the ITS
JPO Program Assessment Coordinator.
2. ITS Deployment Program--Projects carried out under the ITS
Deployment Program will be reviewed as follows:
(a) ITS Integration Program--Projects selected for funding under
section 5208 shall be reviewed, and a limited number will be selected
for national evaluations in accordance with appropriate criteria
described in these guidelines.
Corridor Development and Coordination--Projects resulting
from grants made to the State of Wisconsin to continue activities in
the corridor serving Milwaukee, Wisconsin; Chicago, Illinois; and Gary,
Indiana, in accordance with section 5208(g)(2)(A) and projects
resulting from grants made to the I-95 Northeast ITS Priority
Corridors, in accordance with section 5208(g)(3)(A), may be reviewed
for incorporation in a national evaluation as determined by the
Department's evaluation needs in corridor settings.
(b) Commercial Vehicle ITS Infrastructure Deployment--A limited
number of CVISN deployment sites funded under section 5209 will be
selected for national evaluations in accordance with appropriate
criteria described in these guidelines.
B. Self-Evaluations
All project partnerships are expected to perform locally conducted
self-evaluations.
C. Funding for Project Evaluations
In order to fulfill the mandate specified in section 5204(j)(1)(C)
that these guidelines ``shall establish evaluation funding levels based
on the size and scope of each test or project that ensure adequate
evaluation of the results of the test or project,'' the following
procedures apply:
1. Operational Tests--Funding for evaluations of operational tests
of intelligent infrastructure systems will be provided by the ITS JPO.
2. ITS Deployment Program--(a) Projects funded under the
Intelligent Transportation System Integration Program (section 5208)
and the Commercial Vehicle Intelligent Transportation System
Infrastructure Deployment Program (section 5209) selected for national
evaluations will use a pooled funding mechanism. During each year
authorized by TEA-21, two percent of the amount authorized for the ITS
Deployment Program will be placed into a deployment evaluation fund.
National evaluations for selected projects will be funded by this
account.
(b) All projects will fund locally conducted self-evaluations from
project resources.
D. Selection Criteria for National Evaluation
1. Intelligent Transportation System Integration Program
Projects carried out under section 5208 are designed to accelerate
the integration and interoperability of intelligent transportation
systems in metropolitan and rural areas. Projects resulting from
corridor grants may also be reviewed for inclusion in a national
evaluation. Projects selected for funding should support the goals of
the program as defined in section 5208.
Projects selected for funding will be reviewed by the ITS Program
Assessment Working Group (ITS PAWG) chaired by the ITS JPO Program
Assessment Coordinator. The ITS PAWG is comprised of representatives
from the modal administrations at HQ, U.S. DOT, and field
representatives from the Federal Highway Administration (FHWA) Resource
Centers and Division Offices, to include the Office of Technology
Evaluation and Deployment within the Office of Motor Carrier and
Highway Safety. Federal Transit Administration field representation is
included as well.
The ITS PAWG will consider projects for national evaluation through
successive iterations designed to narrow the list of candidate projects
consistent with available funding and the ITS Program's most compelling
information needs. While the specific considerations for project
selection may vary in any given year of TEA-21 authorization, the
criteria described below will guide the review process.
a. Major consideration will be given to ITS areas for which
insufficient knowledge was developed during the operational test
experience under ISTEA authorization.
Metropolitan ITS Integration Program projects will be assessed for
their potential to provide benefits-related data in areas where
evaluation data are still needed.
b. Rural ITS Integration Program projects will be assessed for
their potential to provide benefits-related data.
It is anticipated that, over the life of TEA-21, the areas
reflecting shortfalls in ITS Program evaluation needs will vary. The
ITS Program reserves the flexibility to develop a working document with
annual updates of these areas of shortfall which would be used to guide
the selection process. These areas will be addressed in annual
Departmental project guidance documents.
2. CVISN Deployment Projects
The FHWA's State CVISN Level 1 deployment strategy consists of
three key steps: Planning; Design; and, Implementation and Deployment.
Step 1, Planning, includes participation in two ITS/Commercial
Vehicle Operations (ITS/CVO) training
[[Page 50858]]
courses (Introduction to ITS/CVO and ITS/CVO Technical Project
Management for Non-Technical Managers) and the development of an ITS/
CVO State business plan. This step is essential to promote ITS/CVO
awareness and coalition building among the State agencies involved in
CVO and with industry.
The focus of Step 2, Design, is for the State to establish its
CVISN project team, including at a minimum a CVISN project manager and
a system architect. Once these individuals have been selected, a State
can participate in the Understanding ITS/CVO Technology training course
and in three CVISN workshops. These activities will assist the State in
developing its CVISN Project Plan and Top-Level Design.
Step 3 is the Implementation and Deployment of CVISN Level 1
capabilities.
Only CVISN sites selected for Step 3 funding will be considered for
national evaluation. The three elements of Level 1 capabilities subject
to national evaluation are:
Safety Information Exchange,
Credentials Administration, and
Electronic Screening.
During each year of TEA-21 authorization, the ITS PAWG will convene
its CVO Subcommittee comprised of subject matter experts from FHWA's
Office of Technology Evaluation and Deployment within the Office of
Motor Carrier and Highway Safety. Based on the ITS/CVO Program's
information needs about Level 1 capabilities in different operating
environments, a limited number of State CVISN deployment sites engaged
in Step 3 (Implementation and Deployment) activities will be selected
for national evaluation.
IV. Selection of Evaluating Organizations
A. General
Subtitle C, section 5204(j)(1)(B) requires that these guidelines
include provisions to ensure objectivity and independence of the
evaluator so as to avoid any real or apparent conflict of interest or
potential influence on the outcome by parties to any such test or
deployment project or by any other formal evaluation carried out under
this subtitle.
B. Provisions for National Evaluations
The required provisions are outlined as follows:
1. Organizations demonstrating technical qualifications are
eligible for selection to perform a national evaluation of designated
operational tests or deployment projects.
2. In the process of submitting necessary documentation
demonstrating qualifications to perform a national evaluation,
interested organizations should include a statement certifying
commitment to the conduct of a completely objective and independent
evaluation, and its commitment to ensuring any subcontracting
organizations will adhere to that standard. It is expected that this
would include a provision for voluntary recusal from evaluating
selected aspects of, or technologies involved in, the project to be
evaluated, if appropriate.
3. For national evaluations conducted under the auspices of U.S.
DOT, the ITS JPO Program Assessment Coordinator will convene selected
members of the ITS PAWG to participate in the selection of an
evaluating organization with consultations as follows:
a. Operational Tests--For operational tests of intelligent
infrastructure systems, the ITS JPO Program Assessment Coordinator, in
consultation with project partners and the ITS PAWG, will select the
evaluating organization. For IVI field operational tests, the selection
of organizations will be conducted in accordance with procedures
established by the IVI Program Manager.
b. ITS Deployment Program--Participants in the evaluating
organization selection process are as follows:
(1) Intelligent Transportation System Integration Program--For
projects in this category selected for national evaluations, the ITS
JPO Program Assessment Coordinator in consultation with project
partners will conduct a selection process. The selected evaluator will
receive direction and oversight by a U.S. DOT representative, and will
be required to document compliance with the ``OBJECTIVITY and
INDEPENDENCE'' requirements imposed by section 5204(j)(1)(B) of
subtitle C, and will forward authenticated certifications described in
IV.B.2 of these guidelines to the designated U.S. DOT representative.
This representative, in coordination with the ITS JPO Program
Assessment Coordinator, will conduct an expeditious review of selected
organizations prior to completion of contract arrangements.
(2) CVISN Project Evaluations--The selection of an evaluation
organization for the conduct of national evaluations of selected CVISN
project sites will be led by the ITS JPO Program Assessment Coordinator
and the ITS PAWG representative from the Office of Technology
Evaluation and Deployment in the Office of Motor Carrier and Highway
Safety who will convene a CVO subcommittee of the ITS PAWG. This
subcommittee will be comprised of the ITS JPO Commercial Vehicle
Operations Coordinator and subject matter experts on the staff of the
Office of Motor Carrier and Highway Safety, HQ, FHWA.
C. Provisions for Self-Evaluations
The guidelines for the conduct of self-evaluations in ITS
Deployment Program projects include the following: (1) The evaluations
should be conducted under the auspices of the project partners; (2) The
partners should form evaluation organizations; and, (3) The process for
forming evaluation organizations should strive to adhere to the
objectivity and independence principles cited in section 5204(j)(1)(B)
of subtitle C.
V. Evaluation--Definition, Process, and Guidelines
A. General--The purposes of this section are to accomplish the
following:
Define evaluation as a management process;
In general terms, differentiate between procedures and
expectations to be encountered in national evaluations and self-
evaluations;
Convey reporting requirements.
These guidelines are not intended to provide detailed procedures
for the conduct of evaluations. Extensive documentation addressing
detailed procedures, to include examples of evaluation plans and
reports will be available at the ITS JPO web site. (See Section V.G.).
B. Definition of Evaluation
Evaluation is the reasoned consideration of how well project goals
and objectives are being achieved. The primary purpose of evaluation is
to cause changes in the project so that it eventually meets or exceeds
its goals and objectives. Evaluation is an essential ingredient to good
project management. Evaluations can be qualitative and quantitative;
however, the best evaluations employ the combination of qualitative and
quantitative information that compare and contrast converging, non-
converging, and diverging evidence to result in a complete diagnosis.
The most effective evaluations occur when goals and objectives are
explicitly stated, are measurable, and are agreed to by all project
partners.
Evaluation should be considered as part of the project development
process that iterates across stages of strategy
[[Page 50859]]
formulation, detailed planning, system design and implementation, data
collection, data analysis, and reporting of results. Evaluations should
be performed by a party who has had no vested interest or stake in the
project itself.
Independence of the evaluator does not mean uninvolvement with the
project. Key roles of the evaluator requiring early evaluator
involvement are: (1) Identification of key stakeholder partners; (2)
eliciting from the partners a meaningful set of goals and objectives
for the project and their relative priorities; (3) obtaining insight
and consensus regarding which measures will best reflect the degree of
success in achieving prioritized goals; and, (4) communicating changes
in goals, objectives, and measures as the project evaluation matures.
Data can be collected either by the partners in the project (as long as
the independent evaluator maintains some oversight of the process) or
by the independent evaluator, or by both. The data analysis phase must
be performed completely independently of the partners; however, draft
results should be shared to obtain partners' expert insights regarding
possible flaws in assumptions or errors in analysis. The best
evaluation results are those that can bring about positive and timely
changes in system operations or maintenance. In such instances,
documentation of results may not be as important as the positive
changes influenced by the results. Nevertheless, documentation has
tremendous importance to the national interest.
A significant goal influencing the approach to ITS Program
evaluation planning is the expansion of the knowledge base among
transportation community professionals. To the extent that policy
makers, planners, engineers, and other influence brokers become better
informed about successful ITS deployment practices, project evaluations
will make significant contributions. Effective information
dissemination techniques will be critical. Information drawn from
interim and final evaluation reports will be summarized in electronic
media normally accessed by transportation professionals.
Project evaluation reports will be crucial to the ITS Program
during the years funded by TEA-21. As the tempo of project deployments
accelerates, newly formed project partnerships preparing to undertake
deployments will need any available experience-based information
relevant to their projects. The need for timely information will
emphasize the value of interim reports capturing valuable, recent,
lessons learned.
C. General Evaluation Process
The following are general steps successfully used to accomplish an
ITS project evaluation for national or local evaluations:
1. Form the Evaluation Team. Each of the project partners and
stakeholders designates one member to participate on the evaluation
team. The program manager should designate an evaluation team leader.
In the interests of conducting an effective evaluation, this team
should interact with the independent evaluator periodically throughout
the project development and deployment. Experience has demonstrated
that formation of this team early in the project is essential to
facilitating evaluation planning along a ``no surprises'' path.
Participation by every project stakeholder is particularly crucial
during the development of the ``Evaluation Strategy.''
2. Develop the Evaluation Strategy. This evaluation strategy
document includes a description of the project to be evaluated and
identifies the key stakeholders committed to the success of the
project. It also relates the purpose of the project to the general goal
areas of an ITS project.
Projects deploying intelligent metropolitan or rural infrastructure
are expected to allocate resources adequate for evaluating the impact
(or impacts) their projects exert in certain major goal areas which can
be pursued through deploying and integrating ITS technologies. ITS goal
areas include:
Traveler Safety
Traveler Mobility
Transportation System Efficiency
Productivity of Transportation Providers
Conservation of Energy and Protection of the Environment
Others as may be appropriate to unique features of a project
A major purpose of the evaluation strategy document is to focus
partner attention on identifying which of the above goal areas has
priority for their project. Partners assign ratings of importance to
goal areas and evaluation priorities and resources are consequently
aligned to the prioritized set. This rating process gives partners
valuable insights regarding areas of agreement and disagreement and
assists in reconciling differences and bolstering common causes.
Each of these goal areas can be associated with outcomes of
deployment that lend themselves to measurement. These outcomes
resulting from project deployment are identified as measures and have
been adopted as useful metrics. The association of goal areas and
measures is depicted as follows:
------------------------------------------------------------------------
Goal area Measure
------------------------------------------------------------------------
Safety....................... Reduction in the Overall Rate of
Crashes.
Reduction in the Rate of Crashes
Resulting in Fatalities.
Reduction in the Rate of Crashes
Resulting in Injuries.
Mobility..................... Reduction in Delay.
Reduction in Transit Time
Variability.
Improvement in Customer
Satisfaction.
Efficiency................... Increases in Freeway and
Arterial Throughput or Effective
Capacity*.
Productivity................. Cost Savings.
Energy and Environment....... Decrease in Emissions Levels.
Decrease in Energy Consumption.
------------------------------------------------------------------------
*A discussion of the distinction between Throughput and Effective
Capacity is included in the ITS Evaluation Resource Guide referred to
in Section V.G. at the end of this document.
The ``few good measures'' in the preceding table constitute the
framework of benefits expected to result from deploying and integrating
ITS technologies. While each project partnership will establish its
unique evaluation goals, the measures serve to maintain the focus of
goal setting on how the project can contribute to reaping the benefits
of one or more of the measures. A sample Evaluation Strategy document
is provided as part of the ITS Evaluation Resource Guide (see Section
V.G. at the end of this document).
[[Page 50860]]
3. Develop the Evaluation Plan. After the goals are identified and
priorities are set by the partners, the evaluation plan should refine
the evaluation approach by formulating hypotheses. Hypotheses are
merely ``if-then'' statements about expected outcomes after the project
is deployed. For example, a possible goal of coordinating
jurisdictions' signal systems is improving safety by reducing rear-end
crashes. If the evaluation strategy included this goal, the evaluation
plan would formulate hypotheses that could be tested. In this case, one
hypothesis might be, ``If jurisdictions coordinate signal timing, then
rear-end collisions will be significantly reduced at intersections at
jurisdictional boundaries.'' An even more aggressive hypothesis might
suggest that such collisions would be reduced by ten percent. The
evaluation plan identifies all such hypotheses and then outlines the
number of different tests that might be needed to test all hypotheses.
In addition to hypotheses regarding system and subsystem
performance, the evaluation plan identifies qualitative studies that
will be performed. The evaluation should address key components of the
project, such as, (but not limited to):
Implications of achieving consistency with the National
ITS Architecture;
Standards implementation;
Consumer acceptance;
Others as appropriate to local considerations;
Institutional issues.
Institutional issues require brief elaboration. An area of special
emphasis in all evaluation endeavors should be the non-technical
factors influencing project performance. ITS projects conducted under
ISTEA were profoundly influenced by considerations such as procurement
practices, contracting policy, organizational structure, and
relationships among major participants such as prime contractors and
their subcontractors. The transportation community stands to reap
significant benefit from understanding how the varied range of non-
technical factors impacts directly on traditional project performance
parameters, such as, cost, schedule, and final functionality.
As these critical aspects of the project are addressed, the value
of the evaluation increases in proportion to its ability to produce
lessons learned that can improve project performance. Thus, the
measures serve as the foundations of the evaluation as project
implementation seeks the best mix of approaches to ensure achieving
some level of benefits described by the measures.
To the extent that projects define evaluation goals derived from
one or more of the few good measures and document the impacts of
deployment on transportation performance in their communities, they
will have made significant contributions to the ITS Program and the
Nation. A sample Evaluation Plan document is provided as part of the
ITS Evaluation Resource Guide (see Section V.G. at the end of this
document).
4. Develop one or more Test Plans. A test plan will be needed for
each test identified in the evaluation plan. A test plan lays out all
of the details regarding how the test will be conducted. It identifies
the number of evaluator personnel, equipment and supplies, procedures,
schedule, and resources required to complete the test. A sample test
plan is provided as part of the ITS Evaluation Resource Guide (see
Section V.G. at the end of this document).
5. Collect and analyze data and information. This step is the
implementation of each test plan. It is in this phase where careful
cooperation between partners and evaluators can save money. By early
planning, it is possible to build into the ITS project capabilities for
automatic data collection. Such data collection can be used by partners
after the evaluation is completed to provide valuable feedback with
regard to the performance of the system. Such feedback can help in
detecting system failures and to improve system performance.
6. Document strategy, plans, results, conclusions, and
recommendations in a Final Report. A sample Final Report document is
provided as part of the ITS Evaluation Resource Guide (see Section V.G.
at the end of this document).
D. Self-Evaluation Process
1. ITS Integration Program Evaluation Procedures
To the extent that resources allow, project partners should strive
to follow the General Evaluation Process outlined in Section V.C. of
this document. Partners should also utilize the ITS Evaluation Resource
Guide described in Section V.G. at the end of this document.
Projects conducting self-evaluation should respond to a minimum of
two reporting requirements.
a. In an attempt to satisfy significant and critical data needs in
the ITS Program, projects should collect and document cost accounting
data to include acquisition, life-cycle, and operations and maintenance
costs capturing both start-up and sustaining cost factors. Reporting of
cost data will be solicited annually for the duration of the deployed
system's (or systems'') life cycle(s). Cost data collection guidelines
are provided as part of the ITS Evaluation Resource Guide (see Section
V.G. at the end of this document).
b. Additionally, self-evaluations should result in one or more of
the following efforts:
Evaluating institutional issues associated with achieving
cooperation among public sector agencies and documenting how they were
overcome.
Providing a brief lessons learned report on the technical
and institutional issues encountered in integrating ITS components.
Providing an evaluation report on the lessons learned in
employing innovative financing or procurement and/or public-private
partnering techniques.
Producing a lessons learned report on the experiences,
challenges, and approaches used in achieving consistency with the
National ITS Architecture and/or implementation of ITS standards.
Producing a case study on the planning process used to
achieve integration into an approved plan and program developed under
an area-wide (statewide and/or metropolitan) planning process which
also complies with applicable State air quality implementation plans.
Providing the appropriate metropolitan planning process
with data generated by ITS technologies and services, and provide a
report on plans or intentions for archiving the data and using it.
2. Commercial Vehicle Intelligent Transportation System Infrastructure
Deployment Projects
All projects participating in the Commercial Vehicle Intelligent
Transportation System Infrastructure Deployment Program are expected to
accomplish the following:
Document lessons learned in the areas of:
--institutional issues
--technical challenges
--employment of innovating financing
--public-private partnering
--achieving consistency with the National ITS Architecture and
implementation of ITS Standards.
Document benefits data.
Collect cost data, especially with regard to operations and
maintenance cost factors, and briefly address their implications for
sustaining the project.
[[Page 50861]]
E. National Evaluations
1. Initial Procedures for Operational Tests of Intelligent
Infrastructure
Upon selection of the organization to conduct a national evaluation
of a designated ITS operational test of intelligent infrastructure, the
ITS JPO Program Assessment Coordinator will establish the initial
communications between involved parties. The detailed procedures for
the conduct of the evaluation and the scope of the evaluating
organization's tasks will be defined in accordance with procedures
established by the ITS JPO Program Assessment Coordinator.
2. Requirements for ITS Integration Program Projects
During the annual project definition and proposal process,
participants in the ITS Integration Program will be offered the
opportunity to commit to cooperate with evaluators in the event of
selection for a national evaluation. This commitment includes
participation in evaluation planning and in-progress reviews to ensure
a consensus-based, successfully implemented national evaluation as
described in sections V.B. and V.C. of these guidelines.
3. Requirements for CVISN Deployment Projects
During the annual project definition and proposal process,
participants in the Commercial Vehicle Intelligent Transportation
System Infrastructure Program will be offered the opportunity to commit
to cooperate with evaluators in the event of selection for a national
evaluation. This commitment includes participation in evaluation
planning and in-progress reviews to ensure consensus-based,
successfully implemented national evaluations.
4. Timing Considerations for ITS Deployment Program Projects Selected
for National Evaluations
Participants in ITS Deployment Program projects selected for
national evaluations may experience a time delay between receipt of
notification for project funding and, in the event of selection for
national evaluation, notification of such selection. Upon notification
of project funding approval, the project participants should proceed
with the preparatory steps required for evaluation. The preparatory
measures will lay the foundation for an effective self-evaluation. In
the event of selection for a national evaluation, that process will
build on this foundation. An evaluation team should be formed and an
evaluation strategy, based upon the example in the ITS Evaluation
Resource Guide, should be developed.
It is anticipated that U.S. DOT notification of selection for
national evaluation will be accomplished prior to a project's
development of an evaluation plan. This will facilitate coordination
between the independent evaluating organization and the project
partners in proceeding with the development of a consensus-based
evaluation plan.
F. Reporting Requirements
This section prescribes reporting procedures for the categories of
evaluations.
1. Projects conducting self-evaluations in the ITS Integration
Program are expected to produce: (1) an annual cost report based upon
guidelines in the ITS Evaluation Resource Guide; and, (2) a final
evaluation report. Projects conducting self-evaluations of the CVISN
are expected to produce a final report. All project partnerships
conducting self-evaluations are expected to submit two camera-ready
reproducible copies and one electronic file to the ITS JPO Program
Assessment Coordinator at: Intelligent Transportation Systems, Joint
Program Office (HOIT-1), Attn: JPO Program Assessment Coordinator, U.S.
Department of Transportation, 400 Seventh St., SW., Washington, D.C.
20590.
Copies which may be required for other addressees will be defined
in annual guidance documents transmitting instructions.
2. Reporting procedures in national evaluations will be defined in
the appropriate documentation governing the contract entered into by
the evaluating organization.
G. References
In lieu of incorporating detailed procedural guidance for the
conduct of evaluations in this document, an ITS Evaluation Resource
Guide has been developed. This comprehensive resource for supporting
evaluation planning is accessible at the ITS JPO web site (http://
www.its.dot.gov) through the Program Assessment/Evaluation Link.
[FR Doc. 99-24363 Filed 9-17-99; 8:45 am]
BILLING CODE 4910-22-P