Guidelines for Transportation Management Systems
5. TMS Maintenance Program
The objective of this chapter is to guide the practitioner in developing a maintenance program. A maintenance program provides a plan on what maintenance is, how it is performed, how it can be budgeted, and why it is needed. It is a document that describes the needs to persons outside the direct department with maintenance responsibilities and provides guidance to those within that department.
This chapter lays out the logistics for producing a maintenance program. Many of the details contained within the plan are available in other chapters of this report. The practitioner will need to adapt the plan to suit the local institutional and organizational conditions and to meet the needs of the Agency's ITS hardware.
As defined in Chapter 1, responsive maintenance is the repair or replacement of failed equipment and its restoration to safe, normal operation. Preventive maintenance is the activity performed at regularly scheduled intervals for the upkeep of equipment.
Maintenance is the upkeep of property and equipment. Within the context of TMS maintenance activities, it covers a wide range of activities — from extreme emergencies, when roadway crashes may have caused state property to fall and block the freeway, through scheduled activities such as cleaning camera housings and backing up hard drives that keep the control systems running.
The topics covered in this chapter mirror the contents of a maintenance program. They include:
The remaining sections of this chapter define how a plan can be created materials in other sections of this document.
Maintenance programs for TMS elements are best considered as part of the larger and more conventional maintenance activities of an Agency. Resurfacing, safety, and rehabilitation programs are all ongoing activities in that all Agencies are involved. Considering TMS maintenance as a critical part of the wider maintenance activities allows such an Agency program to be considered the within the framework of Agency operations.
Having a mission is an effective tool in selling maintenance to individual Agencies and other institutions. The following example is from the Caltrans Division of Maintenance:
A maintenance program should be developed during the initial design stages of an ITS program. Section 8 of the National Architecture for ITS (Version 4) states:
There are significant advantages in emphasizing the maintenance program in the early stages of ITS design. During ITS architectural development, institutional issues and connections between jurisdictions are made and agreements are forged. The maintenance program in many locations operates services across these boundaries and maintenance should be included as a component of the institutional agreements. During the early stages of ITS design, the locations for control centers and depots are considered. The staffing of a maintenance program is critical to keeping the system running.
Early planners of ITS often locate maintenance depots on an expedient basis of what land is available to the state and adjacent to a highway. The needs of the maintenance staff should be included in such plans. Maintenance staff need to access specific hardware often in the middle of the night (snow plows, diggers, cherry pickers, etc.). Staff access staff to maintenance depots needs to be considered in the light of their operations. For example, the Washington State DOT maintenance depot at Hyak on I-90 east of Seattle is at the summit of Snoqualmie Pass; it is there that the big equipment and the central computer are housed. The location is often inaccessible, although the facility is outfitted with beds and a kitchen for long-term stays. Some locations allow the maintenance crews use of Agency vehicles to commute home at night in order to reduce response times to access equipment outside of normal operational hours. Activities and geography need to be accommodated within the staffing issues associated with early stages of ITS planning.
During the early stages of the plan development process, specific ITS devices are selected. The awareness of maintenance difficulties are often now included in the selection of, for example, non-intrusive detectors, ensuring that the maintenance activity does not cut into the road surface. However, this process should be taken further by incorporating the extent of the work, the frequency, and the level of staffing expertise into the device selection criteria. For instance, the maintenance requirements of dynamic message signs might include the ability to:
Getting the maintenance staff involved in the specification writing to include these types of elements will save effort and make the system, when installed, easier to maintain. The concept of ensuring that all activities that affect maintenance should be considered as part of the program will help later with institutional issues. In some cases, there are separate departments that are used for planning and design rather than operations and maintenance. In these instances, it is harder to develop system designs that accommodate maintenance activities and additional efforts should be made to ensure coordination.
The development of a maintenance program requires that there be input from a variety of other plans and budgets within the Agency. It is common for operations and maintenance to be considered together. This results from the fact that operations and maintenance are frequently funded as a single budget item. Operations are more evident than maintenance, because the former have a more immediate impact on traffic and because operational control centers are designed with public relations in mind.
The development of a maintenance program needs to define the mission and vision of the program put into the context of the other programs and activities within the Agency. Getting maintenance into the minds of the related departments and institutions may require a selling the whole program. The intent is to ensure that the maintenance goals and missions are included in the thinking of others. Web pages, short brochures, and full maintenance plans have all been used to achieve this end. In some cases, these plans as formalized (as noted in the Caltrans' example, above; Oregon DOT has a formal maintenance plan). In other cases, the maintenance activities for TMS are combined with operations and budgeted and funded as an integral unit.
A maintenance program is intrinsically linked to the operations and design of the Agency's traffic management center. It is also linked to the other broader aspects of the Agency's programs insofar as the staff is often part of that department. There are unfortunate examples where the operations group has added ITS devices that the separate maintenance group did not, or could not, maintain and the system fell into disrepair. Setting relationships to the TMC staff, the relevant police, fire, and emergency Agencies, as well and those organizations responsible for incident management, will assist in the development of a program. The issue of a formal memorandum of understanding is discussed in Chapter 2 of this report.
As illustrated in the Figure 5-1, a maintenance program requires a close and well-defined relationship with:
A maintenance program needs to be developed in conjunction with the operations group. The program development should, at a minimum, consider:
The operations group within an Agency will need to be involved with the maintenance program development. There are operational aspects that affect the timing and frequency of the maintenance program. There are few aspects of operations that do not affect such a program. Backing-up of operating systems, replacing video displays, coordinating the cleaning operations centers, and site visits all need to be considered as a part of the program. In some circumstances, the operations staff are used to ensure that contractors are on-site at the right time and are following the correct safety procedures. The operations group needs to be in constant audio contact with the maintenance staff and the program needs to ensure that radios or cell phone facilities are planned and budgeted.
Training is an important part of the maintenance program. Maintenance staff training will often include the Agency's safety and operational procedures; the detailed maintenance procedures for the individual devices; and the operation and calibration of test equipment. In addition, the maintenance staff often needs to be trained in the use of differing vehicle types such as cranes, heavy plant equipment, etc. The development of a maintenance program should allocate budget for training new employees, training employees on new technologies and providing refresher courses to keep all staff aware of the safety procedures. Maintenance of data archives, updating various databases, and running configuration management on the various pieces of software in the TMC should also be part of a maintenance program. These elements of the program require specialized computer skills and such staff capability should be included in the program.
Maintenance operations require a range of equipment from the highly specialized electronics used for fault finding and repairing fiber-optics, through large trucks used to spread material in response to spills and inclement weather. A maintenance program needs to include detailed estimates of the entire Agency's requirements, and how they are to be purchased and maintained. The maintenance staff is likely to have to allocate time to the maintenance of their own plant. Lifetimes, salvage values, and replacement also need to be considered as equipment wears out and is lost or broken. This is one aspect that can feed into the decisions concerning the use of contractors. Expensive equipment, such as fusion-splicing equipment to mend fiber-optic, requires trained personnel. In addition, the number of times such facilities are needed is infrequent. Contracting options or keeping such skills on hand by use of retainer payments may form part of the maintenance program.
Other contracting options are described in earlier sections of this report. However, a maintenance program should involve the contracting and purchasing departments of the Agency to ensure that invoicing, payments, contracting options, and all the relevant terms and conditions are accommodated. Some contracting options and payment procedures, particularly those for responsive maintenance, can become complex. These procedures should form part of the maintenance program planning.
The staffing of a maintenance program is often the key factor in its success. Estimates of staffing levels can be found in Chapter 3. TMS's are often an accumulation of many years of growth spread over a wide range of contracts. Maintenance staff tend to accumulate a lot of details concerning the quirks and idiosyncrasies of their particular communications network and the devices being operated. Keeping these staff on-board, and providing them with an attractive career path, is one of the major problems in developing a maintenance program. The Agency management needs to provide a rewarding working environment to ensure that vital staff assets are not lost.
The inclusion of a mission-and-objectives definition is a useful and salutary exercise for the developer of a maintenance plan. By defining the objectives the maintenance operations, the developer of the plan has to provide a balance between the various resources available. Some Agencies insist that all items that break are fixed as soon as possible. Others are continually trying to catch up with the required responsive maintenance. Others consider preventive maintenance a luxury they have no time or budget for. A mission statement that defines the priorities can assist in the development of a plan.
Traditionally, maintenance funding has been the responsibility of the operating Agency. In a few cases, such as Virginia, there is a statutory requirement that maintenance funds come "off the top" of the budget, i.e., that maintenance is funded first and then capital projects. In the case of TMS projects, O&M costs are rarely included beyond the first year of operation. Typically, the capital cost for implementing TMS elements are programmed for one category of funding while the O&M must come from another funding category.
The ISTEA legislation and later TEA-21 legislation have relaxed certain FHWA restrictions on federal aid funding operations and management of TMS activities. CMAQ funds can be used for certain O&M activities. There are still roadblocks to using these categories for TMS. The CMAQ funding is limited to certain qualifying projects and has a three-year time limitation. Federal funding for O&M must compete with capital projects for prioritization. This means it often loses out to the larger more visible projects.
The Volpe National Transportation Systems Center (VNTSC) recently interviewed several cities and MPO's regarding the funding of O&M activities. These interviews indicated a lack of knowledge about the availability of, and guidelines for, use of federal funds for O&M. Those interviewees that had some knowledge of the availability of O&M funding indicated some reluctance to pursue this source of funding because of the competition with capital projects and the level of effort involved (i.e., it takes the same amount of effort to secure funding for a large capital project as it does for a relatively small O&M project).
The relative newness of TMS projects means that many Agencies do not have much experience with maintenance requirements. There are quite a few references (Ref 1, 7) available, however, to assist in developing budgets for TMS maintenance. Also, the maintenance budgets presented by some of the survey sites may be helpful in comparison.
The Caltrans District 7 Maintenance Plan (Ref. 8) is unique in that a detailed methodology is used to develop workload standards for maintenance of each device type. This methodology uses mean time between failure and mean time to repair (MTTR) for corrective maintenance activities, as well as time spent in preventive maintenance. An average travel time to conduct each activity is calculated based on the location of the maintenance facility and the mean distance to all devices of a given type. Using these data, an estimated labor workload can be calculated for each device type (per unit per year).
Funding of maintenance programs has proven to be very troublesome in many Agencies. ITS implementations are typically justified on the grounds that they will reduce congestion and provide information to the public. The many references on the FHWA benefit cost website (Ref. 9) show justification for these expenditures. However, maintenance programs that can be justified as a way to ensure the continuation of these benefits are often harder to sell. The continued operation of hundreds of roadside devices can also be jeopardized by the costs of running the TMC that is using the information these devices provide. It is important to emphasize the relationship between surveillance, operations, and the delivery of data to the public.
The budgeting section of a maintenance program needs to emphasize the fact that operations and maintenance costs can be similar. Generally there are about half as many maintenance positions as there are operator positions within a TMC. This ratio can vary according to the type of ITS, geography, and the specific structure of the various responsibilities. However, this can be used as a starting point in the budgeting process. Later in this chapter, processes for calculating the staffing and inventory levels are described. Inventory, staffing, plant, and vehicles, as well as contracted services, all make significant contributions to the cost of maintenance.
Although the use of Federal funds is allowed for maintenance projects, there are impediments to using these. Some states maintain that federal funding is not available for maintenance.
Since Federal funds are available, the developer of the maintenance program should investigate opportunities with the local FHWA representative.
The funding of some maintenance programs is enhanced by monies from other jurisdictions including local cities and counties that use state maintenance staff for their devices. The state signal shop providing maintenance to county signals is one example of this. In the case of the City of Charleston, the City maintains the signals in the adjacent jurisdictions in return for an annual fee. Similar opportunities are available with toll authorities and large sporting complexes that may have extensive video systems.
Current practice with the funding of maintenance programs is varied. It is difficult to determine the size of individual budgets as there is no common basis on how funding occurs. In some Agencies, operations and maintenance are combined and a group of staff are dedicated to both functions and the budget item for maintenance may be just for keeping inventory current. In other locations, there may be an entirely independent budget for TMS; this seems more common where the maintenance activities are contracted out and therefore a specific budget item is required. Some Agencies do not have separate budgets for TMS's — the upkeep of the devices and the centers become another item within the entire Agencies' budgets.
Within these frameworks, budgets are often allocated by district; structures can be very different between adjacent jurisdictions in the same state. Given this complexity of choices, the developer of the maintenance program should consider multiple approaches to fund maintenance and develop arguments and justifications for the budget that are more liable to be successful in each area. For example, putting long-term warranties to cover maintenance activities on devices may be included in the construction budget. Caltrans installed most of its ramp metering system by ensuring that the standard construction plans for a ramp included all the appropriate devices, power, and communications connections. Connecticut DOT when contracting for DMS, incorporated maintenance contracts in the bid for the signs.
The process for developing an estimate for the budget for a maintenance program includes:
There are two basic options for conducting TMS maintenance: (1) in-house resources, or (2) contract maintenance. For the vast majority of TMS locations cited in the literature, there was some combination of in-house maintenance and contract operations. Our survey also was reflective of this hybrid approach to providing maintenance, but included two extremes. Cincinnati has a turnkey O&M contract that covers all system maintenance. The local Ohio DOT district office was scheduled to assume more responsibility and involvement in the ARTIMIS system beginning in April 2002, and this may change in the future. The Department explained that one of the drawbacks of a turnkey O&M contract is that lack of involvement sometimes means a lack of control.
The other end of the spectrum is the Albany, New York system. Here, the New York State DOT handles practically all system maintenance for this 35-mile freeway system. Some maintenance responsibilities for the TOC are shared with the State Police who are co-located in the same facility. They have recently negotiated an agreement with the State University to provide maintenance on the CCTV cameras. A software maintenance agreement with the vendor of the central system software was also recently executed as part of an overdue upgrade.
The use of, and experience with, device and system warranties was examined in the survey. The general consensus was that while warranties are useful and desirable, they tend to be difficult to administer. Warranties generally cover the replacement of failed hardware components; they do not, however, usually cover the labor costs to repair the components or install replacement hardware. Of course, during the bid process, the installation contractor could theoretically be required to provide the repair labor — doing so, however, can make it difficult for contractors to compete. Retainage of a percentage of the contractor's funds until the end of the warranty period presses the contractor to minimize the warranty length. On the other hand, eliminating the retainage leaves the contractor with less incentive to furnish warranty and repair labor service.
Maintenance contracts sensitive to the needs of contractors are more likely to be successful than are those which are substantially oblivious to these needs. For example, in Virginia, VDOT initially advertised for contract maintenance of a large freeway management system. This was to be the first such contract covering all field devices. The RFP requirements included little detail about the status of existing system devices and lots of stringent requirements for response time and repair time. The few contractors choosing to respond all priced the project much higher than the amount anticipated in the Department's budget, primarily because these contractors perceived the risk to be so high. A subsequent re-advertisement of this maintenance contract was preceded by a pre-bid conference at which many details about the existing system were discussed and feedback solicited from the potential contractors. The result was a much more competitive procurement with prices within the Department's budget.
The program in this area needs to consider the breakdown of operations to determine which, if any, elements of the maintenance procedures should be contracted out. Section 7.2, Options for Contracting, describes the contract options available to Agencies and how to make informed decisions. It also includes a sample contract.
There are several elements to the procedures:
The developer of the maintenance program will need to determine responsibilities for each of these elements. Then a procedures document should be prepared that offers guidance to the maintenance staff and provides a basis for the staff estimating spreadsheet.
Most Agencies do not currently have maintenance plan documents. Agencies often have lists of maintenance procedures and references to maintenance in operations documents. However, dedicated maintenance plans are not readily found.
The maintenance program must specifically identify the TMS elements to be included in preventive maintenance. For corrective maintenance, a predetermined priority list is a minimum requirement. Ideally, a process can be defined for assigning priorities for corrective maintenance as problems occur.
The maintenance program should include descriptions of maintenance actions and frequencies for the following minimum elements:
The information on the specific procedures for maintaining devices is contained in Appendix B. The procedures associated with starting a maintenance action can come from a variety of sources, as illustrated in Figure 5-2 that indicates the typical flow process when dealing with a responsive maintenance activity.
Figure 5-2 Responsive Maintenance Process D
After the Maintenance operator receives notification, he or she opens a trouble report. Examples of such reports are shown in Appendix A. There are existing maintenance programs that keep track of trouble calls and incidents. These features are sometime integrated with the central control software; stand-alone products are also available.
Regardless of the Agency's procedures, the appropriate responder is notified, the responder acts on the response, the responder reports back on the outcome to the Maintenance operator, and the operator verifies that the work is done and updates the trouble report.
Measuring the performance of the maintenance program provides information both on organization and management issues, in addition to the reliability of various ITS devices. Having metrics of the system provides continual feedback on how well the system is operating.
The metrics associated with the structure of the plan could include:
The program developer will need to consider these and possibly other metrics that help the Agency assess the success of the program. Although the list above reflects negative issues only, this is to some extent a consequence of maintenance. There is no good news as if all the maintenance is not performed. The best that can be hoped for is that everything works all the time and minimum expectations are met.
The metrics associated with the individual components could include:
A clear message from all resources and participants involved is that whatever type of maintenance program is selected for a TMS, it must be well-organized and efficient. One of the best tools to help with this is a Maintenance Management System that includes, at a minimum, an inventory or database of all major devices and subsystems within the TMS.
A step-up from a simple inventory based MMS is an Asset Management System (AMS). AMS's are available from a variety of commercial vendors and are widely used in both public and private practice. Typical features will include the use of bar codes or similar identification procedures for tagging each item in the inventory; tracking of MTBF and MTTR measures; comparative analysis of maintenance activities by location, vendor, time of year, etc.; and reporting of various maintenance and inventory characteristics.
In setting up MMS's and AMS's, the following key considerations should be taken into account:
With the maintenance plan developed, approved, and funded, there must be a structured practice for managing the plan. The basics of plan management are similar to other practices and should include:
There are tools, such as commercially available asset management systems, that can help with the operation and management of a maintenance plan. There is also a new User Service within the National ITS Architecture that can help describe and specify a function to support the maintenance of a TMS. This service, "Maintenance and Construction Operations," is introduced in the new version of National ITS Architecture (Version 4.0).
The following are requirements of the user service "Maintenance and Construction Operations" that an ITS device should provide (the numbering scheme is the same used in the National ITS Architecture):
Defined in the National ITS Architecture, the Maintenance and Construction Management Subsystem monitors and manages roadway (and TMS) infrastructure construction and maintenance activities. Representing both public Agencies and private contractors that provide these functions, this subsystem manages fleets of maintenance, construction, or special service vehicles. The Maintenance & Construction Vehicle subsystem resides in maintenance, construction, or other specialized service vehicles or hardware and provides the sensory, processing, storage, and communications functions necessary to support highway maintenance and construction. All types of maintenance and construction vehicles are covered, including bucket trucks, heavy equipment, and supervisory vehicles.
Additionally ten Market Packages, defined by sets of equipment packages required to work together (typically across different subsystems) to deliver specific transportation services and the major architecture flows between them and other important external systems, which support the user service of are given as following:
Additionally, ten ITS "market packages" are identified to support the user service of "Maintenance and Construction Operations":
Not all of these market packages are necessary for every MMS, but demonstrate the range of possibilities. Table 5-2 shows subsystem and equipment packages associated with the Maintenance and Construction market packages.
The specific structure of a maintenance policy is less important than simply having one in the first place. There are practices and procedures that are helpful in implementing the maintenance policies. For instance, if contracting for maintenance, one should recognize what it may take for a maintenance contractor to economically provide maintenance on short-notice or even a full-time basis. This may mean increasing the overall scope of the maintenance contract so that there is enough work to justify assigning contractor personnel to the TMS project. Similarly, the duration of the maintenance contract will have an impact on how many potential bidders it attracts and how the contract is staffed. A longer contract — i.e., two years or more with additional option years — may enable the contractor to relocate staff to the TMS area.
Once there is a general agreement that maintaining the TMS is part of the Agency's policy, several key factors should govern details of the policy. For instance, an important policy objective should be protection of the investment — built on the rule-of-thumb that equipment maintenance is generally less expensive than equipment replacement. If systems fail and basic operations degrade, a significant portion of the initial financial investment is lost.
Configuration Management should be included as part of a maintenance policy; this helps ensure that the original functionality of the system is maintained throughout its life. Management of the Maintenance plan, described in the previous section, like many TMS-related functions, requires level-of-effort on the part of the Agency, regardless of whether it is contracted out or performed in-house. The literature research and site interviews indicated that this work is often added to the duties of existing positions within the Agency, leaving little time to get everything accomplished.
The Level of Service (LOS) encompassing operations and maintenance focuses on providing the traveling public with an operational system and the functions and benefits of that system. Those LOS measures that can be used as part of maintenance policies include:
Each Agency should include in its policy appropriate metrics for the LOS of maintenance. This will allow the effectiveness of the various procedures to be quantitatively evaluated. It will also facilitate the identification and monitoring of trends over time; these can be indicators of improvements or degradation of service.
Staffing and Training
The availability of properly skilled and trained staff is critically important to getting the most out of any TMS and assuring that it meets its intended concept of operations throughout the system's intended life-cycle. Several practices related to this critical need were noted in the literature research and interviews, including:
The basic Systems Engineering Life-Cycle process, described in Chapter 3, concludes with "operations and maintenance." We have extended the engineering process to encompass a parallel maintenance concept that provides a structured procedure for validating and confirming maintenance requirements and execution, thus supporting the original concept of operations.
To validate the process, there must be a close coordination between maintenance and operations. The following is a list of key functions and activities that can help achieve this coordination:
TMS operators should have input into system availability status reports, beyond a simple listing of failed devices. TMS operators need to be able to convey to maintenance personnel the impact that system degradation has on their ability to perform important TMS functions. This will assist the maintenance staff in prioritizing responsive maintenance. Before leaving the site, the maintenance staff should ensure that the TMS operator is actually able to control the device that has been repaired.