Methodology Shootout: Standards vs. Custom Road Performance Measures

Standard performance measures

PCI, IRI, PSI, PASER… if you’re new to pavement evaluation and pavement management, you should know we love acronyms. There are a ton of them in this space. The acronyms listed above represent performance measures that were created by international standards bodies or academic groups and are used all over the world to rate the condition of roads.

At the other end of the spectrum, you have custom performance measures that were created in-house or by consultants and tailored for specific road authorities. In this post, we’ll explore how the two stack up and discuss things to consider when developing your road condition assessment framework.

Here are a few common standard performance measures used to rate different attributes of a road’s condition by jurisdictions of all sizes around the world.

Pavement Condition Index (PCI)

Pavement Condition Index, or PCI, is a composite score calculated from physical distresses that rates a road’s surface condition between 0 and 100, with 100 being perfect. This method is based on ASTM D6433, although many variants exist.

International Roughness Index (IRI)

International Roughness Index, or IRI, is a measure of a road’s roughness reported in units of slope, such as m/km or in/mile. A theoretically perfect smooth surface would have an IRI of 0 m/km, and the number increases as roughness increases. Various components of this method are detailed in ASTM E1926 and ASTM E1364, among others.

Present Serviceability Index (PSI)

Present Serviceability Index, or PSI, is a score between 0, worst, and 5, best, calculated from different variables such as slope variance, rutting, cracking, and patching. PSI was developed by the American Association of State Highway and Transportation Officials.

There is one thing in common with these and other standard performance measures: they were developed by groups whose core function is to create standardized ways of testing everything from road condition, to the material properties of concrete, to a “Standard Practice for Assessing the Tendency of Industrial Boiler Waters to Cause Embrittlement.” Yes, that is a thing.

These standards are generally created by subject matter experts and go through various stages of review before being published.

Custom performance measures

Custom performance measures are assessment methods that were either developed in-house by a road authority or by an outside consultant. We’ll lump them into two broad categories.

Bespoke methods

Bespoke methods are performance measures and methodologies developed specifically for your organization. This would usually be done as part of a consulting project where a firm develops an assessment methodology and strategy based on factors such as your geographic location, road materials, goals, available resources, and more.

At the end of the project, you get a report outlining how to carry out the assessment method that is specifically tailored to your organization’s exact needs and resources.

Proprietary methods

Proprietary methods are assessment methodologies that were developed, owned, and carried out by consulting firms. The consulting firm will rate your road network using its own assessment methodology and provide you with the data.

We will compare standard assessment methodologies and custom assessment methodologies using the following four categories:

• Repeatability: How hard is it to learn or teach someone the method?
• Accuracy and applicability: How well does the data represent the actual condition of your road network?
• Upkeep of the methodology: How often is the methodology reviewed or revised?
• Transportability: How do my average network conditions compare to my neighbour?

Repeatability

The beauty of standards is that they are specifically written by a team of experts to be reproducible. By the time the standard is published, it will have passed through many reviewers’ hands. This back-and-forth review process irons out kinks in the methodology so that when it finally gets to print, it is a fairly concise list of steps that should be reasonably easy to follow for someone with general experience in the field.

For example, anyone with a copy of the PCI standard could read it and be able to assess the section of road outside their house or office. The standard even provides template worksheets you can use, as well as worked examples for various situations.

Due to this repeatability, using standards gives you the power to choose how you want to have your roads assessed. Are you a DIYer who isn’t afraid to get your hands dirty? Learn the method and do the assessment in-house.

Maybe you didn’t like the previous consultant you hired a few years ago to do a network-level PCI scan. No problem. Hire any number of different firms that have the capability to collect the exact same standard data. Regardless of who collects the data or how it is collected, as long as you are following the same method, you will always have a clean historic dataset to compare with previous years.

Custom methodologies are not so cut and dry. First, if a consultant is going to assess your roads using their own proprietary method, that method essentially has no repeatability on your own. This means that when it comes time to assess your roads again, you are locked into hiring that exact firm in order to maintain your historic dataset.

If you happen to not like working with that firm or the data they provide and are forced to use a different assessment method for your next survey, such as switching to PCI, you lose the historic value in the previous datasets because you would be comparing apples to oranges: data collected from two completely different assessment methods.

Bespoke methods that were created specifically for your organization can potentially be easy to learn at first. However, down the road, questions on certain aspects of the method are bound to arise. Those questions can only be answered by people who were directly involved with the development of the method, either the consultant themselves or a colleague who directly oversaw the project.

This might be fine for the first few years, but as time passes, people change roles, retire, leave the organization, and before you know it, you’re left to come up with your own interpretations. That can result in inconsistencies from what may have been done in the past.

With common standards like PCI, IRI, or PSI, you can call a colleague at a neighbouring town to see how they apply the standard, or simply Google the issue at hand and find thousands of resources.

Additionally, if you collected data in-house using your own custom method in the past but would now like a consultant to do the work going forward, you would be on the hook for teaching that consultant your own method. A request for proposals is much cleaner when you can simply reference the standard methodology, such as ASTM D6433, because many consultants will already be familiar with what you want.

Winner: Standards

Accuracy and applicability

Accuracy can be a drawback for international standards because they are designed to be generally applicable under a number of different circumstances and conditions. For example, the ASTM PCI method includes a list of 20 individual distresses. While comprehensive, the problem is that you may never see many of those distresses depending on where you are located and your specific area’s climate.

The second issue associated with general standards is the way they can be interpreted differently depending on where you are located, given differences in distress propagation and degradation.

A 70 PCI score on the East Coast of Canada, where there are vicious freeze-thaw cycles in late winter and early spring, means something vastly different compared to an equivalent road in a place like southern California, where the climate is relatively stable all year round.

The road with a 70 in Eastern Canada could turn into a 50 quickly from accelerated degradation, so preventative maintenance is extremely important. In the southern United States, you might get away with ignoring that section for a few years without much consequence.

The interpretation of IRI data is also heavily dependent on context. The range of expected values you’re working with will be drastically different depending on whether you’re managing highways, county roads, or urban areas.

For example, if you are managing a municipal network and your average network IRI is 2.50 m/km, or 158.4 in/mile, you should be celebrated for having exceptionally smooth roads. On the other hand, a highway manager whose network average is 2.50 m/km, or 158.4 in/mile, would be doing a terrible job. IRI ranges for asphalt highways can be as low as 0.5 m/km, or 31.68 in/mile, and up to around 1.5 to 2.0 m/km, or 95.0 to 126.7 in/mile. Anything higher would be fairly dangerous depending on the design speed of the road.

All things considered, when applying international standards, it is absolutely imperative that you research whichever methodology you choose to ensure you are applying it appropriately for your specific context.

These issues are not a concern if you are using a custom methodology that was designed to fit your exact context, assuming that whoever developed the method did a good job. You don’t have to worry about extra distresses because the only distresses included in your list would be those that actually occur in your area, which would streamline the assessment.

Additionally, there is no guessing when it comes to interpretation because the expected ranges of scores would likely be included in your documentation.

Winner: Custom

Upkeep of the methodology

The clear winner here is international standards because they are reviewed by a panel of experts on a regular basis. After this review, they are either withdrawn, re-approved, or revised.

For longstanding standards that have been used in practice for decades, such as ASTM PCI, subtle changes are rare but do occur. The best part about this as a user of the method is that you don’t have to pay for the costs associated with undertaking these reviews, as they are usually built into the standards body’s processes.

With custom performance measures, you are on the hook for upkeep and maintenance of the methodology. Unless you schedule a review process for your methodology, which you will pay for if you have to hire a consultant every time, the methodology itself could change or be interpreted differently over time as people cycle through the organization.

Even if you do have a built-in review process, your organization may not have the capacity or know-how to effectively carry out the task.

Winner: Standards

Transportability

A trend we’ve been seeing in the infrastructure asset management space recently has been the aggregation of data at a large scale to evaluate how governments are managing public infrastructure. Here in Canada, for example, the Canadian Infrastructure Report Card has compiled state of infrastructure data from municipalities and synthesized it into a digestible report. The first report was originally released in 2012 and was subsequently updated in 2016 and 2019.

Aggregated data like this is not only of interest to taxpayers. It also allows municipal engineers to fairly objectively benchmark themselves against their peers to compare network performance.

A cheeky aside: the key word here is objective, because basing your performance on public complaints might lead you to believe you have the worst roads in whatever location you happen to be in.

In a perfect world, this aggregated data would all have been collected using the same standard methodology, although that is not the case. Whether or not a specific methodology should be mandated is a topic for another day, but if all municipalities in Canada were using the exact same standard to collect condition data, you could theoretically create perfect benchmarks for what condition your network should be in.

We do not live in a perfect world. Even if you are using a standard methodology, there is a chance it could be different from your neighbouring municipalities, making it difficult to compare your performance with your peers. With that said, if you choose a popular methodology like ASTM PCI, chances are better that you will have many colleagues across the country to compare numbers with.

Using a custom methodology means you are siloing yourself on an island. It is impossible, or at least difficult, to compare numbers with your neighbour and create benchmarks when you are using a completely different methodology than everyone else.

Winner: Standards

Recommendations on moving forward

With a 3-1 win using the criteria mentioned above, standard condition assessment methodologies come out as the resounding victor over custom assessment methods.

With all this said, even if you are using standards as a base but want to take things a step further, there is always room for customization after the data has been collected. Many times, organizations will take individual performance measures like IRI and PCI and combine them into overall serviceability ratings weighted based on the importance they put on each metric.

Another thing to note is that at the end of the day, any condition data is far better than no condition data at all. So if you have been using a custom method for years and it is working, my hat is off to you. Stick with it.

Finally, the exception to what we’ve laid out above is very large road authorities like state and provincial DOTs. These organizations can have large teams of analysts managing their data collection efforts, giving them the ability to maintain custom assessment methods and the capacity for corporate knowledge transfer to do so effectively over time.

Only under those circumstances do we think the benefits of a tailored methodology really outweigh the downsides described above.

For organizations with only a few people tasked with managing an entire asset management program, we’ll leave you with one last acronym. The K.I.S.S. rules apply here: keep it simple and stick with the standards.