MassDOT deserves enormous credit for trying to connect its investment decisions with the desired outcomes. It’s a challenging and complicated undertaking, constrained in many ways by federal reporting requirements, limited data, and unverified impact-calculating methodologies. The fact that their first attempt, the very impressive WeMove Massachusetts: Planning for Performance tool, is deeply flawed (for example, defining mobility solely as car travel) is much less important than the Agency’s public willingness to admit those flaws and commit itself to an iterative improvement process. This is something that every public— and private – organization needs to take on, not merely to better serve its stakeholders but also to be better in control of its own fate.
When trying to make investments with impact, there are three major difficulties. First, you must identify and prioritize or “weight” the goals according to their relative importance, selecting carefully among possible Policy-based, User-Experience-based, and Operations-based goals. For example, GreenDOT has declared “mode shift” to be a goal: to increase the number of people who choose to walk, bike or ride public transportation, instead of driving in a Single Occupancy Vehicle. MassDOT also has a goal of keeping roads in a “state of good repair” – which may conflict with the mode shift goal by increasing the attractiveness of car driving relative to other modes. Which goal gets higher priority?
Second is selecting the right metrics to evaluate progress towards each of the goals: things that can be cost-effectively measured, can be influenced by your actions, and that are sufficiently within your scope of control. Not only must good metrics be selected, appropriate numerical targets need to be set that reflect the “goal weighting” priorities as well as safety limits, federal requirements, and other parameters.
And third is figuring out how to model the ways and degrees that different types and amounts of investment will change operations, and that each of those operational changes will impact the metrics. While nationally-accepted formulas already exist for translating road budgets into road improvements and then into increased car mobility (meaning greater speed and volume with fewer delays), doing the same for other modes – transit, bicycling, walking – is a still-evolving practice. (Equally important, and also lacking in predictive tools, is exploring ways to restructure operations and infrastructure to move the metrics without major investment!)
In each category, the way a goal is defined shapes the way it is measured and the actions ultimately taken to achieve it. For example, MassDOT’s Planning for Performance tool’s definition of “mobility” as “the number of hours of delay experienced by the average driver for every 1,000 Vehicle-Miles-Traveled” not only ignores anything related to non-car travel, it also skews the measurement towards Single Occupancy Vehicles since it only counts delays to a driver rather than to all vehicle occupants.
——————
USER EXPERIENCE GOALS
Theoretically, because MassDOT is a public agency, meeting its Policy goals is the organization’s top priority, just as earning profit is the top priority of a business. In reality, as private sector firms have long known those that seek profit at the cost of quality end up with neither. Similarly, public agencies that seek policy compliance over customer satisfaction end up having their budgets cut. In reality, it turns out that meeting customer needs is the road to policy fulfillment, not the opposite. So the most important goals are based on the user experience.
Furthermore, while Policy-based goals are mostly imposed from outside or from the top of an organization downwards, defining User-Experience-based goals requires a bottom-up approach. From a user perspective, a service or product is generally evaluated based on its personal efficiency, convenience, affordability, comfort, and perceived status.
User input has many sources: one-on-one and group discussions, electronic and telephone surveys, formal and informal feedback or complaints (sometimes as reflected through the media), personal experience of MassDOT and MBTA staff and their families, and careful examination of usage data. But some data can be computed from operational statistics: train, trolley, and bus delays can be combined with ridership figures to track reductions in average user delay time or trip time. It is easy to compute the percentage of the regional population living and/or working within a half-mile walk of a transit station or stop. A customer-oriented organization, as MassDOT is trying to become, needs to have embedded systems for regularly monitoring all the sources and relaying the trends to senior management as well as the entire staff.
Underlying user experience is the complicated reality that products and services need to not only work well they also need to feel good, or at least make you feel good about using it — feelings shaped by the service or product’s aesthetic and functional design as much as by its operational smoothness. Some of this comes from things intrinsic to the product or service, the “look and feel.” Some of it has to do with societal-cultural projections: expensive cars are considered desirable not only because they tend to be well made but because they are a visible statement of wealth and the resulting social status. Bus Rapid Transit (BRT) lines need to be not only an efficient commuting option but also have to overcome the cultural stigma of buses being seen as something that only poor people use.
An often overlooked aspect of User-Experience is the working condition of the employees. In a sense, employees are the organization’s internal users. To the extent that an organization relies on the creativity, effort, commitment, and loyalty of its staff, the quality of their daily experience will shape the quality of the experience they provide for their external users. Happier workers create happier customers. Management leads, but the reason that “work to rule” is such a powerful tactic is that official rules simply don’t – and never can – fully describe the complexity of actual work processes; successful production always requires flexibility, resourcefulness, and cooperation at the front line.
POLICY GOALS
Meeting policy mandates is essential for the survival of non-profit and for-profit organizations as well. Policy-based goals emerge from law, court rulings, administrative regulations and decisions, and elections – and they address both external impacts and internal functions. The external aspects are the result of transportation’s broad impact on our entire society, from our economy to our health, from our social/cultural interactions to our housing patterns. The internal aspects address MassDOT’s reality as an organization.
For MassDOT, some key external-impact policy-based goals are reducing greenhouse gas emissions and other pollutants as well as storm-water run-offs and noise, facilitating transit-oriented smart growth, tripling the number of trips taken by transit, walking, and bicycling over the next 15 years, and more. Some of these mandates have numeric targets (e.g. GHG reduction and mode shift), some are described in general terms (e.g. increasing service to Gateway Cities and Environmental Justice or Title VI populations), and some are simply implicit in today’s political context (e.g. generating tax revenue). (The Planning for Performance tool cites the MBTA’s scoring factors as “(1) Safety, Health, Environment; (2) State of Good Repair; (3) Operations Impact; (4) Cost/Benefit; and (5) Legal Commitments.”)
Policy also sets some of MassDOT’s internal goals including limits on debt, reduction of unfunded pension liabilities, improving handicapped access to stations and vehicles, and ending discrimination against non-whites and women. These, too, can have numeric components (e.g. debt limits) or general descriptions (e.g. non-discrimination). They can be derived from MassDOT-specific legislation (e.g. reforming pension rules) or imposed by courts based on general laws (e.g. handicapped access). Or they can combine elements of each.
Remaining aware of the complex and ever-changing spectrum of policy-based goals all is a complicated task in itself. Having a great legal department isn’t enough – management at all levels need ways to stay aware of the existence of these goals and what they are supposed to do to keep the organization moving in the right direction, as well as to be effectively motivated and held responsible for their successful efforts.
OPERATIONAL GOALS
Operations, the day-to-day and long-term functioning of an organization as it produces products or services, appropriately grabs the most attention of nearly everyone within an organization. Operational goals include everything from making the trains run on time to reducing workplace accidents, from maintaining staffing levels to increasing productivity and doing more with less. Operational goals range from the smallest items (e.g. keeping track switches working) to the biggest projects (e.g. getting the Green Line Extension finished “on budget, on time”). Operations are the muscle and bone of an organization, meeting operational goals is what allows progress towards everything else: MassDOT already has strong Greenhouse Gas reduction policy goals, what is now needed are concrete analyses of which operational changes will make what contribution to meeting them.
But operational goals are mostly second tier or intermediate issues. They are vital and deserving of fanatically detailed attention, and the appropriate target of improvement efforts, but they are mostly a means to the end of meeting Policy and User-experience goals. Increasing the number of miles of off-road bike paths is an excellent operational goal; however, increasing the number of bicyclists or bicycle-miles-traveled and the percentage of trips taken by bike, or perhaps increasing the percentage of people who live within a half-mile of a traffic-separated bike facility are really the desired accomplishments. Increasing the percentage of buses in a state of good repair is a vital operational goal; however, reducing the average time-per-trip of bus riders, increasing the percentage of people who use a bus for daily trips, and having a higher level of bus rider satisfaction are what is really wanted.
Even budget amounts or increases, crucial as they may be for allowing change to occur, are a secondary issue from a performance perspective: the amount of money spent on safety is not the point, the reduction in total accidents, injuries, or fatalities is the real goal with special emphasis on more vulnerable populations such as children, the elderly, people pushing strollers or using wheelchairs, bicyclists, runners, and pedestrians. For walkers, better than the number of dollars spent on sidewalk repair is the percentage of the population who feel safely able to get to a local grocery story, workplace, school, or recreational area with a 15 minute walk.
PRIORITIZATION
An organization that fails to achieve enough Policy-based goals jeopardizes its legal ability to exist; failing on too many User-Experience-based goals jeopardizes its financial and political ability to survive. There are some goals that are absolute, or nearly so, and simply cannot be ignored.
But how to prioritize among all the rest? It’s hard for an organization to fully focus on everything all at once – although there should always be systems in place that monitor the full spectrum of concerns to alert people at the appropriate level and department if something is starting to get too far out of whack. One rule-of-thumb is that the more times something appears in your lists the more important it is. A policy goal that is also a key factor in generating positive user-experience seems like a no-brainer.
Another approach is to divide goals into three categories: Easy, Intermediate, and Foundational. Easy goals can be achieved quickly, inexpensively, with little disruption or controversy, and are visible to either staff or the public in order to send a message that change is coming (even if the easy action itself is merely symbolic and doesn’t actually change anything). Intermediate goals take longer, are more expensive or difficult, and (like the Easy category) cause a measurable positive movement towards the attainment of Policy-based, User-Experience, or Operational goals. The payoff occurs soon after the work is done and “bang-for-the-buck” is a useful prioritizing criterion. Most valuable are Intermediate goals that are worthwhile in themselves but also lay the foundation for additional steps.
Foundational goals have a longer time-line before paying off; they often require considerable investment and structural change; they are targeted towards Policy and User-Experience goals; but they can have the most impact – radically changing the way the organization operates, gets financed, or even how it defines its mission. There are few truly Foundational goals, and they often get little attention until a crisis causes the old paradigms to fail. But it is essential for organizational survival to know what the Foundational goals may be, and to do the groundwork that will allow them to be activated when necessary.
Ultimately, prioritization is a subjective, value-driven process. If making customers happy, increasing environmental sustainability, and fostering a more cost-effective transportation system are what you consider most important, then you’ll end up with a different set of actions than if you prioritize reducing traffic congestion, keeping local DPWs happy, and maintaining pavement conditions.
CAUSATION MAPPING AND METRICS
From an investment-to-impact perspective, the biggest challenge actually comes after you identify and prioritize goals: creating a model capable of predicting what changes in which functions will provide the greatest positive change in final results. The first step is creating a “logic map” of linkages, which can be done moving either forward or backward. Mapping forward means starting with a list of Inputs (money, labor, equipment, supplies), showing how they flow through varying Processes (planning, permitting, etc.) to produce Intermediary Outputs (products and services), which then go through a second set of Processes (bidding, construction, maintenance) and/or interact with external factors (the cost of materials, legal challeges) to create final Impacts (aka Results or Outcomes such as “increased number of trips taken by bike”).
This kind of “logic map” can often be more easily created in reverse order – tracing backwards from Impacts to Inputs. But in either direction, it is the starting point for computing the leveraging potential of changes at each step in the flow. The “leveraging potential” of various kinds of Inputs need to be given “weights” showing their relative impact on the following Processes and Outputs. This usually starts with rough estimates that are then tested using known historical data, and then continuously adjusted as the “impact model” is used.
A similar, but often less rigorous, approach comes from the rapidly expanding field of Health Impact Analysis (HIA) which suggest the type and amount of public health impact that would be caused by proposed projects or policies. HIAs can be very detailed or very general, based on existing or new research, written for internal use or public reporting. At their best, the HIA process includes a follow-up analysis to evaluate the accuracy of the predicted impact (assuming the proposed project or policy actually gets implemented).
The mapping of internal processes can also be usefully informed by the Total Quality Management-derived techniques of ISO 9000, Six Sigma, and other process re-engineering strategies.
MOVING FORWARD
Given the state, and nation’s, dangerous underfunding of transportation, MassDOT’s primary purpose is to justify increased funding and help both voters and Legislators grasp the catastrophic economic and public consequences of its absence. But no matter what the future funding scenarios are, performance measurement is a bottom-line requirement for good management and good agency operations. MassDOT is starting down the right road.
———————-
(Many of my specific comments were informed by a previous critique of WeMove written by Matt Danish – who also graciously offered comments on earlier drafts. The remaining mistakes and opinions are, of course, my own responsibility.)
———————-
Related previous postings:
> SLOWING TRAFFIC TO A TARGET SPEED: How To Make Our Streets Safer
> THE ART OF TRANSPORTATION (AND URBAN) PLANNING: Going Beyond the Technical Specs
