Of Benchmarks and Scorecards: Reporting on Multiple Projects

Alex S. Brown, PMP IPMA-C

Two versions of this paper are available. The short version was prepared for a newsletter and is approximately 2,000 words, or six pages. The long version (below) was prepared for a 45-minute presentation, contains some additional details, and runs approximately 3,800 words, or ten pages. Feel free to read either or both versions.

One of the best pieces of advice for any project manager is, “Divide and conquer.” Take a complex, large deliverable, and break it into parts. Deliver it in small pieces, each of which can be managed separately, with as few inter-dependencies as possible. This strategy is an absolute requirement at times. Some efforts are so large that a single project manager cannot reasonably track and manage the entire effort alone. Some efforts are so complex that they need multiple managers, each with different specialized knowledge. Thank goodness for “divide and conquer.”

There is a dark side to “divide and conquer”, however. At some point senior management will ask, “But how are we doing overall?” The individual status reports from each sub-project may not show the progress towards the overall goal. Cross-team problems may be ignored. Multiple formats from multiple authors can be confusing to read.

People want to see the forest, not just the trees. Creating an overall status report is a possible solution.

One approach to an overall status report is simple:

  • collect text descriptions of status and goals from the manager of each sub-project,
  • submit them all to a single person
  • concatenate them into a single status report, editing the result for consistent format and language

This overall report can be extraordinarily helpful, to senior managers and sub-project managers alike. Sometimes it is enough, but a project manager should have additional tools in his or her toolbox.

A second approach is to create an overall schedule for the effort. These schedules can be as simple as a milestone-only schedule, with key sub-project deliverables tracked on a single plan. Dependencies and cross-team milestones can be represented. More complex versions will have a single master schedule that contains all tasks for all sub-projects, including actual and estimated work hours. Scheduling software can help automate creating these master schedules. Creating schedules is often a tool-specific challenge, and in my experience there are easier ways to get clearer information to senior management.

  • Two sub-projects contribute paper reports and one project contributes a phone-in report. All feed into the overall project report. Each Sub-Project Contributes to the Overall Scorecard

Any project can use benchmarks and scorecards, but they are especially useful for projects composed of sub-projects. Each sub-project manager can create a custom scorecard containing critical, relevant data for their effort. Often a useful scorecard can be just a single printed page. Team members can use the scorecards to track progress, gauging concrete results against benchmark numbers. The overall project manager can then collect these scorecards, combine the relevant data from each, and produce an overall scorecard. Usually it does not capture all the metrics of all the sub-projects, but it provides an accurate view of the forest of sub-projects. Benchmarks can be combined to track overall project-level progress. The approach is satisfying because each manager has local control over his or her deliverable, and can set benchmark goals independently. Overall management can still track progress towards the ultimate goal with confidence and precision.

Based upon my experience managing software projects in financial services, this work is never simple or easy. Perhaps in a very mature organization with concrete, well-understood standards it would be. Most of us do not have the privilege of working in such an organization. Using a step-by-step process, though, anyone can establish some consistent standards for a multiple-project effort. If these standards survive the effort, the overall manager can take pride; he or she has increased the maturity of the organization while completing the project.

A note on terminology: Some organizations talk about “programs” composed of multiple “projects”. Other organizations use “program” to mean a whole product line, while still others talk about “projects” composed of multiple “programs”. This paper will avoid the term “program” entirely. Instead “sub-projects” are parts of a “multiple-project effort” or an “overall project”. “Project manager” means any project manager, while “overall project manager” is someone running the overall project. “Sub-project manager” is someone running a sub-project.

Step 1: Determine Senior Management Needs

The target audience for these benchmarks and scorecards is almost always senior management. Most stakeholders are concerned with either day-to-day progress at a low level, or overall completion dates and cost. Management typically wants to see regular progress reports, to ensure that goals will be met.

A meeting is almost always the first step, to establish frequency of reporting, to gather a list of any key data they want to see, and to understand what questions they expect to have answered at each reporting period. At this stage, some reporting goals will be concrete, while some will be vague. Sample goals might be:

  • “Weekly status reports by noon on Monday and a status meeting to discuss at 10 a.m. on Tuesday”
  • “An e-mail if anything seems off, otherwise a monthly meeting”
  • “Whatever numbers seem most important to you at the time, and of course any changes to delivery date”
  • “An up-to-date earned value graph, and control graphs for defects found and fixed”

Almost any combination of information is possible at this point. Of course the overall project manager will have some ideas about metrics that he or she wants to capture, but this meeting is about senior management needs. The project manager can collect other data, so long as management’s needs are met.

Step 2: Plan Sub-Project Metrics

Each sub-project needs metrics to contribute to the overall scorecard. The sub-project should measure project results to show variance, problems, and progress. The sub-project manager should come up with ideas about the metrics that he or she wants to collect. He or she may collect traditional project management metrics like earned value metrics, budget vs. actuals, and schedule slippage. Domain-specific metrics are often also useful, like number of defects opened or closed, feet of wire laid, number of rooms painted, number of code-reviews completed, and so on. The possibilities are infinite. The choice of metrics will depend upon the sub-project manager’s experience with metrics as well as the domain. Some sub-project managers may even decide to collect no metrics of their own. At this point in the process, that is fine.

Step 3: Identify Common Metrics

Once the overall project manager has a list from each sub-project manager, he or she can look for common metrics. Sometimes two managers will call the same metric two different things, or calculate the same basic metric two different ways. For instance, a straight budget calculation of actual costs to-date will usually be greater than an earned-value calculation of actual costs, due to differences in definition of earned and unearned costs. Note areas of possible conflict as well as areas of agreement.

At this stage it is also critical to compare the sub-project manager’s metrics against the senior management needs. The metrics list may meet some management needs completely, while other management needs have no supporting metrics. For the next step, it is critical to analyze the results and develop a strategy to meet management needs.

Step 4: Negotiate with Sub-Project Managers

With a list of common metrics, conflicting metrics, and missing metrics, it is time to talk to each sub-project manager. There will always be some changes to negotiate at this phase. Changes at a sub-project level might include:

  • changing data collection methods (i.e. use an earned-value budget or an accounting budget)
  • adding more data elements to the scorecard
  • renaming data elements
  • changing data collection/reporting tools to achieve consistent results
  • resolving differences in definitions (i.e. what does “50% complete” mean?)

The overall project manager may need to convince some sub-project managers to begin creating a scorecard at all. It is not uncommon for people to manage projects without a scorecard, and the main benefit of a common scorecard is to the overall project manager and to the overall management team. Some people may resist the standardization required.

Any sub-project manager who decided to collect no metrics in step #2 must now agree to collect a set of basic data, in order to participate in the overall project. These negotiations can be difficult. It is critical for the overall project manager to have a simple set of basic data needs, and to know exactly why each piece of data is necessary. Sometimes resistant managers fight metrics at every step of the way. The overall project manager should be prepared with advice or even practical help to begin data collection for each sub-project.

Be aware that gaining their cooperation or acceptance at this phase is not enough. Sometime resistant managers can misreport or report late during the project execution, as a form of passive resistance. The overall project manager needs to understand each sub-project manager; this negotiating period is a perfect time to get to know the overall project team really well. Wherever possible, try to generate goodwill and cooperation among the managers. Avoid creating situations where a single sub-project manager appears to win every negotiation, while everyone else looses.

Step 5: Set Standards

The overall project manager takes the results of all the negotiations and creates a list of standards for the overall project scorecard. The standards should describe who collects what data and when. Responsibilities for each sub-project should be clear and precise. Measurement methods should be exact. For instance, for “percent complete” for tasks, describe your method. Sample methods include:

  • Either 0% or 100% – complete or not yet complete
  • 0% if not started, 50% when started, 100% when complete
  • (100* Act) / (Act + ETC)
  • Different percentages for each phase of completion: 0% not started, 10% in-progress, 75% almost done, 90% ready for review, 100% passed review

Clear standards are critical for consistent reporting. Even the best-intentioned managers will vary in the way that they report information. It is impossible to have 100% consistency for some types of metrics, but a good standards document increases consistency better than any other single tool.

While setting standards, beware of vague standards that are hard to enforce or audit. Ideally, metrics are like the results of scientific experiments: anyone else could use the same methods, observe the same phenomena, and get the same results. The overall project manager should be able to verify the sub-project scorecards, at least within some range of certainty, like plus or minus 20%.

Networked tools for project scheduling, defect tracking, and work tracking simplify reporting considerably. Sometimes the sub-project manager does not even need to report their data to the overall project manager. If the data is networked, the overall project manager can read and report on the data independently. This kind of transparency is a huge benefit in multi-project situations.

Step 6: Create an Overall Project Scorecard

  • A scorecard showing counters for UIs, Classes, and Interfaces. Shows number of items coded, tested, required and remaining, with totals. Data appears in a table and a bar graph. Scorecards can combine tables and graphs to show a snapshot of current status

With a standard document in hand, designing the scorecard is straightforward. A few key questions will drive formatting and layout:

  • What information is most important?
  • What order will we discuss this information at status meetings (if applicable)?
  • How are we delivering the material (on-line, web, e-mail, text-only, print, projector, etc.)?
  • What are the limits of the physical media (color, font size, etc.)?
  • Do we need to highlight any key information?
  • Do any readers need the data in a particular format (graphical, percentage, table of results, etc.)?
  • What formatting or aesthetic standards should we follow?
  • Is this report too expensive to deliver?

In general, a good scorecard flows clearly from one set of information to the next, in the order that management wants to see and discuss it. Data is clear, formatting does not get in the way, and it is easy to scan the page for critical information. Totals are obvious and meaningful.

Time and time again, I have found that management appreciates a one-page scorecard. Sometimes this single page just has summary data. Additional pages can always show the detail. A single-page summary makes it easy to see project status at a glance.

  • Step 7: Create Overall Benchmarks

    • A spreadsheet shows goals vs. actuals on a monthly basis for classes, UIs, and interfaces. Goals extend into the future, beyond the actual values. Below each actual is a variance, highlighted in green if it is positive, red if negative. Benchmarks show progress over time, measured against concrete goals. Color helps show that this project is falling behind, and is ready for new benchmarks.

    Certain scorecard metrics will be expected to steadily increase or decrease over time. These metrics are good candidates for benchmarking. Imagine benchmarks as a snapshot of expected values over time, with one expected value for each reporting period. The most common form of benchmarking is earned value analysis. Managers take the schedule and budget at the start of the project, and write the expected earned value at each period throughout the project. Regular status reports show the actual values compared to the benchmarks.

The overall project manager can use this technique for any metric on the scorecard. Benchmarks can be extraordinarily useful for schedule-constrained projects. Management can see with every report whether work is on schedule, ahead or schedule, or behind schedule, without reviewing hundreds of tasks.

Some advice for creating benchmarks:

  • Only benchmark values that change predictably over time: number of open defects might seem like a good value to benchmark, because it must be zero at the end, but if the value goes up and down every week, it may be a poor choice
  • Benchmark a few values that really matter, that predict project success and show progress: too many benchmarks are confusing
  • Estimate likely progress: take into account vacation time, project phases, and seasonal issues, instead of just assuming a linear progression, like “2 more each week”
  • Publish the benchmarks: benchmarks are a great motivational tool for many teams, encouraging team members to contribute to the overall goals
  • Use sub-project benchmarks: give sub-project managers control over their own benchmarks, and total their goals to set the overall project benchmarks

The overall project manager must understand the purpose of the benchmarks. If there is a prize for the team at each met-or-exceeded benchmark, then set the benchmark goals high. Perhaps the manager sets the benchmarks so that he or she is only 30% sure that the team will make it each week. Perhaps the benchmarks show project completion several weeks ahead of schedule. Communicate these expectations to management and to the teams.

Sometimes benchmarks are a simple monitoring tool. The overall project manager should then set the benchmark at a spot where he or she expects the team to perform. Estimating uncertainty in the estimate is very valuable. Benchmarks can serve as a form of control graph, showing whether results are in control or out of control. The classic control graph has a single result expected over time, with horizontal bars for average, high, and low values. The benchmark graph would have an expected-result line that moves up or down over time, with control bars above and below it. When using this approach, uncertainty usually varies over time, so the uncertainty levels may change with each reporting period. As each goal comes closer in time, the likelihood of reaching the goal becomes more and more certain.

Sub-project managers may or may not choose to create their own scorecards for each value. Over the life of a software project, one sub-project might occasionally find and fix one or two defects, while another sub-project finds and fixes a hundred defects per week. A defect scorecard and benchmark could be useful for the overall project and the second sub-project, but not for the first. Be flexible about applying metrics at the overall and sub-project level.

Step 8: Senior Management Approval

When developing any reporting tool, it is critical to get approval of its target audience. A good project manager will involve the audience throughout the process. When and how to involve senior management always varies by organization, but with a draft scorecard developed and benchmarks set, it is essential to present a proposal to management.

This meeting is typically quite formal. The overall project manager distributes the benchmarks and scorecard ahead of time, then reviews it step by step with management. A face-to-face meeting is recommended, even if the project communication calls for no face-to-face, regular status meetings. People need time to review and discuss the results. Different senior managers may need different information, and it helps them to hear each other defend each part of the report. Removing or changing a scorecard section requires everyone’s approval, and often requires discussion.

Usually there is at least one change needed. Often the organization or format of the document requires changes. Sometimes the benchmark goals are too aggressive, or not aggressive enough. Occasionally the metrics do not meet management needs, and the scorecard goes back to the drawing board. Usually the overall project manager can note the changes in a follow-up e-mail and begin using the new report for the next project-reporting period. Going through this process almost always produces a report that is better than the tools that came before it. Even if it is not perfect, management usually is ready to adopt it.

Step 9: Update Scorecards, Evaluate Actuals vs. Benchmarks

With each reporting period, metrics change. The overall project manager should set a reporting schedule with all sub-project managers, to meet the reporting schedule required by management. Having everyone report all data as of the same date is critical to having a useful overall project scorecard. Assembling and reporting data for a complex project often takes several days. Even after the sub-project managers have finished their reports, it may take a day for the overall project manager to file the overall report. A tightly organized team can improve that time, by meeting or beating reporting deadlines consistently. Data may be old by the time it is published; balance the competing needs for timely data against the needs for accurate, well-reviewed data.

Each sub-project manager may have different reporting responsibilities for each reporting period. Typical responsibilities include:

  • Producing the sub-project scorecard and benchmarks
  • Reviewing and approving data in shared databases
  • Reporting additional data to the overall project manager, via e-mail or phone
  • Meeting with the overall project manager to review and confirm project results

It is useful for the whole project-management team to get together and review the overall results, before publishing them to senior management. This step ensures that no one is surprised by the results in the report. The overall project manager can reconcile any conflicts between sub-project managers and can make last-minute corrections to the scorecards.

Benchmark values change every week. The overall project manager fills in the actuals for the week and compares them against the benchmarks. Any variances should have an explanation.

The overall project manager publishes the scorecards and benchmarks to management. The cycle repeats for every reporting period until the project is complete.

Step 10: Refine and Update

Projects are all unique endeavors, and an overall project is a collection of many unique endeavors. Even the best project manager will have trouble predicting the course of an overall project. Benchmarks are usually the first to change. Changes in staffing levels, unexpected project events (positive or negative), changes to scope, or changes to estimates all should trigger a review of benchmarks. Benchmarks loose their value as a management and motivation tool as soon as they become too easy or too difficult.

Often the project’s metrics change as the project goes through different phases. The overall project manager should monitor overall project activities, to see when a particular metric is no longer relevant or when a new one should be captured. The overall project manager may choose to design a new scorecard for each phase or to keep a single scorecard, but remove and add data elements as the project evolves. Either approach can be successful. Adding new metrics mean that the overall project manager should consider adding a new benchmark goal. Dropping old metrics sometimes means dropping a benchmark.

This step goes hand-in-hand with updating scorecards. It may be triggered anytime after reporting begins on a project. Critical events in the project almost always trigger it, including changes to scope, budget, schedule, or phase.

During project close-out, the use of the scorecards and their evolution through the project often make their way into the lessons learned for the project. The history of scorecards and benchmarks will help the managers of future projects develop their own templates and standards.

Putting Scorecards and Benchmarks to Use

Following these ten steps, a project manager can have an easier transition from managing stand-alone projects to managing projects that contain sub-projects. These techniques are broadly applicable, useful for a wide range of industries. There is some key recommendations for people who are new to these tools:

  1. Keep the metrics simple and easy to understand
  2. Make sure that sub-project metrics “scale up” to the overall project
  3. Do not force metrics that work for one sub-project onto all sub-projects; only use what works
  4. Watch your math, because not all metrics sum up through simple addition; standard deviation adds using squares, for instance – see recommendation #1 to avoid this problem entirely
  5. Explicitly state the uncertainty of the numbers at the start; it is next to impossible to collect data completely accurately or to forecast benchmarks perfectly
  6. When benchmark values and actual values diverge, do not panic; figure out if the project is off-course or if the metrics are off-course and take corrective action
  7. Never, ever adjust the scorecard numbers in order to appear to meet goals; not only is it unethical behavior, it is increasing the projects’ list of problems and threatening the bond of trust with project sponsors
  8. Do adjust metric calculations and benchmarks as needed; do so openly, with the consent of senior managers and sub-project managers
  9. Use the literature to select appropriate metrics; academics have studied and argued over appropriate metrics for decades, and it is expensive to ignore their advice
  10. Only use sophisticated metrics after gaining experience with simpler metrics, and only if the audience of the reports understand the complex metrics thoroughly.

To summarize: FOCUS ON COMMUNICATION. These reports are a tool to communicate progress and status. Be creative, using color, graphs, charts, tables, pictures, or anything that helps communicate. An effective report will generate complete silence around a meeting-room table, followed by grunts of surprise, grunts of “humph,” exclamations of “wow!” and finally cries, “This is great,” or “NOW I understand.” The measure of effectiveness for a scorecard or benchmark is its communication value. It is not enough to have numbers that accurately reflect project progress, they must also communicate that progress to the intended audience. Perhaps the ultimate compliment is when senior management starts telling your peers to use benchmarks and scorecards, and your peers come to you as the in-house expert for advice.

Ultimately the scorecards and benchmarks are about seeing the forest, despite all the trees. Managers who remain focused on communicating the state of the project “forest” will not get lost in the “trees” of numbers in their reports. Giving senior management the right level of vision, supported by accurate, demonstrable facts, is the ultimate goal of the exercise.