Dr. George A. Johnson, CFPIM
Professor of Operations Management, RIT, Retired
A company presently evaluates the performance of its forecasters against a plus/minus 15% standard. That is, if actual results are not more than 15% above or below a forecast, the forecast is rated as acceptable. There are concerns about this approach, since the results of some errors clearly are more serious than others, even if the errors are of the same magnitude. Is there a better way to establish accuracy standards?
There is a better way to view the situation and it involves conceiving the issue as one of process control and improvement. The explanation is something like this.
Forecasting is a process. Its outputs typically are numbers. Some of the numbers represent expected future demand (or maybe future shipments); other numbers represent errors, the inability of the forecasting “model” to make perfect estimates. In this situation we are interested primarily in the second kind of output, forecast error.
Dr. Deming noted there are two important kinds of variation in process results and that it is crucial to understand which kind is being dealt with before taking any corrective action. One type of variation is that which has “common” causes. It is the largely unexplainable “background noise” which is present when a process performs the best it can with its current resources, under normal operating conditions.
Under normal conditions my full-size station wagon will get from 16-18 MPG on the highway, with the variation resulting from such typical, essentially random factors as changes in road pitch, elevation, outside temperature, wind conditions and traffic. To perform significantly better (less variably) than the typical 16-18 MPG, the vehicle’s (i.e., process) characteristics would have to be changed in some major way which requires an owner’s investment. For example, the axle ratio might have to be changed or cruise control installed.
The second kind of variation results from “special” or “assignable” causes. It is variation for which there is a known or knowable reason. If the highway performance of my wagon suddenly drops from the 16-18 MPG range to 10 MPG, I should investigate the unexpected change for its cause. Perhaps I find that this degradation is associated with a new and unusual operating circumstance, the strong smell of gasoline. Inspection reveals that a liquid is escaping from the top of the fuel pump while the engine runs. The fuel pump has failed, partially, resulting in a leak. This additional operating factor explains the extra fuel use which shows up as degraded MPG performance. Replacing the pump, i.e., removing the “special” cause of variation will restore the performance to a stable 16-18 MPG, where only “common” sources of variation are present.
Returning now to the original question, forecast accuracy standards, the plus/minus 15% standard has been set arbitrarily. It seems like a reasonable allowance for error, but is it a rational criterion for accuracy? Let’s explore the issue using the example above.
After extensive experience with my station wagon I have gathered quite a bit of data about its performance. Under normal conditions it averages 17 miles per gallon on the road and has never been observed to go above 18 or below 16 MPG. One could say the expected limits of highway performance are 17 MPG plus/minus 1 MPG. These are the typical results when the car is properly maintained and driven. If I am asked to forecast the average highway MPG for my next trip, I will estimate 17. I will not be surprised, however, if it actually turns out to be anywhere in the interval 16-18 MPG. Now, how does this knowledge play out against the seemingly reasonable but arbitrary plus/minus 15% standard?
The 15% standard translates to .15 x 17 or 2.55 MPG. Thus, “management” will be happy if the actual highway MPG is in the range 17 plus/minus 2.55 MPG or 14.45 - 19.55 MPG. When the fuel pump failed, the highway MPG dropped to 10 MPG. Clearly this was outside both the expected range (16-18 MPG) and the 15% standard range (14.45-19.55 MPG) of results. The forecaster was wrong! He should be admonished for poor forecast accuracy. But wait, the forecaster had no control over the failure of the fuel pump. It was a “special” cause of variation; a one-time event. Fix the special cause and performance should return to normal. It was good to be able to detect the abnormality, but why criticize the forecaster? It wasn’t his fault.
Consider another scenario. Prior to a second trip I forecast 17 MPG, just as above. At trip end, I compute that the actual MPG is 15. How was my forecast accuracy? By the 15% standard, it was OK. My forecast was accurate. I get a pat on the back. But in the recesses of my brain I know that something isn’t quite right. The trip MPG wasn’t in the 16-18 range. A little checking reveals that I unknowingly bought some inferior gasoline at the last stop and this explains the result. This is another example of a “special” cause of variation affecting the process outcome. (I’ll keep this one to myself, though, since management thinks everything is OK.)
In still another scenario, let’s assume that I have a really tired, old car, not as heavy as the station wagon. It also has given me an average of 17 MPG on the road and has never gone above 20 nor below 14 MPG under normal operating conditions. Thus, the normal range of highway MPG is 14-20 with an average of 17. I forecast 17 MPG for my next trip with this car. At trip end I compute the MPG actually is 14. I’m not surprised, because it has done this before under typical operating conditions (i.e., “common” variation).
I’m OK with this result but “management” is not. The actual trip MPG was below the limit of 14.45. I’m going to get criticized. But wait a minute. I operated the car correctly -- did what I was supposed to do -- and the outcome was the result of “common” causes of variation. It’s not fair that I get criticized. The car and I were doing the best we could do under the circumstances.
After being unreasonably criticized a few times, I start to get motivated to coast down hills, sneak a little extra gas into the tank, and play the system in other ways so I don’t get penalized for things I can’t control. This is not the best way to use my time (or company resources), but it will keep me out of trouble.
I think the message is clear. A well-intended but arbitrary standard usually does not correlate well with how a process actually functions under typical conditions. As a result, the standards do not help highlight the right things and often give very misleading impressions. These false impressions can then trigger “management” actions which actually make things worse. A better approach is based in the theory and tools of quality control and improvement.
Instead of emphasizing standards, the main focus should be on when and how to improve the forecasting process as measured by forecast error. Improvement should be immediately pursued whenever an unexpected event (i.e., “special” cause), pushes a forecast result out of its typical range of variation. The root cause should be uncovered and corrective action taken to eliminate or correct it.
Example (1): The special event is a one-time, non-recurring order from outside the set of established customers. It will never occur again. In this case, it probably is best to ignore the demand from this order for purposes of future forecasts.
Example (2): The event is the bankruptcy and permanent cessation of business by a major customer. While this will not occur again, the effect is lasting. In this case, the permanent reduction in expected demand should be taken into account in future forecasts. This may require that the forecast model be permanently reset to a new, lower level prior to the next forecasting cycle.
Since a well-run forecasting process should not have been surprised by the bankruptcy event, there is room for improvement. The possibility of the event should have been picked up by marketing intelligence, by the accounts receivable system sounding an alert, or even by finance keeping an eye on this major customer’s “health” via public reports and stock market figures. This is an example of why forecasting by team usually is more effective than that by an individual and these sources of information should be systematically integrated into the forecasting process for the future.
Many companies have large numbers of items to forecast, so many that it is effectively impossible to monitor all of them, manually. Computers and statistics to the rescue! Exceptions can be detected statistically and brought to the attention of forecasters for investigation. In a statistical sense, an exception is detected by comparing a given cycle’s forecast error for an item against its typical forecast error from the past (only “common” cause variation present). This is how statistical quality control charts work. If the current cycle’s error is significantly larger or smaller than that which is expected, a “flag” is raised. The item’s forecast is then subject to investigation for the root cause of the exceptional error.
Once the output of a forecasting process is stable, i.e., free of “special” cause variation, there may be other reasons to improve the process. In one case, it may be that the amount of “common”-cause-only variation which remains is quite large. This error forces the company to maintain a big investment in safety stock and/or to sacrifice some degree of customer service and/or to reduce operating efficiency to meet its objectives. In another case it may be that a noticeable pattern, perhaps seasonal, has developed in the “common”-cause-only error. This would be an opportunity to explain more of the variation — to remove it from the error variation and make it part of the demand forecast -- by using a more complete model.
Quality improvement tools such as check sheets, Pareto analysis, cause and effect diagrams, flow charts, scatter diagrams, run charts, regression analysis, etc., are useful for these kinds of process improvement projects, even though the application is outside the traditional quality field. It is easy and logical to portray forecast results as products of a systematic process just as physical products are the results of production processes. Hence, quality concepts and tools apply.
Now let’s close the loop to the original standards issue. Holding forecasters to an arbitrary standard is illogical and a lose-lose situation. Doing so can put more variation into the situation than if the standard did not exist in the first place (what Deming refers to as “tampering”). Ratees may change their behavior arbitrarily trying to live with an invalid standards system and/or management may make uninformed or misinformed and, therefore, inappropriate investments in the process, e.g., new software, new forecasters, new organization structure, etc.
It is far better to measure how the forecasting system actually performs statistically, to understand its inherent variation and be able to detect exceptions. Efforts to improve forecast accuracy, i.e., reduce error, can then be directed to the right places for the right reasons using the most appropriate tools and methods. Over time, forecast accuracy should improve without reference to any standards and this will pay off in improved customer service, lower operating expense and lower inventory investment, the real objectives.
Beck, J., “SPC and Selectable Calendars Work Magic for GE Aircraft Engines’ Service Parts Operation,” 1995 Annual International Conference Proceedings, APICS, pp. 477-481.
(Describes the use of several SPC tools for monitoring forecasts.)
Gitlow, H.; Oppenheim, A.; Oppenheim, R., Quality Management: Tools and Methods for Improvement, Second Edition, Irwin, 1995.
(See, especially, the discussion of Deming’s funnel experiment in Chapter 14, which illustrates the concept of “tampering” with processes.)
Ishikawa, Kaoru, Guide to Quality Control. Asian Productivity Organization, 1986. Available from GOAL/QPC, Methuen, MA.
Nolan, T.; Provost, L., “Understanding Variation,” Quality Progress, Vol. 23, No. 5, May 1990, pp. 70-78.
(Excellent discussion of common vs. special causes of variation and the implications for management of processes, systems and people. If you can read but one article, read this one.)
Link to the E/Step Software Web Site