NASA Jet Propulsion Laboratory (JPL) and their use of commercial predictive models has prompted a need for an internal validation study to build confidence in the model results, enabling future mission estimates to be proposed with evidence of substantiation. Parametric (predictive) cost estimates are based on mathematical expressions derived from historical data that relate cost as the dependent variable to selected, independent, cost-driving variables such as weight characteristics and design complexity. Generally, predictive cost models are used to support design-to-cost trades, crosschecking or benchmarking, and long-range planning.
There are a variety of predictive cost models available to NASA for estimating a project cost (NASA Cost Estimating Handbook Version 4.0, Appendix E). NASA provides these models and tools for its cost user community, including tools that NASA has sponsored (PCEC, NICM) and tools for which NASA provides licenses (SEER, TruePlanning).
NASA Jet Propulsion Laboratory (JPL) and their use of commercial predictive models has prompted a need for an internal validation study to build confidence in the model results, enabling future mission estimates to be proposed with evidence of substantiation.
Unison Cost Engineering had the pleasure of working with The JPL Cost Estimation and Pricing (CE&P) Section to support this TruePlanning model validation study. The scope of the validation focused on the following objectives:
- Establish a repeatable process for estimating JPL project costs in TruePlanning
- Validate TruePlanning 2016 against JPL actuals
- Provide recommended mapping for System and Assembly costs objects to the NASA Standard WBS
- Document results and communicate to end-users
- Standardize guidance to ensure application of the tool is consistent across JPL projects
- Provide guidance to external entities when the model is used for assessing JPL projects
By developing and executing a standard approach for modeling projects, 10 JPL missions (including 25 instruments) were validated during this study. The results of the validation study are consistent with the levels of uncertainty seen in proposals and early formulation work. In the future, additional data points can be added to the study to expand the scope of the validation.
The criteria for successful validation was for each project to model within 30% of its historical actuals. This criteria was chosen to represent a percentage that is consistent with model bias experienced at the early formulation stage. This model bias from which we validate to is in line with the average growth experienced by projects during development.
At the end of the study, all ten projects validated within 21% of the project actuals.
The validation study performed by NASA JPL in conjunction with Unison Cost Engineering established a repeatable process for estimating JPL project costs in TruePlanning. In the future, additional data points can be added to the study to expand the scope of the validation.
We presented the results of this validation study at the NASA Cost and Schedule Symposium, and can make the details available to reviewers on an as-needed basis.