Methodological Consistency for Quantitative Analysis and Reporting in Project Delivery System Performance Research
Publication: Journal of Construction Engineering and Management
Volume 148, Issue 9
Abstract
The project-based nature of construction means that a unique team of designers, contractors, and specialty trade partners must be assembled for the delivery of each new asset. The decisions around how these parties are procured and contracted can have downstream consequences for the success of those delivered assets, both in the timeliness and cost-effectiveness of the project, as well as in meeting the long-term needs of the client. Over the last 20 years, this Journal has published several studies that statistically compare cost and schedule performance between two or more project delivery systems. New and emerging alternative project delivery systems have recently been included in these studies. However, greater consistency in the data collection, analysis, and reporting practices are needed in the published literature around the topic of project delivery systems to allow for comparisons of studies, identifying trends over time, and understanding this body of knowledge. This article presents a review of the research methods used in 30 empirical papers published in leading construction engineering and management journals that studied the performance of different project delivery systems. These papers serve as a basis to identify successful practices and propose a set of recommendations for more consistent quantitative research in this field. The authors argue that greater consistency in project delivery research is critically needed, proposing a path forward that will assist researchers in conducting more reliable project delivery research for years to come. The paper is structured around three main categories of recommendations: (1) overcoming bias in project delivery data collection, (2) making a fair comparison, and (3) standardizing the presentation of results. In each category, the authors identify successful practices from published papers and make specific recommendations that provide a consistent standard for future studies. The vision of the authors is that the recommendations advanced in this paper, derived from exemplary cases of published work, will start a dialogue in the construction, engineering, and management community about establishing a standard of practice for quantitative project delivery system research. This article can be used as a resource for peer-reviewers and editors evaluating a project delivery study’s readiness for publication, and for faculty and graduate students looking for consistent guidance in conducting project delivery studies.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, and code generated or used during this study appear in the published article.
Acknowledgments
The authors would like to thank all the researchers that have contributed to the CEM project delivery methods literature over the last three decades, making this work possible and significantly impacting how projects are being delivered today.
References
Allen, L. N. 2001. Comparison of design-build to design-bid build as a project delivery method. Monterey, CA: Naval Postgraduate School Monetary.
Altman, D., D. Machin, T. Bryant, and M. Gardner, eds. 2013. Statistics with confidence: Confidence intervals and statistical guidelines. New York: Wiley.
Chinowksy, P. 2011. “Engineering project organization: Defining a line of inquiry and a path forward.” Eng. Project Organ. J. 1 (1): 3–10. https://doi.org/10.1080/21573727.2010.549611.
Cho, N., M. El Asmar, S. Underwood, and Y. Kamarianakis. 2020. “Long-term performance benefits of the design-build delivery method applied to road pavement projects in the US.” KSCE J. Civ. Eng. 24 (4): 1049–1059. https://doi.org/10.1007/s12205-020-1814-3.
Col Debella, D., and R. Ries. 2006. “Construction delivery systems: A comparative analysis of their performance within school districts.” J. Constr. Eng. Manage. 132 (11): 1131–1138. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:11(1131).
Collins, W., K. Parrish, and G. E. Gibson Jr. 2017. “Defining and understanding ‘small projects’ in the industrial construction sector.” Procedia Eng. 196: 315–322. https://doi.org/10.1016/j.proeng.2017.07.205.
Dainty, A. 2008. “Methodological pluralism in construction management research.” In Advanced research methods in the built environment, edited by Knight, A. and L. Ruddock, 1–13. Oxford, UK: Wiley.
Dao, B., S. Kermanshachi, J. Shane, S. Anderson, and E. Hare. 2017. “Exploring and assessing project complexity.” J. Constr. Eng. Manage. 143 (5): 04016126. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001275.
Demkin, J. A. 2002. The architect’s handbook of professional practice. 13th ed. Hoboken, NJ: Wiley.
Dempster, A. P., N. M. Laird, and D. B. Rubin. 1977. “Maximum likelihood from incomplete data via the EM algorithm.” J. R. Stat. Soc. B 39 (1): 1–22. https://doi.org/10.1111/j.2517-6161.1977.tb01600.x.
DeSarbo, S., P. E. Green, and J. D. Carroll. 1986. “An alternating least-squares procedure for estimating missing preference data in product-concept testing.” Decis. Sci. 17 (2): 163–185. https://doi.org/10.1111/j.1540-5915.1986.tb00219.x.
Donner, A. 1982. “The relative effectiveness of procedures commonly used in multiple regression analysis for dealing with missing values.” Am. Stat. 36 (4): 378–381. https://doi.org/10.1080/00031305.1982.10483055.
El Asmar, M., A. S. Hanna, and W. Y. Loh. 2013. “Quantifying performance for the integrated project delivery system as compared to established delivery systems.” J. Constr. Eng. Manage. 139 (11): 04013012. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000744.
El Asmar, M., A. S. Hanna, and W. Y. Loh. 2016. “Evaluating integrated project delivery using the project quarterback rating.” J. Constr. Eng. Manage. 142 (1): 04015046. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001015.
El Asmar, M., D. Ramsey, G. E. Gibson, and W. Bearup. 2020. “Design-build for transportation projects: Cost and schedule change performance analysis.” J. Leg. Aff. Dispute Resolut. Eng. Constr. 12 (1): 04519030. https://doi.org/10.1061/(ASCE)LA.1943-4170.0000335.
Ellis, R. D., Z. J. Herbsman, and A. Kumar. 1991. Evaluation of the FDOT design/build program. Tallahassee, FL: Florida DOT.
El Wardani, M. A., J. I. Messner, and M. J. Horman. 2006. “Comparing procurement methods for design-build projects.” J. Manage. Eng. 132 (3): 230–238.
Franz, B., and R. Leicht. 2016. “An alternative classification of project delivery methods used in the United States building construction industry.” Construct. Manage. Econ. 34 (3): 160–173. https://doi.org/10.1080/01446193.2016.1183800.
Franz, B., R. Leicht, K. Molenaar, and J. Messner. 2017. “Impact of team integration and group cohesion on project delivery performance.” J. Constr. Eng. Manage. 143 (1): 04016088. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001219.
Franz, B., K. R. Molenaar, and B. A. Roberts. 2020. “Revisiting project delivery system performance from 1998 to 2018.” J. Constr. Eng. Manage. 146 (9): 04020100. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001896.
Gransberg, D. D., and K. R. Molenaar. 2019. “Critical comparison of progressive design-build and construction manager/general contractor project delivery methods.” Transp. Res. Rec. 2673 (1): 261–268. https://doi.org/10.1177/0361198118822315.
Gransberg, D. D., and A. Touran. 2019. Value, benefits, and limitations of qualifications-based selection for airport project delivery. Washington, DC: Transportation Research Board.
Hale, D. R., P. P. Shrestha, G. E. Gibson Jr., and G. Migliaccio. 2009. “Empirical comparison of design/build and design/bid/build project delivery methods.” J. Constr. Eng. Manage. 135 (7): 579–587. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000017.
Hamer, R. M., and P. M. Simpson. 2009. “Last observation carried forward versus mixed models in the analysis of psychiatric clinical trials.” Am. J. Psychiatry 166 (6): 639–641. https://doi.org/10.1176/appi.ajp.2009.09040458.
Han, F., and S. M. Bogus. 2021. “Resilience criteria for project delivery processes: An exploratory analysis for highway project development.” J. Constr. Eng. Manage. 147 (11): 04021140. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002179.
Ibbs, C. W., Y. H. Kwak, T. Ng, and A. M. Odabasi. 2003. “Project delivery systems and project change: Quantitative analysis.” J. Constr. Eng. Manage. 129 (4): 382–387. https://doi.org/10.1061/(ASCE)0733-9364(2003)129:4(382).
Ibrahim, M. W., A. Hanna, and D. Kievet. 2020. “Quantitative comparison of project performance between project delivery systems.” J. Manage. Eng. 36 (6): 04020082. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000837.
Ireland, V. 1984. “Virtually meaningless distinction between nominally different procurement methods.” In Proc., 4th Int. Symp. on Organisation and Management of Construction: Organising and Managing Construction, CIB W65. Chicago: American Bar Association.
Kang, H. 2013. “The prevention and handling of the missing data.” Korean J. Anesthesiol. 64 (5): 402. https://doi.org/10.4097/kjae.2013.64.5.402.
Kim, J. O., and J. Curry. 1977. “The treatment of missing data in multivariate analysis.” Sociol. Methods Res. 6 (2): 215–240. https://doi.org/10.1177/004912417700600206.
Konchar, M., and V. Sanvido. 1998. “Comparison of US project delivery systems.” J. Constr. Eng. Manage. 124 (6): 435–444. https://doi.org/10.1061/(ASCE)0733-9364(1998)124:6(435).
Kwak, Y. H., Y. Chih, and C. W. Ibbs. 2009. “Towards a comprehensive understanding of public private partnerships for infrastructure development.” Calif. Manage. Rev. 51 (2): 51–78. https://doi.org/10.2307/41166480.
Malhotra, N. K. 1987. “Analyzing marketing research data with incomplete information on the dependent variable.” J. Marketing Res. 24 (1): 74–84. https://doi.org/10.1177/002224378702400107.
Migliaccio, G. C., S. M. Bogus, and A. Chen. 2010. “Effect of duration of design–build procurement on performance of transportation projects.” Transp. Res. Rec. 2151 (1): 67–73. https://doi.org/10.3141/2151-09.
Migliaccio, G. C., G. E. Gibson Jr., and J. T. O’connor. 2008. “Changing project delivery strategy: An implementation framework.” Public Works Manage. Policy 12 (3): 483–502.
Minchin, R. E., Jr., X. Li, R. R. Issa, and G. G. Vargas. 2013. “Comparison of cost and time performance of design-build and design-bid-build delivery systems in Florida.” J. Constr. Eng. Manage. 139 (10): 04013007. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000746.
National Research Council. 2010. The prevention and treatment of missing data in clinical trials. Washington, DC: National Academies Press.
Pocock, J. B., C. T. Hyun, L. Y. Liu, and M. K. Kim. 1996. “Relationship between project interaction and performance indicators.” J. Constr. Eng. Manage. 122 (2): 165–176. https://doi.org/10.1061/(ASCE)0733-9364(1996)122:2(165).
Ramsey, D., and M. E. Asmar. 2020. “Cost and schedule performance analysis of transportation public–private partnership projects.” J. Leg. Aff. Dispute Resolut. Eng. Constr. 12 (1): 04519032. https://doi.org/10.1061/(ASCE)LA.1943-4170.0000328.
Rojas, E. M., and I. Kell. 2008. “Comparative analysis of project delivery systems cost performance in Pacific Northwest public schools.” J. Constr. Eng. Manage. 134 (6): 387–397. https://doi.org/10.1061/(ASCE)0733-9364(2008)134:6(387).
Rosner, J. W., A. E. Thal Jr., and C. J. West. 2009. “Analysis of the design-build delivery method in air force construction projects.” J. Constr. Eng. Manage. 135 (8): 710–717. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000029.
Scharfstein, D. O., J. Hogan, and A. Herman. 2012. “Randomized trials in orthopaedic surgery: Advances and future directions: On the prevention and analysis of missing data in randomized clinical trials: The state of the art.” J. Bone Joint Surg. Am. 94 (S1): 80. https://doi.org/10.2106/JBJS.L.00273.
Shane, J. S., T. Becker, C. Roberts, M. Kiggins, and M. van Buren. 2020. “Owner and contractor choices and dilemmas: Contemporary project delivery options.” In Proc., Construction Research Congress 2020: Project Management and Controls, Materials, and Contracts, 1256–1265. Reston, VA: ASCE.
Sherratt, F., and R. Leicht. 2020. “Unpacking ontological perspectives in CEM research: Everything is biased.” J. Constr. Eng. Manage. 146 (2): 04019101. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001734.
Shrestha, P. P. 2020. “Legal issues and lessons learned with project delivery systems.” J. Leg. Aff. Dispute Resolut. Eng. Constr. 12 (3): 02020002. https://doi.org/10.1061/(ASCE)LA.1943-4170.0000410.
Shrestha, P. P., J. T. O’Connor, and G. E. Gibson Jr. 2012. “Performance comparison of large design-build and design-bid-build highway projects.” J. Constr. Eng. Manage. 138 (1): 1–13. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000390.
Sinharay, S., H. S. Stern, and D. Russell. 2001. “The use of multiple imputation for the analysis of missing data.” Psychol. Methods 6 (4): 317. https://doi.org/10.1037/1082-989X.6.4.317.
Sullivan, J., M. El Asmar, J. Chalhoub, and H. Obeid. 2017. “Two decades of performance comparisons for design-build, construction manager at risk, and design-bid-build: Quantitative analysis of the state of knowledge on project cost, schedule, and quality.” J. Constr. Eng. Manage. 143 (6): 04017009. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001282.
Thompson, S. K. 2012. Wiley series in probability and statistics sampling. Hoboken, NJ: Wiley.
Tran, D., J. Cameron Lampe, S. Bypaneni, and K. Molenaar. 2016. “An empirical comparison of cost growth between highway design-bid-build and design-build projects by project size.” In Proc., Construction Research Congress 2016, 2029–2038. Reston, VA: ASCE.
Tran, D. Q., G. Diraviam, and R. E. Minchin Jr. 2018. “Performance of highway design-bid-build and design-build projects by work types.” J. Constr. Eng. Manage. 144 (2): 04017112. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001437.
Williams, G. H., Jr. 2003. An evaluation of public construction contracting methods for the public building sector in Oregon using data envelopment analysis. Portland, OR: Portland State Univ.
Wisniewski, S. R., A. C. Leon, M. W. Otto, and M. H. Trivedi. 2006. “Prevention of missing data in clinical research studies.” Biol. Psychiatry 59 (11): 997–1000. https://doi.org/10.1016/j.biopsych.2006.01.017.
Yussef, A., G. E. Gibson Jr., M. El Asmar, and D. Ramsey. 2019. “Quantifying FEED maturity and its impact on project performance in large industrial projects.” J. Manage. Eng. 35 (5): 04019021. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000702.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 148 • Issue 9 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Oct 22, 2021
Accepted: Apr 13, 2022
Published online: Jun 24, 2022
Published in print: Sep 1, 2022
Discussion open until: Nov 24, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Ramy Khalef, Islam H. El-adaway, Performance of Project Delivery Methods for US Airport Projects: A Holistic Quantitative Risk-Based Assessment through Perceptions of Associated Industry Experts, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-14924, 150, 10, (2024).
- Fareed Salih, Radwa Eissa, Islam H. El-adaway, Data-Driven Analysis of Progressive Design Build in Water and Wastewater Infrastructure Projects, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13824, 150, 1, (2024).
- Charles T. Jahren, Patrick X. W. Zou, John E. Taylor, Edward J. Jaselskis, Second Special Collection on Research Methodologies in Construction Engineering and Management, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13129, 149, 2, (2023).