Concerns about Methodology Used in Real-World Experiments on Transport and Transport Safety
Publication: Journal of Transportation Engineering
Volume 133, Issue 1
Abstract
Real-world experiments in which some experimental units (e.g., people or roadway sites or vehicles) are randomized to treatment and others to no treatment are increasingly regarded as practicable and desirable. This is the case for complex educational and social welfare interventions as well as for therapeutic drug trials. The present paper discusses the relevance of such experiments to transport and transport safety research. When compared with the more common designs of research, both the advantages and the disadvantages may seem substantial, and decisions will require good judgment. Drawing lessons from medical and social welfare research, the paper aims to be a resource when the relative merits of randomized experimentation and other methods are being debated.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
The Centre for Automotive Safety Research receives core funding from both the Department of Transport and Urban Planning (South Australia) and the Motor Accident Commission (South Australia). The views expressed in this report are those of the writer and do not necessarily represent those of the University of Adelaide or the sponsoring organizations.
References
Basile, A. J. (1962). “Effect of pavement edge markings on traffic accidents in Kansas.” Bulletin No. 308, Highway Research Board 80–86.
Benekohal, R. F., and Hashmi, A. M. (1992). “Procedures for estimating accident reductions on two-lane highways.” J. Transp. Eng., 118(1), 111–129.
Bird, S. M., and Fairweather, C. B. (2004). “Criminal justice on trial. What works for young offenders.” Significance, 1(2), 55–58.
Burtless, G. (1995). “The case for randomized field trials in economic and policy research.” Journal of Economic Perspectives, 9(2), 63–84.
Chalmers, I., Sackett, D., and Silagy, C. (1998). “Cochrane Collaboration.” Encyclopedia of biostatistics, Vol. 2, Wiley, Chichester, U.K., 764–767.
Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide, Institute of Education Sciences, U.S. Dept. of Education, Washington, D.C.
Cochrane Collaboration. (2004). “What is the Cochrane Collaboration?” ⟨http://www.cochrane.org/docs/descrip.htm⟩.
Cochrane Injuries Group Driver Education Reviewers. (2001). “Evidence based road safety: The Driving Standards Agency’s schools programme.” Lancet, 358(9277), 230–232.
Council, F. M. et al. (1980). “Accident research manual.” Rep. from the Highway Safety Research Center, Univ. of North Carolina, National Technical Information Service, Springfield, Va.
Department of Transport. (1986). Accident investigation manual, Vol. 2, Royal Society for the Prevention of Accidents, Birmingham, U.K.
Dunford, F. W. (1987). “Random assignment: Practical considerations from field experiments.” Proc., Survey Research Methods Section, American Statistical Association, 64–69, ⟨http://www.amstat.org/sections/srms/Proceedings/y1987.html⟩.
Elvik, R., and Vaa, T., eds. (2004). The handbook of road safety measures, Elsevier, Amsterdam, The Netherlands.
Farrington, D. P. (1998). “Evaluating ‘Communities that care’. Realistic scientific considerations.” Evaluation, 4(2), 204–210.
Farrington, D. P. (2003). “British randomized experiments on crime and justice.” Annals of the American Academy of Political and Social Science, 589, 150–167.
Feinstein, A. R., and Horwitz, R. I. (1997). “Problems of the ‘evidence’ of ‘evidence-based medicine’.” Am. J. Med., 103(6), 529–535.
Fitz-Gibbon, C. (2004). “Editorial: The need for randomized trials in social research.” J. R. Stat. Soc. Ser. A (Stat. Soc.), 167(1), 1–4.
Freedman, D., Pisani, R., Purves, R., and Adhikari, A. (1991). Statistics, 2nd Ed., Norton, New York.
Gotzsche, P. C. (1996). “Blinding during data analysis and writing of manuscripts.” Control Clin. Trials, 17(4), 285–293.
Grossman, J. B. (1994). “Evaluating social policies: Principles and U.S. experience.” The World Bank Research Observer, 9(2), 159–180.
Hauer, E. (2004). “The harm done by tests of significance.” Accid. Anal Prev., 36(3), 495–500.
Hawe, P., Shiell, A., and Riley, T. (2004). “Complex interventions: How ‘out of control’ can a randomised controlled trial be?” Br. Med. J., 328(7455), 1561–1563.
Heckman, J. J., and Smith, J. A. (1995). “Assessing the case for social experiments.” Journal of Economic Perspectives, 9(2), 85–110.
House of Representatives Standing Committee on Transport and Regional Services. (2004). National road safety—Eyes on the road ahead, Parliament of the Commonwealth of Australia, Canberra, Australia, ⟨http://www.aph.gov.au/house/committee/trs/roadsafety/report/fullreport.pdf⟩.
Hummer, J. E. (1993). “Discussion of procedures for estimating accident reductions on two-lane highways.” J. Transp. Eng., 119(3), 486–487.
Hutchinson, T. P., and Meier, A. J. (2004). “Evidence-based road safety policy? Evidence-based transport policy? A discussion of randomised experimentation and meta-analysis in the evaluation of interventions.” Papers of the 27th Australasian Transport Research Forum, M. A. P. Taylor and P. M. Tisato, eds., Transport Systems Centre, Univ. of South Australia, Adelaide.
Leigh, A. (2003). “Randomised policy trials.” Agenda, 10(4), 341–354.
Meinert, C. L. (1998). Clinical trials, overview. Encyclopedia of biostatistics, Vol. 1, Wiley, Chichester, U.K., 698–713.
Mosteller, F., and Boruch, R., eds. (2002). Evidence matters. Randomized trials in education research, Brookings Institution Press, Washington, D.C.
Musick, J. V. (1960). “Effect of pavement edge marking on two-lane rural state highways in Ohio.” Bulletin No. 266, Highway Research Board 1–7.
Newman, J., Rawlings, L., and Gertler, P. (1994). “Using randomized control designs in evaluating social sector programs in developing countries.” The World Bank Research Observer, 9(2), 181–201.
Oakley, A., Strange, V., Toroyan, T., Wiggins, M., Roberts, I., and Stephenson, J. (2003). “Using random allocation to evaluate social interventions: Three recent U.K. examples.” Annals of the American Academy of Political and Social Science, 589, 170–189.
Pawson, R., and Myhill, A. (2001). “Learning lessons: Enhancing evaluation through research review.” Rep. No. 507, TRL, Crowthorne, U.K.
Pawson, R., and Tilley, R. (1998). “Caring communities, paradigm polemics, design debates.” Evaluation, 4(1), 73–90.
Peterson, A. V., Mann, S. L., Kealey, K. A., and Marek, P. M. (2000). “Experimental design and methods for school-based randomized trials: Experience from the Hutchinson Smoking Prevention Project (HSPP).” Control Clin. Trials, 21(2), 144–165.
Petrosino, A., Turpin-Petrosino, C., and Buehler, J., (2003). “ ‘Scared straight’ and other juvenile awareness programs for preventing juvenile delinquency.” Campbell Review Update I. The Campbell Collaboration Reviews of Intervention and Policy Evaluations(C2-RIPE), Campbell Collaboration, Philadelphia.
Pope, C. (2003). “Resisting evidence: The study of evidence-based medicine as a contemporary social movement.” Health: An Interdisciplinary Journal for the Social Study of Health, Illness and Medicine, 7(3), 267–282.
Prescott, R. J., et al. (1999). “Factors that limit the quality, number and progress of randomised controlled trials.” Health Technol. Assess, 3(20).
Research on Road Traffic. (1965). Her Majesty's Stationery Office, London.
Riccio, J. A., and Bloom, H. S. (2001). Extending the reach of randomized social experiments: New directions in evaluations of American welfare-to-work and employment initiatives. Working Paper on Research Methodology, Manpower Demonstration Research Corporation, New York, ⟨http://www.mdrc.org/Reports2001/newdirections-methodologywrkppr/Methodology-NewDirections.pdf⟩.
Sherman, L. W., ed. (2003). “Misleading evidence and evidence-led policy: Making social science more experimental.” Annals of the American Academy of Political and Social Science, 6–19.
Sutton, A. J., Abrams, K. R., Jones, D. R., Sheldon, T. A., and Song, F. (2000). Methods for meta-analysis in medical research, Wiley, Chichester.
Tamburri, T. N., Hammer, C. J., Glennon, J. C., and Lew, A. (1968). “Evaluation of minor improvements.” Highway Research Record No. 257, 34–79.
Thompson, S. G. (1998). “Meta-analysis of clinical trials.” Encyclopedia of biostatistics, Vol. 4, Wiley, Chichester, 2570–2579.
van den Eynde, J., Veno, A., and Hart, A. (2003). “They look good but don’t work: A case study of global performance indicators in crime prevention.” Evaluation and Program Planning, 26(3), 237–248.
Watson, L., Small, R., Brown, S., Dawson, W., and Lumley, J. (2004). “Mounting a community-randomized trial: Sample size, matching, selection, and randomization issues in PRISM.” Control Clin. Trials, 25(3), 235–250.
Whitehead, A. (2002). Meta-analysis of controlled clinical trials, Wiley, Chichester, U.K.
Wittes, J. (1998). “Randomized treatment assignment.” Encyclopedia of biostatistics, Vol. 5, Wiley, Chichester, U.K., 3703–3711.
Information & Authors
Information
Published In
Journal of Transportation Engineering
Volume 133 • Issue 1 • January 2007
Pages: 30 - 38
Copyright
© 2007 ASCE.
History
Received: Jun 14, 2005
Accepted: Mar 7, 2006
Published online: Jan 1, 2007
Published in print: Jan 2007
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.