Bibliography

Bibliography

1

G. E. P. Box. Science and Statistics. Journal of the American Statistical Association, 71(356):791–799, 1976. URL: https://www.tandfonline.com/doi/abs/10.1080/01621459.1976.10480949, doi:10.1080/01621459.1976.10480949.

2

National Research Council and others. Convergence: Facilitating transdisciplinary integration of life sciences, physical sciences, engineering, and beyond. National Academies Press, 2014.

3

Sondoss Elsawah, Tatiana Filatova, Anthony J Jakeman, Albert J Kettner, Moira L Zellner, Ioannis N Athanasiadis, Serena H Hamilton, Robert L Axtell, Daniel G Brown, Jonathan M Gilligan, and others. Eight grand challenges in socio-environmental systems modeling. Socio-Environmental Systems Modelling, 2:16226–16226, 2020.

4

Yacov Y Haimes. Risk modeling of interdependent complex systems of systems: theory and practice. Risk analysis, 38(1):84–98, 2018.

5

Dirk Helbing. Globally networked risks and how to respond. Nature, 497(7447):51–59, 2013.

6

Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices. Environmental modelling & software, 114:29–39, 2019.

7

Daniel Wirtz and Wolfgang Nowak. The rocky road to extended simulation frameworks covering uncertainty, inversion, optimization and control. Environmental Modelling & Software, 93:180–192, 2017.

8

Roger Cooke and others. Experts in uncertainty: opinion and subjective probability in science. Oxford University Press on Demand, 1991.

9

Enayat A Moallemi, Jan Kwakkel, Fjalar J de Haan, and Brett A Bryan. Exploratory modeling for analyzing coupled human-natural systems under uncertainty. Global Environmental Change, 65:102186, 2020.

10

Warren E Walker, Poul Harremoës, Jan Rotmans, Jeroen P Van Der Sluijs, Marjolein BA Van Asselt, Peter Janssen, and Martin P Krayer von Krauss. Defining uncertainty: a conceptual basis for uncertainty management in model-based decision support. Integrated assessment, 4(1):5–17, 2003.

11

Andrea Saltelli, Philip B Stark, William Becker, and Pawel Stano. Climate models as economic guides scientific challenge or quixotic quest? Issues in Science and Technology, 31(3):79–84, 2015.

12

Hoshin V. Gupta, Thorsten Wagener, and Yuqiong Liu. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrological Processes: An International Journal, 22(18):3802–3813, 2008. Publisher: Wiley Online Library.

13

Antonia Hadjimichael, Julianne Quinn, and Patrick Reed. Advancing Diagnostic Model Evaluation to Better Understand Water Shortage Mechanisms in Institutionally Complex River Basins. Water Resources Research, 56(10):e2020WR028079, 2020. URL: http://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020WR028079 (visited on 2020-10-16), doi:10.1029/2020WR028079.

14

Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. Global Sensitivity Analysis: The Primer. Wiley-Interscience, Chichester, England ; Hoboken, NJ, 1 edition edition, February 2008. ISBN 978-0-470-05997-5.

15

Keith Beven. Towards a coherent philosophy for modelling the environment. Proceedings of the royal society of London. Series A: mathematical, physical and engineering sciences, 458(2026):2465–2484, 2002.

16

Naomi Oreskes, Kristin Shrader-Frechette, and Kenneth Belitz. Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences. Science, 263(5147):641–646, February 1994. URL: https://science.sciencemag.org/content/263/5147/641 (visited on 2020-04-15), doi:10.1126/science.263.5147.641.

17

Keith Beven. Prophecy, reality and uncertainty in distributed hydrological modelling. Advances in water resources, 16(1):41–51, 1993.

18

Keith Beven and Andrew Binley. The future of distributed models: Model calibration and uncertainty prediction. Hydrological Processes, 6(3):279–298, 1992. doi:10.1002/hyp.3360060305.

19

Yaman Barlas and Stanley Carpenter. Philosophical roots of model validation: Two paradigms. System Dynamics Review, 6(2):148–166, 1990. doi:10.1002/sdr.4260060203.

20

Stephen Toulmin. From form to function: philosophy and history of science in the 1950s and now. Daedalus, pages 143–162, 1977. Publisher: JSTOR.

21

George B. Kleindorfer, Liam O'Neill, and Ram Ganeshan. Validation in simulation: Various positions in the philosophy of science. Management Science, 44(8):1087–1099, 1998. Publisher: INFORMS.

22

Sibel Eker, Elena Rovenskaya, Michael Obersteiner, and Simon Langan. Practice and perspectives in the validation of resource management models. Nature communications, 9(1):1–10, 2018.

23

Yaman Barlas. Formal aspects of model validity and validation in system dynamics. System Dynamics Review: The Journal of the System Dynamics Society, 12(3):183–210, 1996. Publisher: Wiley Online Library.

24

Thomas H. Naylor and Joseph Michael Finger. Verification of computer simulation models. Management science, 14(2):B–92, 1967. Publisher: INFORMS.

25

Keith J Beven. On hypothesis testing in hydrology: why falsification of models is still a really good idea. Wiley Interdisciplinary Reviews: Water, 5(3):e1278, 2018.

26

Hoshin V Gupta, Martyn P Clark, Jasper A Vrugt, Gab Abramowitz, and Ming Ye. Towards a comprehensive assessment of model structural adequacy. Water Resources Research, 2012.

27

Praveen Kumar. Typology of hydrologic predictability. Water Resources Research, 2011. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2010WR009769 (visited on 2020-04-15), doi:10.1029/2010WR009769.

28

Grey S. Nearing, Benjamin L. Ruddell, Andrew R. Bennett, Cristina Prieto, and Hoshin V. Gupta. Does Information Theory Provide a New Paradigm for Earth Science? Hypothesis Testing. Water Resources Research, 56(2):e2019WR024918, 2020. doi:10.1029/2019WR024918.

29

Hoshin Vijai Gupta, Soroosh Sorooshian, and Patrice Ogou Yapo. Toward improved calibration of hydrologic models: Multiple and noncommensurable measures of information. Water Resources Research, 34(4):751–763, 1998. URL: http://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/97WR03495 (visited on 2020-04-07), doi:10.1029/97WR03495.

30

Francesca Pianosi and Thorsten Wagener. Understanding the time-varying importance of different uncertainty sources in hydrological modelling using global sensitivity analysis. Hydrological Processes, pages 3991–4003, November 2017. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/hyp.10968%4010.1111/%28ISSN%291099-1085.Kieth-Beven, doi:10.1002/hyp.10968@10.1111/(ISSN)1099-1085.Kieth-Beven.

31

Charles Rougé, Patrick M. Reed, Danielle S. Grogan, Shan Zuidema, Alexander Prusevich, Stanley Glidden, Jonathan R. Lamontagne, and Richard B. Lammers. Coordination and Control: Limits in Standard Representations of Multi-Reservoir Operations in Hydrological Modeling. Hydrology and Earth System Sciences Discussions, pages 1–37, November 2019. URL: https://www.hydrol-earth-syst-sci-discuss.net/hess-2019-589/, doi:https://doi.org/10.5194/hess-2019-589.

32

David W. Cash, William C. Clark, Frank Alcock, Nancy M. Dickson, Noelle Eckley, David H. Guston, Jill Jäger, and Ronald B. Mitchell. Knowledge systems for sustainable development. Proceedings of the national academy of sciences, 100(14):8086–8091, 2003. Publisher: National Acad Sciences.

33

Dave D. White, Amber Wutich, Kelli L. Larson, Patricia Gober, Timothy Lant, and Clea Senneville. Credibility, salience, and legitimacy of boundary objects: water managers' assessment of a simulation model in an immersive decision theater. Science and Public Policy, 37(3):219–232, April 2010. Publisher: Oxford Academic. URL: https://academic.oup.com/spp/article/37/3/219/1626552 (visited on 2020-05-12), doi:10.3152/030234210X497726.

34

Andrea Saltelli and Silvio Funtowicz. When all models are wrong. Issues in Science and Technology, 30(2):79–85, 2014. Publisher: JSTOR.

35

Thorsten Wagener and Francesca Pianosi. What has Global Sensitivity Analysis ever done for us? A systematic review to support scientific advancement and to inform policy-making in earth system modelling. Earth-Science Reviews, 194:1–18, July 2019. URL: https://www.sciencedirect.com/science/article/pii/S0012825218300990 (visited on 2021-08-30), doi:10.1016/j.earscirev.2019.04.006.

36

Steve Bankes. Exploratory Modeling for Policy Analysis. Operations Research, 41(3):435–449, June 1993. URL: https://pubsonline.informs.org/doi/abs/10.1287/opre.41.3.435 (visited on 2018-09-11), doi:10.1287/opre.41.3.435.

37

Christopher P. Weaver, Robert J. Lempert, Casey Brown, John A. Hall, David Revell, and Daniel Sarewitz. Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks. Wiley Interdisciplinary Reviews: Climate Change, 4(1):39–60, 2013. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/wcc.202 (visited on 2019-10-01), doi:10.1002/wcc.202.

38

Andrea Saltelli, Stefano Tarantola, Francesca Campolongo, and Marco Ratto. Sensitivity analysis in practice: a guide to assessing scientific models. Volume 1. Wiley Online Library, 2004.

39

Emanuele Borgonovo and Elmar Plischke. Sensitivity analysis: a review of recent advances. European Journal of Operational Research, 248(3):869–887, 2016.

40

O Rakovec, Mary C Hill, MP Clark, AH Weerts, AJ Teuling, and R Uijlenhoet. Distributed evaluation of local sensitivity analysis (delsa), with application to hydrologic models. Water Resources Research, 50(1):409–426, 2014.

41

Andrea Saltelli and Paola Annoni. How to avoid a perfunctory sensitivity analysis. Environmental Modelling & Software, 25(12):1508–1517, 2010.

42

Yong Tang, Patrick Reed, Thibaut Wagener, and K van Werkhoven. Comparing sensitivity analysis methods to advance lumped watershed model identification and evaluation. Hydrology and Earth System Sciences, 11(2):793–817, 2007.

43

Nicholas AS Hamm, Jim W Hall, and MG Anderson. Variance-based sensitivity analysis of the probability of hydrologically induced slope instability. Computers & geosciences, 32(6):803–817, 2006.

44

Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why so many published sensitivity analyses are false: a systematic review of sensitivity analysis practices. Environmental modelling & software, 114:29–39, 2019.

45

Benjamin P Bryant and Robert J Lempert. Thinking inside the box: a participatory, computer-assisted approach to scenario discovery. Technological Forecasting and Social Change, 77(1):34–49, 2010.

46

Andrea Saltelli and Stefano Tarantola. On the relative importance of input factors in mathematical models: safety assessment for nuclear waste disposal. Journal of the American Statistical Association, 97(459):702–709, 2002.

47

Barry Anderson, Emanuele Borgonovo, Marzio Galeotti, and Roberto Roson. Uncertainty in climate change modeling: can global sensitivity analysis be of help? Risk analysis, 34(2):271–293, 2014.

48

Emanuele Borgonovo. Sensitivity analysis with finite changes: an application to modified eoq models. European Journal of Operational Research, 200(1):127–138, 2010.

49

Andrea Saltelli, Marco Ratto, Terry Andres, Francesca Campolongo, Jessica Cariboni, Debora Gatelli, Michaela Saisana, and Stefano Tarantola. Global sensitivity analysis: the primer. John Wiley & Sons, 2008.

50

Neil R Edwards, David Cameron, and Jonathan Rougier. Precalibrating an intermediate complexity climate model. Climate dynamics, 37(7):1469–1482, 2011.

51

Francesca Pianosi, Keith Beven, Jim Freer, Jim W Hall, Jonathan Rougier, David B Stephenson, and Thorsten Wagener. Sensitivity analysis of environmental models: a systematic review with practical workflow. Environmental Modelling & Software, 79:214–232, 2016.

52

RC Spear and GM Hornberger. Eutrophication in peel inlet—ii. identification of critical uncertainties via generalized sensitivity analysis. Water research, 14(1):43–49, 1980.

53

Jon C Helton, Jay Dean Johnson, Cedric J Sallaberry, and Curt B Storlie. Survey of sampling-based methods for uncertainty and sensitivity analysis. Reliability Engineering & System Safety, 91(10-11):1175–1209, 2006.

54

Ronald Aylmer Fisher. Design of experiments. Br Med J, 1(3923):554–554, 1936.

55

JD Herman, PM Reed, and T Wagener. Time-varying sensitivity analysis clarifies the effects of watershed model formulation on model behavior. Water Resources Research, 49(3):1400–1414, 2013.

56

Carolina Massmann, Thorsten Wagener, and Hubert Holzmann. A new approach to visualizing time-varying sensitivity indices for environmental model diagnostics across evaluation time-scales. Environmental modelling & software, 51:190–194, 2014.

57

An Van Schepdael, Aurélie Carlier, and Liesbet Geris. Sensitivity analysis by design of experiments. In Uncertainty in Biology, pages 327–366. Springer, 2016.

58

Nicholas Metropolis and Stanislaw Ulam. The monte carlo method. Journal of the American statistical association, 44(247):335–341, 1949.

59

John Norton. An introduction to sensitivity assessment of simulation models. Environmental Modelling & Software, 69:166–174, 2015.

60

Douglas C Montgomery. Design and analysis of experiments. John wiley & sons, 2017.

61

George EP Box and J Stuart Hunter. The 2 k—p fractional factorial designs. Technometrics, 3(3):311–351, 1961.

62

Izabella Surowiec, Ludvig Vikstrom, Gustaf Hector, Erik Johansson, Conny Vikstrom, and Johan Trygg. Generalized subset designs in analytical chemistry. Analytical chemistry, 89(12):6491–6497, 2017.

63

Michael D McKay, Richard J Beckman, and William J Conover. A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics, 21(1):239–2451, 1979.

64

Boxin Tang. Orthogonal array-based latin hypercubes. Journal of the American statistical association, 88(424):1392–1397, 1993.

65

Ishaan L Dalal, Deian Stefan, and Jared Harwayne-Gidansky. Low discrepancy sequences for monte carlo simulations on reconfigurable platforms. In 2008 International Conference on Application-Specific Systems, Architectures and Processors, 108–113. IEEE, 2008.

66

SK Zaremba. The mathematical basis of monte carlo and quasi-monte carlo methods. SIAM review, 10(3):303–314, 1968.

67

Sergei Kucherenko, Daniel Albrecht, and Andrea Saltelli. Exploring multi-dimensional spaces: a comparison of latin hypercube and quasi monte carlo sampling techniques. arXiv preprint arXiv:1505.02350, 2015.

68

Bertrand Iooss, Loïc Boussouf, Vincent Feuillard, and Amandine Marrel. Numerical studies of the metamodel fitting and validation processes. arXiv preprint arXiv:1001.1049, 2010.

69

Ruichen Jin, Wei Chen, and Agus Sudjianto. An efficient algorithm for constructing optimal design of computer experiments. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 37009, 545–554. 2003.

70

Max D Morris and Toby J Mitchell. Exploratory designs for computational experiments. Journal of statistical planning and inference, 43(3):381–402, 1995.

71

Jeong-Soo Park. Optimal latin-hypercube designs for computer experiments. Journal of statistical planning and inference, 39(1):95–111, 1994.

72

Ilya M Sobol. Uniformly distributed sequences with an additional uniform property. USSR Computational Mathematics and Mathematical Physics, 16(5):236–242, 1976.

73

Il'ya Meerovich Sobol'. On the distribution of points in a cube and the approximate evaluation of integrals. Zhurnal Vychislitel'noi Matematiki i Matematicheskoi Fiziki, 7(4):784–802, 1967.

74

Max D Morris. Factorial sampling plans for preliminary computational experiments. Technometrics, 33(2):161–174, 1991.

75

RI Cukier, CM Fortuin, Kurt E Shuler, AG Petschek, and JH Schaibly. Study of the sensitivity of coupled reaction systems to uncertainties in rate coefficients. i theory. The Journal of chemical physics, 59(8):3873–3878, 1973.

76

Andrea Saltelli, Stefano Tarantola, and KP-S Chan. A quantitative model-independent method for global sensitivity analysis of model output. Technometrics, 41(1):39–56, 1999.

77

Ilya M Sobol. Global sensitivity indices for nonlinear mathematical models and their monte carlo estimates. Mathematics and computers in simulation, 55(1-3):271–280, 2001.

78

Jonathan D Herman, Harrison B Zeff, Jonathan R Lamontagne, Patrick M Reed, and Gregory W Characklis. Synthetic drought scenario generation to support bottom-up water supply vulnerability assessments. Journal of Water Resources Planning and Management, 142(11):04016050, 2016.

79

PCD Milly, Julio Betancourt, Malin Falkenmark, Robert M Hirsch, Zbigniew W Kundzewicz, Dennis P Lettenmaier, and Ronald J Stouffer. Stationarity is dead: whither water management? Earth, 4:20, 2008.

80

Edoardo Borgomeo, Christopher L Farmer, and Jim W Hall. Numerical rivers: a synthetic streamflow generator for water resources vulnerability assessments. Water Resources Research, 51(7):5382–5405, 2015.

81

Manuel Herrera, Sukumar Natarajan, David A Coley, Tristan Kershaw, Alfonso P Ramallo-González, Matthew Eames, Daniel Fosas, and Michael Wood. A review of current and future weather data for building simulation. Building Services Engineering Research and Technology, 38(5):602–627, 2017.

82

Daniel S Wilks and Robert L Wilby. The weather generation game: a review of stochastic weather models. Progress in physical geography, 23(3):329–357, 1999.

83

JR Lamontagne and JR Stedinger. Generating synthetic streamflow forecasts with specified precision. Journal of Water Resources Planning and Management, 144(4):04018007, 2018.

84

Sanghamitra Medda and Kalyan Kumar Bhar. Comparison of single-site and multi-site stochastic models for streamflow generation. Applied Water Science, 9(3):67, 2019.

85

Brian R Kirsch, Gregory W Characklis, and Harrison B Zeff. Evaluating the impact of alternative hydro-climate scenarios on transfer agreements: practical improvement for generating synthetic streamflows. Journal of Water Resources Planning and Management, 139(4):396–406, 2013.

86

Daniel P Loucks and Eelco Van Beek. Water resource systems planning and management: An introduction to methods, models, and applications. Springer, 2017.

87

Scott Steinschneider, Sungwook Wi, and Casey Brown. The integrated effects of climate and hydrologic uncertainty on future flood risk assessments. Hydrological Processes, 29(12):2823–2839, 2015.

88

Richard M Vogel. Stochastic watershed models for hydrologic risk management. Water Security, 1:28–35, 2017.

89

Richard M Vogel and Jery R Stedinger. The value of stochastic streamflow models in overyear reservoir design applications. Water Resources Research, 24(9):1483–1490, 1988.

90

Emanuele Borgonovo. Sensitivity analysis of model output with input constraints: a generalized rationale for local methods. Risk Analysis: An International Journal, 28(3):667–680, 2008.

91

Bertrand Iooss and Paul Lemaître. A review on global sensitivity analysis methods. In Uncertainty management in simulation-optimization of complex systems, pages 101–122. Springer, 2015.

92

Francesca Campolongo and Roger Braddock. The use of graph theory in the sensitivity analysis of the model output: a second order screening method. Reliability Engineering & System Safety, 64(1):1–12, 1999.

93

Roger A Cropp and Roger D Braddock. The new morris method: an efficient second-order screening method. Reliability Engineering & System Safety, 78(1):77–83, 2002.

94

Jon C Helton. Uncertainty and sensitivity analysis techniques for use in performance assessment for radioactive waste disposal. Reliability Engineering & System Safety, 42(2-3):327–367, 1993.

95

Gemma Manache and Charles S Melching. Identification of reliable regression-and correlation-based sensitivity measures for importance ranking of water-quality model parameters. Environmental Modelling & Software, 23(5):549–562, 2008.

96

F Pappenberger and Keith J Beven. Ignorance is bliss: or seven reasons not to use uncertainty analysis. Water resources research, 2006.

97

Jerome H Friedman and Nicholas I Fisher. Bump hunting in high-dimensional data. Statistics and Computing, 9(2):123–143, 1999.

98

Leo Breiman, Jerome Friedman, Charles J Stone, and Richard A Olshen. Classification and regression trees. CRC press, 1984.

99

Yoav Freund, Robert Schapire, and Naoki Abe. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence, 14(771-780):1612, 1999.

100

Leo Breiman. Bagging predictors. Machine learning, 24(2):123–140, 1996.

101

George M Hornberger and Robert C Spear. Approach to the preliminary analysis of environmental systems. J. Environ. Mgmt., 12(1):7–18, 1981.

102

Robert J Lempert, David G Groves, Steven W Popper, and Steve C Bankes. A general, analytic method for generating robust strategies and narrative scenarios. Management science, 52(4):514–528, 2006.

103

David G Groves and Robert J Lempert. A new analytic method for finding policy-relevant scenarios. Global Environmental Change, 17(1):73–85, 2007.

104

Keith Beven and Andrew Binley. Glue: 20 years on. Hydrological processes, 28(24):5897–5918, 2014.

105

Roberta-Serena Blasone, Jasper A Vrugt, Henrik Madsen, Dan Rosbjerg, Bruce A Robinson, and George A Zyvoloski. Generalized likelihood uncertainty estimation (glue) using adaptive markov chain monte carlo sampling. Advances in Water Resources, 31(4):630–648, 2008.

106

SA Cryer and PL Havens. Regional sensitivity analysis using a fractional factorial method for the usda model gleams. Environmental modelling & software, 14(6):613–624, 1999.

107

Pengfei Wei, Zhenzhou Lu, and Xiukai Yuan. Monte carlo simulation for moment-independent sensitivity analysis. Reliability Engineering & System Safety, 110:60–67, 2013.

108

Peter Young. Data-based mechanistic modelling, generalised sensitivity and dominant mode analysis. Computer Physics Communications, 117(1-2):113–129, 1999.

109

Andrea Saltelli. Making best use of model evaluations to compute sensitivity indices. Computer physics communications, 145(2):280–297, 2002.

110

Andrea Saltelli. Sensitivity analysis for importance assessment. Risk analysis, 22(3):579–590, 2002.

111

Toshimitsu Homma and Andrea Saltelli. Importance measures in global sensitivity analysis of nonlinear models. Reliability Engineering & System Safety, 52(1):1–17, 1996.

112

Gregory J McRae, William R Goodin, and John H Seinfeld. Development of a second-generation mathematical model for urban air pollution—i. model formulation. Atmospheric Environment (1967), 16(4):679–696, 1982.

113

Andrea Saltelli and Ricardo Bolado. An alternative way to compute fourier amplitude sensitivity test (fast). Computational Statistics & Data Analysis, 26(4):445–460, 1998.

114

MA Vazquez-Cruz, R Guzman-Cruz, IL Lopez-Cruz, O Cornejo-Perez, I Torres-Pacheco, and RG Guevara-Gonzalez. Global sensitivity analysis by means of efast and sobol’methods and calibration of reduced state-variable tomgro model using genetic algorithms. Computers and Electronics in Agriculture, 100:1–12, 2014.

115

Benjamin Auder and Bertrand Iooss. Global sensitivity analysis based on entropy. In Safety, reliability and risk analysis-Proceedings of the ESREL 2008 Conference, 2107–2115. 2008.

116

Farkhondeh Khorashadi Zadeh, Jiri Nossent, Fanny Sarrazin, Francesca Pianosi, Ann van Griensven, Thorsten Wagener, and Willy Bauwens. Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the swat model. Environmental Modelling & Software, 91:210–222, 2017.

117

Francesca Pianosi and Thorsten Wagener. A simple and efficient method for global sensitivity analysis based on cumulative distribution functions. Environmental Modelling & Software, 67:1–11, 2015.

118

Ronald Aylmer Fisher and others. Statistical methods for research workers. Statistical methods for research workers., 1934.

119

Loıc Brevault, Mathieu Balesdent, Nicolas Bérend, and Rodolphe Le Riche. Comparison of different global sensitivity analysis methods for aerospace vehicle optimal design. In 10th World Congress on Structural and Multidisciplinary Optimization, WCSMO-10. 2013.

120

GEB Archer, Andrea Saltelli, and IM Sobol. Sensitivity measures, anova-like techniques and the use of bootstrap. Journal of Statistical Computation and Simulation, 58(2):99–120, 1997.

121

Art B Owen. Variance components and generalized sobol'indices. SIAM/ASA Journal on Uncertainty Quantification, 1(1):19–41, 2013.

122

Emanuele Borgonovo. Measuring uncertainty importance: investigation and comparison of alternative approaches. Risk analysis, 26(5):1349–1361, 2006.

123

Emanuele Borgonovo. A new uncertainty importance measure. Reliability Engineering & System Safety, 92(6):771–784, 2007.

124

Elmar Plischke, Emanuele Borgonovo, and Curtis L Smith. Global sensitivity measures from given data. European Journal of Operational Research, 226(3):536–550, 2013.

125

Jiri Nossent, Pieter Elsen, and Willy Bauwens. Sobol’ sensitivity analysis of a complex environmental model. Environmental Modelling & Software, 26(12):1515–1525, 2011. URL: https://www.sciencedirect.com/science/article/pii/S1364815211001939, doi:https://doi.org/10.1016/j.envsoft.2011.08.010.

126

Jon Herman and Will Usher. Salib: an open-source python library for sensitivity analysis. Journal of Open Source Software, 2(9):97, 2017.

127

Hoshin V Gupta, Thorsten Wagener, and Yuqiong Liu. Reconciling theory with observations: elements of a diagnostic approach to model evaluation. Hydrological Processes: An International Journal, 22(18):3802–3813, 2008.

128

Keith Beven and Jim Freer. Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the glue methodology. Journal of hydrology, 249(1-4):11–29, 2001.

129

Cameron McPhail, HR Maier, JH Kwakkel, M Giuliani, A Castelletti, and S Westra. Robustness metrics: how are they calculated, when should they be used and why do they give different results? Earth's Future, 6(2):169–191, 2018.

130

John D Sterman. System dynamics modeling: tools for learning in a complex world. California management review, 43(4):8–25, 2001.

131

John M Anderies, Jean-Denis Mathias, and Marco A Janssen. Knowledge infrastructure and safe operating spaces in social–ecological systems. Proceedings of the National Academy of Sciences, 116(12):5277–5284, 2019.

132

Rachata Muneepeerakul and John M Anderies. The emergence and resilience of self-organized governance in coupled infrastructure systems. Proceedings of the National Academy of Sciences, 117(9):4617–4622, 2020.

133

Antonia Hadjimichael, Patrick M Reed, and Julianne D Quinn. Navigating deeply uncertain tradeoffs in harvested predator-prey systems. Complexity, 2020.

134

Julianne D Quinn, Patrick M Reed, and Klaus Keller. Direct policy search for robust multi-objective management of deeply uncertain socio-ecological tipping points. Environmental modelling & software, 92:125–141, 2017.

135

S. R. Carpenter, D. Ludwig, and W. A. Brock. Management of Eutrophication for Lakes Subject to Potentially Irreversible Change. Ecological Applications, 9(3):751–771, August 1999. URL: http://onlinelibrary.wiley.com/doi/10.1890/1051-0761(1999)009[0751:MOEFLS]2.0.CO;2/abstract, doi:10.1890/1051-0761(1999)009[0751:MOEFLS]2.0.CO;2.

136

Julianne Quinn. Julianneq/Lake_problem_dps. December 2017. original-date: 2017-02-06T18:33:54Z. URL: https://github.com/julianneq/Lake_Problem_DPS (visited on 2021-06-14).

137

David Hadka. Project-Platypus/Rhodium. 2017. original-date: 2015-10-29T18:08:43Z. URL: https://github.com/Project-Platypus/Rhodium (visited on 2021-06-14).

138

Stephen R. Carpenter, William A. Brock, Carl Folke, Egbert H. van Nes, and Marten Scheffer. Allowing variance may enlarge the safe operating space for exploited ecosystems. Proceedings of the National Academy of Sciences, 112(46):14384–14389, November 2015. URL: http://www.pnas.org/content/112/46/14384 (visited on 2017-08-18), doi:10.1073/pnas.1511804112.

139

Mustafa Hekimoğlu and Yaman Barlas. Sensitivity analysis for models with multiple behavior modes: a method based on behavior pattern measures. System Dynamics Review, 32(3-4):332–362, 2016.

140

Patrick Steinmann, Willem L Auping, and Jan H Kwakkel. Behavior-based scenario discovery using time series clustering. Technological Forecasting and Social Change, 156:120052, 2020.

141

Steven C Bankes, Robert J Lempert, and Steven W Popper. Computer-assisted reasoning. Computing in Science & Engineering, 3(2):71–77, 2001.

142

Robert J. Lempert, Steven W. Popper, and Steven C. Bankes. Shaping the Next One Hundred Years. RAND Corporation, 2003. URL: https://www.rand.org/pubs/monograph_reports/MR1626.html (visited on 2017-09-14).

143

Jonathan R Lamontagne, Patrick M Reed, Robert Link, Katherine V Calvin, Leon E Clarke, and James A Edmonds. Large ensemble analytic framework for consequence-driven discovery of climate change scenarios. Earth's Future, 6(3):488–504, 2018.

144

Brian C O’Neill, Elmar Kriegler, Keywan Riahi, Kristie L Ebi, Stephane Hallegatte, Timothy R Carter, Ritu Mathur, and Detlef P van Vuuren. A new scenario framework for climate change research: the concept of shared socioeconomic pathways. Climatic change, 122(3):387–400, 2014.

145

Warren E Walker, Marjolijn Haasnoot, and Jan H Kwakkel. Adapt or perish: a review of planning approaches for adaptation under deep uncertainty. Sustainability, 5(3):955–979, 2013.

146

Suraje Dessai, Mike Hulme, Robert Lempert, and Roger Pielke Jr. Climate prediction: a limit to adaptation. Adapting to climate change: thresholds, values, governance, 64:78, 2009.

147

Jonathan D Herman, Patrick M Reed, Harrison B Zeff, and Gregory W Characklis. How should robustness be defined for water systems planning under change? Journal of Water Resources Planning and Management, 141(10):04015012, 2015.

148

Robert J Lempert. Robust decision making (rdm). In Decision making under deep uncertainty, pages 23–51. Springer, Cham, 2019.

149

Jan H Kwakkel and Marjolijn Haasnoot. Supporting dmdu: a taxonomy of approaches and tools. In Decision Making under Deep Uncertainty, pages 355–374. Springer, Cham, 2019.

150

BC Trindade, PM Reed, and GW Characklis. Deeply uncertain pathways: integrated multi-city regional water supply infrastructure investment and portfolio management. Advances in Water Resources, 134:103442, 2019.

151

Julianne D Quinn, Patrick M Reed, Matteo Giuliani, Andrea Castelletti, Jared W Oyler, and Robert E Nicholas. Exploring how changing monsoonal dynamics and human pressures challenge multireservoir management for flood protection, hydropower production, and agricultural water supply. Water Resources Research, 54(7):4638–4662, 2018.

152

DF Gold, PM Reed, BC Trindade, and GW Characklis. Identifying actionable compromises: navigating multi-city robustness conflicts to discover cooperative safe operating spaces for regional water supply portfolios. Water Resources Research, 55(11):9024–9050, 2019.

153

JR Lamontagne, PM Reed, G Marangoni, K Keller, and GG Garner. Robust abatement pathways to tolerable climate futures require immediate global action. Nature Climate Change, 9(4):290–294, 2019.

154

Antonia Hadjimichael, Julianne Quinn, Erin Wilson, Patrick Reed, Leon Basdekas, David Yates, and Michelle Garrison. Defining Robustness, Vulnerabilities, and Consequential Scenarios for Diverse Stakeholder Interests in Institutionally Complex River Basins. Earth's Future, 8(7):e2020EF001503, 2020. URL: https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/2020EF001503 (visited on 2020-07-13), doi:10.1029/2020EF001503.

155

Harris Drucker and Corinna Cortes. Boosting decision trees. Advances in neural information processing systems, pages 479–485, 1996.

156

Kevin P Murphy. Machine learning: a probabilistic perspective. MIT press, 2012.

157

Kelsey L. Ruckert, Gary Shaffer, David Pollard, Yawen Guan, Tony E. Wong, Chris E. Forest, and Klaus Keller. Assessing the Impact of Retreat Mechanisms in a Simple Antarctic Ice Sheet Model Using Bayesian Calibration. PLOS ONE, 12(1):e0170052, January 2017. doi:10.1371/journal.pone.0170052.

158

B. Efron and R. Tibshirani. Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy. Statistical Science, 1(1):54–75, February 1986. doi:10.1214/ss/1177013815.

159

Ryan L. Sriver, Robert J. Lempert, Per Wikman-Svahn, and Klaus Keller. Characterizing uncertain sea-level rise projections to support investment decisions. PLOS ONE, 13(2):e0190641, February 2018. URL: http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0190641 (visited on 2021-06-09), doi:10.1371/journal.pone.0190641.

160

Kelsey L. Ruckert, Yawen Guan, Alexander M. R. Bakker, Chris E. Forest, and Klaus Keller. The effects of time-varying observation errors on semi-empirical sea-level projections. Climatic Change, 140(3):349–360, February 2017. URL: https://doi.org/10.1007/s10584-016-1858-z (visited on 2021-06-09), doi:10.1007/s10584-016-1858-z.

161

Neil R Edwards, David Cameron, and Jonathan Rougier. Precalibrating an intermediate complexity climate model. Clim. Dyn., 37(7-8):1469–1482, 2011. URL: http://dx.doi.org/10.1007/s00382-010-0921-0, doi:10.1007/s00382-010-0921-0.

162

Alexis Boukouvalas, Pete Sykes, Dan Cornford, and Hugo Maruri-Aguilar. Bayesian Precalibration of a Large Stochastic Microsimulation Model. IEEE Transactions on Intelligent Transportation Systems, 15(3):1337–1347, June 2014. doi:10.1109/TITS.2014.2304394.

163

David Makowski, Daniel Wallach, and Marie Tremblay. Using a Bayesian approach to parameter estimation; comparison of the GLUE and MCMC methods. Agronomie, 22(2):191–203, 2002. Publisher: EDP Sciences.

164

Mahyar Shafii, Bryan Tolson, and Loren Shawn Matott. Uncertainty-based multi-criteria calibration of rainfall-runoff models: a comparative study. Stochastic Environmental Research and Risk Assessment, 28(6):1493–1510, August 2014. doi:10.1007/s00477-014-0855-x.

165

Keith Beven and Jim Freer. Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology. Journal of hydrology, 249(1-4):11–29, 2001. Publisher: Elsevier.

166

Jasper A. Vrugt and Keith J. Beven. Embracing equifinality with efficiency: Limits of Acceptability sampling using the DREAM(LOA) algorithm. Journal of Hydrology, 559:954–971, April 2018. doi:10.1016/j.jhydrol.2018.02.026.

167

Jery R. Stedinger, Richard M. Vogel, Seung Uk Lee, and Rebecca Batchelder. Appraisal of the generalized likelihood uncertainty estimation (GLUE) method. Water Resources Research, 2008. doi:10.1029/2008WR006822.

168

Christian Robert and George Casella. Monte Carlo Statistical Methods. Springer Science & Business Media, March 2013. ISBN 978-1-4757-3071-5.

169

Christian P. Robert. The Metropolis–Hastings Algorithm. In Wiley StatsRef: Statistics Reference Online, pages 1–15. American Cancer Society, 2015. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat07834 (visited on 2021-06-14), doi:10.1002/9781118445112.stat07834.

170

James M. Flegal, Murali Haran, and Galin L. Jones. Markov Chain Monte Carlo: Can We Trust the Third Significant Figure? Statistical Science, 23(2):250–260, May 2008. Publisher: Institute of Mathematical Statistics. URL: https://projecteuclid.org/journals/statistical-science/volume-23/issue-2/Markov-Chain-Monte-Carlo--Can-We-Trust-the-Third/10.1214/08-STS257.full (visited on 2021-06-14), doi:10.1214/08-STS257.

171

Carla Currin, Toby Mitchell, Max Morris, and Don Ylvisaker. Bayesian Prediction of Deterministic Functions, with Applications to the Design and Analysis of Computer Experiments. Journal of the American Statistical Association, 86(416):953–963, December 1991. Publisher: Taylor & Francis _eprint: https://www.tandfonline.com/doi/pdf/10.1080/01621459.1991.10475138. URL: https://www.tandfonline.com/doi/abs/10.1080/01621459.1991.10475138 (visited on 2021-06-14), doi:10.1080/01621459.1991.10475138.

172

Jerome Sacks, William J. Welch, Toby J. Mitchell, and Henry P. Wynn. Design and Analysis of Computer Experiments. Statistical Science, 4(4):409–423, 1989. Publisher: Institute of Mathematical Statistics. URL: https://www.jstor.org/stable/2245858 (visited on 2021-06-14).

173

Roger G. Ghanem and Pol D. Spanos. Spectral Stochastic Finite‐Element Formulation for Reliability Analysis. Journal of Engineering Mechanics, 117(10):2351–2372, October 1991. Publisher: American Society of Civil Engineers. URL: https://ascelibrary.org/doi/abs/10.1061/%28ASCE%290733-9399%281991%29117%3A10%282351%29 (visited on 2021-06-14), doi:10.1061/(ASCE)0733-9399(1991)117:10(2351).

174

Dongbin Xiu and George Em Karniadakis. The Wiener–Askey Polynomial Chaos for Stochastic Differential Equations. SIAM Journal on Scientific Computing, 24(2):619–644, January 2002. Publisher: Society for Industrial and Applied Mathematics. URL: https://epubs.siam.org/doi/abs/10.1137/S1064827501387826 (visited on 2021-06-14), doi:10.1137/S1064827501387826.

175

Angelo Ciccazzo, Gianni Di Pillo, and Vittorio Latorre. A SVM Surrogate Model-Based Method for Parametric Yield Optimization. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 35(7):1224–1228, July 2016. Conference Name: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems. doi:10.1109/TCAD.2015.2501307.

176

W. Andrew Pruett and Robert L. Hester. The Creation of Surrogate Models for Fast Estimation of Complex Model Outcomes. PLOS ONE, 11(6):e0156574, June 2016. Publisher: Public Library of Science. URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0156574 (visited on 2021-06-14), doi:10.1371/journal.pone.0156574.

177

John Eason and Selen Cremaschi. Adaptive sequential sampling for surrogate model generation with artificial neural networks. Computers & Chemical Engineering, 68:220–232, September 2014. URL: https://www.sciencedirect.com/science/article/pii/S0098135414001719 (visited on 2021-06-14), doi:10.1016/j.compchemeng.2014.05.021.

178

Dirk Gorissen, Luciano De Tommasi, Karel Crombecq, and Tom Dhaene. Sequential modeling of a low noise amplifier with neural networks and active learning. Neural Computing and Applications, 18(5):485–494, June 2009. URL: https://doi.org/10.1007/s00521-008-0223-1 (visited on 2021-06-14), doi:10.1007/s00521-008-0223-1.

179

Jenný Brynjarsdóttir and Anthony O'Hagan. Learning about physical parameters: the importance of model discrepancy. Inverse Problems, 30(11):114007, October 2014. Publisher: IOP Publishing. URL: https://doi.org/10.1088/0266-5611/30/11/114007 (visited on 2021-06-14), doi:10.1088/0266-5611/30/11/114007.

180

Michael Betancourt. A conceptual introduction to Hamiltonian Monte Carlo. arXiv preprint arXiv:1701.02434, 2017.

181

Radford M. Neal. MCMC using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2, 2011.

182

Matti Vihola. Robust adaptive Metropolis algorithm with coerced acceptance rate. Statistics and Computing, 22(5):997–1008, September 2012. URL: https://doi.org/10.1007/s11222-011-9269-5 (visited on 2021-06-14), doi:10.1007/s11222-011-9269-5.

183

Perry de Valpine, Daniel Turek, Christopher J. Paciorek, Clifford Anderson-Bergman, Duncan Temple Lang, and Rastislav Bodik. Programming With Models: Writing Statistical Algorithms for General Model Structures With NIMBLE. Journal of Computational and Graphical Statistics, 26(2):403–413, April 2017. Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10618600.2016.1172487. URL: https://doi.org/10.1080/10618600.2016.1172487 (visited on 2021-06-14), doi:10.1080/10618600.2016.1172487.

184

NIMBLE Development Team. NIMBLE: MCMC, Particle Filtering, and Programmable Hierarchical Modeling. May 2021. URL: https://zenodo.org/record/4829693 (visited on 2021-06-14), doi:10.5281/zenodo.4829693.

185

Stan Development Team. Stan Modeling Language Users Guide and Reference Manual. 2021. URL: https://mc-stan.org/docs/2_27/stan-users-guide/index.html (visited on 2021-06-14).

186

John Salvatier, Thomas V. Wiecki, and Christopher Fonnesbeck. Probabilistic programming in Python using PyMC3. PeerJ Computer Science, 2:e55, April 2016. Publisher: PeerJ Inc. URL: https://peerj.com/articles/cs-55 (visited on 2021-06-14), doi:10.7717/peerj-cs.55.

187

Hong Ge, Kai Xu, and Zoubin Ghahramani. Turing: A Language for Flexible Probabilistic Inference. In International Conference on Artificial Intelligence and Statistics, 1682–1690. PMLR, March 2018. ISSN: 2640-3498. URL: http://proceedings.mlr.press/v84/ge18b.html (visited on 2021-06-14).

188

Andrew Gelman and Donald B. Rubin. Inference from Iterative Simulation Using Multiple Sequences. Statistical Science, 7(4):457–472, November 1992. Publisher: Institute of Mathematical Statistics. URL: https://projecteuclid.org/journals/statistical-science/volume-7/issue-4/Inference-from-Iterative-Simulation-Using-Multiple-Sequences/10.1214/ss/1177011136.full (visited on 2021-06-14), doi:10.1214/ss/1177011136.

189

Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. Sequential Monte Carlo samplers. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(3):411–436, 2006. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1467-9868.2006.00553.x. URL: http://rss.onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-9868.2006.00553.x (visited on 2021-06-14), doi:10.1111/j.1467-9868.2006.00553.x.

190

Arnaud Doucet, Simon Godsill, and Christophe Andrieu. On sequential Monte Carlo sampling methods for Bayesian filtering. Statistics and Computing, 10(3):197–208, July 2000. URL: https://doi.org/10.1023/A:1008935410038 (visited on 2021-06-14), doi:10.1023/A:1008935410038.

191

Jane Liu and Mike West. Combined Parameter and State Estimation in Simulation-Based Filtering. In Arnaud Doucet, Nando de Freitas, and Neil Gordon, editors, Sequential Monte Carlo Methods in Practice, Statistics for Engineering and Information Science, pages 197–223. Springer, New York, NY, 2001. URL: https://doi.org/10.1007/978-1-4757-3437-9_10 (visited on 2021-06-14), doi:10.1007/978-1-4757-3437-9_10.

192

Stefano Cabras, Maria Eugenia Castellanos Nueda, and Erlis Ruli. Approximate Bayesian Computation by Modelling Summary Statistics in a Quasi-likelihood Framework. Bayesian Analysis, 10(2):411–439, June 2015. Publisher: International Society for Bayesian Analysis. URL: https://projecteuclid.org/journals/bayesian-analysis/volume-10/issue-2/Approximate-Bayesian-Computation-by-Modelling-Summary-Statistics-in-a-Quasi/10.1214/14-BA921.full (visited on 2021-06-14), doi:10.1214/14-BA921.

193

Jarno Lintusaari, Michael U. Gutmann, Ritabrata Dutta, Samuel Kaski, and Jukka Corander. Fundamentals and Recent Developments in Approximate Bayesian Computation. Systematic Biology, 66(1):e66–e82, January 2017. URL: https://doi.org/10.1093/sysbio/syw077 (visited on 2021-06-14), doi:10.1093/sysbio/syw077.

194

Mikael Sunnåker, Alberto Giovanni Busetto, Elina Numminen, Jukka Corander, Matthieu Foll, and Christophe Dessimoz. Approximate Bayesian Computation. PLOS Computational Biology, 9(1):e1002803, January 2013. Publisher: Public Library of Science. URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1002803 (visited on 2021-06-14), doi:10.1371/journal.pcbi.1002803.

195

Edwin T. Jaynes. Probability theory: the logic of science. Washington University St. Louis, MO, 1996.

196

Andrew Gelman, Daniel Simpson, and Michael Betancourt. The Prior Can Often Only Be Understood in the Context of the Likelihood. Entropy, 19(10):555, October 2017. Number: 10 Publisher: Multidisciplinary Digital Publishing Institute. URL: https://www.mdpi.com/1099-4300/19/10/555 (visited on 2021-06-14), doi:10.3390/e19100555.

197

Christian Robert. The Bayesian choice: from decision-theoretic foundations to computational implementation. Springer Science & Business Media, 2007.

198

Andrew Gelman, Xiao-Li Meng, and Hal Stern. Posterior Predictive Assessment of Model Fitness via Realized Discrepancies. Statistica Sinica, 6(4):733–760, 1996. Publisher: Institute of Statistical Science, Academia Sinica. URL: https://www.jstor.org/stable/24306036 (visited on 2021-06-14).

199

Andrew Gelman, Aki Vehtari, Daniel Simpson, Charles C. Margossian, Bob Carpenter, Yuling Yao, Lauren Kennedy, Jonah Gabry, Paul-Christian Bürkner, and Martin Modrák. Bayesian Workflow. arXiv:2011.01808 [stat], November 2020. arXiv: 2011.01808. URL: http://arxiv.org/abs/2011.01808 (visited on 2021-06-14).

200

Andrew Gelman and Cosma Rohilla Shalizi. Philosophy and the practice of Bayesian statistics. British Journal of Mathematical and Statistical Psychology, 66(1):8–38, 2013. _eprint: https://bpspsychub.onlinelibrary.wiley.com/doi/pdf/10.1111/j.2044-8317.2011.02037.x. URL: https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1111/j.2044-8317.2011.02037.x (visited on 2021-06-14), doi:10.1111/j.2044-8317.2011.02037.x.

previous

Glossary