A rigorous scientific process is essential to forming sound conclusions that can inform evidence-based decision-making. This process starts with defining a research question, assessing what level of information is needed and then critically assessing how that information should be obtained (see Table 2.1 and Hayes et al., 2019). Evidence can be obtained from a variety of sources, ranging from expert opinion, through ad-hoc data collection, then well-designed observational surveys, and finally to randomised controlled experiments. Well-designed experiments/surveys that are targeted to the research question are often more expensive than other options, and in certain circumstances (e.g. an inability to randomly allocate sample units to treatment/control groups), may be unavailable. The other sources of information, however, may be adequate depending on the research question and situation (see Leek and Peng, 2015). Table 2.1 for example provides a brief overview of a hierarchy of research questions and the types of data that are appropriate to answer them.
Table 2.1: Different types of research questions (adapted from Leek and Peng, 2015 and Hayes et al 2019)
| Research Type | Description | Example Question | Examples of adequate data sources | Complexity |
|---|---|---|---|---|
| Descriptive associations | Summaries of observed data | What is happening within our sample? | Expert judgement*, and all forms of controlled/uncontrolled trials, and observational studies with or without a representative sample | Simple |
| Exploratory | Identify trends and relationships within the sample1 | What correlates with reef die-back in the sample? |  |
|
| Inferential | Extending the patterns in the sample to the population from which the sample was taken | What is the status of species X in a marine protected area? | Expert judgement*, all forms of randomised controlled trials, and observational studies with a representative sample | |
| Predictive | Predict the values at unsampled locations based on sampled data | What assemblage is likely to be found in this location? | ||
| Causal | Identify the reason for a particular association | Are the management actions having an effect? | Expert judgement*, all forms of randomised controlled trials and observational studies with a representative sample so long as the effects of potentially confounding variables can be controlled. | Complex |
1There is no way to tell if the sample’s associations are the same as the population’s
*Expert judgement will likely be influenced by a variety of well known heuristics biases. Attempts should be made to control for these during any elicitation exercise.
Observational studies using data from well-designed surveys (e.g. surveys that ensure samples are representative of the population of interest) are able to answer all types of research questions, and are sometimes the only source of adequate information (Table 2.1). These research questions include those concerning status and trends of biological populations and ecological metrics.
Observational data are generated by scientists observing the current state of the system, whether it be an altered system (e.g. after the establishment of a reserve or an industry) or not (e.g. a baseline survey). The key attribute of an observational study is that there is no attempt to intentionally alter parts of the system for the sole purpose of quantifying effects. Rather, the data is gathered and analysed in a manner that provides information on what the system is like (its status), how it is evolving (its trends) and what may be responsible for these trends (its causative factors). As an example, if baseline/ground-truth surveys are conducted inside and outside previously established no-take marine park boundaries, then they would generate observational data. This chapter discusses the requirements for appropriate statistical design for this observational data.
Causal research questions (attributing observed changes to specific causes) are the most difficult questions to answer. In this case (ideally randomised) controlled experiments are typically recommended, but in this context there are usually limiting factors whose discussion is beyond the scope of this manual (see Hayes et al., 2019). Causal questions require special care and are usually more demanding in terms of the resources needed to answer them. Here we focus on (marine) observational surveys, and in particular the design of surveys. Whilst the topics discussed in this section are relevant to investigating causal relationships, other considerations associated with the analysis of observational data would also be required to be addressed before undertaking causal research (and we do not deal with these issues here). For more information on the evidence hierarchy, and a more thorough description of the different design types for marine ecology, see Hayes et al. (2019).
A key concern in this scientific process is ensuring that survey data are trustworthy and fit-for-purpose (i.e. can answer the research question). To this end, it is important that surveys and monitoring programs are designed and implemented in such a way that the resulting data are: (i) appropriate for the research question under consideration; (ii) representative of the population under investigation so that (for example) the sample mean is generalisable to the population mean; and (iii) information rich so that uncertainty around inferences is reduced as much as survey budgets will allow. We focus here on survey designs that will help ensure environmental monitoring programs deliver data with these characteristics.