Review of 'Growing interdisciplinary research capacity for sustainable development: Self-reported evaluation'

This review focuses on the preprint titled ‘Growing interdisciplinary research capacity for sustainable development: Self-reported evaluation’, authored by Fiona Culhane, Victoria Cheung, and Melanie Austen [1, 2].

(1) General comments

The objective of the authors’ manuscript is to quantitatively and/or qualitatively validate the effectiveness of the interdisciplinary collaborative approach, specifically the ‘learning-by-doing’ method, implemented in the Blue Communities (BC) project. This validation is accomplished through surveys and questionnaires administered to participants. The ultimate goal is to gain insights and draw lessons from the research findings, with the aim of enhancing and advancing capacity building through interdisciplinary collaboration in various other present and future cases.

However, the current manuscript needs improvement in several aspects from both internal and external validity perspectives. I hope that the authors will find my comments in this review helpful and incorporate them to enhance the next version of their work. If there are any comments that may be deemed unacceptable due to my misunderstanding or lack of comprehension, the authors are free to disregard them. Moreover, since some comments regarding data presentation and other aspects have already been provided in another review [3], I will refrain from repeating them here. In fact, I concur with many of the points raised in Ref. [3]. I recommend referring to it as well and utilising its suggestions to improve the manuscript.

 (2) Internal validity perspective

The authors’ decision to publicly share the collected data [2], along with the questionnaire used, is commendable in terms of promoting transparency and reproducibility. This practice is highly recommended as it enhances the value of open peer-review.

Regarding the adequacy of the questionnaire’s development, I will not address that issue in this review. However, it appears that the authors may not be fully utilising the obtained data. Each respondent answered multiple questions, encompassing individual-level, team-level, and organisation-level aspects, as well as questions regarding attributes and demographics like gender, country of affiliation, and research career stage. As a result, numerous cross-tabulations or regression analyses could be conducted among these question responses.

For instance, attributing a specific perception exhibited by early career researchers in response to a particular question solely to the brevity of their career can be misleading. In reality, it could be influenced by the research environment they are situated in or even their predisposition. Additionally, exploring the correlation between responses to individual-level questions and those related to team-level or organisation-level inquiries would be beneficial. By further harnessing the response data in this manner, it is worth considering methodologies that can yield more dependable and conclusive analytical outcomes.

Regression analysis is a useful tool that can provide valuable insights beyond simple cross-tabulation. However, it is essential to ensure that the regression model satisfies various conditions necessary for causal inference. While acknowledging the limitations, conducting regression analysis can yield helpful insights. Using the CSV data publicly available in Ref. [2], I present below an example of regression analysis that I have attempted and found suggestive. I hope you find it beneficial.

In this example, I will outline a method for capturing the factors or drivers of perception formation regarding the effectiveness of the BC project at the individual level using regression analysis. This is likely a significant concern for the authors as well. The authors have collected response data for various aspects (e.g. ‘Finding relevant literature’, ‘Critically reviewing the literature’, etc.) in terms of ‘personal current success or skill level’ (rated on a scale of 1–9, hereafter denoted as ‘x’) and the perception of how each aspect changed as a result of the BC project (rated on a 5-point scale from ‘much worse’ to ‘much better’, hereafter denoted as ‘y’). Therefore, a simple regression model can be formulated with ‘y’ as the response variable and ‘x’ along with other variables such as gender, country of affiliation, employment type, and research career stage as explanatory variables. If the variable ‘y’ is coded as ordered integers, such as 1–5 or –2 to +2, an ordered logit model would be suitable for the regression analysis.

Considering that the number of valid observations (respondents) is relatively small (around 50), it is not feasible to include a large number of explanatory variables. Thus, it becomes crucial to carefully select the most important explanatory variables. With regards to the research career variable, it would be preferable to treat it as individual dummy variables based on their respective values rather than as a single ordered variable. However, this approach may result in an excessive number of explanatory variables compared to the number of observations, making it impractical for this study. Therefore, it is important to strike a balance between the number of explanatory variables and the available sample size in order to ensure the reliability and validity of the regression analysis.

When actually conducting the regression analysis at the individual level, it becomes evident that, in the case of most y-variables, the corresponding x-variables or country of affiliation (either individually or in combination) demonstrate statistical significance, while other variables do not. This process enables a more reliable comprehension of the factors that influence changes in each y-variable. By meticulously interpreting the outcomes from both qualitative and empirical perspectives, deeper insights into the effectiveness of the BC project can be obtained. Consequently, this would further augment the value of the paper by providing a more comprehensive understanding of the project’s impact.

If conducting regression analysis poses challenges, I suggest conducting more detailed descriptive analyses as a minimum. To gain further insights, it would be beneficial to create histograms for different attributes such as gender and country of affiliation, and overlay them for comparison, for both the x and y variables defined earlier. Additionally, creating scatterplots with corresponding x and y pairs as axes would provide valuable information. To account for the discrete nature of the responses, consider introducing jittering. Varying the marker styles in the scatterplot based on attributes like gender and country of affiliation and overlaying them for comparison is also recommended. These visual representations will enhance the understanding of the data and facilitate comparisons across different attributes.

Performing these analyses, including overlaid histograms and scatterplots, will indeed offer a clearer understanding of the distributions of perceptions (x and y) across different attributes, as well as the associations between x and y within various attributes. These insights can yield valuable findings and contribute to a more comprehensive understanding of the data. Furthermore, conducting these analyses can serve as valuable preparation for the previously mentioned regression analysis. I highly encourage the authors to explore these visualisations if feasible, as they can provide valuable insights and enhance the overall analysis of the data.

(3) External validity perspective

Overall, I get the impression that the authors’ manuscript resembles more of a report on the organisational activities or activity records specifically of the BC project, rather than a research paper that contributes to the academic knowledge base or provides lessons for other (future) cases.

In the Discussion and Conclusion sections, there are instances where the authors attempt to extrapolate their analysis and interpretations to general theories related to career development, the academic environment, or interdisciplinary collaboration, in an effort to draw significant implications. However, there seem to be logical leaps in many parts of the manuscript. For example, generalising specific comments from certain individuals in the open-ended responses and using them to justify the overall evaluation of the BC project or attempting to derive universal conclusions lacks sufficient credibility.

It is important to ensure that the conclusions drawn in the manuscript are supported by robust evidence and rigorous analysis. Additionally, generalisations should be made cautiously, considering the limitations of the study and the specific context of the BC project. Providing a clear rationale and using appropriate references or theoretical frameworks can strengthen the credibility and reliability of the manuscript’s conclusions.

From that perspective, I suggest revisiting the Discussion and Conclusion sections and carefully examining the descriptions regarding the level of external validity. If the goal is to produce impactful content that is relevant to a broad audience, as advocated by UCL Open: Environment, it is desirable to uncover valuable insights beyond the specific BC project. By doing so, this paper will become even more valuable as a publication in the journal, as it will offer insights and lessons that can be applied to a wider range of contexts and contribute to the broader academic knowledge base.

(4) Miscellaneous

1. It is recommended to include the actual number of respondents alongside the response rate in Tables 1 and 2. This will provide readers with a better understanding of the sample size and the proportion of participants who responded to the survey.
2. In my view, the numbers indicated with ‘%’ next to the bar graphs in Figures 1–5 should be removed. These numbers can be misleading and confusing since they do not correspond to the length of the bars.
3. When stating phrases like ‘Most respondents felt...’, it is advisable to quantify the extent of ‘most’ using the format like ‘X out of Y respondents (Z%)’. This will provide a quantitative representation and enhance the clarity of the statement.
4. In Section 2.2 (Questionnaire), it is necessary to provide more specific and detailed explanations about the methods of questionnaire development, distribution, and collection, including the survey duration.
5. In the Supplementary Material and the file named ‘Survey_Questions.pdf’, each question item should be labelled with a number or symbol for individual identification, and it is strongly recommended to ensure a one-to-one correspondence between each response in the data file.
6. In the response data (CSV file), it seems that responses related to age groups and sectors of affiliation have been removed. If there is a deliberate reason for omitting these responses, it should be mentioned in the document to avoid any confusion or misinterpretation.
7. It would be advantageous to provide a more compelling rationale for the relative superiority of the ‘learning-by-doing’ approach compared to other approaches. While it is generally expected that any approach implementing in a project could yield positive outcomes, the key lies in demonstrating how the benefits derived from the ‘learning-by-doing’ approach outweigh those that would have been obtained through alternative approaches. This will strengthen the argument and provide a clearer understanding of why the ‘learning-by-doing’ approach is recommended.

(5) Overall impression

This manuscript has the potential to significantly enhance its academic and societal value by conducting a more comprehensive analysis considering both internal and external validity. To achieve this, it would be beneficial to provide meticulous descriptions of the approach employed to draw conclusions, ensuring transparency and clarity. Additionally, improving the methods of data visualisation, presentation, and delivery will contribute to a more effective communication of the research findings. By implementing these enhancements, the overall quality and impact of the paper can be greatly improved, leading to a more valuable contribution to the academic and societal discourse.

References

[1] Culhane, Fiona E. and Cheung, Victoria and Austen, Melanie (2022). Self-reported Change in Research Capacity Following Participation in an Interdisciplinary Research Project, 2017-2021. https://doi.org/10.14324/111.444/000189.v1

[2] Culhane, Fiona E. and Cheung, Victoria and Austen, Melanie (2022). Self-reported Change in Research Capacity Following Participation in an Interdisciplinary Research Project, 2017-2021. [Data Collection]. Colchester, Essex: UK Data Service. https://doi.org/10.5255/UKDA-SN-856101

[3] Washbourne, Carla-Leanne (2023). Review of ‘Growing interdisciplinary research capacity for sustainable development: Self-reported evaluation’. https://doi.org/10.14293/S2199-1006.1.SOR-SOCSCI.AHPMPZ.v1.RMKJUG

Keisuke Okamura

Washington D.C., USA

15^th July 2023