From the journal Environmental Science: Atmospheres Peer review history

14-Jul-2023

Dear Dr Simon:

Manuscript ID: EA-ART-06-2023-000092
TITLE: Evaluating reduced-form modeling tools for simulating ozone and PM2.5 monetized health impacts

Thank you for your submission to Environmental Science: Atmospheres, published by the Royal Society of Chemistry. I sent your manuscript to reviewers and I have now received their reports which are copied below.

After careful evaluation of your manuscript and the reviewers’ reports, I will be pleased to accept your manuscript for publication after revisions.

Please revise your manuscript to fully address the reviewers’ comments. When you submit your revised manuscript please include a point by point response to the reviewers’ comments and highlight the changes you have made. Full details of the files you need to submit are listed at the end of this email.

Please submit your revised manuscript as soon as possible using this link :

*** PLEASE NOTE: This is a two-step process. After clicking on the link, you will be directed to a webpage to confirm. ***

https://mc.manuscriptcentral.com/esatmos?link_removed

(This link goes straight to your account, without the need to log in to the system. For your account security you should not share this link with others.)

Alternatively, you can login to your account (https://mc.manuscriptcentral.com/esatmos) where you will need your case-sensitive USER ID and password.

You should submit your revised manuscript as soon as possible; please note you will receive a series of automatic reminders. If your revisions will take a significant length of time, please contact me. If I do not hear from you, I may withdraw your manuscript from consideration and you will have to resubmit. Any resubmission will receive a new submission date.

The Royal Society of Chemistry requires all submitting authors to provide their ORCID iD when they submit a revised manuscript. This is quick and easy to do as part of the revised manuscript submission process. We will publish this information with the article, and you may choose to have your ORCID record updated automatically with details of the publication.

Please also encourage your co-authors to sign up for their own ORCID account and associate it with their account on our manuscript submission system. For further information see: https://www.rsc.org/journals-books-databases/journal-authors-reviewers/processes-policies/#attribution-id

Environmental Science: Atmospheres strongly encourages authors of research articles to include an ‘Author contributions’ section in their manuscript, for publication in the final article. This should appear immediately above the ‘Conflict of interest’ and ‘Acknowledgement’ sections. I strongly recommend you use CRediT (the Contributor Roles Taxonomy, https://credit.niso.org/) for standardised contribution descriptions. All authors should have agreed to their individual contributions ahead of submission and these should accurately reflect contributions to the work. Please refer to our general author guidelines https://www.rsc.org/journals-books-databases/author-and-reviewer-hub/authors-information/responsibilities/ for more information.

I look forward to receiving your revised manuscript.

Yours sincerely,
Dr. Stephen Klippenstein
Associate Editor, Environmental Science: Atmospheres

************

Reviewer 1

This article provides an important contribution by introducing two reduced-complexity modeling (RCM) tools and evaluating them against full-form models and other RCMs. RCMs are increasingly being used for a wide variety of applications, and the introduction of new tools and evaluations across models are highly valuable to a variety of users interested in the responses of air pollution and health to emissions changes. The paper is well written with clear explanations. I just have a few substantive comments that should be addressed and noticed one typo. I look forward to seeing this paper published and these tools made available to the public.

1. Not tracking the contribution of ammonia is a major shortcoming. Many models show ammonia to be especially potent in contributing to PM formation, yet it is often neglected in air pollution control strategies. If it is not practical to add ammonia to the species assessed, then this paper and any users’ guide for this model should note its absence, especially since this model will be used by the U.S. EPA and others working in air quality management.
2. The “assumption that particulate sulfate can be scaled to SO2 emissions, and particulate nitrate can be scaled to NOx emissions” (line 198-199) is problematic, especially since ammonium nitrate formation is often limited by ammonia rather than nitrate. This deserves discussion. If possible, a model sensitivity case should be run to test if particulate nitrate really does scale with NOx emissions.
3. Why were different values of a statistical life used? (line 311)
4. The strong overpredictions of power plant impacts on ozone and sulfate (line 329) are concerning. Is that likely due to the concentrated nature of power plant plumes, in which ozone production efficiency may be lower due to the concentrated NOx? Are problems arising in part by ignoring the titration of ozone by NOx in the initial plume?

Minor edits:
Line 351: As --> A

Reviewer 2

Simon et al. present a comparison of air quality models of varying complexity. They quantify air quality changes and downstream economic benefits from hypothetical emissions scenarios developed to support air quality regulations. They apply 2 full-complexity chemical transport models and multiple reduced complexity models that are publicly available. In addition, they develop a new reduced complexity model, SABAQS. The work is ambitious and important, largely because of RCMs’ growing popularity in many applications.

SABAQS builds on existing RCMs by its grounding in source apportionment derived directly from CAMx. In addition, SABAQS is able to assess changes in O3 concentrations from changes in precursor emissions. It would be helpful to include a statement about what SABAQS is intended to be used for. The comparisons here for nationwide sector-specific regulations leave open the question of whether SABAQS is appropriate for individual states assessing air quality benefits of implementing the rules only in their state, for example, or for comparing benefits of two control scenarios on different industries at a local/regional scale? Do the authors have recommendations for which applications are appropriate? Will the model be available to run outside EPA?

A general comment is that the models are being compared based on quantities—i.e., change in air pollution, health impacts, or economic benefit—that cannot be measured. CAMx & CMAQ are assumed to be the ground truth because they include more complete parameterizations of physics & chemistry, but these models are often assessed against observations using the same factor of 2 metric used to assess RCMs in this work. I think it would be useful to address head-on that a) these values are important for developing environmental policies but b) there is no independent evaluation possible against observations. If possible, it would be useful to discuss the policy implications of a factor of two uncertainty in monetized benefit estimates.

Finally, I found it difficult to assess how much the emissions and meteorology years used in each scenario and model (Table 1) influenced the results. As I understand, meteorology and underlying emissions in each RCM are frozen to the model year for which each was developed. While it is clear the comparisons here are designed to test the models out of the box, I think more clarity is needed on the meteorology years for each model. Is there evidence that meteorological conditions or baseline chemistry conditions are more important for each RCM?

More specific
I suggest replacing the terms under/over-predicted with “biased high/low” for clarity throughout.

86: “Finally” is confusing

156: “morality”

155: I believe it is important to include the specific value of ß used and its units

158-161: it is unclear to me how this scaling was done—can you list a scaling factor used?

228: Is “source apportionment case” the same as the base case? It would be helpful to have a list of all CAMx OSAT/PSAT runs performed and their inputs.

225-237: I recommend the authors consider a more succinct way to present this information, possibly in a table

Table 1: I recommend noting which scenarios were developed for this and which previously.

298-303: The word “values” used throughout this paragraph is not specific—what does it refer to?

328-329: magnitude is higher (bias is negative)

359-362: my interpretation is that the difference in evaluation statistics for CPPP & EGU Sensitivity A are not very large, which makes sense given that emissions reductions are similar between the two.

Figures 1-5: the maps show concentrations over ocean areas. Were this included in the evaluation?

Table 3: are CMAQ & CAMx run in brute force scenarios for these comparisons? [I believe the answer is yes as described in lines 277-285, but it would help to have a clarifying statement]. In general, it would be helpful to have a list/table of all CTM and RCM model runs used in the analysis.

Table 4: it would be helpful to make a statement about uncertainty in these results. Since the only component of Eqn 1 that includes uncertainty is the health impacts effect coefficient that is consistent across models, I think it should be straightforward to make a general statement about the relative width of confidence intervals around these point estimates. Of course, concentrations changes are highly uncertain too, but I think the cross-model comparisons do a fair job of establishing uncertainties between models.

Table 3 & 4 & 5: These results may be better communicated in figures to go along with the tables.

514-515: true, but (from Table 5), some models/scenarios did not meet the factor of 2 standard for the O3 + PM2.5 benefits.

544-547: I wonder if this statement is more appropriate at the beginning of the manuscript

Referee: 1

This article provides an important contribution by introducing two reduced-complexity modeling (RCM) tools and evaluating them against full-form models and other RCMs. RCMs are increasingly being used for a wide variety of applications, and the introduction of new tools and evaluations across models are highly valuable to a variety of users interested in the responses of air pollution and health to emissions changes. The paper is well written with clear explanations. I just have a few substantive comments that should be addressed and noticed one typo. I look forward to seeing this paper published and these tools made available to the public.
Response: Thank you

Not tracking the contribution of ammonia is a major shortcoming. Many models show ammonia to be especially potent in contributing to PM formation, yet it is often neglected in air pollution control strategies. If it is not practical to add ammonia to the species assessed, then this paper and any users’ guide for this model should note its absence, especially since this model will be used by the U.S. EPA and others working in air quality management.
Response: The following clarification was added to the description of the SABAQS methodology: “This method does not include capabilities for tracking PM2.5 impacts from ammonia emissions.” In addition, we have added the following text to the results section “While we do not explicitly assess causes of SABAQS biases in this analysis, it is possible that some of the nitrate biases in San Joaquin Valley may result from the fact that SABAQS method does not include nonlinear chemistry impacts from ammonia emissions. Despite this limitation of scaling nitrate impacts linearly to NOx emissions, national nitrate biases are relatively small.”

The “assumption that particulate sulfate can be scaled to SO2 emissions, and particulate nitrate can be scaled to NOx emissions” (line 198-199) is problematic, especially since ammonium nitrate formation is often limited by ammonia rather than nitrate. This deserves discussion. If possible, a model sensitivity case should be run to test if particulate nitrate really does scale with NOx emissions.
Response: A reduced form model like SABAQS is, by definition, a simplified representation of the physical and chemical processes simulated in full photochemical models and therefore will not be able to fully account for nonlinear atmospheric processes. Table 2 and Figures 2, S-2, S-7, and S-12 show exactly the “sensitivity case” that the reviewer is asking for. The SABAQS results were derived using linear scaling and the comparisons against full CAMx model outputs show how that linear scaling relates to results using full non-linear chemistry that includes ammonia interactions. As shown in Table 2, across the emissions scenarios nitrate bias ranges from 1.5% to 45% compared to full CAMx despite the simplicity of the linear scaling approach. As noted above, we have added a sentence to the methods section clarifying that ammonia emissions impacts are not accounted for in SABAQS as well as statements in the results about ammonia interactions being a potential reason for the underestimated nitrate impacts in the San Juaquin Valley.

Why were different values of a statistical life used? (line 311)
Response: The different VSL values are due to different future years for the various emissions cases. We have now clarified this in the text.



The strong overpredictions of power plant impacts on ozone and sulfate (line 329) are concerning. Is that likely due to the concentrated nature of power plant plumes, in which ozone production efficiency may be lower due to the concentrated NOx? Are problems arising in part by ignoring the titration of ozone by NOx in the initial plume?
Response: We do not believe that the overpredictions described for the CPPP scenario are due to nonlinearities in the power plant plumes since it is unlikely that the 12km CAMx simulation would capture plume chemistry. Therefore, differences between SABAQS and CAMx are unlikely the effect of plume chemistry. In addition, Figures 1 and 4 show that the ozone and sulfate impacts are not limited to grid cells directly next to EGU sources. We also note that the overprediction was much more pronounced for the CPPP scenario than for the other two EGU sensitivities. While we don’t know precisely why the CPPP scenario had larger biases, the CPPP case represents a projection of the future year with high coal power plant utilization while Scenario A compares the higher coal projection to a lower coal projection and Scenario B compares 3 years of emissions change in a lower coal projected future. The higher coal projections are based on older emissions models and are not as representative of actual coal and gas utilization that has occurred in the last 10 years as the newer projections which account for more real-world retirements of coal-fired power plants. So, while we don’t know exactly why the SABAQS model has larger biases for the CPPP scenario, it does appear that SABAQS biases are much less pronounced for the two additional EGU sensitivity cases which are more representative of actual current-day power plant emissions.

Line 351: As --> A
Response: corrected, thank you.

Referee: 2

Simon et al. present a comparison of air quality models of varying complexity. They quantify air quality changes and downstream economic benefits from hypothetical emissions scenarios developed to support air quality regulations. They apply 2 full-complexity chemical transport models and multiple reduced complexity models that are publicly available. In addition, they develop a new reduced complexity model, SABAQS. The work is ambitious and important, largely because of RCMs’ growing popularity in many applications.
Response: Thank you

SABAQS builds on existing RCMs by its grounding in source apportionment derived directly from CAMx. In addition, SABAQS is able to assess changes in O3 concentrations from changes in precursor emissions. It would be helpful to include a statement about what SABAQS is intended to be used for. The comparisons here for nationwide sector-specific regulations leave open the question of whether SABAQS is appropriate for individual states assessing air quality benefits of implementing the rules only in their state, for example, or for comparing benefits of two control scenarios on different industries at a local/regional scale? Do the authors have recommendations for which applications are appropriate? Will the model be available to run outside EPA?
Response: In this paper we are trying to illustrate the SABAQS approach but not telling others what models to use. The specificity of SABAQS depends on the specificity of the inputs. If states want to apply SABAQS, or a similar method, they should develop input datasets that are appropriate to the spatial scale and sector-specificity of their policies. The conclusions section already discusses how SABAQS precision will depend on how well the spatial resolution of the source apportionment tags match the emissions changes from the policy scenario. We have added the following sentence to conclusions section “Anyone applying SABAQS to assess air quality benefits of local, regional, or national air quality policies needs to carefully design source apportionment tags to appropriately replicate the types of policies being assessed.”

While we are not providing public software tool at this time, the methodology is fully described in the methods section of this paper so anyone in the public would be able to replicate these calculations for their own assessments.

A general comment is that the models are being compared based on quantities—i.e., change in air pollution, health impacts, or economic benefit—that cannot be measured. CAMx & CMAQ are assumed to be the ground truth because they include more complete parameterizations of physics & chemistry, but these models are often assessed against observations using the same factor of 2 metric used to assess RCMs in this work. I think it would be useful to address head-on that a) these values are important for developing environmental policies but b) there is no independent evaluation possible against observations. If possible, it would be useful to discuss the policy implications of a factor of two uncertainty in monetized benefit estimates.
Response: The reviewer is correct that this a big challenge but photochemical models are the best point of comparison. We think that presenting this type of analysis is an important first step because most reduced form models have not undergone this type of evaluation in the literature.

We have added the following statement to the methods section: “We note that we use both SABAQS and the full-form models to estimate changes in air pollution from hypothetical emissions perturbations. Given the hypothetical nature of these emissions perturbations it is not possible to validate models against measured resulting air pollution changes. While we use CAMx and CMAQ, which include complex representations of physical and chemical atmospheric processes, to ground-truth SABAQS estimates, we acknowledge that CMAQ and CAMx themselves have uncertainties.”

We have removed the sentence of the conclusions that tied the factor of 2 performance to being appropriate for screening assessment of health impacts. We have changed text in the results section to clarify that the factor of 2 performance is similar to performance reported elsewhere in the literature rather than a bright-line benchmark for policy applications.

Finally, I found it difficult to assess how much the emissions and meteorology years used in each scenario and model (Table 1) influenced the results. As I understand, meteorology and underlying emissions in each RCM are frozen to the model year for which each was developed. While it is clear the comparisons here are designed to test the models out of the box, I think more clarity is needed on the meteorology years for each model. Is there evidence that meteorological conditions or baseline chemistry conditions are more important for each RCM?
Response: Meteorology years are provided in Table 1 and in the new Table S-1. The 2026fj CPPP sensitivity is meant to test these impacts. We have added caveat to the discussion of 2026fj CPPP sensitivity that those results provide assessment of different underlying modeling but don’t disentangle to what degree the changes are from differences in base year emissions inventories vs meteorology: “This analysis does not provide sufficient information to determine how much differing underlying meteorology versus emissions years between SABAQS and CAMx impacted these results.” We note that while swapping base year meteorology and emissions in 2026fj sensitivity do degrade SABAQS performance, the difference in performance from the case with matching base year meteorology and emissions is relatively small.

I suggest replacing the terms under/over-predicted with “biased high/low” for clarity throughout.
Response: Due to the fact that we are comparing deltas from reduced form models to full-form modeling and most of these deltas are decreases in pollutant concentrations & health impacts, we think “biased high/low” is ambiguous. In contrast, saying that the impact is underpredicted or overpredicted makes it clear whether the magnitude of the changes are too large or too small. We have gone back and made edits throughout in an attempt to make sure all instances of this language in the text are as clear as possible.

86: “Finally” is confusing
Response: We have deleted “Finally”

156: “morality”
Response: Fixed typo

155: I believe it is important to include the specific value of ß used and its units
Response: The beta coefficient is unitless but we have now added beta values for the ozone and PM2.5 concentration-response functions to the text.

158-161: it is unclear to me how this scaling was done—can you list a scaling factor used?
Response: The text has been modified to clarify this point: “Elemental carbon was used as a surrogate for all components of primary PM2.5 and used to represent the full amount of primary PM2.5 emissions. This was done to remove potential influence of secondarily formed organic aerosol on health damages reported for emission control scenarios modeled with photochemical grid models. Health impacts related to elemental carbon were linearly scaled proportionately to the total amount of primary PM2.5 emissions to elemental carbon emissions.” And “Elemental carbon was used as a surrogate for all components of primary PM2.5 and BPT values derived from elemental carbon were applied to total primary PM2.5 emissions.”

228: Is “source apportionment case” the same as the base case? It would be helpful to have a list of all CAMx OSAT/PSAT runs performed and their inputs.
Response: We have added table S-1 in the supporting information to describe all modeling cases. The text now points to Table S-1 to clarify.

225-237: I recommend the authors consider a more succinct way to present this information, possibly in a table
Response: This information has been moved into the new Table 2.

Table 1: I recommend noting which scenarios were developed for this and which previously.
Response: The information has been added in a new column to Table 1.

298-303: The word “values” used throughout this paragraph is not specific—what does it refer to?
Response: We have added “BPT” before “values” in this paragraph to clarify.

328-329: magnitude is higher (bias is negative)
Response: We have clarified this sentence by adding “magnitude of”: SABAQS NMB is higher for sulfate and ozone at -49.4% and -79.3% indicating that SABAQS shows a larger magnitude of impact from CPPP on these two pollutants than CAMx”.

359-362: my interpretation is that the difference in evaluation statistics for CPPP & EGU Sensitivity A are not very large, which makes sense given that emissions reductions are similar between the two.
Response: Table 2 shows that the SABAQS orverprediction of magnitude of sulfate impacts is much larger for CPPP case than for either sensitivity case A or sensitivity case B.

Figures 1-5: the maps show concentrations over ocean areas. Were this included in the evaluation?
Response: The following sentence has been added to the description of the statistics calculations: “All statistics are calculated based on grid cells covering contiguous US land locations (i.e. excluding grid cells outsides of the US or over purely water grid cells).”

Table 3: are CMAQ & CAMx run in brute force scenarios for these comparisons? [I believe the answer is yes as described in lines 277-285, but it would help to have a clarifying statement]. In general, it would be helpful to have a list/table of all CTM and RCM model runs used in the analysis.
Response: Text has been added to clarify that CMAQ and CAMx impacts are from brute force runs: “Table 4 provides a comparison of ozone and speciated PM2.5 health impacts derived from the SABAQS air quality surfaces in comparison to those derived from the full-form modeling tools using brute force emissions changes (Table S-1).” Table S-1 has also been added to the supporting information that lists all model runs used for full form modeling and reduced from modeling analysis.

Table 4: it would be helpful to make a statement about uncertainty in these results. Since the only component of Eqn 1 that includes uncertainty is the health impacts effect coefficient that is consistent across models, I think it should be straightforward to make a general statement about the relative width of confidence intervals around these point estimates. Of course, concentrations changes are highly uncertain too, but I think the cross-model comparisons do a fair job of establishing uncertainties between models.
Response: We have added the following text to the description of the benefits calculations in the methods section: “The estimated number and economic value of air pollution-attributable deaths and illnesses are subject to sources of uncertainty that we were unable to characterize quantitatively. Key sources of uncertainty include: the projected changes in the number and distribution of individuals exposed to air pollution in the future; the extent to which modeled air quality changes represent a reasonable surrogate for population exposure; the baseline rates of death and disease experienced by these populations; and, future changes in income, which in turn affect individual willingness to pay to reduce the risk of premature death.”

Table 3 & 4 & 5: These results may be better communicated in figures to go along with the tables.
Response: We have plotted results from Tables 4 and 5 in two new figures (Figure 6 and Figure 7). We have moved Tables 4 and 5 into the supplemental information.

514-515: true, but (from Table 5), some models/scenarios did not meet the factor of 2 standard for the O3 + PM2.5 benefits.
Response: This sentence has been deleted.

544-547: I wonder if this statement is more appropriate at the beginning of the manuscript
Response: This statement has been moved to the end of the introduction. The order of the conclusions section has also been updated for better flow.

26-Jul-2023

Dear Dr Simon:

Manuscript ID: EA-ART-06-2023-000092.R1
TITLE: Evaluating reduced-form modeling tools for simulating ozone and PM2.5 monetized health impacts

Thank you for submitting your revised manuscript to Environmental Science: Atmospheres. I am pleased to accept your manuscript for publication in its current form. I have copied any final comments from the reviewer(s) below.

You will shortly receive a separate email from us requesting you to submit a licence to publish for your article, so that we can proceed with the preparation and publication of your manuscript.

You can highlight your article and the work of your group on the back cover of Environmental Science: Atmospheres. If you are interested in this opportunity please contact the editorial office for more information.

Promote your research, accelerate its impact – find out more about our article promotion services here: https://rsc.li/promoteyourresearch.

We will publicise your paper on our Twitter account @EnvSciRSC – to aid our publicity of your work please fill out this form: https://form.jotform.com/211263048265047

How was your experience with us? Let us know your feedback by completing our short 5 minute survey: https://www.smartsurvey.co.uk/s/RSC-author-satisfaction-energyenvironment/

By publishing your article in Environmental Science: Atmospheres, you are supporting the Royal Society of Chemistry to help the chemical science community make the world a better place.

With best wishes,

Dr. Stephen Klippenstein
Associate Editor, Environmental Science: Atmospheres