A Free-Market Energy Blog

Where Wind Studies Go Wrong: Cullen in AEJ (Part I)

By Kent Hawkins -- February 13, 2014

“The nature of the short-term operation of an electricity system is more like that of a machine than a market.”

A paper published by Joseph Cullen in the American Economic Journal: Economic Policy (November 2013), “Measuring the Environmental Benefits of Wind-Generated Electricity” [1] is important in two regards. First, using Texas data, it shows that even with notable emissions savings attributed to wind, the highly subsidized cost of wind is exceeded only by high estimates of the social costs of pollution.

Secondly and perhaps more importantly, his paper provides an opportunity to illustrate where wind-performance analyses fall short. This is the subject of this two-part post today and tomorrow, and is independent of the issue of carbon dioxide social benefits versus social costs.

Professor Cullen first determines how much electricity production of other generator types is offset by the presence of wind plants in the grid using a reduced form econometric model based on “…observed behavior and current market conditions.” The time frames for production are 15 minute intervals and two hour ahead forecasting by market participants. The market-oriented approach is exemplified by the following quote:

“When low marginal cost wind-generated electricity enters the grid, higher marginal cost fossil fuel generators will reduce their output.” (emphasis added)

This assumption that the suppliers of generation resources make these decisions is questionable because the nature of the short term operation of an electricity system is more like that of a machine than a market. The electricity system has been described as, “… the largest and most complex machine ever made.” Usually the production offset by wind is that which is most easily and quickly varied by the system operator regardless of fuel source or cost. Further, depending on the generation fleet online profile at the time, this can be other emissions free generation, such as hydro or even in some cases nuclear, which admittedly is not easily varied.

This is not to say there is not a wholesale electricity market, but that critical short term electricity system operational considerations are less dependent on decisions by the market participants and more dependent on the system operator matching supply and demand in real time using available online dispatchable generation resources.

So, although the operation of the electricity system is intended to provide the lowest wholesale cost from available choices, the overall reliability of the system is the paramount consideration. Other factors impacting operational decisions include: (1) government mandates to accept wind plant production whenever it occurs; (2) the profile of generation plants in use at the time (this typically varies considerably throughout the 24 hour day, and can be a limited subset, particularly at night); and (3) as already indicated, which of the available online generation plants can most easily be varied to meet the erratic output of wind plants. The Cullen analysis does not take into account the over-riding reliability consideration, as explained at the bottom of page 111.

“Finally, the econometric approach cannot comment on wind-induced reliability, or congestion issues that engineering approaches are geared to address.”

Having determined the production offsets, the paper suggests that “… it is straight forward to calculate emissions offsets by wind.” The emphasis is added because the “straight forward” claim is very questionable. So this review will put both sets of results to the test, and they will be found wanting.

Can opposition to wind plant implementations be encouraged by the results nonetheless? I suggest not without considerable caution for a number of reasons:

  • As introduced above, the paper uses an econometric modelling technique involving regression claiming to control for other influencing factors to ultimately lead to a determination of emissions savings from wind. This is a questionable approach in the absence of supporting proof of a clear cause and effect relationship between wind electricity production and claimed emissions and electricity production offsets. An example of such proof will be provided. Other corroboration used by the author will be shown to be not useful.
  • The analysis is based on questionable or incomplete data, especially emissions data, both in the approach to calculate the reported emissions savings, and in other reports used to support the “robustness” of this paper’s results.
  • The treatment of wind costs is incomplete.
  • As acknowledged by Cullen, the results of the sample used, in this case the Texas ERCOT electricity system, are not likely transferrable to other jurisdictions or different levels of wind penetration.
  • Other acknowledged shortcomings of the analysis are too important to support a complete understanding of the effect of introducing wind plants into electricity systems.
  • The use of questionable claims/opinions do not contribute to the overall dependability and hence usefulness of the study.

Part I of this series addresses the following in more detail:

  • Analysis approach
  • Robustness claims

Analysis Approach

A summary of Cullen’s approach is:

“Utilizing information on production decisions [by market participants] in 15-minute intervals on the Texas electricity grid, I estimate the response of each generator to exogenous changes in wind power. Realizing that wind power production is not completely random, I control for factors that may drive the incentives for electricity production, which may also be correlated with wind power production. The resulting quasi-experimental residual variation is then used to identify a substitution coefficient for each generator on the grid. Importantly, I show that failing to control for impact that wind has on the dynamic process of electricity production overestimates the production offsets. These production offsets then translate directly into emission offsets using generator emission rates.” (emphasis added)

This approach may work if all the factors in determining production and emissions offsets are accounted for, which I suggest is not an easy task, and has not been convincingly and completely done by anyone. Cullen does include a number of factors, but are important ones missed?

One way to test for this is to provide two finely-grained (5 minute intervals or less) time series plots for wind production versus both fossil fuel production offsets and emissions offsets claimed for wind. If these plots clearly show, in timing and relative size, that a very close relationship (or mirror image in the case of negative correlation), then there is merit to the approach. In the absence of a demonstration of this strong cause and effect relationship, the degree of mismatch is an indication of the impact of other factors not modelled.

In the case of the Kaffine, McBee and Lieskovsky study referred to by Cullen, because it employed the same approach “using different data sources”, when such time series plots were provided, there was little indication of a cause and effect relationship between wind production and the residual emissions claimed as a consequence of wind presence.

As a quick test of the possible causal links between wind and fossil fuel electricity production I looked to a chart I had previously prepared for Texas using EIA data.[2] It includes the period analyzed in Cullen’s paper, that is 2005-2007, and is shown in Figure 1. This chart focuses on the changes in electricity production on a yearly basis for fossil fuel plants, wind plants and net imports as an indicator of plausible cause and effect.


Figure 1 shows that in five of the eight years the annual change in wind production is in the same direction (both up or down) as that for fossil fuel, which suggests that wind is not the major factor in offsetting fossil fuel generation. However, for five of these years the change in net imports is in the opposite direction as changes in fossil fuel production. Note the relative sizes as well. This suggests that imports are more likely a factor than wind in offsetting fossil fuel electricity production.

Do not be confused by the use of the term net imports, in the context of a net exporter for the time period in Cullen’s analysis. Net imports are used here because imports are an additive generation source and are thus shown as positive in the chart. When net import changes are positive, year over year, this means there was an increase in imports relative to exports.

This test may not be conclusive, but it reinforces that further verification as described above would be appropriate.

In the absence of such corroboration, the results remain suspect because the sophistication of the approach is not a guarantee of accuracy. See more comment under the Robustness Claim (below) and Interstate Trade in Electricity sections in Part II.

There is interesting information bearing on the impact of cycling fossil fuel plants, whether for just “normal” frequency regulation (load variation only) or the additional requirement of balancing wind. [3] On a short term basis, such as minutes (or less), both have the same type of system impact. This is referred to in a KEMA [4] report, “Emissions Comparison for a 20 MW Flywheel-based Frequency Regulation Power Plant” in which a reference is made to another KEMA report on the impact of such cycling on the fuel consumption and hence emissions of fossil fuel plants. Attempts to get a copy of the referenced report have not been successful, but Dr. Kees lePair, in the Netherlands, claims to have had access to it and describes its findings in his paper “Wind turbines increase fossil fuel consumption & CO2 emission” as follows:

“Recently we received some information concerning a fuel flow recording of a coal fired generator during cycling. The generator running stationary for some time at 100% of its optimal capacity reduced its output to 80% and up again to 100%. The whole cycle took place in one hour. The total fuel consumption during that period was 1,2% more than it would have been had the machine continued running at 100%. It was suggested that for a CCGT this outcome should have been 1%.”

Given the combined effect of: (1) the more frequent cycling of fossil fuel plants than KEMA observed, whether normal frequency regulation operations or balancing wind plants, and (2) an overall reduced electricity output, and therefore less efficient operation due to periods of part loading in the fossil fuel plants as a result, it is difficult to escape the expectation of very little emissions savings at best over a normal steady operation of the fossil fuel plants meeting the demand alone. In other words, wind presence likely increases emissions overall.

In the first paragraph in the “Robustness” section of Cullen’s paper he expresses some concern about using averages of emission rates and acknowledges emissions effects missed, but counters this with the claim that as the number of fossil fuel plants supporting this cycling requirement increases, the changed output of each would be eased thus reducing the fuel and emissions impacts. This misses the important consideration that the frequency of such cycling is at least, if not more than, as important as the extent of the changes required. Further, grid topology could also restrict wind’s impact to subsets of fossil fuel plants.

Robustness Claim

In the “Robustness” section the paper looks to the Katzenstein and Apt analysis “Air Emissions Due to Wind and Solar Power” for confirmation of results. Most readers appear to miss the many caveats that Katzenstein and Apt correctly identify. For example:

“As discussed in the Supporting Information, the emission and heat rate data we obtained for the gas turbines did not cover all combinations of power and ramp rate. We therefore further constrain the model to operate only in regions of the power-ramp rate space for which we have data.” (emphasis and link added)


“…and caution that we have made no attempt to ensure the stability of an electrical grid. Grid dynamic response may somewhat change our results.” (It is suggested that “somewhat” is an understatement)

“Realistically, displaced generators will differ from the generators providing fill-in power and would produce different results.”

With respect to the last quote above, there is, or will be, a trend to introduce different types of generator plants (less expensive and more flexible) to balance wind as wind penetration increases. One example would be the increased use of gas turbine (aka GT or SCGT) plants versus combined cycle gas plants (CCGT). The gas turbine plants produce about 50% more emissions per MWh than combined cycle gas plants, the latter being designed to operate in steady state base load or intermediate electricity production roles. This is a major and often overlooked factor that contributes to reducing wind emissions offsets. See additional comments on this in the Questionable Data section in Part II.

Further, the impact of low operating cost of wind plants on wholesale markets, in part due to subsidization, makes more expensive operating cost, but more reliable, generation plant investments less attractive, thus putting at risk long range capacity needs. This is a serious matter for longer term electricity system reliability.

This is not to say that all wind costs are as indicated by wholesale market bidding, which is based only on the cost at the wind plant site, because there are substantial additional electricity systems costs which are incurred solely because of wind’s presence. This is discussed further in Part II.

From the Katzenstein and Apt paper Supporting Information addendum:

“Therefore, the results seen in Table 1 of the main paper, obtained from using the full time series of the 5 data sets (see Table S6), estimates only the emission reductions for the conditions that existed during the periods when the data were collected. Ideally, a significant number of high time-resolution independent power plant outputs would be used in our simulations. However we did not have access to such a data set, only to the 5 data sets described.” (emphasis added)

The wind data sets were for the relatively short periods of 15, 84, 240 and 370 hours. The fifth data set was for solar PV. This speaks to the limited data impact on conclusive results.

In summary, the Katzenstein and Apt paper does not contribute to robustness for the Cullen paper, due to the limitations of both.

Part I has shown that Cullen’s approach and robustness claims are questionable. Part II will look in more detail at the reliability of the available data, the interstate trade in electricity considerations, and discuss briefly some of Cullen’s acknowledged caveats and some other claims made.

[1] There is an earlier version which may be accessed at http://www.u.arizona.edu/~jcullen/Documents/measuringwind.pdf , but no attempt has been made to confirm that it is an identical document.

[2] EIA http://www.eia.gov/electricity/data/state/ Go to first spreadsheet entitled “Net Generation by State by Type of Producer by Energy Source (EIA-906, EIA-920, and EIA-923)” for Wind data, http://www.eia.gov/electricity/state/texas/ Table 10 for Net Interstate Trade and Table 4 for fossil fuel data.

[3] The short term (minutes or less) variation of load and wind in combination produces a greater range of variance than either individually, with a bias towards greater instances of higher variance. An analysis of the Bonneville Power Administration (BPA) in the US northwest electricity production by fuel source at 5 minute intervals shows this. Note that this assumes an arithmetic summation of all wind production, and grid restrictions might exacerbate this experience on a more localized basis.

[4] For information on KEMA, a Netherlands based international consulting firm, start with this and note that DNV acquired 74.3% of KEMA shares. The new company name is DNV KEMA.


  1. Where Wind Studies Go Wrong: Cullen in AEJ (Part I) » coalitionoffreedom.com  

    […] KHawkins “The nature of the short term operation of an electricity system is more like that of a […]


  2. Michael Goggin, AWEA  

    Kent, you wasted a lot of words speculating about questions that have already been answered. The Cullen piece is further confirmation of the results found in NREL’s comprehensive analysis that conclusively answered this question using real-world emissions data for every power plant in the Western U.S. The result? Wind energy reduces CO2 emissions by 1190 pounds/MWh on average. Cycling’s impact on the efficiency of fossil-fired power plants is a “negligible” 0.2%, or about 2 pounds/MWh. PJM’s wind integration study further confirmed those results. And now Cullen finds similar results for ERCOT, though he used a different methodology.

    Even the second fossil-funded Bentek report, which directly contradicts the first Bentek “report,” also found that wind produces comparable or larger emissions reductions in ERCOT and MISO. The second report went wrong when it applied that methodology to CAISO and the Pacific Northwest, as it failed to account for these factors:
    1. How wind energy is used to store water behind hydroelectric dams, where it is used to offset fossil generation later.
    2. How wind energy reduces emissions from fossil generation imports from neighboring regions.

    Both are major factors in California and the Pacific Northwest, but not in ERCOT and MISO. That type of analysis works in a state with almost no imports or hydro, like Texas.

    Finally, the Katzenstein and Apt piece is widely ridiculed in the field because it completely misrepresents how the power system operates. It incorrectly assumes a fossil generator is directly paired with a wind generator, while in the real world the grid operator only cares about accommodating the total variability of all sources of supply and demand. Wind is a very small contributor to that because most variability is canceled out by the far larger variability of load and the sizable schedule deviations of conventional power plants. Apt’s assumption is comparable to assuming that you need a battery or dedicated power plant attached to your house to accommodate every change in your electricity usage, while in reality those changes are obviously canceled out by changes in others’ usage.

    Not sure how many more “Parts” you plan to include in your piece, but until you can provide a substantive refutation of the NREL and PJM analyses, you can cut your losses and not waste ink on matters that have already been settled.

    Michael Goggin,
    American Wind Energy Association


  3. rbradley  

    Michael: Your critique is really the ‘argument from authority’ more than a point-by-point rebuttal. As shown in the climate-change debate, Internet review of published articles is tougher and more thorough than peer review for the journal.


  4. KHawkins  

    I have already responded to many of your comments on a number of occasions, including the comments sections of:


    With respect to the Bentek study you referred to, I pointed out the problem with that analysis in Part I of this series above.

    With respect to the Pacific Northwest electricity system, there is much more about the effect of wind there than the simple concept you suggest of wind energy being used to store water behind electricity dams. To assist you, some of the complications will be touched upon in Part II of this series.

    Also, contrary to your suggestion, the matter of interstate transactions in electricity is of consequence in Texas, notwithstanding its relative isolation. This is also covered in Part II.

    I seriously doubt your assertion the Katzenstein and Apt piece is “widely ridiculed in the field.” It has made a notable contribution to knowledge in this area, but is not conclusive. I won’t comment on your very questionable analogy.

    In the critical short term period of minutes or less in electricity systems, wind variability exacerbates the net load that other electricity systems elements have to respond to. Net load is the combined effect of load variation and wind variation, and is often used as a measure of the impact of wind on electricity systems. In real time, an important time frame for electricity systems operation, this combination is not the square root of the sum of the squares. It is a simple arithmetic one.

    It’s possible you are also being confused on this by looking at the fairly predictable rise in demand to daytime peaks, typically twice a day, which is met by scheduled intermediate generation plants.

    Finally, as I have told you repeatedly before, you have not proved your case or disproved mine.


  5. Kent Hawkins  

    Further to your critique of the Katzenstein and Apt paper, you claim that it “… assumes a fossil generator is directly paired with a wind generator.”

    Unfortunately, as you state it, this might be misunderstood to mean that you are talking about a single fossil fuel motor-generator set and a single wind turbine. However, it is possible that this is what you meant.

    Katzenstein and Apt are very clear on this. Their first paragraph that states, “We model a wind or solar plus gas system using measured 1-min time-resolved emissions and heat rate data from two types of natural gas generators, and power data from four wind plants and one solar plant.”

    Also you claim that “…while in the real world the grid operator only cares about accommodating the total variability of all sources of supply and demand.”

    In a general sense you are right. However, “in the real world”, the grid is usually comprised of subsets that have constraints in interconnections and balancing often occurs within these subsets. Katzenstein and Apt’s modelling is a good example of what this might entail.

    In any event, a system wide analysis is not necessary to illustrate the effect of wind turbines in an electricity system. A system wide approach may conceal more than it reveals.


Leave a Reply