In the same month that the Environment Department (DEFRA) announced significant funding cuts for bodies previously under the Business Resource Efficiency & Waste Programme (BREW) (see p 4 ), it has emerged that there are questions over how it can judge their relative effectiveness in cutting waste and pollution.
There is no common methodology to gauge the environmental benefits achieved by these bodies, which include the Waste & Resources Action Programme (WRAP) and Envirowise. And DEFRA admits it is "cautious about using [such] results alone" to compare their relative success.
BREW launched in 2005 to return £284 million from increases in landfill tax to business through programmes to improve resource efficiency (ENDS Report 358, p 49 ). That funding draws to an end in April. Although no targets were set for the programme, DEFRA did produce some metrics to measure its achievements and value for money: waste diverted from landfill; reduction of greenhouse gases; virgin material saved; reduced hazardous waste arisings; decreased water use; costs savings to business; and new business sales. It was left open to each delivery body to report against as many or as few of these as it wanted.
As ENDS went to press, DEFRA was about to issue a report on the achievements of the programme, but there are questions about the reliability of the data. ENDS understands there have been thorny debates on how to exactly attribute the benefits of work done by each delivery body. Equally contentious is how to account for the fact that benefits may persist over time.
In March, Parliamentary Under-Secretary of State for the environment Joan Ruddock said in response to a parliamentary question that figures had to "be viewed with caution since delivery bodies report according to a range of methodologies". This concern was raised to ENDS by several of the delivery bodies themselves in the run up to the budget cuts.
There are also concerns that DEFRA has made little attempt to standardise the assessment of claimed achievements, and as a result reporting practices vary wildly. For example, the Carbon Trust will not attribute savings achieved from its advice if regulation may have driven them too. It does not include potential savings achieved through seminars and training events either.
WRAP, in contrast, has created an "attribution methodology" for working out the benefits of its actions if other things could have had an impact, such as regulation. And it does explore whether it can attribute to itself benefits from seminars and workshops. When applying the methodology, WRAP asks firms whether they would have changed their resource use if WRAP had not been involved, and whether WRAP’s involvement made them make the change sooner than they would have done otherwise. The achievement claimed is then "down-rated", based on the company’s response.
If the firm says it made the change "a lot sooner" because of WRAP, it claims 40% of the resulting waste or carbon reductions. It claims 20% for "a little sooner". Confusingly, these claimed savings are adjusted further by questions about whether firms made changes "better" because of WRAP’s help.
When looking at the persistence of benefits, WRAP reports a five-year cumulative figure. A risk rating is assigned to this indicating the likelihood of the project continuing to deliver results. For example, a red rating is "very risky" and WRAP only claims a third of the predicted results for those five years in such cases.
The Carbon Trust’s persistence figures vary according to the type of technology or action. Impacts are estimated to persist anywhere between a few years to over 25 years.
Yet another methodology is seen with the National Industrial Symbiosis Programme (NISP), which conducts a phone interview as part of a third-party verification exercise. In it, firms are asked what benefits were achieved due to NISP. A firm unable to come up with concrete figures is presented with NISP estimates and asked whether they "sound reasonable".
Firms are then asked to give a rating from one to five on the degree to which NISP helped them achieve these benefits. But unlike WRAP, NISP does not adjust the figures based on what they say. The persistence of the benefits is also based on questions in phone interviews. In the absence of information, NISP assumes a decline over five years to zero.
Envirowise declined to outline the exact approach it uses to work out the savings achieved by its work. It would only say that its performance figures are based on the amount of help businesses feel they received from the body.
In a statement to ENDS, DEFRA admitted that while there is general guidance now in place on attribution and persistence, it is still aware of "inconsistencies which need to be addressed". The different approaches to attribution "could serve to overstate the results achieved", it added. It is "cautious about using the results alone to compare the relative success of different delivery bodies".
DEFRA added that guidance on attribution and persistence needs to be developed further and that it is "working with delivery bodies to achieve this and implement the agreed approach".