Risk Assessment Epilogue: Have a bad case of Anecdotes? Better call an Epidemiologist

In my series on risk assessment I ended by noting that policy-makers can’t only rely on risk assessments when it comes to establishing whether contaminants are actually having an effect on a population. The reason for this is that risk assessment is a probabilistic field.  It provides probabilities and likelihoods that adverse events will occur.  But populations aren’t static nor are they uniform or homogeneous. What that means is that risk assessment predictions can be easily trumped by factors out of the risk assessor’s control. As my blog title indicates, I am a Professional Chemist, but I am also a Professional Biologist and an Environmental Scientist. My undergraduate degree included studying population ecology where I was first required to study statistics. I also have graduate-level training in statistics. Even with all that education and training, I know that when you really want to determine whether your handful of anecdotes actually represents a trend to be worried about, then it is time to call in a real professional and that often means an epidemiologist. Many of you have heard of the field but for those who haven't, epidemiology is the study of the distribution and determinants of health-related states or events (including disease), and the application of this study to the control of diseases and other health problems (ref WHO).

Epidemiology is a fascinating field and one that takes a lot of education and experience to do well. Unfortunately, it is also a field rife with amateurs wandering around in a Dunning-Kruger haze. Who hasn’t read about Jenny McCarthy with her self-styled degree from the University of Google? Using only her Google skills she claimed to be able to tease globally relevant autism research from her personal experience with her son.  No one can deny Ms. McCarthy’s personal experiences but from a policy perspective it isn't enough. Anecdotal data can be a real problem because anecdotal data can be incredibly compelling. It can tug at the heartstrings and cause otherwise rational individuals and news organizations, (like say the Toronto Star) to go places where they best not go. While the Toronto Star/HPV story is still emerging, it is nowhere near the worst recent case of good intentions gone wrong in the field of homespun epidemiology. In my opinion that award has to go to the ongoing Fort Chipewyan cancer story.

For those of you not familiar with this story, a local family physician in Fort Chipewyan noticed what he felt was a “cluster” of unexpected cancers coming through his local practice. Based on this small batch of anecdotal data he raised an international alarm against oil sands development in the region (ref). The authorities, doing as would be expected, brought in a team of professional epidemiologists who did a comprehensive analysis and eventually established that no such cancer clusters existed. Now you would think this would represent an open-and-shut case right? Well you would be wrong. As is typically the case when health risks are drawn into the political world, the spinners started spinning. The anecdotal data was described as “an ongoing tragedy” and was broadcast around the globe almost immediately, because that is what extravagant claims do. Meanwhile, the comprehensive work of collecting, compiling and analyzing the data had to be completed. An initial report was prepared in 2009 and the final report completed in 2014 (presented in full at the Alberta Health Cancer page). This report completely vindicated the public health officials and debunked the built-up claims of the activists (as described in the CBC follow-up report :“Higher cancer rates not found in oil sands community, study shows”). That being said the original reports from the local doctor were broadcast in 2003 so for the better part of a decade the Alberta Government and the various oil sands producers in the area had to deal with public-relations firestorm after public relations firestorm based on a small sample of anecdotal data and a self-schooled “expert”.

As described, the problem faced in the field is that doing good science can take a long time. For those of you unfamiliar with what an epidemiological study entails here is a short summary. Initially health records (some still in paper form) have to be processed, input into information systems and anonymized (for ethical and privacy reasons). Now we are not talking about one or two records here but rather thousands of records and the records need to include all sorts of accompanying information needed for the analyses. The data must then be examined and confounding variables (like age, smoking habits etc...) have to be considered since any detailed statistical analyses have to be able to control for those confounding variables. What do I mean by that? Well we all know that smokers are more susceptible to certain types of cancers, so if your population has a lot of smokers you have to recognize that some cancers will be related to the smoking and not the oil sands. All this work takes time. In the end the waiting is typically worth it. In this case, as described by Dr. James Talbot, Alberta’s Chief Medical officer, the rates of cancers were comparable with the rest of the Province (ref). Of particular interest, the cancers reported as being an issue by the local physician included cervical cancer (which is primarily caused by a virus not environmental causes) and bile duct cancer (which also has strong non-environmental risk factors). Any reporter worth his/her salt could have looked up details of these two cancers and perhaps included those useful details in their reports, but instead it took years to thoroughly demonstrate that the "oil sands caused cancer clusters" were no such thing. Ultimately for the oil sands producers and the public health officials it was a pretty pyrrhic victory. Sure in the end the oil sands were proved to not be at fault, but like the Olympic gold medallist who gets his medal taken away from him a decade later, the gold medal had long since been used by the activists in their campaigns and the all-important financial rewards of increased fundraising for activist groups had been achieved.

You would think that a decade’s worth of empirical study with thousands of hours of work and millions of dollars of effort would buy you some peace of mind for a little while...but this is the oil sands so that is simply not going to happen. Mere months after the Alberta Health report came out a new study was produced, this time by a group of ecologists (One River web site with reports). The McLachlan report is a tremendous piece of work and for anyone interested in the field it is well worth the time to download and peruse. Unfortunately, it does not do what all the activists who cite it or journalists who wrote about it claim it does. The report is another grand example of the power of the anecdote. For outsiders this should not have been a surprise after all, the lead researcher was quoted as saying:

“I am not a physician. I am originally an ecologist by training. What do ecologists do? They look for links between organisms and their environments. What we have done is to link environmental wildlife and human health. Something that reductionist science would never be able to do.”

and

"The study was holistic in approach. It examined the linkages between the environment, wildlife, diet and human health. In that way it complements and resonates with Traditional Knowledge which is similarly holistic ."

I suppose I am a bit old school but whenever I hear a scientist using the term “reductionist science” in that manner while repeating the word “holistic” my ears perk up. In this case my concerns were apparently well deserved.  The report represents a fascinating blend of ethnocultural and ecological research but it is a survey report pure and simple. It purports itself to be an attempt to “characterize the impacts of upstream industrial activity associated with the Athabasca Oil Sands for wildlife, environmental and especially human health as it affects the MCFN and ACFN” but the report suffers from a severe lack of data with so few analytical results that the statistics and conclusions have little or no statistical power. The report provides a terrific snap-shot into community beliefs and illness records but does so without the benefit of context or addressing confounding variables. The data is expository in nature with broad chemical conclusions built on a mere handful of samples (for PAHs they relied on 3 beavers samples, 4 moose, 23 ducks and 8 muskrats). The report details PAH concentrations  from samples collected in areas downgradient of the oil sands developments but, as I described in a previous post, PAHs are naturally occurring in the area as a result of the presence of surficial oil sands (some of which are being mined) as well as yearly forest fire events. From my reading of the work, no effort was made to obtain background samples. Background samples would be samples collected from upstream of the oil sands developments, but within the area of the oil sands themselves. Background samples allow scientists to establish the natural concentrations of contaminants in the region. It is impossible to establish whether the samples presented in the report have concentrations that are significantly higher than the background concentrations if you have never collected a background sample. Moreover, what little data collected is not presented in useful form. The authors depend on means with error bars rather than providing raw data and admit that in the case of the moose PAHs the results were skewed by a single highly impacted animal. After all this, the concentration of PAHs that were considered “very high” by the authors were actually half the Health Canada safe dose. As I described in my previous post, given all the conservatism built into a Health Canada safe dose, if your highest value is half that safe dose then you are doing pretty well.

So why am I harping on this one study? Well it is the type of work that generates headlines “Oil sands pollution linked to higher cancer rates in Fort Chipewyan for first time: study” (Financial Post) and can be hyped by activist organizations: “'Alarming' New Study Finds Contaminants in Animals Downstream of Oilsands” (Desmog Canada). However, when viewed from a risk assessment and epidemiological perspective can be discounted as an initial survey that is in dire need of follow-up and a more comprehensive sampling plan. It represents exactly the type of study that the news and activists love and policy-makers have to fear. The study will drive public perception for years without sufficient data to support that drive.  From a risk communication perspective, it represents a gold mine for activists, with its headline making quotations from first nations members, but has no underlying substance to support those quotations. Most disappointingly, until a carefully designed study is carried out to either confirm or debunk the results, it will be the last word on the subject whenever anyone wants to make a headline or solicit funds.