Got around to reading the primary source. I am surprised there isn't more skepticism from this group, but it may have to do with not having time to read the paper. A few reasons to be skeptical of the results (some are stronger than others):
1) First, required reading for anyone interpreting medical studies: http://jama.jamanetwork.com/article.aspx?.articleid=201218 TLDR; is that ~40% of _randomized_ (generally considered highest level of quality) studies and maybe more are proven wrong by subsequent ones resulting from publication bias. Consider that this would not have been published in a high profile journal if it showed no effect, hence pub bias. In other words, we're not seeing the similarly large analyses that show no effect because they're not published.
2) The adjusted analyses have confidence intervals on the odds ratio that overlap with 1.0 except for the highest dose group. There is evidence of a dose response relationship, but there is a problem in how they measure dose response. Cumulative dose biases towards people who have been on medications for a long time. People who are able to take medications (ie are older) for a long time are at greater risk for dementia.
3) It is problematic that they assume any effect on their endpoint is a class effect (ie true of all anticholinergics). The relationship with one medication may be driving this effect entirely. I would like to see, for example, oxybutynin pulled out of the analysis which is never prescribed for allergies and more often for neurological conditions, which could be an unmeasured confounder depending on how they modeled the comorbidities.
4) Controlled analyses are difficult to do well. Most of the time it controls for linear effects, meaning synergistic effects (which are probably involved in development of dementia) are not captured unless explicitly modeled. I do not see that they did this here.
5) I am further suspicious of the way they obtained fill data. As you are noting, many OTC anticholinergics are not "dispensed" in a way typical of prescription anti-cholinergics. If I bought Benadryl at CVS or even if I bought the pre-packaged bottle at the Group Health pharmacy, it's pretty clear to me that neither of these would be reflected in the data. This skews heavily toward prescription anti-cholinergics, which tend to be given for psych/neuro indications which themselves have an association (though likely not a causal one) with dementia. Again, unclear if they modeled all of these.
Overall, I share the sentiment that medical literature needs to be more open. I think it's unfortunate that current state makes it hard to confirm analyses like these with a great degree of certainty.
Completely agree with reproducibility of research. Would be great to further improve such algorithms using data collected from the public--selection issues and inaccuracies aside.
The algorithm is in the appendix of the article the tool links to.
Input fields are limited via javascript so may not have worked if you have it disabled. You're right we should do it server side as well.
Definitely. But with still mostly paper recording, limited data standards, and particularly onerous privacy requirements (justifiably so), it is not a foregone conclusion that the big data trend will impact medicine anytime soon.
I'm glad you made that point. We are very aware of how important user privacy is for this sensitive information. We want to begin the conversation with users while we're young so that we don't mess it up when we're big (a la facebook).
As a general principal, we will only reflect data back that has been fully anonymized. In fact, we don't collect personally identifiable information (that's why there are only 3 choices for age right now). We are building HIPAA-compliant software (even though it's not legally applicable to us yet). We have a team of advisors, including privacy experts, but honestly, we believe the best ethics panel will come from the users and are very interested in feedback in this respect.
> we believe the best ethics panel will come from the users
Not when it comes to HIPAA compliance. This isn't about finding the best ethical code of conduct for privacy (which can be tricky), but simply abiding by existing and well-defined rules; all users agreeing you're a paragon of virtue doesn't matter much if you break said law once it does apply to you.
Please understand I've no wish to rain on your parade; it's just that I know all too well dealing with HIPAA can cause some headaches, but that's part of the game when working in anything connected to healthcare in the US.
That's true, for HIPAA compliance, there is no negotiation and we will meet that standard.
But there inevitably will be some user concerns that fall outside of HIPAA compliance. So, we see HIPAA+HITECH as a minimum requirement. We don't expect it to be sufficient, however, and that's where user feedback, the "user ethics panel" if you will, comes in.
We were getting a lot of traffic in a short amount of time that day, so the site performance was lacking. You might have better luck trying again (maybe wait a few hours until this new HN traffic dies down).
I welcome the opinion of a legal expertise on your later point, but we're taking the appropriate measures based on the advice we've received.
1) First, required reading for anyone interpreting medical studies: http://jama.jamanetwork.com/article.aspx?.articleid=201218 TLDR; is that ~40% of _randomized_ (generally considered highest level of quality) studies and maybe more are proven wrong by subsequent ones resulting from publication bias. Consider that this would not have been published in a high profile journal if it showed no effect, hence pub bias. In other words, we're not seeing the similarly large analyses that show no effect because they're not published.
2) The adjusted analyses have confidence intervals on the odds ratio that overlap with 1.0 except for the highest dose group. There is evidence of a dose response relationship, but there is a problem in how they measure dose response. Cumulative dose biases towards people who have been on medications for a long time. People who are able to take medications (ie are older) for a long time are at greater risk for dementia.
3) It is problematic that they assume any effect on their endpoint is a class effect (ie true of all anticholinergics). The relationship with one medication may be driving this effect entirely. I would like to see, for example, oxybutynin pulled out of the analysis which is never prescribed for allergies and more often for neurological conditions, which could be an unmeasured confounder depending on how they modeled the comorbidities.
4) Controlled analyses are difficult to do well. Most of the time it controls for linear effects, meaning synergistic effects (which are probably involved in development of dementia) are not captured unless explicitly modeled. I do not see that they did this here.
5) I am further suspicious of the way they obtained fill data. As you are noting, many OTC anticholinergics are not "dispensed" in a way typical of prescription anti-cholinergics. If I bought Benadryl at CVS or even if I bought the pre-packaged bottle at the Group Health pharmacy, it's pretty clear to me that neither of these would be reflected in the data. This skews heavily toward prescription anti-cholinergics, which tend to be given for psych/neuro indications which themselves have an association (though likely not a causal one) with dementia. Again, unclear if they modeled all of these.
Overall, I share the sentiment that medical literature needs to be more open. I think it's unfortunate that current state makes it hard to confirm analyses like these with a great degree of certainty.