Vaccine Effectiveness, Data, and the CDC

I bet you don't know how the CDC 'calculates' influenza vaccine effectiveness.* I also bet once you do, you'll be horrified. (Here's a moderately technical link which inspired my outrage.)

In general, vaccine effectiveness is this: Out of 100 people exposed to a disease, how many will get it?  If all of those 100 people have been fully vaccinated, now how many will get it?

For fairly obvious reasons, this is quite hard to measure.  For one, you don't need a hundred people; you need more like a few thousand people.  Also, how can you tell if they've gotten an illness or not?  Basically only by doing a culture/ titre/ etc. every week.  This rapidly becomes awkward, expensive, and difficult to accomplish (would you go be cultured every week for six months?  Me neither.)  The most captive populations - hospital nurses, nursing home patients, and so on - are, for various reasons, not a good sample population - age homogeneity and uncommonly high exposure levels play a role.

The easiest way to measure effectiveness is to compare populations before and after universal-vaccination campaigns.  But for a lot of things, that's impractical.

The flu vaccine changes every year, because viruses mutate like wildfire, so the vaccine has to keep up.  Also, an enormous amount must be produced before the fall, so there are a lot of educated guesses involved.

So what the CDC actually does is this: They grab 3000-some people who have already showed up at the doctor's office - outpatient only! - culture them for influenza, and then ask if they've gotten a vaccine or not.  While I understand the constraints, this is still the worst sample in ever.  For one, it's going to skew toward people with insurance, and it will almost certainly underestimate effectiveness - everyone who wasn't sick enough to go to the doctor is automatically excluded.  Then, there's the inclusion criteria: People presenting with a respiratory infection.  Everyone who showed up at the doctor with a cold was enrolled.  And then there's reliability.  Only one site included an immunization registry (my state has one; they generally capture 95% of flu vaccine admins, which is pretty good), so they're relying on self-reporting, which is not terribly accurate.

 So I don't know what they're actually measuring, but I assure you that it is not vaccine effectiveness.

* Unless I already told you this. My outrage was vocal.