Risk analysis and estimation…


I hadn’t planned on writing a followup on my risk post, but one comment compelled me to write some more. I hope it isn’t too preachy.


J writes:


My girlfriend is a doctor. She deals with an inordinate number of motorcyclists every day, from every walk of life. The actual level of risk appears to be very high from the evidence I’ve seen.


n the final statement, J is asserting that the risk is very high, and he’s also asserting that he’s basing his conclusion on evidence. There’s also an implicit assertion that his conclusion applies to me.


I believe that is an unsupported assertion, and I’d like to discuss why.


The branch of science that deals with risk analysis is know as epidemiology. To pull a definition from a state website:


“Epidemiology is the study of the occurrence and distribution of illnesses and injuries in human populations.”


If you read a research paper that says something like, “smoking increases your chance of lung cancer by 45%“, chances are that the work was done by an epidemiologist.


The reason that we do studies in the first place is that it’s well known that drawing conclusions from individual experience is often problematic. The whole point of the scientific method is to approach things in an objective and repeatable manner (which doesn’t always happen, but that’s another subject).


Just to pick a few of the factors that could invalidate J’s conclusion (which I’ll restate as “Eric shouldn’t ride motorcycles because my girlfriend the doctor see’s lots of injured motorcyclists“) (and I should note that I’m not asserting that any of these factors are valid or invalid, just listing what are possible sources of error):



  1. The doctor may be filtering based on preconception of motorcyclists being more risky

  2. The doctor may be giving a summary that is weighted towards more injuries than actually occur

  3. The hospital may get a higher percentage of motorcycle injuries than other hospitals in the area.

  4. The motorcyclists in the region may ride more miles than in other regions.

  5. The region around the hospital may have a higher percentage of motorcyclists compared to a larger population

  6. The average motorcyclist age in the region may be younger

  7. The region may have less effective motorcycle safety training programs

  8. The region may have motorcycle dealers who focus on selling more powerful motorcycles

  9. There may be motorcycle clubs who increase the incidence of risky behavior

  10. Law enforcement may not be enforcing against risky behavior.

Because of factors such as these, anecdotal data isn’t very useful to draw conclusions from (though it is useful to come up with good research topics). You need a real study that looks at motorcycle injuries, looks at the causes of those injuries, looks at the demographics of the situation, and then does some analysis to identify correlations between risk factors and injuries.


(Aside: Epidemilology is a requirement when you’re looking at low-level effects, such as environmental cancer rates, danger from EMF radiation, etc. If not, you can’t separate clustering due to random distribution (which is by definition not uniform) from a real effect))


So, once you’ve done that research, you should have a lot of information saying how much more likely motorcyclists are to be injured or killed based on a number of factors. And then you can apply those risk factors to a specific situation, and come up with an estimate on how risky an activity is compared to a different activity.


The literature isn’t great in this area, and more study is needed. Many motorcycle accidents come from the actions of other drivers, but I don’t know of a recent good study in that area. On single-vehicle accidents, we have the following:


This study states that on a per-mile basis, a motorcyclist is 3x as likely to be injured, and 16x as likely to be killed. But it also lists some risk factors (look to the study for all of them):


  • Helmet use among fatally injured motorcyclists below 50 percent
  • High blood alcohol levels are a major problem among motorcycle operators
  • Almost two thirds of the fatalities were associated with speeding as an operator contributing factor in the crash
  • Almost 60 percent of motorcyclist fatalities occur at night
  • Braking and steering maneuvers possibly contribute for almost 25 percent of the fatalities
  • Almost one third of the fatally injured operators did not have a proper license

  • I always wear a helmet. I don’t drink before I ride. I rarely ride at night. I’m well trained and understand how to properly brake and steer, and I practice a bunch (or I did when I rode more). I have a proper license.


    So I don’t have a lot of the risk factors that the accident-involved motorcyclist has, so it’s unlikely that the 3x and 16x factors apply to my risk as compared to the general population.


    Of course, my risk as a car driver is also lower because of good habits, so the relative factors could be the same, or could even be worse.


    So, what’s my point in all of this? Well, two things.


    The first is that many people make the mistake of assessing risk based upon the overall societal attitude towards a activity and/or anecdotal data (“I knew a guy who…“) rather than any factual basis. If you to to an emergency room with a leg injury, you will get a different response based on whether you say you were skiing or skydiving.


    My second point is that it takes a good study to tell you how you can reduce your risk, so being better informed really pays off.


    Comments (23)

    1. Lee says:

      The best way to evaluate risk of a certain vehicle over another is to look at the insurance costs associated with them.

      Actuarians spend years in college learning about risk assesments and how to evaluate them. Insurance companies spend millions of dollars to evaluate the risks of certain activities so that they can plan and price their premiums accordingly.

      If you apply for insurance, you can tell what kinds of things that the insurance companies consider to be risky by what questions they ask you.

      There are typically four different types of insurance for vehicles:

      1. Insurance for damage to your vehicle.

      2. Insurance for damage caused by your vehicle to other things.

      3. Insurance for damage to you.

      4. Insurance for damage caused by your vehicle to other people.

      In this case you would want to compare number 3 between a motorcycle policy and a policy on say a honda civic or something similar.

    2. Eric Gunnerson says:

      Lee,

      I thought about bringing up the actuarial angle, but decided not to. But since you did…

      Insurance companies are in the business of measuring risks accurately (or, accurately enough that they can make money), but my experience is that there are still societal factors at work. Some insurance companies won’t want to insure certain bikes for any price, because they don’t want that kind of business. One common reason that they don’t want that kind of business is because their actuarial models aren’t good enough in that area, and it’s not worth it to them to take risk that they don’t need to.

      But, I agree it is possible to do a comparison between two policies, though that will only show you the relative risk difference of the a large segment of the population (say, Male 40 year old motorcycle riders) rather than personal risk, which is likely to be different.

    3. As an aside, I have a question about risk analysis. Specifically with the kinds of statements like "smoking increases your chance of lung cancer by 45%." Lemme make up a few more neutral ones, as examples only: "5th graders who go to bed before 9pm on school nights score 10% higher on standardized tests", "men who have high cholesterol have a higher risk to get prostate cancer", "users of Microsoft systems experience 30% more daily crashes than VMS systems."

      Most of these seem reasonable on the surface, I suppose, but I can’t imagine how the studies that found these statistics excluded the human element.

      The "5th graders" stat may reflect the fact that parents who give their kids an earlier bedtime may take more interest in their kids’ studies and that results in better grades. From a personal perspective, most parents I know that let their kids roam free at night really don’t pay much attention to schoolwork and would bear this out. It has almost nothing to do with sleep.

      The "cholesterol" stat, may suggest a link between cholesterol and prostate cancer but may merely suggest that men who take better care of themselves (eat better, monitor their chemistry) may not engage in other activities which would tend to trigger prostate cancer.

      And finally (and most absurdly) the Windows and VMS stat may just reflect that the few VMS machines still running in the world aren’t being used with "risky" software, and doesn’t speak at all to the stability of Windows or VMS.

      So how do you weed out the "outside" factors in all of this? For making of things like actuary tables I can see where a behavior is an *indicator* of a risk, but nothing more than that. A behavior might be an indicator of another behavior which is actually the real risk.

    4. Lee says:

      "So how do you weed out the "outside" factors in all of this? "

      This is a difficult process and there are several extremely complicated statistical formulas used to determine the validity of correlations such as these.

      I just recently finished a Statistics course for my CS degree and there were two examples of this that the instructor used:

      1. It has been shown that cities that have fitness centers have a higher crime rate, is it thus safe to assume that fitness centers attract crime?

      The answer was that there was no correlation, that fitness centers usually appear in larger cities, and the larger the city the more change of having crime in general.

      2. A study in Florida was commisioned by the DOT to propose raising the speed limit. The study found that more accidents occured at speeds below 65 miles per hour and thus raising the speed limit to 70 on Interstates would have little to no affect on the number of accidents.

      The instructor said that a further study was commisioned and found that the reason there were more accidents at speeds below 65 wasn’t because of the speed, but because of the weather. When the weather was poor, people would drive slower, however there were also more accidents when the weather was poor so a false correlation was drawn.

      So even though complicated statistical formulas can be used to determine correlations (which are still only not 100% accurate), it is impossible to tell as a reader of such a statistic without familiarity of the factors included and excluded from the study.

      (As a side note there is also a large section of the text that was dedicated to ethical use of statistics, since it is so easy to manipulate a statistical analysis to show favor to either side of an issue).

    5. Radu Grigore says:

      Clinton: Those studies give information about correlation of two factors A and B, NOT about cause->effect relation. They do not say "going to bed early increases school grades", they simply say "going to bad early and getting high grades are correlated". In practice strong correlation is an indication of a cause-and-effect relation. It can be A->B, B->A, C->A / C->B or some other combination. If you understood them as claiming a cause and effect relation then you are simply misreading them.

    6. Radu Grigore says:

      Clinton: Those studies give information about correlation of two factors A and B, NOT about cause->effect relation. They do not say "going to bed early increases school grades", they simply say "going to bad early and getting high grades are correlated". In practice strong correlation is an indication of a cause-and-effect relation. It can be A->B, B->A, C->A / C->B or some other combination. If you understood them as claiming a cause and effect relation then you are simply misreading them.

    7. Lee says:

      "If you understood them as claiming a cause and effect relation then you are simply misreading them."

      I don’t think that he is misreading them, I think that most statistics written like that are meant to be misread.

      That goes back to what I mentioned about ethics in Statistics, it is easy to promote either side of an issue with the same statistical analysis, and the public eats it up.

      When they report a statistic like the above on television or in print, the large majority of people (Americans especially) just accept it as fact. It becomes part of the endless tome of "They say…" facts. (As in "They say that riding motorcycles is 10 times more likely to kill you than jumping off a bridge…") Most people can’t remember where they heard this fact or who did the study, but it doesn’t matter because it is part of the "They say…" portion of our brain that can only be modified by another "They say…" statistic presented in the Mass Media.

    8. C# programmer says:

      Eric,

      You can pretty well assess the risks of riding motorcycle just by looking at the physics involved. Even if after accounting for all the factors you stated (e.g. alcohol, driving at night), the chance of _accident_ involving motorcycle is the same as chance of accident involving a car, the _damage_ to your health due to motorcycle accident is much higher.

      There are several factors that put a motorcyclist at higher risk than a driver of a car. When a car collides with another vehicle, it protects you from direct collision (and modern cars are specifically designed for that). The car takes on the damage due to collision. The only thing someone inside a car needs to worry about is colliding with something inside a car, but seatbelts and airbags minimize the damage. Even if two modern cars collide head-on at high speed, the chance of fatal injury is small.

      Now, consider what happens when a motorcycle collides with a car. The speed vector of the motorcycle changes due to collision while the motorcyclist continues moving in the same direction. Therefore motorcyclist is "separated" from motorcycle and is essentially "thrown" in the direction the motorcycle was moving before the collision. Motorcyclist immediately looses balance and collides with something else. There is a high chance that motorcyclist finds himself/herself in the nearby lane in front of another moving car.

      Also, since a motorcycle is much lighter than a car, the change of speed vector of motorcycle due to collision with a car is much higher compared to the case when a car collides with another car (the law of conservation of momentum). Heavier vehicle causes more damage to the lighter vehicle than vehicle of the same mass.

      If I want to assess risks of jumping out of the window, I don’t need statistics proving me that it is more dangerous than using an elevator. Just a bit of physics and common sense.

      Also you wrote "Many motorcycle accidents come from the actions of other drivers."

      Well, that makes riding a motorcycle a less desirable activity, not more desirable. If many accidents are caused by other drivers, then there is nothing you can do about it. You can’t prevent those accidents by improving your skills, not driving under influence of alcohol etc.

    9. jaybaz [MS] says:

      C# programmer: It’s even more complex than that.

      I decided to blog about it:

      http://blogs.msdn.com/jaybaz_ms/archive/2004/06/22/163045.aspx

    10. Radu Grigore says:

      "I don’t think that he is misreading them, I think that most statistics written like that are meant to be misread."

      I’m afraid I don’t follow. Is there a cause-effect relation or just a correlation between your two sentences? :)

      "When they report a statistic like the above on television or in print, the large majority of people (Americans especially) just accept it as fact."

      Well, it IS a fact. The problem is that people infer much more than it is actually being said in the claim. This is an education issue similar to the one pointed out by Eric.

    11. Eric,

      One of your eariler posts got me to consider getting a motorcycle, so I spent some time looking for safety statistics. The URL is long gone from my bookmarks, but I found the statistics for various modes of transportation put up by some Australian government agency.

      Motorcycle riders have 42 times more fatalities per kilometer than car drivers. Of those, roughly 50 percent drover without a helmet, and roughly 50 percent of crashes were caused by other drivers.

      I don’t know how those probabilities are correlated, but for the sake of the estimate let’s say that they are not.

      This leaves us with the rider that always wears a helmet and never makes mistakes. In that case he is still ten times (42/2/2 = ~ 10) more likely to get killed per kilometer traveled than a car driver, simply because of other people’s mistakes.

      We all have different risk tolerances and different fun/risk ratios. But I’d say that it’s a pretty safe assumption that you are more likely to get killed on a bike then in a car.

      (Fun fact: riding a bicycle and walking are even more dangerous than riding a motorcycle.)

      Dejan

    12. Adam Miller says:

      C# Programmer,

      Eric never stated that riding a motorcycle is more safe than driving a car. He is merely pointing out the fact that alot of motorcycle injuries are byproducts of carelessness (in one form or another) and that there is things you can do to minimize the risk of injury. You can’t take the risk out of anything but you can always do what you can to minimize it. Some might say if you want to minimize the risk of motorcycle injury then don’t ride them. If you feel that way then please, don’t ride them. But life could be pretty boring without a little bit of risk.

      Adam

    13. Lee says:

      "problem is that people infer much more than it is actually being said in the claim"

      My point was that statistics like this are presented because they want us to infer more into it.

      Take the oft-quoted toothpaste statistic "4 out of 5 dentists prefer…" They want us to infer that 80% of all dentists prefer this brand, even though they aren’t saying that directly. (and can’t because it isn’t true)

      Getting back to the motorcycle saftey question though, I think it all comes down to what we consider to be risky, which is usually what we are unfamiliar with.

      For example, I have a two-car garage and the torsion spring above the door recently went bad.

    14. Lee says:

      Sorry, got cut off…

      For example, I have a two-car garage and the torsion spring above the door recently went bad.

      Everything that I read said that working on the spring myself was a death wish, I should cash in my life insurance now, etc…

      Then I found a website that showed you how to do it, and said sure, it’s dangerous to do this if you aren’t careful, but it is no more dangerous than changing your tire. If you carelessly let your leg go under the axel and then accidently knock your jack over, you’ll probably have to amputate. However if you are familiar with what you are doing and are careful about it, it isn’t a problem.

    15. Interesting discussion. I’ve had similar discussions (or arguments) with people regarding the relative risks of smoking pipes and/or cigars. Certain people think that I’m silly for asserting that the risks of smoking pipes and/or cigars are substantially smaller than those of smoking cigarettes. They seem to be operating on the premise that "smoking is smoking". But the epidemiology of pipe and cigar smoking is quite different, since one doesn’t inhale.

      Unfortunately, like motorcycling, the studies available on pipe/cigar smoking are limited, and many do not control for alcohol consumption. Given that alcohol is a known carcinogen, and one of the main risks cited for cigar/pipe smoking is mouth/throat cancer, it’s quite difficult with such studies to figure out whether the risk is coming from the smoking or the alcohol, or whether perhaps the combination of the two is greater than simply combinatory.

      The point is not that pipe and cigar smoking are safe, but rather that my experience matches with yours…lots of people are more than willing to hold forth about "risks" on subjects on which they have little or no evidence for.

    16. Scott says:

      G. Andrew:

      Alcohol is often labeled as a cocarcinogen (e.g. There is a higher percentage of oral cancer in people who both smoke and drink vs. people who just smoke but the risk of lung cancer seems to be the same in either case.). But that’s neither here nor there. The risks are for different cancers.

      Eric: The biggest problem with J’s point is that’s it’s inherently biased. An ER doctor is more likely to see a motorcylce accident victim than say a Dermatologist would. A Plastic surgeon might conclude that skateboarding is much more dangerous than riding a motorcylce based on the number of skateboard vs. motorcycle victims they see. In other words, if she were another type of doctor or working in a different medical field her view might be different. If she were a coroner she might say that motorcycle accidents have a much higher mortality rate than other motor vehicle accidents. If he would have said "my girlfriend is a clerk in an insurance office and she sees a lot more motorcycle accidents than car accidents." His point might have more validity. I’d think almost every motorcycle accident involves a vehicle OTHER than another motorcycle. In that case, every motorcycle accident is also an automoble accident.

      The fact is that whoever was the cause of a motorcycle accident once it happens to you the effective risk is 100%.

      I spend my days building an application to mine cancer patient data looking for trends, possible candidates for treatment protocols, etc… This kind of stuff fascinates me.

    17. Barry Gervin says:

      http://objectsharp.com/Blogs/barry/archive/2004/06/24/Risk.aspx

      It’s good to see that you’ve followed some of MSF Risk Management Principles. You’ve calculated your Risk Exposure accurately based on Probabilities (driving during day, owning a license, etc) and Impact and you’ve also worked to lower your risk by wearing a helmet and not drinking.

      Project Management and Motorcylces. Who’d have thunk?