You are here:

Advanced Math/Deciding which expert to follow when experts have a disagreement

Advertisement


Question
QUESTION: My question is how to calculate probability when confronted by experts having an argument about a particular subject (in cases where there are only 2 possible answers, such as yes/no questions).
For example: A, B and C are all expert doctors. When each of them (individually) gives a diagnosis (in a yes/no question), the chance of accuracy is 90%, or 9/10. In a case where A and B argue for a certain treatment, and C argues against, what is the chance that A and B are correct? Originally I worked out that the answer is 9/10, but when constructing the general formula I realised that if 11 doctors were arguing against 10 doctors the answer would be the same - 9/10 - which is counter-intuitive and wrong. I then realised that the more doctors who are at loggerheads over a question indicates that this question is more difficult than the average one, in which case the chance of accuracy of all doctors goes down. (90% is only the success rate for the average question, not for the more difficult ones.)

My question is: given these facts, is there any way to give a formula which will tell me the probability of accuracy given the number of doctors on both sides of the issue (assuming that all doctors have the same expertise, that they answer 90% of questions correctly)?

If you could help me I would appreciate it!

ANSWER: This problem has a couple of parts and I hope I can help you out. I'll start by showing you the way to calculate the probability of M doctors giving the right diagnosis out of N, given the probability p of a correct diagnosis for each doctor individually. I think we can use this to answer the final question of the proportion of doctors saying yea and nay that gives the most accurate overall diagnosis.

As you probably realize, this problem uses the binomial distribution which gives the probability of M events occuring out of N for a probability per event of p. The formula is

P(M,N;p) = { N!/[M!(N-M)!] }(p^M)(1-p)^(N-M)

(it is hard to duplicate equations with this editor so I recommend you look up the binomial distribution).

Using your numbers as an example, the probability of 2 doctors making the correct diagnosis and a 3rd make a wrong one is

{ 3!/(2!1!) }(0.9)^2(0.1)^1 = 3(0.81)(0.1) = 0.24.

This is the probability that A and B are correct since, in fact, they did make the right diagnosis.

This can be compared to the situation where all 3 doctors make the right diagnosis where P(3,3;0.9) = (0.9)^3 = 0.73. These numbers tells us something about what is going on. Given the high probability of a correct diagnosis for each doctors, it makes sense that the probability of all 3 doctors making the right diagnosis is higher than if only 2 do. It may be surprising that the probability of all 3 giving the right diagnosis is not 9/10, but this makes sense when you consider that the probabilities correspond to discrete events. The probabilities of all the possible outcomes are (which you can calculate using the formula):

P(3,3;0.9) = 0.73
P(2,3:0.9) = 0.24
P(1,3:0.9) = 0.027
P(0,3;0.9) = 0.001.

Thus the highest probability is for all 3 doctors making the right diagnosis, which makes sense intuitively given the high individual success probability of 0.9. So now we can try and answer the question of the accuracy of the group diagnosis given the proportion of doctors giving the correct diagnosis individually. Another way to state this is to ask what proportion gives the highest probability, i.e., maximize P(M,N;p) with respect to (wrt) M. Because of the discrete nature of the function P, we can't just take the derivative wrt M, set it equal to 0 and solve for M. Instead, we need to solve for M such that

P(N,M-1;p) ≤ P(N,M;p) and
P(N,M+1;p) ≤ P(N,M;p)

in other words, the optimal value of M gives a higher probability than the cases for M 1. Working through the formulas, I get that

Np ≤ M ≤ (N+1)p.

For a large number of doctors, the highest probability is for the number of doctors to be ~ Np for p ≥ 0.5. Thus 9-out-of-10 is reasonably accurate.

---------- FOLLOW-UP ----------

QUESTION: Thank you for your swift reply. But my question still stands: if 1001 doctors told you yea, and 1000 doctors told you nay, are you seriously telling me that you would be just as sure of "yea" as you would be if 2 out of 3 doctors told you yea and one nay! I know that probability can sometimes be counter-intuitive, but this seems plain wrong, and I gave the reason for why it is wrong - that the probability of 90% is not uniform across the whole range of questions, but in fact only represents an average. The more difficult a question is, the lower the probability of success(accuracy). So the more who say nay, indicate that the question is more difficult (that is why so many doctors are in disagreement), hence the chance is much lower than 90%. But I now need a formula which can reflect this point. I hope I'm making myself clear - I'm only an amateur mathematician and although I generally understood what you wrote, I could probably write the problem better if I knew the correct notation. Any more help would be greatly appreciated!

Answer
Thanks for the follow-up. I think the confusion is coming from making an a priori assumption about the accuracy of the individual doctor's prowess. You are, rightfully, concerned about including the difficulty of a diagnosis in the analysis and point out that the accuracy would not be uniform across the range of questions (afflictions to be diagnosed). However, you then make the statement at the end of your question "assuming that all doctors have the same expertise, that they answer 90% of questions correctly". As I think you can see, these assumptions are conflicting.

I agree that it is an interesting goal to tie the accuracy of a group diagnosis to the intrinsic difficulty of the individual diagnosis, but this line of analysis is obviated by making an a priori assumption of the individual doctor's accuracy. The analysis using a given individual's accuracy is valid but may not be what you are really looking for. So let's go on to the real question of what to believe if a group of doctor's give a certain proportion of yes and no diagnoses. In the following, upper case words refer to real mathematical terms.

It seems to me that the crux of this problem is first to estimate the EXPECTED VALUE, EV, of the probability of a correct diagnosis, and then go from there. Absent any additional information about the prowess of individual doctors (say, that doctor A is great and his diagnosis should be given greater weight vs. doctor B who is a known hack), the EV of the probability p = correct diagnosis is

p_bar = EV(p) = ∑(over j)・rj・(pdf)j = ∑(over j)Ij/N   <-- honest-to-god definition of an EV

where j = the jth doctor, rj is the jth doctor's response, the sum is over N doctors, (pdf)j is the jth value of the PROBABILITY DENSITY FUNCTION and Ij is an INDICATOR FUNCTION which is just a fancy way of specifying the pdf of the response of the doctors: Ij = 1 if correct and 0 if wrong. Thus, for a given affliction, if M doctors answered yes (and of course N-M answered no) then the EV estimate would be p_bar = M/N.

In the above, each doctor is weighted equally so that each response is weighted the same. This weight is 1/number of responses = 1/N. Thus we are assuming a UNIFORM DISTRIBUTION for the responses. It is perfectly feasible to instead make this weighting reflect the varying prowess of the doctors by specifying a non-uniform pdf, i.e., (pdf)j ≠ constant, but this would have to be specified A PRIORI (a separate analysis would have to be done using test cases for the doctors).

So the question is, how much a priori information are you going to assume before polling the doctors? If you know p_bar for a given affliction (are dead certain), then you use the previous analysis as is. However, it sounds like you are interested in the case where you are not sure how difficult the diagnosis is, meaning you don't know p_bar all that well, and would like to know what the chances are of a correct diagnosis given a the proportion of yes responses from a group. A worthy question and one I think I can answer with a little more work.

I'd like to help you further. Please email me at rpatton@mathscienceguy.com and I will take it from there.

Thanks again,

Randy

Advanced Math

All Answers


Answers by Expert:


Ask Experts

Volunteer


randy patton

Expertise

college mathematics, applied math, advanced calculus, complex analysis, linear and abstract algebra, probability theory, signal processing, undergraduate physics, physical oceanography

Experience

26 years as a professional scientist conducting academic quality research on mostly classified projects involving math/physics modeling and simulation, data analysis and signal processing, instrument development; often ocean related

Publications
J. Physical Oceanography, 1984 "A Numerical Model for Low-Frequency Equatorial Dynamics", with M. Cane

Education/Credentials
M.S. MIT Physical Oceanography, B.S. UC Berkeley Applied Math

Past/Present Clients
Also an Expert in Oceanography

©2016 About.com. All rights reserved.