Errors in Screening Procedures

ERRORS IN SCREENING PROCEDURES

Any screening procedure has a characteristic error rate. This is inevitable because in high-throughput screening it is necessary to compromise with some accuracy or precision to achieve the requisite speed. Thus when a large number of compounds are carried through a particular screen, some of the compounds are classified incorrectly. A screen may be used in an absolute sense, so that compounds that pass a certain criterion are termed positives, whereas those that fail to meet the criterion are termed negatives. Compounds that pass, but should have failed, are false positives. In general, false positives are tolerable, if they are not too numerous, because they will be rectified later. Compounds that fail, but should have passed, are false negatives. False negatives are lost forever if the failure eliminates them from further testing.

All screening procedures are based on assumptions of analogy. They have different degrees of relevance or predictability. Studies in phase II clinical trials predict the results with high probability in large clinical trails. But even here there is the possibility of false-positive or false-negative results. The relevance of a test is much less in early pharmacological tests, such as used in high-throughput screening. Generally, the relevance is inversely proportional to the simplicity of the test.

In any case, one is confronted with the problem of false-positive results (type I errors) and false-negative results (type II errors).

In each step, two sources of error for false-positive results have to be taken in to account:

1.     a = error of the first type due to the model

2.     α = error of the first type due to statistics

In the error of the first type, a compound is considered to be active, but is actually ineffective. This type of error is clarified during further development, after negative clinical trials at the latest.

However, there are two sources of error for false-negative results:

1.     b = error of the second type due to the model

2.     β = error of the second type due to statistics

In the error of the second type, a compound is considered to be ineffective, but is actually effective.

This type of error will never be clarified; an effective drug has just been missed. Perhaps another investigator will test this compound under different aspects.

The statistical errors derive from the fact that a phar-macological test is performed only several times or in a limited number of animals. One can specify the probability that a decision made is incorrect, that is, a drug candidate is erroneously identified as effective when it is actually ineffective. Usually this risk is set to 5% (P < 0.05) and is called the statistical error of the first type or type I error. The error of the second type or type II error is connected to the type I error by statistical rules.

Usually, screening is performed sequentially. Tests in high-throughput screening are followed by tests in isolated organs, then in small animals, and special tests in higher animals, until the compound is recommended for further development and for studies in human beings. From each step, not only errors of type I, but also from type II, arise. As a consequence, many effective compounds are lost.

There are two ways to circumvent this obstacle: (1) to increase the number of compounds entering the screening procedure dramatically, hope for a reasonable number of true positives, and accept a high rate of false-negative results (White 2000) as followed in the ultra high-throughput screening; or (2) to perform tests with high relevance, meaning tests with high predictive value in whole animals at an early stage (Vogel and Vanderbeeke 1990).

The literature on high-throughput screening includes some publications dealing with false-negative results (Jones and King 2003; Colland and Daviet 2004; Heller-Uszynska and Kilian 2004).

Zhang et al. (1999, 2000) studied the role of false-negative results in high-throughput screening procedures. They presented a statistical model system that predicts the reliability of hits from a primary test as affected by the error in the assay and the choice of the hit threshold. The hit confirmation rate, as well as false-positive (representing substances that initially fall above the hit limit but whose true activity is below the hit limit) and false-negative (rep-resenting substances that initially fall below the hit limit but whose true activity is in fact greater than the hit limit) rates have been analysed by computational simulation. The Z-factor and the Zi-factor were introduced to characterize the reliability of high-throughput assays.

The problem of type II errors, that is, false-negative results, also exists in many other physiological and phar-macological studies (Martorana et al. 1982; Bar- ros et al. 1991; Sandkühler et al. 1991; Waldeck 1996; Williams et al. 1997). For example, Pollard and Howard (1986) reinvesti-gated the staircase test, a well-accepted primary screening method for anxiolytics, and found several false-negative results for clinically active anxiolytics.