Improving Performance in Detection Systems: Exploring a Parallel-judgment Structure
Ignacio J Martinez-Moyano, Ph.D.
Argonne National Laboratory
Abstract:
In this paper, we describe a model of two judges in parallel. We expand work related to the
identification of threats by analyzing the effectiveness of including a second judge in the process
and identifying its effects on performance and error rates. In addition, we explore the
implications for learning under uncertainty.
Introduction
Improving performance in detection systems is one important challenge in dealing with terrorism and
insurgencies. Terrorist activities are based on covert operations and make use of deception to hide intent and
possible detection before action occurs. The intelligence community and law enforcement personnel struggle to
improve detection systems to counter activities of terrorists and insurgents across the world. In the decision
sciences and learning disciplines, most notably in the cases of cyber threats and attacks, selection-detection
processes have been studied as part of an effort to increase the accumulated knowledge of how to detect those
elements that belong to a distribution of interest (signal) when such elements are embedded in a larger
population of elements that are not of interest (noise).
The challenge of identifying the members of a distribution of interest (signal) in a certain population belongs to a
broad class of problems known as selection-detection problems. Frequently, this class of problems is addressed
with signal-detection theory (Egan, 1975; Green & Swets, 1966; Swets, 1992). In this type of problem, a
mechanism needs to be used to make a judgment concerning the likelihood that the element belongs to either
the signal or noise distribution to determine appropriate action (e.g., keep or discard, let through or stop, arrest
or not, accept or reject). The said mechanism can be automated or human-based. The performance of the judge
(automatic or human) determines the adequacy of the detection-selection process. A perfect detection-
selection process should be capable of identifying the members of both distributions (signal and noise) without
errors. Figure 1 shows how the two distributions typically overlap each other, making the matter of selection-
detection anything but trivial.
This manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory (“Argonne”). Argonne, a U.S. Department of Energy
Office of Science laboratory, is operated under Contract No. DE-ACO2-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-
up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and
display publicly, by or on behalf of the Government.
Noise Signal
Distribution Distribution
\ /
Probability density
Judgment
Figure 1—Selection-Detection Problem Characteristics
A selection-detection problem is trivial when the targeted population (signal) is clearly distinguishable from the
non-targeted one (noise). Figure 2-A presents a graphical representation of this case: the no-uncertainty case. In
this case, the two distributions do not overlap, allowing for a level of judgment (along certain dimensions or a
composite of dimensions) that clearly determines where one distribution ends and the other starts. The no-
uncertainty case exists in the following example: the task, in a population of 100 items composed of squares and
circles whose shapes are fully visible, is to separate (detect and select) the squares from the circles. The
distribution of circles and the distribution of squares are clearly distinguishable and, therefore, a judge (human,
mechanical, or electronic) could easily separate the two types of items without any errors. In the context of the
identification of terrorists, this challenge would be equivalent to identifying a characteristic of individuals such
that terrorists would be unmistakably and clearly identifiable from other individuals. (If such a characteristic or
bundle of characteristics exists, it is not yet known, making the identification of terrorists an important
challenge). In reality, most detection-selection problems are complex and difficult to address. Judges have to
rely on multiple, fallible information cues about the distributions to attempt to distinguish members of the
signal from the noise (Hammond, 1996, 2000). The detection-selection problem becomes more difficult as the
area of overlap of the distributions increases. Figure 2-B shows a graphical representation of a small level of
uncertainty in this type of problem. As the figure shows, the two distributions overlap, creating the possibility of
error. Figure 2-C shows a higher level of uncertainty by presenting a larger area of overlap of the distributions.
As shown in these figures, the larger the area of overlap, the more difficult it is for a judge to distinguish which
elements belong to which distribution. Thus, a judge making a judgment under conditions of uncertainty in a
detection-selection task will unavoidably face the possibility that two types of error will occur: false positives
and false negatives.
No Uncertainty Smaller Uncertainty Greater uncertainty
Non-Terrorists Terrorists
Non-Terrorists Terrorists Non-Terrorists Terrorists Distribution Distribution
2| Distribution Distribution 2] Distribution Distribution | || 2|
3 3 3
Fa 2 z
3 3 3
é é é
Judgment odgment Judgment
Figure 2—Uncertainty Configurations
A judge makes a false-positive error when he, she, or it mistakenly identifies a member of the noise distribution
as belonging to the signal distribution. This error is also identified as a false alarm. Depending on the context,
this type of error might not be as costly as a false-negative error. False-negative errors happen when a judge (or
a selection-detection system) mistakenly identifies a member of the signal distribution as belonging to the noise
distribution. In the context of terrorism, this type of error equates to allowing a terrorist through a security
checkpoint (at an airport or border crossing point) instead of arresting the individual, thus allowing him or her to
carry out his/her mission. False-negative errors tend to be more costly than false-positive errors. In the context
of terrorism, a false-negative error can have devastating consequences, generating a large amount of economic
damage and casualties. Therefore, in this context, judges normally tend to err on the side of incurring more-
than-needed false-positive errors to prevent false-negative errors from occurring. Adding to the complexity of
the problem, some terrorists are deliberately trying to sneak through defenses to learn about the defense
systems and to be able to report back to their groups. In these cases, terrorists do not exploit the penetration of
the defenses immediately, making it more difficult for judges to identify when errors happen. In this sense, and
not only in the context of terrorism, many detection-selection processes are forced to operate under conditions
of limited and conditional feedback (Ghaffarzadegan, 2008).
In many countries, and in particular in the United States, selection-detection processes with an important
human-based judgment component are widely used in the prevention of terrorism. In particular, and among
many others, the Transportation Security Administration and the Bureau of U.S. Citizenship and Immigration
Services of the Department of Homeland Security (DHS) deploy tens of thousands of individuals (judges) every
day to selection-detection posts at airports, maritime ports, roads, and border-crossing points who are charged
with the task of identifying and stopping malicious individuals from harming the people and institutions of the
United States. In general, these judges have the ability to make a call individually but also rely on the assessment
of their peers to conduct their very important work. Policies and procedures are in place to increase the
probability that they will successfully carry out their detection and selection tasks.
We have elsewhere explored the fundamental components of iterative detection-selection processes in a single-
judge structure and have identified these components to be judgment, learning, and decision-making processes
(Martinez-Moyano, Conrad, & Andersen, 2007; Martinez-Moyano, Rich, Conrad, Stewart, & Andersen, 2006). In
this paper, we use formal modeling to learn about the effects that parallel judgment has on the performance of
detection-selection processes. In order to explore this idea, we are expanding on previous work and building a
dual-judge structure in which the judgment process occurs in parallel, thereby introducing a multi-view of the
detection problem. Expanding previous research (Martinez-Moyano, Rich, Conrad, Andersen, & Stewart, 2008),
in this model we represent a situation in which two judges are exposed to the same set of information cues
about a certain phenomena and they (together) become a higher-level judge that decides about the likelihood
that a case belongs to either the signal or noise distributions.
In the next section, we explain the main parts of the structure of the model that we use to explore the dual-
judge system. Later, we present the simulation scenarios used in this investigation and, in the last part of the
paper; we discuss the findings, present conclusions, and advance our thinking on future avenues for research.
Model Structure
Judgment Process
The dual-judge model is organized around three main processes: judgment, decision making, and learning. In
this model, we integrate constructs from social judgment theory (Brunswik, 1943, 1956; Hammond, 1996;
Hammond & Stewart, 2001), signal detection theory (Green & Swets, 1966; Macmillan & Creelman, 2005), and
outcome-based learning (Erev, 1998; Erev, Gopher, Itkin, & Greenshpan, 1995) in a system dynamics framework
(Forrester, 1961; Richardson & Pugh, 1981; Sterman, 2000) to capture the process in which two judges make
assessments about a series of cases with respect to their likelihood of representing a threat, thus triggering
defensive action.
Judge A Bias
Reliability
JUDGE A> Judge A Weight of
Environment sTédae A Sans
Natural Propensity Reliability i
Judge A Weight of
Inherent «Information Knowledge about Judgment of Distal -«— Information Cue 2
Unpredictability of the Cue b> Information Cue 1 Variable JUDGE A
i —_
Environment» ee \ ‘Tuige A Weight of
«Measurement Error of \ Information Cue 3
‘True Weight of ‘Information Cue 1> ‘Ny
Information Cue 1 <
— am «Information Knowledge abo Adjusted Judgment of Adjusted Judgment of
‘ x Cue 2 Formatior Distal Varaible JUDGE A _ Distal Varaible JUDGE B
Tras Webhtae — ue Information Cue my
Information Cue 2
«Measurement Error of
True Weight of Information Cue 2>
Information Cue 3
Weight of <Information Knowledge about Judgment of Distal *
Unpredictability of the Cue 3> Information Cue 3 Variable JUDGE B Judge 8 Weight of
Environment *e
Information Cue 1
«Measurement Error of
Information Cue 3> Judge B Bias Judge B Weight of
Information Cue 2
Judge 8 Judge B Weight of
Reliability Weight Information Cue 3
Figure 3—Judgment Process
In Figure 3 (above), we show how we structure the judgment process in the model. In the model, two judges
(Judge A and Judge B) make judgments about the nature of a distal variable. Both judges use the same
information vector (information cues about the distal variable) and have the same judgment policies (bias,
reliability, and relative weights for information cues). In this sense, the two judges are symmetrical (later, we will
relax this condition). The symmetry of the judges is used to represent an extreme case — desirable to many in
the security industry — in which all judges present in a security checkpoint share the exact same policies (in
practice, however, no two judges have identical judgment policies). We advance the identical-judges scenario as
desirable given our experience in identifying high levels of training and on-the-job mentoring as pervasive and
highly regarded in many agencies as a good way to get officers and inspectors to improve their judgment skills
and to ensure minimum standard levels of practice.
As explained earlier, the judges share the exact same information vector about the environment. In addition, to
relax the full-symmetrical assumption, in the model we introduce a two-stage judgment mechanism that
operates as follows: first, we allow the judges to integrate information about the environment and create a
judgment of what they think the distal variable is. Once the “first” judgment is produced, the judges produce an
“adjusted” judgment in which two information cues are used: their own judgment of the distal variable (the
“first” judgment) and information about the other judge’s judgment of the distal variable. The judgment process
of each judge is characterized according to social judgment theory as a function of the information cues about
the phenomenon under study and a number of parameters that capture the individual characteristics of the
judge. (In Table A and Table B of Appendix 1 of the paper, we present these parameters and their base values).
In general, the judgment process is characterized as a weighted average of the information vector available to
the judge and complemented with bias and reliability parameters. The judgment equation is of the form:
y=Yte
Where Y is the judgment of the distal variable, Y is anestimate of Y, and e is an indicator of degree of
reliability of the judge (and, in the model of the environment, represents the inherent unpredictability of the
environment).
Y captures the judgment process by combining information about the distal variable and judgment bias. Yis of
the form:
Y=b,X,+b,X,+b,X,+b,X,+b.X,+k
Where Y is an estimate of Y, 5, is the weight of information cue n on the judgment, X,, is the information
cue n, and k is a bias term.
Using a two-stage judgment process, we model the possibility of the judges’ mutual influence in the detection-
selection task. This element is an important part of the cognitive process that multiple individuals (e.g., agents,
monitors, judges, screeners) face on an everyday basis. When a two-stage judgment process is institutionalized,
5
it provides the basis for teamwork synergy to emerge. In many contexts, including law enforcement and the
military, a multi-judgment environment is fostered to ensure that adequate calls are always made. For example,
in difficult cases in medical radiology, it is common to have a tiered system in which multiple physicians
independently examine the same body of evidence (image) and then make a judgment as to what it is that the
image shows. Once these multiple opinions are generated, the physicians share their views, creating the
opportunity to adjust their assessments by incorporating additional information coming from the expert
judgment of their peers. In this way, junior radiologists learn about the process and find ways to fine-tune their
abilities by seeing what senior radiologists do. In previous work, we have explored the simpler one-stage
judgment process and found it to be a fair representation of what judges do in detection-selection tasks in
experimental settings (see Martinez-Moyano et al., 2007) when working alone. However, very important
detection-selection processes are carried out by judges working in groups (formally or informally) on a daily
basis, making the need for insights about this process salient.
Additionally, in this section of the model, we characterize the environment that is being presented to the judges.
We model the judgment process and the environment in a symmetrical way. Both models share the same
structure but are parametrically different. In this way, in the model we have information about the environment
that we use to compare to what the judges are able to identify and integrate. In our model, we have information
about the true state of the environment and about what the judges perceived of the environment. With this
information we can measure judgment accuracy and overall performance of the detection-selection process. In
particular, we vary the inherent predictability of the environment to assess the benefits of having a dual-judge
structure and other judgment policies. The “Weight of Unpredictability of the Environment” captures the
inherent predictability of the environment in the model. A zero (0) weight represents a perfectly predictable
environment with no stochasticity (and formulated linearly). A weight of 1 represents an environment that is as
unstable as it can possibly be, given a certain distribution of stochastic noise in its predictability. We model an
environment with a low base rate of incidence of signal (13.6% parametrized in the model via a criterion
threshold of 61 in a normal distribution with a minimum of 0, a maximum of 100, a mean of 50, and a standard
deviation of 10).
The judges, therefore, integrate information about the environment and about other judges’ judgments to
create a judgment that is then compared to a decision threshold that determines whether defense action is
granted or not. When their judgment of likelihood is higher than their decision threshold, action is triggered.
Thus, action is the result of the coupling of judgment and decision making.
Decision-making Process
After judgments are generated, these need to be used in a decision-making process to determine whether their
level is sufficient to grant action. This process is captured in the model by comparing the level of the adjusted
judgment with that of a decision threshold that determines the cutoff of what is acceptable and what is not (see
Figure 4).
Multijudge Proposed
Action (Logical OR)
Multijudge Proposed
Action (Logical AND)
Proposed Action
Criterion
‘Threshold JUDGE A Proposed Action
a JUDGE B
Event yon Decision Threshold
Decision Threshold} Tnitial Judge A
for Judge A
Change to Decision _Decision Threshold
Threshold for Judge A ——-Tnitial Judge @
True
Negatives
Influence of False
Negatives on Decision
Threshold Judge A Decision Threshold
for Judge B
Influence of True
Negatives on Decision
Net Influence of
Outcome on DecThre Threshold Judge A Change to
Judge A Decision
‘Threshold for
Infiuence of True Positives Judge B
False Positives on Decision Threshold
Influence of False Judge A
Positives on Decision
Threshold Judge A
fa)
Net Influence of
Influence of False ‘Outcome on DecThre
Negatives on Decision Judge 8
Positives on Decision
Threshold Judge B ————— -
Influence of True Threshold Judge 8
Negatives on Decision Influence of True Positives
Threshold Judge 8 0" Decision Threshold
Tudge B
Influence of False
Figure 4—Decision and Learning Processes
Each judge has his or her own decision threshold that he/she uses for action-related determinations. In this way,
the decision threshold is as instrumental in the determination of action as the judgment is. The coupling of these
two elements is what allows judges to infer whether action is needed or not. After decisions are made and
outcomes are experienced, the potential to update the decision thresholds exists. For example, in the context of
the identification of terrorists, let us suppose that an agent evaluating cases assesses a case with an index of 45
(in a 100-point scale). Should this raise red flags and trigger action? The answer will depend on the decision
threshold that determines what is “normal” and what is not. If in this case, the agent has a decision threshold of
60 points, then the case is considered “normal” and no action is granted. However, if the agent operates with a
decision threshold of 35 points, then the case is above the cutoff point and should be acted upon. In this sense,
the decision-making process is very straightforward and clean. However, several potential problems exist that
contaminate the process, making certainty problematic (or inexistent). First, the original 45-point assessment is
based on fallible information cues and is subject to imperfect judgment and biases. Also, the decision threshold
level (60 or 35) is also subject to biases and uncertain indicators based on outcome identification. Therefore, if
the judge with the 60-point threshold does not engage in defensive action, he might incur error in letting a
terrorist walk. Alternatively, when the judge with the 35-point threshold intervenes to stop the suspect, he
might be detaining an innocent individual. Further, the judge might make the correct call for the wrong reasons
creating a potentially disastrous learning environment. For example, let us suppose that the original 45-point
assessment was wrong and that the true level was 65 points. Also, let us suppose that the judge mistakenly used
the 60-point threshold when in fact he should have used a 75-point threshold that matches the base rate of
occurrence of the phenomenon. In this case the judge has incorrectly assessed the case and has incorrectly
updated her decision threshold while at the same time, by not engaging in a defensive action, the judge is
choosing a correct course of action (as the real assessment is lower than the real threshold). As the outcome of
this decision is successful, the judge will incorrectly reinforce his beliefs about how to assess cases and how to
update his decision threshold. Accidental success becomes the origin of superstitious (and quite mistaken)
learning.
Learning Process
Independent of what type of error is more costly (false positives vs. false negatives); minimizing error happens
through a fundamental learning mechanism that links outcomes of the decision process with updated levels of
the decision threshold (Erev, 1998). As outcomes are experienced and errors are recognized, judges modify the
level of the decision threshold to find the level that creates higher levels of performance. This is not the only
learning mechanism that judges use to improve performance. Other mechanisms, which are not present in the
model, include improving: the judgment process (including reliability, biases, function forms, etc.), selection
technology (i.e., pushing the signal and noise distributions further apart), and quality of the information received
about the environment, etc.
Updating the decision threshold is complicated for judges, because feedback from their decisions is sometimes
difficult to obtain and in many cases is not close in time or space to the original decision (Weaver & Richardson,
2002). In the model, we use an outcome-feedback learning process to update the judge’s decision threshold. In
this sense, judges learn by doing in repeated iterations. Each judge has his or her own value matrix that he or
she uses to modify the decision threshold. A value matrix captures the value that each type of outcome has for
the judge. Correct and incorrect decisions yield results that the judge identifies and uses to adjust the decision
threshold. In general, the absence of errors reinforces the use of the current decision threshold, while
recognition of errors causes the judge to change the level. Depending on the context, the different types of error
might have very different values and effects on changes to the decision threshold.
Additionally, in the model we use a number of flags that allow us change configurations of the judges and of the
judgment and decision-making processes to identify conditions that have the potential to improve performance
of detection-selection processes. The most important flags used in the model are explained in Table C of
Appendix 1. Additional details of the learning theory implemented in this model can be found in (Martinez-
Moyano et al., 2006).
Simulation Scenarios
In order to understand the impact of changes to the characterization of judges, the judgment process, and the
environment on the performance of the detection-selection processes we modeled, we designed 34 different
scenarios to be used. For all simulated scenarios, the base rate of incidents remains constant (13.6%). In Table 1,
we show these scenarios, the variables that we use to characterize them, and the values used in the different
simulations. We use the model to learn more about the impacts that the different symmetry levels of the
judges, different conditions of the environment, different rates of learning, and different conditions of
coordination among judges have on performance metrics of the detection-selection process.
The predictability of the environment is characterized at two levels: perfect and unstable. A perfect environment
is one that is fully predictable (i.e., no stochasticity is built in), and an unstable environment is 50% predictable
(i.e., it is modeled by using a stochastic distribution around the perfect environment). We anticipate that
modifications to the inherent predictability of the environment will have an impact on process performance as
articulated in proposition 1.
Proposition 1: Under conditions of perfect predictability of the environment,
lower error rates and better performance will be achieved.
In Figure 5, we show scatter plots of the judgments of Judge A relative to the actual measure of the distal
variable for runs B (perfect environment) and | (unstable environment). The judgments of a perfectly reliable
judge would correlate better with the environment in both cases. Note that the spread of the judgments is
higher in Figure 5-B than in Figure 5-A.
Scatter Plot Scatter Plot
100 100
15
50 30
25 j 25
o 0 2% 3% 40 SO 6 7 80 9 100 o 0 20 3 4 SO 60 7 80 9 100
Distal Variable Distal Variable
Adjusted Judgment of Distal Varaible JUDGE A: B Adjusted Judgment of Distal Varaible JUDGE A:T ==»
A B
Figure 5—Dispersion of Judgemnt
The frequency of learning is varied at three levels: normal, high, and low. Frequency of learning is contolled with
a sampling mechanism that creates the condition for different levels of exposure to the results of the judgment
process. In high-frequencey cases, judges receive four times as much feedback as in the normal case. In the low-
frequency case, they rceive one-fourth of the feedback they would normally receive. The differing of frequency
levels captures the possibility of having different frequencies of feedback provided to judges and estimating
what impact(s) that would have on performance and error rates. We anticipate that frequency of learning will
influence error rates and performance levels as articulated in proposition 2.
Proposition 2: Under conditions of higher frequency of feedback, lower error
rates and better performance will be achieved.
Judgment symmetry is varied by means of judge characterization parameters related to judgment policies,
function forms, bias, initial decision thresholds, and judgment reliability. We create two basic conditions:
symmetrical judges and asymmetrical ones. Symmetrical judges are identical in every way, including in their
level of judgment reliability. The judges in the model are not perfectly reliable (although they do have the same
level of unreliability in judgment). Asymmetrical judges differ in judgment reliability (i.e., the reliability of
Judge A is higher than the reliability of Judge B) and in the initial levels of their decision thresholds (i.e., Judge A
starts with a higher decision threshold [56] than Judge B [25]). The parametrization of asymmetrical judges
creates scenarios in which two distinct judges interact. Judge A becomes the more reliable judge of the two, and
Judge B becomes the more prudent at the start of the simulation by having a lower initial decision threshold.
Judgment reliability is desirable in judgment processess, and judgment prudency (i.e., having low decision
thresholds) ensures the avoidance of critical errors. In real detection-selection systems like those set up across
the United States (and in many other countries) to prevent terrorists from gaining access to critical
infrastructure (such as through air travel), thousands of human judges engage their abilities every day to protect
the public. In these systems, it is highly likely to find several different judges interacting, some who are more
reliable than others and some who are more prudent (i.e., less tolerant to uncertainty) than others. Our
characterization of the asymmetrical judges in the model is designed to capture these differences in a combined
manner. It is clear to us that a different combination of judgment characteristics can be identified and codified
for judges. Also, we anticipate that changing these characteristics may change the results presented here.
However, we are confident that using the parameters we chose will allow us to identify certain conditions under
which asymmetry in judgment may benefit/hinder performance in detection-selection systems.
Additionally, in order to be able to capture the value of having more than one judge in the decision process, we
introduce a coordination parameter that links results of the pairing of judgment with decision tresholds and
action triggering. As in the dual-judge model, we have two potentially different opinions related to launching a
defense action, and we created two conditions, or policies, on how to determine whether action is granted. The
first one is the no-agreement-is-needed policy. In this policy, when either one of the judges determines that
defensive action is needed, an action is launched. Alternatively, in the agreement-is-needed policy, defensive
action is launched only when both judges agree that it is needed. This second policy imposes a more stringent
control on action triggering. We anticipate that symmetry in judgment and coordination in decision making will
have an effect on performance as articulated in propositions 3 and 4.
Proposition 3: Decision-making coordination will yield lower error rates and
better performance independently of judgment symmetry.
Proposition 4: Under conditions of asymmetrical judgment, higher variability of
error rates and performance will be observed.
Lastly, in order to test the effect of the two-stage judgment process, we designed scenarios in which
symmetrical and asymmetrical judges use information about the other judge in their judgment process to create
adjusted judgments before determining whether action is needed (by comparing the adjusted judgment with
their decision threshold). These scenarios (runs AA through HH) are explored in both environmental conditions
(perfectly predictable and unstable) but only at the normal frequency of learning. Coordination of decision
10
making is also varied in these conditions. We anticipate that the use of a two-stage judgment process will have
an impact on performance as articulated in proposition 5.
Proposition 5: Under conditions of a two-stage judgment process, lower error
rates and better performance will be achieved.
Table 1—Scenario Characterization
# [10 | RunName Variables
5 E <|-s
é s =|
§ # Z| EB) e)e |<
BS fef/B/e je /s |es/SlE |e
/E/f8 |e | |ze]el]s ls
cc a 2 ig c . Sa ce a 3
s|s |e] s]e | 2 | a £lsi/2/2 |
2/3 |= /s}/2 |2 | gf) B)s}s |s
elgl/2 /2]s)/2 |2 /2 Jz] 2]e]8 8
e| o s/s) 2 & = E Gj a
eFele |Z )2/2 /2 /2 Jalelele fs
e| 2] 2 ~ | &| o & wo} s |] sf] < 2
f}el= |S) 2} |e ]¢8 2)2/2)/8 | 8
=| s/o |#/s/e |e 18 Elg/3f/es |e
og} S| 5 = <z/ s = a ala a 3 =
1_[ A _| Base o | 1 [ofowslosl i [ils [sel o 0
2 [| B | Symmetrical Judges (Agreement not slo | 2 ]o]o2 ou] 1 [af se] se] o 0
Needed) E
3 | C | Symmetrical Judges (Agreement €) 2/0 | 2 ]afom os] 1 [alse] se] o 0
Needed) e
4 | D | Symmetrical Judges (Agreement not | § 0 | 2 | 0] 025 | 025 | 025 [1] 56] se] 0 0
Needed) 2z| 5
5 | © | Symmetrical Judges (Agreement s| ={ 0 | 2 [4 | 025 [025] 025 [1] se] se | o 0
Needed) 2
6 | F | Symmetrical Judges (Agreement not | & o | 2 /ofo2s fou} 2 |[1{s6/] se fo 0
Needed) 3
7 | G | Symmetrical Judges (Agreement S[o [2 ]afow fos) 2 [if se | se] o 0
Needed)
8 [ H | Based oe |i [olo fos] 1 [1] s6|s6| 0 0
9 | 1 | Symmetrical Judges (Agreement not sl oe | 2 |o]o2 fou] 1 [af s6 | se] o 0
Needed) E
10 | J | Symmetrical Judges (Agreement §| 2[oe [2 [1/0 foul 1 [ifse|] sel o 0
Needed) &
ii | K | Symmetrical Judges (Agreementnot | £ o6 | 2 | 0] 025 | 02 | 02 [1 | s6 | 56] 0 0
Needed) Els
12 | L | Symmetrical Judges (Agreement 2| Ef ose | 2 [1 [025] 025 | 025 [1] 56 [56] 0 0
Needed) ccd
13 | M | Symmetrical Judges (Agreement not | 5 oe | 2 [o]o2 |ous] 2 [1] s6 | se] o 0
Needed) 3
14 | _N_ | Symmetrical Judges (Agreement Sfoe [2 ]ifo fos) 2 [if se] se] o 0
Needed)
15 | © | Asymmetrical Judges (Agreement z|_| 0 | 2 [0] 025 | os 1 [ij[s]alo 0
not Needed) o| @
16 | P | Asymmetrical Judges (Agreement =| sf 0 [2 || 02 | os 1 [ifs ]a]o 0
Needed) c
17 | Q | Asymmetrical Judges (Agreement & o | 2 [0] 02] os | o2 [1] 56] 2] 0 0
not Needed) gis
18 | R | Asymmetrical Judges (Agreement 5/ =f 0 | 2 [1] 02s] os | 02 [1a | s6 | 2a] o 0
Needed) =
11
19 | S ] Asymmetrical Judges (Agreement 0 025 ] 05 2 56] 25] 0 0
not Needed) Pa
20 | T | Asymmetrical Judges (Agreement S[ 0 025 | 0S 2 56 [25 | 0 0
Needed)
21 | U_ | Asymmetrical Judges (Agreement _| 06 025 | 05 1 36 | 25 | 0 0
not Needed) é
22 | V_ | Asymmetrical Judges (Agreement z| 8] 06 0.25 | 05 1 56 | 25 | 0 0
Needed) g
23 | W | Asymmetrical Judges (Agreement 5 06 025 | 05 | 025 56 | 25 | 0 0
not Needed) =| 5
24 | X_ | Asymmetrical Judges (Agreement S|) x] 06 025 | 05 | 025 56 | 25 | 0 0
Needed) 2
25 | Y | Asymmetrical Judges (Agreement a 06 025 | OS 2 56 | 25 | 0 0
not Needed) 5S] 3
26 | Z | Asymmetrical Judges (Agreement S06 025 | 05 2 56 | 25 | 0 0
Needed)
27 | AA | Symmetrical Judges (Using other 2 0 025 [02 | 1 36 | 56 | 05 | 05
3
Judgment) o
28 | BB | Symmetrical Judges (Using other — —=| 20 025 [02 | 1 36 | 56 | 05 | 05
Judgment) =| €
29 | CC | Asymmetrical Judges (Using other &| s| 0 0.25 | 0S 1 se | 25 | 025 | 05
Judgment) S
30 | DD | Asymmetrical Judges (Using other 5 0 025 | 05 1 36 | 25 | 025 | 05
&
Judgment)
31 | EE | Symmetrical Judges (Using other 06 025 [02 | 1 56 | 56 | 05 | 05
Judgment)
32 | FF | Symmetrical Judges (Using other «| _| 06 025 | 025 | 2 36 | 56 | 05 | 05
Judgment) 2; é
33 | GG | Asymmetrical Judges (Using other 2| 8] 06 0.25 | 05 1 56 | 25 | 0.25 | 05
Judgment) 2
34 | HH | Asymmetrical Judges (Using other 06 025 | 0S 1 56 | 25 | 025 | 05
Judgment)
Results and Discussion
In the model we use stochasticity to generate information cues about the environment and about judgment
predictability. In order to understand more clearly the simulation results derived from the different simuation
scenarios described in the previous section we simulated each scenario 2,000 times by using different seeds for
the stochastic processess driving the behavior of the information cues, the predictability of the environment,
and judgment reliability. In Table 2, we report results for performance metrics of the detection-selection
process: the decision threshold for Judge A, decision threshold for Judge B, selection rate, error rate, sensitivity
(proportion of elements belonging to the signal distribution that are acted upon), and specificity (proportion of
elements belonging to the noise distribution that are correctly ignored).
12
Table 2—Results (2,000 simulations)
# [io | l l | | | l l
Decision Threshold for Judge A Decision Threshold for Judge B Selection Rate Error Rate
5 5 5 &
. S | » © & | 2 < 8 2 < 8 2
£/5| 38 a 8 £/ 3] 38 a fal £ & & a fal ie & & 2 fal
S|s)]s =) 5 S|s)/e 2/5 = 2 = = a S 2 = = a
es ee ee 031 | 01909 | o.19 | 00341 J 002 | 016 | 00693 | 0.07 | 0.0193
2 |B J s9 | 65 | 6211 | 62 | 1.014 [so | 65 | 6211 | 62 | 1.014 J 01 0.33 | 0.2029 | 0.2 | 0.0348 0.03 | 0.16 | 0.0742 | 0.07 | 0.0171
3 |_c [sz [63 | 5946 | 59 | 09774 [57 | 63 | 5946 | 59 | 09774 | 0.07 03 | 01763 | 017 | 0.0342] 001 | 012 | 0.0533 [ 0.05 | 0.0178
4 [| d [60 | 66 | 62.64 | 63 | 0.9626 | 60 | 66 | 62.64 | 63 | 0.9626 § 0.1017 | 0.2183 | 0.1592 | 0.1588 | 0.0172 | 0.0223 | 0.0794 | 0.0482 | 0.0471 | 0.0092
5 |_& [s7 [63 | 5982 | 60 | 09514 [57 | 63 | 5982 | 60 | 0.9514 § 0.0967 | 0.2109 | 0.1522 | 0.1513 | 0.0173 J 0.0173 | 0.0769 | 0.0427 | 0.0421 | 0.0094
6 | F [s7 | 65 | 6086 | 61 | 1126 ff s7 | 65 | 6086 | 61 | 1126 J 01 0.42 | 0.2375 | 0.24 | 0.0510 0.02 | 0.24 | 01018 | 0.1 | 0.0263
7 |G [se | 2 | 5870 | 59 | 09936 [se [ 62 | 5870 | s9 | 09936 f 004 | 036 | 01944 | 02 | o0s0af 0 016 | 0.0646 [ 0.06 | 0.0257
8 | H [53 | 65 | s958 | 60 | 1593 53 | 65 | 5958 | 60 | 1593 J 012 | 034 | 02136 | 021 | 0.0355 § 004 | 025 | 01320 | 0.13 | 0.0321
9 | 1 [se | 66 | 6084 | 61 | 1573 [se | 66 | cosa | 61 | 1573 Ff 013 | 0.36 | 0.2261 | 0.22 | 0.03589 0.04 | 0.27 | 0.1356 | 0.13 | 0.0318
10 | 3 953 | 63 | saa7 | sa | 1se3 J 53 | 63 [sai7[ 58 | 1563 J 009 | 033 | 01994 | 02 | 003529 004 | 024 | 01239 | 012 | 0.0321
ai [kK [se | 67 | 61.21 | 61 | 1556 J 56 | 67 | 61.21 | 61 | 1556 J 0.1339 | 0.2506 | 0.1905 | 0.1910 | 0.0184 J 0.0669 | 0.1861 | 0.1228 | 0.1215 | 0.0165
a2 [ t [sa [6s | sai | sa | 1520 J 54 | 65 | saai | 58 | 1520 J 0.1265 | 0.2431 | 0.1836 | 0.1836 | 0.0184 [ 0.0669 | 0.1712 | 0.1196 | 0.1191 | 0.0166
13 [mM Js5 | 65 | 59.92 | 60 | 1512 J 55 | 65 [59.92 | 60 | 1512 J 008 | 0.42 | 0.2541 | 0.26 | 0.0528 0.02 | 032 | 0.1480 | 014 | 0.0452
14 [Nn [53 | 63 | 57.75 | 5a | 1490 J 53 | 63 | 57.75 | 58 | 1490 [| 004 | 038 | 02106 | 02 | 005119 0 03 | 01272 | 012 | 0.0462
15 | 0 J 86 | 95 | 9069 | 91 | 1495 J 55 | 64 | 59.69 | 60 | 1495 J 036 | 059 | 0.4887 | 0.49 | 0.03469 03 0.43 | 0.3605 | 0.36 | 0.0194
ie | P 958 | 64 [6091 | 61 | 1092 [27 | 33 | 2991 | 30 [ 1092 [ o1 031 | 01909 | 019 | 00341 f 002 | 016 | 00693 | 0.07 | 0.0193
17 | a [39 | 98 | 93.29 | 93 | 1477 [58 | 67 | 62.29 | 62 | 1.477 J 0.1786 | 0.2903 | 0.2353 | 0.2357 | 0.0170 f 0.1166 | 0.2034 | 0.1544 | 0.1538 | 0.0128
as [| R 9 58 | 64 | 6133 | 61 | 1065 J 27 | 33 [ 3033 | 30 | 1.065 ff 0.0992 | 0.2158 | 0.1560 | 0.1563 | 0.0171 [ 0.0198 | 0.0967 | 0.0520 | 0.0521 | 0.0104
a9 | s 77 | 90 | 84.45 | 84 | 1815 J 46 | 59 [53.45 [ 53 | 1815 J 052 | 088 | 0.7093 | 07 | 0.04719 042 | 0.68 | 0.5698 | 056 | 0.0365
20 | 7 956 | 63 | 5982 | 60 | 1124 J 25 | 32 | 2882 [ 29 | 1124 J 006 04 [02167 | 022 | 00508} 0 022 | 0.0868 [ 0.08 | 0.3276
21 | U 84 | 95 | 8953 | 90 | 1688 J 53 | 64 | 5853 | so | 1688 J 039 | 065 | 05131 | 0.51 | 0.03629 03 0.49 | 0.3777 [| 0.38 | 0.0265,
22 |v 953 | 65 | 5958 | 60 | 1593 J 22 | 34 [ 285 [ 29 | 1593 J o12 | 034 | 02136 / o21 | 00355 f 004 | 025 | 01320 | 013 | 0.2435
23 | w J 86 | 98 | 91.71 | 92 | 1.731 J 55 | 67 | 60.71 | 61 | 1.731 [0.2059 | 0.3275 | 0.2662 | 0.2655 | 0.0183 [0.1464 | 0.2704 | 0.2013 | 0.2009 | 0.0169
24 | x [55 | 65 | 59.90 | 60 | 1580 J 24 | 34 [ 2890 | 29 | 1580 ff 0.1290 | 0.2481 | 0.1873 | 0.1885 | 0.0184 [ 0.0604 | 0.1786 | 0.1238 | 0.1240 | 0.0169
25 | y [77 | 91 | 83.50 | 835 | 1838 J 46 | 60 | 5250 | 525 | 1838 f 054 | 088 | 0.7258 | 0.72 | 0.0491 f 0.42 07 | 0.5582 | 0.56 | 0.0390
26 | z 954 | 64 | 5887 | 59 | 1515 J 23 | 33 | 27.87 | 28 | 1515 J 0.06 04 | 0.2330 | 0.24 | 0.0523 J 0.02 03 | 01396 | 014 | 0.0462
27 | aA | 58 | 64 | 60.72 | 61 | 09198 J 58 | 64 | 60.72 | 61 | os198f 0.09 | 031 | 01890 / 0.19 | 0.0345 002 | 0.15 | 0.0600 | 0.06 | 0.0162
28 | BB 58 | 64 | 6072 | 61 | osi98 J 53 | 64 [6072 [ 61 | os198f 009 | 031 | 01890 / 019 | 00345 002 | 015 | 00600 | 006 | 0.0162
29 | cc J 86 | 94 | 90.29 | 90 | 1306 J 55 | 63 | 59.29 | so | 1306 J 037 | 059 | 0.4846 | 0.48 | 0.03469 03 04 | 0.3470 | 0.35 | 0.0153
30 | bb f 57 | 64 | 6081 | 61 | 1029 J 26 | 33 [ 2981 | 30 | 1029 J oo8 | o31 | 01899 | o19 | 00346 002 | 015 | 00660 | 0.06 | 0.0183
31 | ce J 55 | 64 | 59.45 | so | 1556 J 55 | 64 | 59.45 [ so | 1556 J 011 | 034 | 0.2123 | 0.21 | 0.0356 0.02 | 0.26 | 0.1273 | 0.13 | 0.0321
32 | FF 55 | 64 | 5945 | 59 | 1556 J 55 | 64 [59.45 / 59 | 1556 J o11 | 034 | 0.2123 | 021 | 0.03569 002 | 026 | 01273 | 013 | 0.0321
33 | Go J a5 | 94 | 8918 | 89 | 1569 J 54 | 63 | sais | 58 | 1569 | 04 0.64 | 0.5096 | 0.51 | 0.0364 0.3 0.46 | 0.3649 | 0.36 | 0.0241
34 | HH 54 | 65 | 5953 | 60 | 1583 f 23 | 34 [ 2853 [ 29 | 1583 J o11 | 034 | 02130 | 021 | 00353 002 | 026 | 01304 | 013 | 0.0319
13
Table 2—Results (2,000 simulations) (Cont.)
# [io I l I I I l l
Sensitivity Specificity
Min
Max
Mean
Median.
stDev
Min
Max
Mean
Median.
stDev
Le |
Leo |
Ce[e fo | 1 [oss | 1 | 005s |
[otf os _|
Po | _1 | 0000 | oaie: | 0.1352 |
Tso |
lL |
Po [1 | 0965 |
fo [1 [os7s2 | 1 | 00617 |
[-osas | 1 [oas39 | 1 _| 00568 |
[os | 1 [3852 |—1 | 00331 |
[02 | 1 | 07364 [0.7391 | 00966 02 | 0764 | 09012 | 09024 | 00226 |
os | 1 | 0.9052 | 0.9130 | 00580 |
We report all characteristics of the distribution of results for these variables (minimum, maximum, mean,
median, and standard deviation) and discuss the results. The different metrics chosen to capture performance
will be discussed. Figure 6 shows the results obtained for selection rates and error rates across all simulation
scenarios.
Runs A through G, O through T, and AA through DD (all in shadowed boxes), belong to the case of the perfectly
predictable environment. It is intersting to see that selection rates are not very different in the first block of
simulations (A—G) with the perfect environment than in the second block (H—N) with the unstable environment.
14
An exception is noted in the case of runs F and M, where a spike in selection occurs in the case of low frequency
of learning and no coordination needed. Error rates, however, in the same two initial blocks vary dramatically
(three fold). This result confirms our thinking that under conditions of perfect predictability of the environment,
lower error rates, and better performance will be achieved (proposition 1). Higher selection rates, however, can
have a negative impact on performance as this represents additional effort and cost. When asymmetrical
judgment is introduced (blocks O—P and U-Z), it is quite evident that both rates experience significant increases
and change dramatically when coordination of action is present. Also, it is interesting to see that in runs T and V,
the standard deviation of the error rate changes significantly from all other runs. An explanation for this result is
not evident to us at this time.
Selection and Error Rates
EAA :
NAb cee
a8 cope fF Gye ts « t mM nfo Paes thu v wx AA BB CC Dol EE FF GG HH
Mean (Intensity Units)
Standard Deviation (Intensity Units)
‘Simulation Scenarios (2,000 Simuation Runs each Scenario)
Selection Rate Mean —+ErrorRateMean =®-Selection Rate StDev © — Error Rate StDev
Figure 6—Selection and Error Rates Results
As presented in proposition 2, the frequency of learning influences performance. Selection rates and error rates
change with the different levels of frequency of feedback. In Figure 7, we show the impact of the different levels
of frequency on the temporal dynamics of selection rates and error rates. Noteworthy is the fact that the lower
the frequency of feedback, the higher the variability in rates experienced (see Figure 7, lower graphs). In the
graph presented in Figure 7 (lower right), a very tight behavior (compared to the others) is observed. This graph
(runs D and K) captures the high-frequency case. This result was not anticipated by the authors.
Environment and Response
0s ]
0375
E 025
PY
0.125
0
0 25 3 76 101
Time (Period)
Selection Rate :B. ——+—+— Error Rate: B ———
Selection Rate :| -——2——— Error Rate :| ——$—
15
Environment and Response Environment and Response
0s I 0s
0.375 0375
E 0.25 f E 0.25
hi
0.125 0.125
0 0
0 25 3 76 01 0 25 3 76 01
Time (Period) Time (Period)
Selection Rate :F Error Rate :F Selection Rate :) ——4+——+— Error Rate :D ss —
Selection Rate : M Error Rate :M Selection Rate :K_ ——s——2— Exot Rate :K —$<—$—
Figure 7— Effect of Frequency of Learning on Performance
The decision threshold of the judges is another good metric of performance of the process. If the judges can
learn about the distal variable and find the decision threshold that minimizes error, the detection-selection
process will yield a minimum error rate at the most acceptable cost. In Figure 8, we present the results for the
decision thresholds of Judges A and B. We also present results of variability of the results obtained. As shown in
Figure 8, in runs A through N, both judges have identical decision thresholds as these runs are parametrized for
symmetrical judges. In runs O through Z, asymmetrical judges are simulated. The results follow a particular trend
by showing the influence of coordination in decision making. The decision thresholds show a significant change
in level when coordination is present. Also, under conditions of an unstable environment, it is noteworthy to
acknowledge the significant increase in variability in the determination of the decision threshold (see standard
deviation of runs U-Z as compared to that of runs O-T). This effect is also evident when comparing runs AA-DD
(perfect environment) with runs EE—-HH (unstable environment). The fact that the decision thresholds of both
judges separate from run O forward is a function of the parametrization of the asymmetrical-judges condition.
Additional research is needed to explore the conditions under which, after a definitional split, convergence may
occur (several additional feedback mechanisms might be needed for this result to occur).
Decision Threshold
100 ? . 2
nt i “
NVAVAV AVAVANAFAWaYiANE
TUAW SYAW/auN
- VIVE MI Nd IN
E
Mean (Intensity Units)
AA 8B cc oof ee FF OGG HH
Standard Deviation (Intensity Units)
Simulation Scenarios (2,000 Simuation Runs each Scenario)
—e-JudgeA Mean —-JudgeBMean ~@-JudgeAStDev Judge B StDev
Figure 8—Decision Threshold Results
16
In detection-selection processes, two metrics are used to capture performance in a general way: sensitivity and
specificity. Sensitivity refers to the ability of the judge to correctly identify members of the signal distribution as
a proportion of the total membership. The equation for sensitivity is of the form:
1 TP.
Sensitivity, = > —
(IP + EN)
where 7P represents true-positive outcomes; FN represents false-negative outcomes; and 1 is the number of
cases examined. When sensitivity is 1, no false-negative outcomes are generated, and 100% of the members of
the signal distribution are correctly identified.
Specificity refers to the ability of the judge (or system) to correctly identify members of the noise distribution as
a percentage of the total membership. The equation for sensitivity is of the form:
A n TN,
Specificity, = 2 TG, + FR)
where 7N represents true-negative outcomes; FP represents false-positive outcomes; and n is the number of
cases examined. When specificity is 1, no false-positive outcomes are generated, and 100% of the members of
the noise distribution are correctly identified.
In Figure 9, we show the results for sensitivity and specificity for all scenarios simulated. As shown in Figure 9,
the sensitivity metric changes much more than the specificity metric as changes to environmental predictability
are introduced (see the difference in behavior between the blocks of runs A-G and H-N). An important drop in
sensitivity is clear in block H-N when compared to block A-G. Of note is the fact that low frequency of feedback
(runs F-G and M-N) seem to improve the sensitivity metric while, at the same time, degrade the specificity
metric independently of environmental predictability. Also, variability of the sensitivity metrics is higher (and
much more volatile) than that of the specificity metric across all scenario runs. This result is interesting in that
the sensitivity metric is the one that captures elements related to the signal distribution that, potentially and
depending on the context, are much more costly and valuable than members of the noise distribution
(characteristic captures with the specific metric). This result is important because under conditions of high
uncertainty and low base rates (as in the case of the identification of terrorists), small changes in detection of
members of the signal distribution (i.e., the population of terrorists) will have a very important effect on the
performance of the detection-selection system. Alternately, high levels of specificity are indicative of the ability
to avoid unnecessary inconveniences to members of the noise distribution (non-terrorists distribution). This
result, although very desirable and potentially important, might not be crucial to the mission of identification of
members of the signal distribution.
17
wan
wy, / K1/| Se
Mean (Intensity Units)
XT,
o
Lp
LI
«ea
oa \ MJ
<
=
Standard Deviation (Intensity Units)
‘Simulation Scenarios (2,000 Simuation Runs each Scenario)
SensitivityMean —Specificity Mean -®-SensitivityStDev —Specificity StDev
Figure 9—Sensitivity and Specificity Results
The 34 scenarios simulated provide a very rich source of information to further analyze how performance of
detection-selection systems can be improved. In Appendix 2, we compare and contrast different sets of runs and
scenario characterizations that will be analyzed in future research.
The dual-judge model allowed us to identify the finding that, under conditions of high uncertainty, having a
dual-judge model yields better results than found in case of using a single-judge and captures in a better way the
conditions that human judges face in detection-selection processes. However, the dual-judge process, under
conditions in which there is no coordination, has the potential to perform extremely poorly and with high levels
of variability. In multi-judge environments, it appears that coordination of action is very useful in achieving
better results. Also, contrary to our expectations for the findings, higher levels of frequency of feedback might
not generate only good results. Additional research into these results is needed.
Acknowledgements
This work was funded in part by the U.S. Department of Homeland Security.
References
Brunswik, E. 1943. Organismic achievement and environmental probability. Psychological Review, 50: 255-272.
Brunswik, E. 1956. Perception and the Representative Design of Psychological Experiments. Berkeley, CA:
University of California Press.
Egan, J. P. 1975. Signal Detection Theory and ROC Analysis. New York: Academic Press.
Erev, |., Gopher, D., Itkin, R., & Greenshpan, Y. 1995. Toward a Generalization of Signal Detection Theory to N-
Person Games: The Example of Two-Person Safety Problem. Journal of Mathematical Psychology, 39: 360-375.
Erev, |. 1998. Signal detection by human observers: A cutoff reinforcement learning model of categorization
decisions under uncertainty. Psychological Review, 105: 280-298.
18
Forrester, J. W. 1961. Industrial Dynamics. In C. Heyel (Ed.), The Encyclopedia of Management: 313-319. New
York: Reinhold Publishing Company.
Ghaffarzadegan, N. 2008. Effect of Conditional Feedback on Learning, System Dynamics Conference. Athens,
Greece.
Green, D. M. & Swets, J. A. 1966. Signal Detection Theory and Psychophysics. New York: John Wiley.
Hammond, K. R. 1996. Human Judgment and Social Policy: Irreducible Uncertainty, Inevitable Error, Unavoidable
Injustice. New York: Oxford University Press.
Hammond, K. R. 2000. Judgments Under Stress. New York: Oxford University Press.
Hammond, K. R. & Stewart, T. R. (Eds.). 2001. The Essential Brunswik: Beginnings, Explications, Applications.
New York: Oxford University Press.
Macmillan, N. A. & Creelman, C. D. 2005. Detection Theory: A User's Guide. New York: Lawrence Erlbaum
Associates.
Martinez-Moyano, I. J., Rich, E. H., Conrad, S. H., Stewart, T., & Andersen, D. F. 2006. Integrating Judgment and
Outcome Decomposition: Exploring Outcome-based Learning Dynamics, International Conference of the System
Dynamics Society. Nijmegen, The Netherlands: System Dynamics Society.
Martinez-Moyano, I. J., Conrad, S. H., & Andersen, D. F. 2007. An Outcome-Based Learning Model to Identify
Emerging Threats: Experimental and Simulation Results. Paper presented at the 40th Annual Hawaii
International Conference on System Sciences (HICSS'07), Hawaii, HI.
Martinez-Moyano, I. J., Rich, E., Conrad, S., Andersen, D. F., & Stewart, T. R. 2008. A behavioral theory of insider-
threat risks: A system dynamics approach. ACM Trans. Model. Comput. Simul., 18(2): 1-27.
Richardson, G. P. & Pugh, A. L., Ill. 1981. Introduction to System Dynamics Modeling. Waltham MA: Pegasus
Communications, Inc.
Sterman, J. D. 2000. Business Dynamics: Systems Thinking and Modeling for a Complex World. Boston MA: Irwin
McGraw-Hill.
Swets, J. 1992. The Science of Choosing the Right Decision Threshold in High-Stakes Diagnostics. American
Psychologist, 47(4): 522-532.
Weaver, E. & Richardson, G. P. 2002. Threshold Setting and the Cycling of a Decision Threshold. System
Dynamics Review, 22(1): 1-26.
19
Appendix 1—Characterization of Judges and Model Flags
Table A—Judge A Parameters
Weight
reliable judge as it cancels a stream of stochastic noise introduced to the judgment process
{a judge that assesses the evidence in a perfectly consistent way over time). A weight of 1
represents a judge that is as unstable as it can possible be given a certain distribution of
stochastic noise in its reliability of judgment (a weight of 1 lets through 100% of the
reliability noise into the judgment process). The stochastic characterization of the reliability
noise is captured elsewhere.
Variable Name Description Default Units
Value
Decision Threshold This parameter captures the initial condition of the decision threshold for Judge A. The 56 Intensity
Initial Judge A range of this variable is between 0 and 100. A threshold of 0 represents a super-vigilant
judge that will scrutinize everything, while a threshold of 100 represents a super-trusting
judge that will not trigger any action ever.
Judge A Weight of This parameter captures how important the judgment of Judge B is for Judge A’s judgment. 0 Dmal
Judgment of Judge B | This formulation generates a double judgment process in which first the judge uses
environmental information cues to make a judgment about the distal variable and then, ina
second judgment, incorporates information about the other judge’s judgment. When this
weight is 0, the judge is not using the other judge's judgment; when this weight is 1;
Judge A is using Judge B's judgment to make his judgment (disregarding his own judgment).
Influence of False This parameter captures the value that information about a false-negative outcome has on “1 | Dmni/évent
Negatives on Decision | the way in which Judge A adjusts his/her decision threshold. When the value is zero, no
Threshold Judge A adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of False This parameter captures the value that information about a false-positive outcome has on A Dmni/Event
Positives on Decision _| the way in which Judge A adjusts his/her decision threshold. When the value is zero, no
Threshold Judge A adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of True This parameter captures the value that information about a true-positive outcome has on 0 Dmni/Event
Positives on Decision _ | the way in which Judge A adjusts his/her decision threshold. When the value is zero, no
Threshold Judge A adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of True This parameter captures the value that information about a true-negative outcome has on 0 | Dmnievent
Negatives on Decision | the way in which Judge A adjusts his/her decision threshold. When the value is zero, no
Threshold Judge A adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Judge A Weight of This parameter captures how important it is for Judge A Information Cue 1 in the (/6) Dmanl
Information Cue 1 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 1is not used by Judge A. The weights in this model are normalized to 1.
Judge A Weight of This parameter captures how important it is for Judge A Information Cue 2 in the (2/6) Dmal
Information Cue 2 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 2 is not used by Judge A. The weights in this model are normalized to 1.
Judge A Weight of This parameter captures how important it is for Judge A Information Cue 3 in the G/6) Dmal
Information Cue 3 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 3 is not used by Judge A. The weights in this model are normalized to 1.
Judge A Bias This parameter captures a bias in judgment for Judge A. In theory, bias can range between 0 Intensity
-100 and +100. When this variable is 0, it represents an unbiased judge. The judgment of
likelihood is a variable that ranges from 0 to 100, and as the bias component has an
additive effect on judgment (capped at the frontiers of its range), the bias parameter can
make the judgment extreme in the range independently of the other components of
judgment. This parameter captures the possibility of having extremely biased judges that,
independent of the evidence provided to them, will judge the likelihood of a phenomenon
according to their prior beliefs.
Judge A Reliability This parameter captures how reliable Judge A is. A zero (0) weight represents a perfectly O21 Dmanl
20
Judge A Reliability
Error StdD
This parameter captures the standard deviation of Judge A’s reliability noise stream. The
judgment of the distal variable varies between 0 and 100. Therefore, the higher the
standard deviation for the noise stream, the higher the uncertainty and reliability of the
judgment process. This parameter captures (in part) the inherent stability and reliability of
the cognitive process of the judge when making repeated judgments. When this parameter
is 0, the judge is perfectly reliable (always uses the information cues in the exact same
way). When this parameter is 100 (when paired with minimum and maximum of -100 and
+100 in a truncated distribution and with a weight of 1 for the judge's reliability), the judge
is completely erratic and unreliable in his/her judgment. Articulated differently, the
judgment is random, independent of the information cues about the distal variable.
10
Intensity
Judge A Reliability
Error Min
This parameter captures the minimum value for Judge A’s reliability noise stream. This
value truncates the results of the stochastic noise to this minimum value. The noise stream
is determined to be centered on 0 (also a parameter). The combination of this parameter
with the Error Max parameter generates the range of variability of the reliability noise
stream for Judge A.
Intensity
Judge A Reliability
Error Max
This parameter captures the maximum value for Judge A’s reliability noise stream. This
value truncates the results of the stochastic noise to this maximum value. The noise stream
is determined to be centered on 0 (also a parameter). The combination of this parameter
with the Error Min parameter generates the range of variability of the reliability noise
stream for Judge A.
30
Intensity
21
Table B—Judge B Parameters
Variable Name Description Default Units
Value
Decision Threshold This parameter captures the initial condition of the decision threshold for Judge B. The 56 Intensity
Initial Judge B range of this variable is between 0 and 100. A threshold of 0 represents a super-vigilant
judge that will scrutinize everything, while a threshold of 100 represents a super-trusting
judge that will not trigger any action ever.
Judge B Weight of This parameter captures how important the judgment of Judge A is for Judge B’s judgment. 0 Dmal
Judgment of Judge A __| This formulation generates a double judgment process in which first the judge uses
environmental information cues to make a judgment about the distal variable and then, in
a second judgment, incorporates information about the other judge’s judgment. When this
weight is 0, the judge is not using the other judge’s judgment; when this weight is 1;
Judge B is using Judge A’s judgment to make his judgment (disregarding his own
judgment).
Influence of False This parameter captures the value that information about a false-negative outcome has on a Dmni/Event
Negatives on Decision | the way in which Judge B adjusts his/her decision threshold. When the value is zero, no
Threshold Judge B adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of False This parameter captures the value that information about a false-positive outcome has on 1 Dmni/Event
Positives on Decision _|_the way in which Judge B adjusts his/her decision threshold. When the value is zero, no.
Threshold Judge B adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of True This parameter captures the value that information about a true-positive outcome has on 0 Dmni/Event
Positives on Decision _|_ the way in which Judge B adjusts his/her decision threshold. When the value is zero, no
Threshold Judge B adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Influence of True This parameter captures the value that information about a true-negative outcome has on 0 Dmni/Event
Negatives on Decision | the way in which Judge B adjusts his/her decision threshold. When the value is zero, no.
Threshold Judge B adjustment is made to the threshold. When the value is positive, the threshold is adjusted
up (making the judge’s behavior more trusting). When the value is negative, the threshold
is adjusted down (making the judge’s behavior more vigilant).
Judge B Weight of This parameter captures how important it is for Judge B Information Cue 1 in the (1/6) Dmnl
Information Cue 1 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 1 is not used by Judge B. The weights in this model are normalized to 1.
Judge B Weight of This parameter captures how important it is for Judge B Information Cue 2 in the (2/6) Dmal
Information Cue 2 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 2 is not used by Judge B. The weights in this model are normalized to 1.
Judge B Weight of This parameter captures how important it is for Judge B Information Cue 3 in the G/6) Dmal
Information Cue 3 conformation of the judgment of the distal variable. When this weight is 0, Information
Cue 3 is not used by Judge B. The weights in this model are normalized to 1.
Judge B Bias This parameter captures a bias in judgment for Judge B. In theory, bias can range between 0 Intensity
~100 and +100. When this variable is 0, it represents an unbiased judge. The judgment of
likelihood is a variable that ranges from 0 to 100, and as the bias component has an
additive effect on judgment (capped at the frontiers of its range), the bias parameter can
make the judgment extreme in the range independently of the other components of
judgment. This parameter captures the possibility of having extremely biased judges that,
independent of the evidence provided to them, will judge the likelihood of a phenomenon
according to their prior beliefs.
Judge B Reliability This parameter captures how reliable Judge B is. A zero (0) weight represents a perfectly 02 Dmal
Weight
reliable judge as it cancels a stream of stochastic noise introduced to the judgment process
(a judge that assesses the evidence in a perfectly consistent way over time). A weight of 1
represents a judge that is as unstable as it can possible be, given a certain distribution of
stochastic noise in its reliability of judgment (a weight of 1 lets through 100% of the
reliability noise into the judgment process). The stochastic characterization of the
reliability noise is captured elsewhere.
22
Judge B Reliability Error
stdD
This parameter captures the standard deviation of Judge B’s reliability noise stream. The
judgment of the distal variable varies between 0 and 100. Therefore, the higher the
standard deviation for the noise stream, the higher the uncertainty and reliability of the
judgment process. This parameter captures (in part) the inherent stability and reliability of
the cognitive process of the judge when making repeated judgments. When this parameter
is 0, the judge is perfectly reliable (always uses the information cues in the exact same
way). When this parameter is 100 (when paired with minimum and maximum of -100 and
+100 in a truncated distribution and with a weight of 1 for the judge's reliability), the judge
is completely erratic and unreliable in his/her judgment. Articulated differently, the
judgment is random, independent of the information cues about the distal variable.
10
Intensity
Judge B Reliability Error
Min
This parameter captures the minimum value for Judge B’s reliability noise stream. This
value truncates the results of the stochastic noise to this minimum value. The noise stream
is determined to be centered on 0 (also a parameter). The combination of this parameter
with the Error Max parameter generates the range of variability of the reliability noise
stream for Judge B.
Intensity
Judge B Reliability Error
Max
This parameter captures the maximum value for Judge B’s reliability noise stream. This
value truncates the results of the stochastic noise to this maximum value. The noise
stream is determined to be centered on 0 (also a parameter). The combination of this
parameter with the Error Min parameter generates the range of variability of the reliability
noise stream for Judge B.
30
Intensity
23
Table C—Flags
Variable Name Description Default Units
Value
‘Same Size Change for When this switch is on (1), the same size of change in the decision 0 Dmnl
Judges Flag threshold jump is used for both judges to create identical behavior.
Same Judge Reliability Flag_| When this switch is on (1), the same reliability is used for both judges to 0 Dmnl
create identical behavior.
‘Same Decision Threshold _| If this flag is O, the judges use their own decision threshold in the decision 0 Dmnt
Flag process. When this flag is 1, Judge B uses Judge A's decision threshold in
his/her decision process, disregarding his own value matrix.
Judges Involved Flag When this flag is 1, just one judge is active (Judge A). When this flag is 2, 2 Dmnt
two judges are active (A & B).
‘Agreement for action This flag implements the "judges-agreement-is-needed-for-action” policy 0 Dmnl
needed Policy Flag
Dynamic Threshold When this flag is 0, the threshold is static (no change). When the flag is 1, 1 Dmnt
activation Flag for Judge A | the threshold is dynamic (changes as other variables change in the
simulation).
Dynamic Threshold When this flag is 0, the threshold is static (no change). When the flag is 1, 1 Dmnl
activation Flag for Judge B_ | the threshold is dynamic (changes as other variables change in the
simulation).
Weight of Unpredictability | This flag captures how inherently predictable the environment is. A zero (0) 0 Dmnl
of the Environment
weight represents a perfectly predictable environment as it cancels a
stream of stochastic noise introduced to the criterion model. A weight of 1
represents an environment that is as unstable as it can possible be, given a
certain distribution of stochastic noise in its predictability. The stochastic
characterization of the predictability noise is captured elsewhere
(0.866025404 for 50%, 0.6 for 80% predictability, Perfect 0).
24
Appendix 2—Additional Graphs of Results of Runs
Decision Threshold for Judge A Error Rate
70 06
65 0.45
5 60 E 03
z rv oe
eee
55
0.15
50
0 25 31 76 01 °
Tame (Period) 0 8 3 76 701
Decision Threshold for Judge A: A| +} 49 Time (Period)
Decision Threshold for Judge A :B Error Rate :A Error Rate :C —-——oo
Decision Threshold for Judge A :C Error Rate :B
Sensitivity Specificity
1 7 T a 1 SSS
075 0.75
Eos Eos
4 4
025 0.25
° °
0 25 3 76 01 0 25 3 76 701
Tame (Period) ‘Time (Period)
Sensitivity: 4-4-4 Sensitivity :C Specificity :A Specificity :C —-—s—s—=—
Sensiivty:B —2—2—2—=
Specificity :B —2—2—2—=
Figure 10—Symmetrical Judges Perfect Environment
Decision Threshold for Judge A. Error Rate
70 06
65 o4s
30 Zoos
~ cl + .
55
ous
0
0 25 SI 76 101 0
Tine (Period) iy 35 31 76 701
Decision Threshold for Judge A: H/ t+ pp Tine (Period)
Decsion Threshold for Jxge A :1 Error Rate: ror Rate :} ee
Decision Threshok! for Jule A 2} Error Rae :1
25
Sensitivity Specificity
1 1
075 5 0.75
as Eos
025 025
° 0
o 25 3I 76 iol ty 25 3 76 ToL
Time (Period) Time (Period)
Sensitivity Sensi :J Specificity :H Speciiciy:) —e—e—o—o
Sensitivity :1 ——2—2—2—2
Specificity :1 222 2
Figure 11—Symmetrical Judges Unstable Environment
Selection Rate
Error Rate
1 1
ors o7s
Eos Zoos
02s oas
0 o
0 ra 7 73 TOI 0 ra si 7 Tol
‘Time (Period) ‘Time (Period)
Selection Rate —@———s———>—
Sensitivity Specificity
1 T 1
075 0.75
os J os
025 025
° 0
o 25 3 76 71 a 25 3 76 Toi
‘Time (Period) ‘Time (Period)
Sensitivity :A Sensitivity :0 Spicy :A Speciity ; —t——
Senstivay Sonstivay P Specy Spesity
Seativty C——
Specitcity ;C ———
Figure 12—Asymmetrical Judges Perfect Environment
26
Selection Rate
Error Rate
1 1
075 0.75
os 2 os
a cage em |
0.25 0.25
° °
0 25 3 76 01 0 25 31 76 01
Tame (Period) ‘Time (Period)
Selection Rate :) —+—+— Selection Rate :U Error Rate :O Enor Rate:U ———
Selection Rate :P 2——2——2— Section Rate :V Error Rate :P Error Rate: V+
Sensitivity Specificity
1 1
ons | oe o7s
Eos Eos
4 4
0.25 0.25
° 0
° 25 ST 76 701 0 25 31 76 Tot
Tine (Period) Time (Period)
Semsitviy :0 ++ — Sensitivity :U Specitcty :0 Specifcty:U =» —
Sensitivity :P- —2——a a Sensitivity :V Specicty :P Speciicity:V —<——e
Figure 13—Asymmetrical Judges Perfect/Unstable Environment
Selection Rate Error Rate
1 1
075 0.75
E os 2 os
4 4
0.25 0.25
’ ees geal oe a
0 25 3 76 01 0 25 st 76 01
Tame (Period) Time (Period)
Selection Rate :AA —+——+— Section Rate :CC —-—o——
Selection Rate : BB. ——2——=— Selection Rate : DD
Figure 14—Symmetrical/Asymmetrical Judges Perfect Environment
27
Selection Rate
Error Rate
1
ors ons
os Boos
o2s as
° Pr UN Rs ea ld
5 rc ar 76 1 Q ra 31 76 ToL
Tie (Per) Tine (Pe)
Scksioa Raw EE —+—+— Schein Rae:GG —#—#— = EmrRae:BE 4+ + Eras GG ee
SceabeRe FF se Sebtin Ra sE Re BF
Exor Rate :HH ———_——
Figure 15—Symmetrical/Asymmetrical Judges Unstable Environment
Selection Rate Error Rate
06 02
04s 0.5
E03 E ol
& &
os 0.05
° 0
o 25 3 76 Tol 0 25 31 76 To
Tame (Period) Time (Period)
Selection Rate : AA. —+——+— Selection Rate :EE ss Error Rate : AA. ——+——+— Error Rate :EE $s»
Selection Rate : BB. ——2——=>— Selection Rate : FE ——-——— Error Rate :BB 22 =~ Error Rate :FF ——————
Sensitivity Specificity
1 ee 1
075 0.75
E os Eos
& &
025 0.25
° 0
o 25 3 76 Tol iy 25 3I 76 To
Tame (Period) Time (Period)
Sensvty:AA| +++ — Senstvy EE > Specitcty :EE =e
Sensiivty:BB_ —2—>——>— Sensitivity :FF ——+—— Specitcty :FF
Figure 16—Symmetrical Judges with additional Information Perfect/Unstable Environment
28
Selection Rate Error Rate
ons 075
0s Eos
02s 025
; | ET
Q 3 3 76 v1 3 25 3 76 v1
Tine Pon) Tine Ps)
Sebvionfae:CC —+—+— Sette: ——#—ErorRae:CC + rfatesg ——e—9—
Schabefin:DD ee Sete. ERD eee rf)
Sensitivity Specificity
‘ Ce 1 = a
ons 073
a 3 ———
Boos Boos
i i
02s 025
0 °
Q 2 3 76 v1 0 25 ar 76 vt
‘Time (Period) ‘Time (Period)
Seniniy:0C +++ Semis @—e—e— Specs iy 100
Sesthty:DD = Sent Spay b> ey
Figure 17—Asymmetrical Judges with additional Information Perfect/Unstable Environment
29
