Friam

Validation

Classic

List

Threaded

2 messages Options

Friam mailing list

Validation

This is a multi-part message in MIME format.

------=_NextPart_000_0008_01C31554.26DF9950
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi all,

The lecture notes Keith refers to highlight these different types of
validation:
* Internal validity - error free code
* Parameter validity - parameters match
* Process validity - processes fits
* Face validity - right type of things
* Pattern validity - pattern matches observed
* Value validity - values match
* Theoretical validity - theory fits
(Carley, 2002)

We also have Owen's "docking" validity. Is this a neat way of tying =
together
the first three in the above list? Are there any other type of validity =
that
leap to people's attention?

Robert

P.S. I might be inviting a chicken-and-egg argument here, but I see
validation as being just one of the things that a methodology should
address. For me, the methodology should also cover:
* Ontological aspects (Carl - I understand this as defining the
objects & concepts that can exist, and what relationships can exist =
between
them. Is that about right?).
* Appropriate level of prescription (generic vs. specific (e.g.
template- or design-based).
* "Philosophical" aspects (sounds a bit pretentious but I can't think
of a better label. Basically do we need to assume an instrumentalist =
view, a
positive realist view, etc or does it even matter? Personally, I think =
it
does - I'm sure this will inform discussion about degree of =
correspondence
between reality and the (micro-level) rules embedded in an ABM).

------=_NextPart_000_0008_01C31554.26DF9950
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.0.4630.0">
<TITLE>Validation</TITLE>
</HEAD>
<BODY>


<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Hi all,</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">The lecture notes Keith refers =
to highlight these different types of validation:</FONT>

<UL>
<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Internal validity - error free =
code</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Parameter validity - parameters =
match</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Process validity - processes =
fits</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Face validity - right type of =
things</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Pattern validity - pattern =
matches observed</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Value validity - values =
match</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Theoretical validity - theory =
fits</FONT></LI>
</UL>
<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">(Carley, 2002)</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">We also have Owen's =
"docking" validity. Is this a neat way of tying together the =
first three in the above list? Are there any other type of validity that =
leap to people's attention?</FONT></P>

<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Robert</FONT>
</P>

<P><FONT SIZE=3D2 FACE=3D"Trebuchet MS">P.S. I might be inviting a =
chicken-and-egg argument here, but I see validation as being just one of =
the things that a methodology should address. For me, the methodology =
should also cover:</FONT></P>

<UL>
<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Ontological aspects (Carl - I =
understand this as defining the objects & concepts that can exist, =
and what relationships can exist between them. Is that about =
right?).</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">Appropriate level of =
prescription (generic vs. specific (e.g. template- or =
design-based).</FONT></LI>

<LI><FONT SIZE=3D2 FACE=3D"Trebuchet MS">"Philosophical" =
aspects (sounds a bit pretentious but I can't think of a better label. =
Basically do we need to assume an instrumentalist view, a positive =
realist view, etc or does it even matter? Personally, I think it does - =
I'm sure this will inform discussion about degree of correspondence =
between reality and the (micro-level) rules embedded in an =
ABM).</FONT></LI>
</UL>
</BODY>
</HTML>
------=_NextPart_000_0008_01C31554.26DF9950--

Friam mailing list

Validation

This is a multi-part message in MIME format.

------=_NextPart_000_0004_01C31565.53DE65B0
Content-Type: text/plain;
charset="us-ascii"
Content-Transfer-Encoding: 7bit

Validation >>Ontological aspects (Carl - I understand this as defining the
objects & concepts that can exist,
>>and what relationships can exist between them. Is that about right?).

Seems right to me. Though be careful that the definition should not exclude
"processes".

The ontolology validation issue is situated in the client relationship. Far
from being fixed, our
understanding of the ontology of the things we are being asked to model
often changes frequently
over the course of the effort. It seldom works to force ontology creation
into an early discovery
phase of the project; it's an ongoing thing. Worse, when the "Face
Validity" (hope I'm using that
term right) changes, it affects all the other types of validation. It seems
that
some form of annoation-based ontology mechanism might be in order, so that
the modelers
can "mark-up" the ontology in some easy and formal way that merges
gracefully with the
other kinds of validation, such that the complete "suite" of validation
techniques doesn't have
to be entirely redone on each markup. (This is kind of a grail, I don't
know how to do this).

Separate from ontology issues, I think the "docking" validation is
interesting and it says a lot
if you can do it, but it would need to be automated somehow if we were to
expect it to be
consistently applied.

Strongly typed languages and some form of units formalism might be useful,
though my experience
has been that modelers tend to find these overly constraining. A validation
methodology will
do no good if nobody uses it.

Robert, could you elaborate on what you mean by
"Appropriate level of prescription (generic vs. specific (e.g. template- or
design-based)."?

Thanks,
Carl

-----Original Message-----
From: [hidden email] [mailto:[hidden email]]On Behalf Of
Robert Holmes
Sent: Thursday, May 08, 2003 11:23 AM
To: [hidden email]
Subject: [FRIAM] Validation

Hi all,

The lecture notes Keith refers to highlight these different types of
validation:

a.. Internal validity - error free code
b.. Parameter validity - parameters match
c.. Process validity - processes fits
d.. Face validity - right type of things
e.. Pattern validity - pattern matches observed
f.. Value validity - values match
g.. Theoretical validity - theory fits
(Carley, 2002)

We also have Owen's "docking" validity. Is this a neat way of tying
together the first three in the above list? Are there any other type of
validity that leap to people's attention?

Robert

P.S. I might be inviting a chicken-and-egg argument here, but I see
validation as being just one of the things that a methodology should
address. For me, the methodology should also cover:

a.. Ontological aspects (Carl - I understand this as defining the
objects & concepts that can exist, and what relationships can exist between
them. Is that about right?).
b.. Appropriate level of prescription (generic vs. specific (e.g.
template- or design-based).
c.. "Philosophical" aspects (sounds a bit pretentious but I can't think
of a better label. Basically do we need to assume an instrumentalist view, a
positive realist view, etc or does it even matter? Personally, I think it
does - I'm sure this will inform discussion about degree of correspondence
between reality and the (micro-level) rules embedded in an ABM).

------=_NextPart_000_0004_01C31565.53DE65B0
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD><TITLE>Validation</TITLE>
<META http-equiv=3DContent-Type content=3D"text/html; =
charset=3Dus-ascii">
<META content=3D"MSHTML 6.00.2715.400" name=3DGENERATOR></HEAD>
<BODY>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003> =20
>><FONT color=3D#000000><FONT face=3D"Trebuchet MS">Ontological =
aspects (Carl=20
- I understand this as defining the objects & concepts that can =
exist,=20
</FONT></FONT></SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003><FONT=20
color=3D#000000><FONT face=3D"Trebuchet MS">  >>and what =
relationships=20
can exist between them. Is that about right?).</FONT><FONT=20
face=3D"Times New Roman" size=3D3> </FONT></FONT></SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>Seems=20
right to me.  Though be careful that the definition should not =
exclude=20
"processes".</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>The=20
ontolology validation issue is situated in the client =
relationship.  Far=20
from being fixed, our</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>understanding of the ontology of the things =
we are=20
being asked to model often changes frequently</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>over=20
the course of the effort.  It seldom works to force ontology =
creation into=20
an early discovery </SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>phase=20
of </SPAN></FONT><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>the project; it's an =
ongoing thing.  Worse,=20
when the "Face Validity" (hope I'm using that</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>term=20
right) changes, it affects all the other types of validation.  It =
seems=20
that</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>some=20
form of annoation-based ontology mechanism might be in order, =
 so that=20
the modelers</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>can=20
"mark-up" the ontology in some easy and formal way that merges =
gracefully with=20
the</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>other=20
kinds of validation, such that the complete "suite" of validation =
techniques=20
doesn't have </SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>to be=20
entirely redone on each markup.  (This is kind of a grail, I don't =
know how=20
to do this).</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>Separate from ontology issues, I think the =
"docking"=20
validation is interesting and it says a lot</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>if you=20
can do it,  but it would need </SPAN></FONT><FONT face=3DArial =
color=3D#0000ff=20
size=3D2><SPAN class=3D100074618-08052003>to be automated somehow if we =
were to=20
expect it to be </SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>consistently applied.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>Strongly typed languages and some form of =
units=20
formalism might be useful, though my experience</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>has=20
been that modelers tend </SPAN></FONT><FONT face=3DArial color=3D#0000ff =

size=3D2><SPAN class=3D100074618-08052003>to find these overly =
constraining.  A=20
validation methodology will</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003>do no=20
good if nobody uses it.</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003>Robert, could you elaborate on what you mean =
by=20
</SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003><FONT=20
face=3D"Trebuchet MS">"<FONT size=3D2>Appropriate level of prescription =
(generic vs.=20
specific (e.g. template- or =
design-based)."?</FONT></FONT></SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN=20
class=3D100074618-08052003></SPAN></FONT> </DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003><FONT=20
face=3D"Trebuchet MS">Thanks,</FONT></SPAN></FONT></DIV>
<DIV><FONT face=3DArial color=3D#0000ff size=3D2><SPAN =
class=3D100074618-08052003><FONT=20
face=3D"Trebuchet MS">Carl</FONT> </DIV>
<DIV></SPAN></FONT> </DIV>
<BLOCKQUOTE>
<DIV class=3DOutlookMessageHeader dir=3Dltr align=3Dleft><FONT =
face=3DTahoma=20
size=3D2>-----Original Message-----<BR><B>From:</B> =
[hidden email]=20
[mailto:[hidden email]]<B>On Behalf Of </B>Robert=20
Holmes<BR><B>Sent:</B> Thursday, May 08, 2003 11:23 AM<BR><B>To:</B>=20
[hidden email]<BR><B>Subject:</B> [FRIAM] =
Validation<BR><BR></FONT></DIV>
<P><FONT face=3D"Trebuchet MS" size=3D2>Hi all,</FONT> </P>
<P><FONT face=3D"Trebuchet MS" size=3D2>The lecture notes Keith refers =
to=20
highlight these different types of validation:</FONT>=20
<UL>
<LI><FONT face=3D"Trebuchet MS" size=3D2>Internal validity - error =
free=20
code</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Parameter validity - =
parameters=20
match</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Process validity - =
processes=20
fits</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Face validity - right type =
of=20
things</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Pattern validity - pattern =
matches=20
observed</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Value validity - values =
match</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Theoretical validity - =
theory=20
fits</FONT> </LI></UL>
<P><FONT face=3D"Trebuchet MS" size=3D2>(Carley, 2002)</FONT> </P>
<P><FONT face=3D"Trebuchet MS" size=3D2>We also have Owen's "docking" =
validity. Is=20
this a neat way of tying together the first three in the above list? =
Are there=20
any other type of validity that leap to people's attention?</FONT></P>
<P><FONT face=3D"Trebuchet MS" size=3D2>Robert</FONT> </P>
<P><FONT face=3D"Trebuchet MS" size=3D2>P.S. I might be inviting a =
chicken-and-egg=20
argument here, but I see validation as being just one of the things =
that a=20
methodology should address. For me, the methodology should also=20
cover:</FONT></P>
<UL>
<LI><FONT face=3D"Trebuchet MS" size=3D2>Ontological aspects (Carl - =
I=20
understand this as defining the objects & concepts that can =
exist, and=20
what relationships can exist between them. Is that about =
right?).</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>Appropriate level of =
prescription=20
(generic vs. specific (e.g. template- or design-based).</FONT>=20
<LI><FONT face=3D"Trebuchet MS" size=3D2>"Philosophical" aspects =
(sounds a bit=20
pretentious but I can't think of a better label. Basically do we =
need to=20
assume an instrumentalist view, a positive realist view, etc or does =
it even=20
matter? Personally, I think it does - I'm sure this will inform =
discussion=20
about degree of correspondence between reality and the (micro-level) =
rules=20
embedded in an ABM).</FONT> </LI></UL></BLOCKQUOTE></BODY></HTML>

------=_NextPart_000_0004_01C31565.53DE65B0--