Friam

Simulation and policy-making

Classic

List

Threaded

10 messages Options

Robert Holmes

Simulation and policy-making

Here's a quote from a letter in this week's Economist about how to use big
economic models:

The root of many problems lies not in the [economic] models themselves but
in the way in which they are used. Too often we ask "What will happen?",
trapping us into the mug's game of prediction, when the real question should
be: "Given that we cannot predict, what is our best move today?"...Instead
of determining the "best" model that solves optimal strategies we should
instead seek the most "robust" model that achieves a given level of
"goodness" across myriad models and uses assumptions consistent with known
facts. My colleagues and I use such methods to address intractable policy
issues fraught with arguments over which model is "right", what assumptions
are valid and what is the nature of the good? This method makes the decision
to be informed part of the analysis itself and the results are more readily
accepted by policymakers.

That was from a senior economist at RAND. So it looks as if at least some
members of the current military-industrial complex are using simulations in
a sensible and valid way (certainly with more sense and validity than a lot
of the academic papers I see).

So if 'valid' simulations are being used to give the 'wrong' answers, what
does that tell us about simulation? Is there ever any hope of objectivity
(I'll give away the answer to that: no) or do all social simulations -
political or economic - inevitably reflect the prejudices of their author or
funder?

Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/friam_redfish.com/attachments/20060808/8319a729/attachment.html

Frank Wimberly

Simulation and policy-making

Lately I have been doing simulations of wildfires. The program that I
have been using (Farsite) is based on first principles of fire physics.
Assuming that the inputs are correct (elevation, slope, aspect, fuel,
canopy, wind, temperature, humidity and a few others) I suspect the
estimate of the fire area as a function of time, and even the footprint
represents a fairly accurate prediction.

Maybe there is a definition of "complex system" that is related to the
question of what kinds of predictions simulations of them can yield?

Frank

---
Frank C. Wimberly
140 Calle Ojo Feliz??????????????(505) 995-8715 or (505) 670-9918 (cell)
Santa Fe, NM 87505???????????wimberly3 at earthlink.net
-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Robert Holmes
Sent: Tuesday, August 08, 2006 1:17 PM
To: FRIAM
Subject: [FRIAM] Simulation and policy-making

Here's a quote from a letter in this week's Economist about how to use
big economic models:
The root of many problems lies not in the [economic] models themselves
but in the way in which they are used. Too often we ask "What will
happen?", trapping us into the mug's game of prediction, when the real
question should be: "Given that we cannot predict, what is our best move
today?"...Instead of determining the "best" model that solves optimal
strategies we should instead seek the most "robust" model that achieves
a given level of "goodness" across myriad models and uses assumptions
consistent with known facts. My colleagues and I use such methods to
address intractable policy issues fraught with arguments over which
model is "right", what assumptions are valid and what is the nature of
the good? This method makes the decision to be informed part of the
analysis itself and the results are more readily accepted by
policymakers.

That was from a senior economist at RAND. So it looks as if at least
some members of the current military-industrial complex are using
simulations in a sensible and valid way (certainly with more sense and
validity than a lot of the academic papers I see).

So if 'valid' simulations are being used to give the 'wrong' answers,
what does that tell us about simulation? Is there ever any hope of
objectivity (I'll give away the answer to that: no) or do all social
simulations - political or economic - inevitably reflect the prejudices
of their author or funder?

Robert

Marcus G. Daniels-3

Simulation and policy-making

In reply to this post by Robert Holmes

Quoting Robert Holmes <robert at holmesacosta.com>:

> So if 'valid' simulations are being used to give the 'wrong' answers, what
> does that tell us about simulation? Is there ever any hope of objectivity
> (I'll give away the answer to that: no) or do all social simulations -
> political or economic - inevitably reflect the prejudices of their author or
> funder?

Validated simulations, by definition, reproduce something that the authors (or
funders) deem relevant as a performance metric. But that's not a problem with
models or simulations, assuming the metrics are documented. If the authors or
funders are prone to choosing easy, low dimensional things to fit, they just
need to be more ambitious.

Robert Holmes

Simulation and policy-making

Oh I thank RAND are probably plenty ambitious in what they simulate for the
US govt. Just check out their research areas:
http://www.rand.org/research_areas/

Robert

On 8/8/06, mgd at santafe.edu <mgd at santafe.edu> wrote:

>
> Quoting Robert Holmes <robert at holmesacosta.com>:
>
> > So if 'valid' simulations are being used to give the 'wrong' answers,
> what
> > does that tell us about simulation? Is there ever any hope of
> objectivity
> > (I'll give away the answer to that: no) or do all social simulations -
> > political or economic - inevitably reflect the prejudices of their
> author or
> > funder?
>
> Validated simulations, by definition, reproduce something that the authors
> (or
> funders) deem relevant as a performance metric. But that's not a problem
> with
> models or simulations, assuming the metrics are documented. If the
> authors or
> funders are prone to choosing easy, low dimensional things to fit, they
> just
> need to be more ambitious.
>
> ============================================================
> FRIAM Applied Complexity Group listserv
> Meets Fridays 9a-11:30 at cafe at St. John's College
> lectures, archives, unsubscribe, maps at http://www.friam.org
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/friam_redfish.com/attachments/20060808/b577b087/attachment.html

Douglas Roberts-2

Simulation and policy-making

Re: simulation and policy-making, a project that my group is working on at
the request of the current Washington administration is helping to do just
that. At the request of a consortium of representatives from the White
House, Dept of Treasury, DHS, Dept. of State, and a few other cabinet-level
political types, we have run numerous simulation experimental designs to
establish the bounds of the effectiveness of various intervention strategies
for containing an H5N1 pandemic, should it occur in the US. We are using
three simulation codes: EpiSims, Epicast, and one from the Imperial College
in the UK. The name of the project is "Models of Infectious Disease Agent
Study" (MIDAS), and it is funded by NIH. See

http://www.hsph.harvard.edu/press/releases/press02202006.html and
http://usinfo.state.gov/gi/Archive/2005/Aug/08-339612.html

or do a google search on "MIDAS bird flu policy" for more info.

--Doug

On 8/8/06, Robert Holmes <robert at holmesacosta.com> wrote:

>
> Oh I thank RAND are probably plenty ambitious in what they simulate for
> the US govt. Just check out their research areas:
> http://www.rand.org/research_areas/
>
> Robert
>
>
> On 8/8/06, mgd at santafe.edu <mgd at santafe.edu> wrote:
> >
> > Quoting Robert Holmes <robert at holmesacosta.com>:
> >
> > > So if 'valid' simulations are being used to give the 'wrong' answers,
> > what
> > > does that tell us about simulation? Is there ever any hope of
> > objectivity
> > > (I'll give away the answer to that: no) or do all social simulations -
> > > political or economic - inevitably reflect the prejudices of their
> > author or
> > > funder?
> >
> > Validated simulations, by definition, reproduce something that the
> > authors (or
> > funders) deem relevant as a performance metric. But that's not a
> > problem with
> > models or simulations, assuming the metrics are documented. If the
> > authors or
> > funders are prone to choosing easy, low dimensional things to fit, they
> > just
> > need to be more ambitious.
> >
> > ============================================================
> > FRIAM Applied Complexity Group listserv
> > Meets Fridays 9a-11:30 at cafe at St. John's College
> > lectures, archives, unsubscribe, maps at http://www.friam.org
> >
>
>
> ============================================================
> FRIAM Applied Complexity Group listserv
> Meets Fridays 9a-11:30 at cafe at St. John's College
> lectures, archives, unsubscribe, maps at http://www.friam.org
>
>

--
Doug Roberts, RTI International
droberts at rti.org
doug at parrot-farm.net
505-455-7333 - Office
505-670-8195 - Cell
-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/friam_redfish.com/attachments/20060808/c79b7e63/attachment-0001.html

McNamara, Laura A

Simulation and policy-making

Computational social science doesn't lend itself to V&V the way that physics-based mod-sim does, so creativity in V&V is required.

But when it comes to policy making, then, the question is, "What does it mean to use modeling and simulation tools appropriately in producing knowledge that will feed into policy decisions?" The answer to this question is going to vary depending on the time scale for the decision (longer term planning vs. shorter term, highly tactical decision making).There are probably lots of other issues as well that I'm not thinking of because I really need another cup of coffee after a long night of lightning, thunder and terrified dogs. That said, I think what Doug described below - experimental design, different modeling and simulation packages, expertise and care - is a great start.

Those of you attending FRIAM on the 18th of August... I'll be there with my Sandia buddy, Tim Trucano, and we're thinking about models, V&V, uncertainty, and policy-making pretty carefully these days because of some work we're pursuing with the Department of Defense. We'd love to engage in a good solid discussion about this topic with the FRIAM brain trust, so bring your ideas along.

Laura

-----Original Message-----
From: [hidden email] on behalf of Douglas Roberts
Sent: Tue 8/8/2006 7:40 PM
To: The Friday Morning Applied Complexity Coffee Group
Subject: Re: [FRIAM] Simulation and policy-making

Re: simulation and policy-making, a project that my group is working on at
the request of the current Washington administration is helping to do just
that. At the request of a consortium of representatives from the White
House, Dept of Treasury, DHS, Dept. of State, and a few other cabinet-level
political types, we have run numerous simulation experimental designs to
establish the bounds of the effectiveness of various intervention strategies
for containing an H5N1 pandemic, should it occur in the US. We are using
three simulation codes: EpiSims, Epicast, and one from the Imperial College
in the UK. The name of the project is "Models of Infectious Disease Agent
Study" (MIDAS), and it is funded by NIH. See

http://www.hsph.harvard.edu/press/releases/press02202006.html and
http://usinfo.state.gov/gi/Archive/2005/Aug/08-339612.html

or do a google search on "MIDAS bird flu policy" for more info.

--Doug

On 8/8/06, Robert Holmes <robert at holmesacosta.com> wrote:

--
Doug Roberts, RTI International
droberts at rti.org
doug at parrot-farm.net
505-455-7333 - Office
505-670-8195 - Cell

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 5037 bytes
Desc: not available
Url : /pipermail/friam_redfish.com/attachments/20060809/055d5c70/attachment-0001.bin

Robert Holmes

Simulation and policy-making

On 8/9/06, McNamara, Laura A <lamcnam at sandia.gov> wrote:
>
> Computational social science doesn't lend itself to V&V the way that
> physics-based mod-sim does, so creativity in V&V is required...

I agree, and I think the approach that RAND take is as good as any. My
concern though is that any social science model is inevitably subjective at
a deep, deep level. You want a simulation that shows it's a good idea to
invade Iraq? No problem, I'll interview a bunch of experts, code up
realistic micro-rules and give you a simulation that shows yes, that's a
sensible policy. You want a simulation that shows it's not a good idea to
invade Iraq? No problem, I'll just interview a different set of experts, get
some different micro-rules in there and voila, I've shown invasion is a Bad
Thing.

Like I said, I'm getting more and more convinced that social science ABMs
just project the prejudices of their authors/funders. Or does anyone have an
example of an objective 'uncorrupted' social science ABM?

Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/friam_redfish.com/attachments/20060809/54a0e931/attachment.html

Marcus G. Daniels-3

Simulation and policy-making

Quoting Robert Holmes <robert at holmesacosta.com>:

> You want a simulation that shows it's a good idea to
> invade Iraq? No problem, I'll interview a bunch of experts, code up
> realistic micro-rules and give you a simulation that shows yes, that's a
> sensible policy. You want a simulation that shows it's not a good idea to
> invade Iraq? No problem, I'll just interview a different set of experts, get
> some different micro-rules in there and voila, I've shown invasion is a Bad
> Thing.

Two ways to deal with that problem.

1) Data

Experts can suggest micro-rules, but then provide examples when those micro-
rules were used and other cases where different micro-rules were used, and
historical accounts of what happened in each case. In other words do
retrodiction on a past similar system that had a known result. If there are
no such systems, and no way to change plans or modify a plan (based on early
data), then indeed the forecasts are suspect and vulnerable to manipulation.

2) Peer review

Not all experts will agree on micro-rules. Get a range of micro-rules from
different experts and compute the range of possibility for all of them.

Marcus

Phil Henshaw-2

Simulation and policy-making

In reply to this post by Douglas Roberts-2

Cool! Now there's a meaningful global modeling task, an event map with
a basically simple dynamic model, that involves mixing a core of
advanced eco-system dynamics with a limited cross section of
interdisciplinary autonomous agents. It's a very demanding but
conceptually realistic task that can be kept to what's working.

Not to be discounted is the nearly ideal information output intended.
The product of that effort would be used to feed a sophisticated global
public health system with better images and contingency plans for what
they might run into, widely disseminating the learning and generating
volumes of quality guiding feedback. That kind of purpose is set up
for greatly advancing our ability to live on earth.

Those who want to use the tools of systems inquiry for secretly
generating new kinds of weapons for central authorities to interfere
with what interests them, won't actually learn much and will cause great
harm.

Phil Henshaw ????.?? ? `?.????
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
680 Ft. Washington Ave
NY NY 10040
tel: 212-795-4844
e-mail: pfh at synapse9.com
explorations: www.synapse9.com <http://www.synapse9.com/>

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On
Behalf Of Douglas Roberts
Sent: Tuesday, August 08, 2006 9:40 PM
To: The Friday Morning Applied Complexity Coffee Group
Subject: Re: [FRIAM] Simulation and policy-making

Re: simulation and policy-making, a project that my group is working on
at the request of the current Washington administration is helping to do
just that. At the request of a consortium of representatives from the
White House, Dept of Treasury, DHS, Dept. of State, and a few other
cabinet-level political types, we have run numerous simulation
experimental designs to establish the bounds of the effectiveness of
various intervention strategies for containing an H5N1 pandemic, should
it occur in the US. We are using three simulation codes: EpiSims,
Epicast, and one from the Imperial College in the UK. The name of the
project is "Models of Infectious Disease Agent Study" (MIDAS), and it is
funded by NIH. See

http://www.hsph.harvard.edu/press/releases/press02202006.html and
http://usinfo.state.gov/gi/Archive/2005/Aug/08-339612.html

or do a google search on "MIDAS bird flu policy" for more info.

--Doug

On 8/8/06, Robert Holmes <robert at holmesacosta.com> wrote:

Oh I thank RAND are probably plenty ambitious in what they simulate for
the US govt. Just check out their research areas:
http://www.rand.org/research_areas/

Robert

On 8/8/06, mgd at santafe.edu < mgd at santafe.edu <mailto:mgd at santafe.edu> >
wrote:

Quoting Robert Holmes <robert at holmesacosta.com>:

> So if 'valid' simulations are being used to give the 'wrong' answers,
what
> does that tell us about simulation? Is there ever any hope of
objectivity
> (I'll give away the answer to that: no) or do all social simulations -
> political or economic - inevitably reflect the prejudices of their
author or
> funder?

Validated simulations, by definition, reproduce something that the
authors (or
funders) deem relevant as a performance metric. But that's not a
problem with
models or simulations, assuming the metrics are documented. If the
authors or
funders are prone to choosing easy, low dimensional things to fit, they
just
need to be more ambitious.

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

============================================================
FRIAM Applied Complexity Group listserv
Meets Fridays 9a-11:30 at cafe at St. John's College
lectures, archives, unsubscribe, maps at http://www.friam.org

--
Doug Roberts, RTI International
droberts at rti.org
doug at parrot-farm.net
505-455-7333 - Office
505-670-8195 - Cell

-------------- next part --------------
An HTML attachment was scrubbed...
URL: /pipermail/friam_redfish.com/attachments/20060809/9f4c649d/attachment-0001.html

Douglas Roberts-2

Simulation and policy-making

Phil,

Not quite, unfortunately. EpiSims, and other similar ABMs can all too
easily be used to identify weaknesses and potential exploits of social
infrastructures. We did studies for the US DHS that demonstrated exactly
this a couple of years ago when I still worked at LANL. One example was
when we simulated the release of a weaponized aerosol pneumonic plague
disease agent in a certain busy subway station during a simulated rush hour
in a simulated Chicago with a simulated population of 6.2 million people...

--Doug

On 8/9/06, Phil Henshaw <sy at synapse9.com> wrote:

>
>
>
> Those who want to use the tools of systems inquiry for secretly generating
> new kinds of weapons for central authorities to interfere with what
> interests them, won't actually learn much and will cause great harm.
>
>
>
> Phil Henshaw ????.?? ? `?.????
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 680 Ft. Washington Ave
> NY NY 10040
> tel: 212-795-4844
> e-mail: pfh at synapse9.com
> explorations: www.synapse9.com
>
> -----Original Message-----
> *From:* friam-bounces at redfish.com [mailto:friam-bounces at redfish.com] *On
> Behalf Of *Douglas Roberts
> *Sent:* Tuesday, August 08, 2006 9:40 PM
> *To:* The Friday Morning Applied Complexity Coffee Group
> *Subject:* Re: [FRIAM] Simulation and policy-making
>
> Re: simulation and policy-making, a project that my group is working on at
> the request of the current Washington administration is helping to do just
> that. At the request of a consortium of representatives from the White
> House, Dept of Treasury, DHS, Dept. of State, and a few other cabinet-level
> political types, we have run numerous simulation experimental designs to
> establish the bounds of the effectiveness of various intervention strategies
> for containing an H5N1 pandemic, should it occur in the US. We are using
> three simulation codes: EpiSims, Epicast, and one from the Imperial College
> in the UK. The name of the project is "Models of Infectious Disease Agent
> Study" (MIDAS), and it is funded by NIH. See
>
> http://www.hsph.harvard.edu/press/releases/press02202006.html and
> http://usinfo.state.gov/gi/Archive/2005/Aug/08-339612.html
>
> or do a google search on "MIDAS bird flu policy" for more info.
>
> --Doug
>
> On 8/8/06, Robert Holmes <robert at holmesacosta.com> wrote:
> >
> > Oh I thank RAND are probably plenty ambitious in what they simulate for
> > the US govt. Just check out their research areas:
> > http://www.rand.org/research_areas/
> >
> > Robert
> >
> >
> > On 8/8/06, mgd at santafe.edu < mgd at santafe.edu> wrote:
> > >
> > > Quoting Robert Holmes <robert at holmesacosta.com>:
> > >
> > > > So if 'valid' simulations are being used to give the 'wrong'
> > > answers, what
> > > > does that tell us about simulation? Is there ever any hope of
> > > objectivity
> > > > (I'll give away the answer to that: no) or do all social simulations
> > > -
> > > > political or economic - inevitably reflect the prejudices of their
> > > author or
> > > > funder?
> > >
> > > Validated simulations, by definition, reproduce something that the
> > > authors (or
> > > funders) deem relevant as a performance metric. But that's not a
> > > problem with
> > > models or simulations, assuming the metrics are documented. If the
> > > authors or
> > > funders are prone to choosing easy, low dimensional things to fit,
> > > they just
> > > need to be more ambitious.
> > >
> > > ============================================================
> > > FRIAM Applied Complexity Group listserv
> > > Meets Fridays 9a-11:30 at cafe at St. John's College
> > > lectures, archives, unsubscribe, maps at http://www.friam.org
> > >
> >
> >
> > ============================================================
> > FRIAM Applied Complexity Group listserv
> > Meets Fridays 9a-11:30 at cafe at St. John's College
> > lectures, archives, unsubscribe, maps at http://www.friam.org
> >
> >
>
>
> --
> Doug Roberts, RTI International
> droberts at rti.org
> doug at parrot-farm.net
> 505-455-7333 - Office
> 505-670-8195 - Cell
>
> ============================================================
> FRIAM Applied Complexity Group listserv
> Meets Fridays 9a-11:30 at cafe at St. John's College
> lectures, archives, unsubscribe, maps at http://www.friam.org
>
>