Main contributions of my research
Presents an approach to measure human performance that minimizes the effect of motor and perceptual components in action. This allows to attribute observed behavior to cognitive factors.
Introduces the idea of an Ideal Performer (IP), an extension of the ideal observers used in perceptual research (Geisler, W. Sequential ideal-observer analysis of visual discriminations. Psychological Review 96:267-314,1989; Barlow, H.B. The absolute efficiency of perceptual decisions. Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences 290: 71-82,1980). By using the ideal performer as a benchmark to characterize human performance, it allows to identify and quantify the strategy that subjects use to control a virtual object. Subjects' performance- limiting motor and perceptual noises can also be examined.
Presents a method to infer the features of the internal model that subjects acquire about the dynamics of a virtual object.
Investigates
the validity of this estimate, and argues for an internal model that contains
abstract, object-related knowledge.
The Internal model of a physical system
Skillful
(fast
and accurate) manipulation of an object
requires knowledge about how the object reacts to our actions. For example,
consider the following mechanical system:
It
consist of a mass that is attached to a piston. The whole arrangement is
solidly fixed to a rigid wall. The mass can move sideways, by pushing or
pulling it horizontally. The piston contains a viscous fluid that
opposes the movement of the mass.
| Let us consider now the task of aligning the mass to the position of a target (vertical line in the figures to the right of this paragraph) as fast and as accurately as possible. We would achieve this by pulling the mass to the right. If we are completely inexperienced about the system, it is not likely that we will perform this alignment very accurately. The movement of the system is not simple. The mass' momentum will cause the system to continue moving for a certain time after we push or pull the mass. Also, the faster the mass moves, the stronger the piston opposes its movement. |
|
To
complete this task quickly and accurately we need to know how the system
reacts to our actions. In what follows, this knowledge about the system
will be called the INTERNAL MODEL
that a human operator acquires about the system. It is "internal"
because it resides in the central nervous
system of the operator. It is a "model" because it represents
(or models) the features of the physical system.
Internal models of virtual objects
Instead
of using a real physical system as the one shown above, my research used
virtual objects. These are computer-generated objects that are created
by rendering a mathematical model of a physical system. The computer calculates
in real-time the state of the virtual object and produces information that
a human operator experiences. This is very convenient for many reasons.
For example, it is not necessary to recalibrate the parameters of the virtual
objects, and it is simple to implement them.
Why to study internal models of virtual objects?
Many important brain functions are related to internal models:
LEARNING
can be seen as the process that acquires internal models about objects
in the world.
MEMORY
can be posed as dealing with the storage and retrieval of internal models.
MOTOR
CONTROL uses internal models of objects (and
of body parts) to execute behavior.
COGNITION
operates on internal models to adjust behavior to changes in the environment.
PERCEPTION
provides the channels to acquire information about internal
models.
Comprehensive description and understanding of the features of an internal model promises a better footing to understand these brain functions. For example, I believe that it is simpler to understand learning if we understand what is being learned.
Using
virtual objects instead of physical ones simplifies the experimental task.
Of course, I also expect to learn something about what happens in the real
world by using realistic virtual objects.
Using dynamics to describe internal models
A dynamical system can be succinctly defined as one whose future state depends on its past and current states. Differential equations are frequently used to describe how this state changes, for example, over time, or space, or other important variables. For example, for the mass-piston system described above, the following differential equation relates the position of the mass (x) over time (t), and relates it to its velocity (dx/dt), its acceleration (d2x/dt2), to the viscous resistance of the piston (b), to the value of the mass (m), and to a force F(t) that pulls the mass horizontally :
This equation is found by using Newton's second law. With it we can compute precisely what the position of the mass will be if, for example, we pull it to the right for 1.5 seconds with a given force. We can also know the velocity of the mass and its acceleration at any time we wish. If we have such a description, then we can predict the future state of the system.
Similar
descriptions can be found for many different physical systems. But
we are not able to write down a differential equation to describe
a cognitive process such as an internal model. Cognitive processes
are clearly dynamical systems. Is it possible to describe them, for example,
by using a differential equation?
| The
following heuristic suggests that this is possible in principle. Consider
a human operator that interacts with the piston-mass system (right panel).
Assume that this operator has had lots of practice in using the system,
and as a result he/she is very skilled in moving the position of the mass
to any place he/she wishes.
Both the operator's knowledge about the object (his/her internal model of the object) and the differential equation that describes the dynamics of the object (shown on top of right panel for convenience) contain information about how the system behaves. In the best possible case, the operator's internal model would contain the same information about the object that the differential equation has. This suggests that the differential equation (or another description of the dynamics of the physical system) could be used to describe the operator's internal model. |
 |
Problems to isolate
the behavioral contribution of internal models
An
internal model resides in the central nervous system of the operator. We
have to infer its features by observing the operator's behavior when he/she
interacts with a physical system.
| But
the operator's behavior has several complex components (right panel).
Complex sensory signals (blue arrows) that are related to the dynamics
of the physical system have to reach the operator's central nervous system
if an internal model of the system has to be acquired. This implies
that inappropriate sensory information could result in an inaccurate internal
model. For example, if the operator can barely see how the system reacts
to his/her pulling it, it is difficult to learn how to operate the
system.
On the other hand, the operator's brain has to generate motor control signals (green arrow) to control the object. If limitations in the operator's motor system (maybe he/she is a bit slow to react to the reactions of the system) cause poor control performance, even a very accurate internal model won't help. |
 |
These
complexities have other consequences. Performance can be limited by cognitive
factors (poor internal model, poor learning ability), and/or sensory and
motor ones.
Isolating internal
models
| Subjects
execute simple motor actions and experience optimal perceptual information
My research proposed a paradigm to tackle these problems by minimizing the effect of subject's motor control components and by simplifying and optimizing the sensory signals that the subject experiences. Subjects
in my paradigm interact with a simple virtual object that consists
of a horizontal line presented in a computer monitor (right panel). This
line moves sideways depending on the subject's control actions (see
below).
|
 |
|
To minimize motor planning and execution, subject's control actions are limited to pressing two buttons. Pressing the right button (shown in yellow , right panel) moves the virtual object to the right. Pressing the left button (shown in red), moves the virtual object to the left. Learning to operate these buttons is trivial. The signals from the buttons do not depend on how hard the subject presses them. This means that subjects' do not receive any force-feedback/kinesthetic information about the movement of the virtual object , and have to rely on visual information only. For a short description of the virtual forces generated by the buttons, click here. |
 |
Learning to control
the virtual object
Subjects
are instructed to use the control buttons to align the virtual object to
the position of a target as fast and as accurately as possible. The target
consists of a small vertical line that appears above the virtual object.
The target stays in the same position during ten seconds, and then changes
position randomly. The control buttons For a short demonstration of the
task, click here. Bill Geisler suggested
that the task is similar to parking a car. James Lackner suggested that
the virtual object is an approximation to a muscle.
Subjects are instructed about the task and the use of the equipment. Subjects were given six practice trials, and they proceeded to complete 240 experimental trials. They had frequent breaks to minimize fatigue. As training progresses, subjects performance improves. The following images show samples of subject A.S. at different stages of training.
The figure above shows the position of the virtual object over time (red curve, left panel). This panel shows also the position of the target, which stays constant during the trial. The right panel shows the control signal (red curve) that the subject used to move the virtual object. In this trial, the subject overshoots the position of the target (left panel). The subject notices this and releases the control button (blue arrows) .
Let us continue examining what happens in this trial. The blue arrows in the following image highlight that after releasing the control button the subject makes several short button presses to move the virtual object back to the target. These button presses did not move the object much.
As a result (following figure), the subject presses the button for a longer time to finally bring the object into alignment to the target.
As training progresses (image below), the subject improved performance, taking less control actions to reach the target.
Measuring subject's
performance: ideal performers
The experimental task was defined precisely enough to allow for detailed computer simulation of an ideal performer. These ideal performers are optimal control systems that know
Object's
dynamics
The
state of the object
An
optimal way to align the object to the target.
The ideal performers were used as benchmarks to evaluate subjects' performance and its improvement with practice. They allowed to identify the strategy that subjects used to control the virtual object. Because the ideal performers solve the same task that subjects do, they allow to test for different hypothesis about the impact of perceptual and motor components on subjects performance.
By using an ideal performer it was possible to estimate subjects' internal model in steady state. The validity of this estimate was supported by a second, transfer experiment.
The details can be found in my doctoral dissertation and in the paper we submitted for publication.
Currently
I am working in applying my performance measures and ideal performers
to study how subjects' interaction with a virtual object is influenced
by force-feedback provided with a haptic interface. The paradigm described
in this document allows to obtain a baseline for human performance when
there is no force-feedback input and when subjects perform simple
motor planning and execution.