Data analysis - StarJack

From TACWiki
Jump to: navigation, search

The game begins with unparameterised SJ_START (1) and ends with SJ_END (19) followed by SJ_ENDSCREEN_CONTINUE (7). The game also is considered to have ended if the begin code for any other mini-game occurs. The game comprises multiple iterations of three phases: the maze task (which can be used as a measure of executive function), the Embedded Figures Test (where reaction times measure perceptual disembedding), and the "Sally-Anne" test of theory-of-mind.

Each level of the maze task begins with parameterised SJ_START (2) and ends with parameterised SJ_END (19). Each level of the maze task is interrupted by invocations of the Embedded Figures Test, and punctuated by an invocation of the Sally-Anne test. These latter two tests also can be invoked directly from the menu, without going through the maze test.

The Embedded Figures Test begins with SJ_EFT_START (49) and ends with SJ_EFT_END_FAIL (44) or SJ_EFT_END_SUCCESS (50).

The Sally-Anne test begins with SJ_SAT_START (52) and ends with SJ_SAT_END (53).

Following is a list of all the event codes, then a summary of the relevant behavioural and electrophysiological derived measures to extract:

Event codes (see autism_collaborative/Game/StarJack/EventCode.cs, autism_collaborative/Game/StarJack/Screen/SallyAnneScreen.cs),
autism_collaborative/Game/StarJack/HackingPhase/HackingPhaseScreen.cs, autism_collaborative/Game/StarJack/Dungeon/DungeonScreen.cs
1 SJ_START The game has begun.
2 SJ_START
    Level=Level_M|tutorialLevel Room=Room_N
The player has entered a new maze level.


Note that distinct numeric event codes (1 and 2) but one and the same string event descriptor (SJ_START) is used to denote the start of the game as a whole and to denote the start of an individual level of the maze within the game. The difference is the presence or absence of the parameters.

7 SJ_MENU_INPUT_MouseClick The player has clicked the mouse to select a sub-portion of the game.
7 SJ_ENDSCREEN_CONTINUE The player has clicked the mouse to continue past the game end screen, exiting the game.


(Note same numeric code as SJ_MENU_INPUT_MouseClick above.)

Maze Task Events
9 SJ_Cloak_Start In the maze, the player's avatar has activated the invisibility cloak.
10 SJ_Cloak_End In the maze, the player's invisibility cloak has deactivated on reaching the end of its allowed time.
11 SJ_TrapPlaced In the maze, the player's avatar has set a trap for the guards.
12 SJ_GuardTrapped
    (X, Y)=display coordinates
A trap previously set by the player has captured a guard.
13 SJ_Caught_TeleportToStart The player has been seen and caught by a guard or a camera, and teleports back to the starting position.
14 SJ_INPUT_Esc The player has pressed the ESCAPE key to pause the game.
15 SJ_MENU_INPUT_Enter The player has pressed the enter key to select a sub-portion of the game.
19 SJ_END
    Level=LevelN
In the maze, the player has reached the end of the current level.
23 SJ_INPUT_Move_Up In the maze, the player has pressed the up-arrow to move up.
24 SJ_INPUT_Move_Down In the maze, the player has pressed the down-arrow to move down.
25 SJ_INPUT_Move_Right In the maze, the player has pressed the right-arrow to move right.
26 SJ_INPUT_Move_Left In the maze, the player has pressed the left-arrow to move left.
33 SJ_INPUT_Interact In the maze, the player is interacting with an object or entity.
34 SJ_INPUT_LeftClick
    (X, Y)=display coordinates
The player has clicked the left mouse button at the given display coordinates.
Embedded Figures Test Events
38 SJ_EFT_Stimulus_Presented
    Key=KeyN Figure=FigureM
In the Embedded Figures Test, the player has been presented with a pair comprising a complex search figure and a smaller search target.
39 SJ_EFT_HIT In the Embedded Figures Test, the SJ_EFT_INPUT_Match event immediately preceding was a correct response.
40 SJ_EFT_REJECT In the Embedded Figures Test, the SJ_EFT_INPUT_NoMatch event immediately preceding was a correct response.
41 SJ_EFT_FALSE_ALARM In the Embedded Figures Test, the SJ_EFT_INPUT_Match event immediately preceding was an incorrect response.
42 SJ_EFT_MISS In the Embedded Figures Test, the SJ_EFT_INPUT_NoMatch event immediately preceding was an incorrect response.
44 SJ_EFT_END_FAIL The player has given incorrect responses on three trials, ending the Embedded Figures Test.
46 SJ_EFT_INPUT_NoMatch In the Embedded Figures Test, the player has indicated that the target does not occur within the given figure.
47 SJ_EFT_INPUT_Match In the Embedded Figures Test, the player has indicated that the target occurs within the given figure.
48 SJ_EFT_INPUT_Proceed The player has started a trial within the Embedded Figures Test.
49 SJ_EFT_START The Embedded Figures Test has begun.
50 SJ_EFT_END_SUCCESS The player has completed the Embedded Figures Test with fewer than three incorrect responses.
Theory-of-Mind Task Events
52 SJ_SAT_START The theory-of-mind test (a.k.a. the "Sally-Anne test") has begun.
53 SJ_SAT_END The theory-of-mind test (a.k.a. the "Sally-Anne test") has ended, after feedback has been given to the player.
55 SJ_SAT_Video_Stop The currently active video has ended, and the player is being prompted to have it repeated or to continue.


See details of the <A HREF="http://www.AutismCollaborative.org/wiki_aux/SJ_SAT_video_timecodes.txt">video event timings.

56 IGNORE (VideoReplay) (This code should not be being transcribed onto the parallel port and thus should not be appearing in the EEG log.)
58 SJ_SAT_INPUT_Continue The player has pressed the space bar, signalling readiness to view the next video stimulus.
(This same numeric code also can appear with the notation "IGNORE" in which case it should not be being transcribed to the parallel port and this should not be appearing in the EEG log.)
59 SJ_SAT_VideoStart_CargoDrop
    Planet={TOP, RIGHT, BOTTOM, LEFT}
The video showing the friendly space captain depositing the cargo on one of the planets has started.
60 SJ_SAT_VideoStart_Pirate_Drop
    Planet={TOP, RIGHT, BOTTOM, LEFT}
The video showing the pirate depositing the cargo on another planet has started. The PLANET parameter will always differ from that of the previous SJ_SAT_VideoStart_Pirate_Theft.
61 SJ_SAT_VideoStart_Pirate_Theft
    Planet={TOP, RIGHT, BOTTOM, LEFT}
The video showing the pirate's theft of the cargo has started. The PLANET parameter will always be the same as that of the previous SJ_SAT_VideoStart_CargoDrop.
62 SJ_SAT_VideoStart_Pirate_NoTheft
    Planet={TOP, RIGHT, BOTTOM, LEFT}
The video showing the pirate's scan (but no theft) of the cargo has started. The PLANET parameter will always be the same as that of the previous SJ_SAT_VideoStart_CargoDrop.
64 SJ_SAT_INPUT_Response_Top In response to a request to plot an intercept course, the player has pressed the up-arrow key, indicating a course to the TOP planet.
65 SJ_SAT_INPUT_Response_Bottom In response to a request to plot an intercept course, the player has pressed the down-arrow key, indicating a course to the BOTTOM planet.
66 SJ_SAT_INPUT_Response_Left In response to a request to plot an intercept course, the player has pressed the left-arrow key, indicating a course to the LEFT planet.
67 SJ_SAT_INPUT_Response_Right In response to a request to plot an intercept course, the player has pressed the right-arrow key, indicating a course to the RIGHT planet.
68 SJ_SAT_FirstOrder_NoTheft_Correct In the control condition in which no theft has occurred (probability 1/3), the player's response to the first-order question ("Where will you send a ship to meet up with the captain?") has correctly indicated the planet on which the cargo was originally cached by the captain (and on which it remains cached).


Note that the numeric event code associated correctly with this event, 68, also is used by SJ_SAT_FirstOrder_NoTheft_Incorrect below. This error is the result of a bug in SallyAnneScreen.cs.

68 SJ_SAT_FirstOrder_NoTheft_Incorrect In the control condition in which no theft has occurred, the player's response to the first-order question ("Where will you send a ship to meet up with the captain?") has indicated a planet other than that on which the cargo was originally cached by the captain (and on which it remains cached).


Note that the numeric event code associated with this event, 68, is incorrect and ought to be 69. This error is the result of a bug in SallyAnneScreen.cs.

70 SJ_SAT_FirstOrder_Scan_Correct In the scenario in which the admiral scans the pirate stealing and re-caching the cargo (probability 1/3), and reports this information to the captain, the player's correct response to the first-order question ("Where will you send a ship to meet up with the captain?") actually indicates higher-order theory-of-mind: the captain believes that the pirate believes that the captain believes that the cargo remains in the planet where the captain had originally cached it, thus the player's answer indicating the same PLANET as in the preceding SJ_SAT_VideoStart_CargoDrop and SJ_SAT_VideoStart_Cruiser_View_Theft is correct.


Note that the numeric event code associated with this event, 70, is incorrect and ought to be 73. This error is the result of a bug in SallyAnneScreen.cs.

70 SJ_SAT_FirstOrder_Theft_Correct In the scenario in which the pirate steals and re-caches the cargo (and the admiral cannot scan this activity and thus can't report anything to the captain) (probability 1/3), the player has correctly responded to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating that the captain will go to the planet on which she originally cached the cargo — that is, the same PLANET as in the preceding SJ_SAT_VideoStart_CargoDrop.


Note that the numeric event code associated correctly with this event, 70, also is used by SJ_SAT_FirstOrder_Scan_Correct above. This error is the result of a bug in SallyAnneScreen.cs.

71 SJ_SAT_FirstOrder_Theft_IncorrectTrueLocation In the scenario in which the pirate steals and re-caches the cargo (and the admiral cannot scan this activity and thus can't report anything to the captain), the player has responded incorrectly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating the true location of the cargo rather than the location believed by the captain.
72 SJ_SAT_FirstOrder_Theft_IncorrectOtherLocation In the scenario in which the pirate steals and re-caches the cargo (and the admiral cannot scan this activity and thus can't report anything to the captain), the player has responded incorrectly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating one of the two planets that harbours neither the captain's original cache nor the pirate's subsequent cache. (This should hardly ever happen, and when it does happen it's as likely to be a consequence of keypress error as any other factor.)
74 SJ_SAT_FirstOrder_Scan_IncorrectFakeLocation In the scenario in which the admiral scans the pirate stealing and re-caching the cargo, and reports this information to the captain, the player has responded incorrectly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating the captain's original hiding place (as though the admiral had not reported to the captain the pirate's new hiding place).
75 SJ_SAT_FirstOrder_Scan_IncorrectOtherLocation In the scenario in which the admiral scans the pirate stealing and re-caching the cargo, and reports this information to the captain, the player has responded incorrectly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating one of the two planets that harbours neither the captain's original cache nor the pirate's subsequent cache. (This should hardly ever happen, and when it does happen it's as likely to be a consequence of keypress error as any other factor.)
76 SJ_SAT_SecondOrder_NoTheft_Correct In the control condition in which the pirate scans but does not steal the cargo, the player has responded correctly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating the planet on which the captain cached the cargo.


Note that the numeric event code associated correctly with this event, 76, also is used by SJ_SAT_SecondOrder_NoTheft_Incorrect below. This error is the result of a bug in SallyAnneScreen.cs.

76 SJ_SAT_SecondOrder_NoTheft_Incorrect In the control condition in which the pirate scans but does not steal the cargo, the player has responded incorrectly to the first-order question ("Where will you send a ship to meet up with the captain?") by indicating one of the three planets that did not harbour the cargo.


Note that the numeric event code associated with this event, 76, is incorrect and ought to be 77. This error is the result of a bug in SallyAnneScreen.cs.

78 SJ_SAT_SecondOrder_Scan_Correct In the scenario in which the admiral scans the pirate stealing and re-caching the cargo, and reports this information to the captain, the player has responded correctly to the second-order question ("Where will you send a ship to intercept the pirate?") by indicating the planet on which the captain had originally cached the cargo. Because the pirate wants to find the captain, and the pirate believes (incorrectly) that the captain believes that the cargo remains where the captain originally cached it, the pirate will indeed go to the planet where the cargo was originally cached.


Note that the numeric event code associated with this event, 78, is incorrect and ought to be 81. This error is the result of a bug in SallyAnneScreen.cs.

78 SJ_SAT_SecondOrder_Theft_Correct In the scenario in which the pirate steals and re-caches the cargo (and the admiral cannot scan this activity and thus can't report anything to the captain), the player has responded correctly to the second-order question ("Where will you send a ship to intercept the pirate?") by indicating the planet on which the captain had originally cached the cargo. Because the pirate wants to find the captain, and the pirate believes (correctly) that the captain believes that the cargo remains where the captain originally cached it, the pirate will indeed go to the planet where the cargo was originally cached.


Note that the numeric event code associated correctly with this event, 78, also is used by SJ_SAT_SecondOrder_Scan_Correct above. This error is the result of a bug in SallyAnneScreen.cs.

79 SJ_SAT_SecondOrder_Theft_IncorrectTrueLocation In the scenario in which the pirate steals and re-caches the cargo (and the admiral cannot scan this activity and thus can't report anything to the captain), the player's response to the second-order question ("Where will you send a ship to intercept the pirate?") incorrectly reports the planet to which the cargo was moved by the pirate, instead of the planet on which the captain originally cached the cargo. Although the captain wants to find the cargo, the pirate wants to find the captain. Thus the pirate will move not to the planet on which he cached the cargo, but to the planet that he (correctly) believes that the captain believes the cargo is on, that is, the planet on which the captain originally cached the cargo.
82 SJ_SAT_SecondOrder_Scan_IncorrectTrueLocation In the scenario in which the admiral scans the pirate stealing and re-caching the cargo, and reports this information to the captain, the player's response to the second-order question ("Where will you send a ship to intercept the pirate?") incorrectly reports the planet to which the cargo was moved by the pirate, instead of the planet on which the captain originally cached the cargo. Although the captain wants to find the cargo, the pirate wants to find the captain. Thus the pirate will move not to the planet on which he cached the cargo, but to the planet that he (incorrectly) believes that the captain believes the cargo is on, that is, the planet on which the captain originally cached the cargo.
83 SJ_SAT_SecondOrder_Scan_IncorrectOtherLocation n the scenario in which the admiral scans the pirate stealing and re-caching the cargo, and reports this information to the captain, the player's response to the second-order question ("Where will you send a ship to intercept the pirate?") incorrectly reports one of the two planets that harbours neither the captain's original cache nor the pirate's subsequent cache. (This should hardly ever happen, and when it does happen it's as likely to be a consequence of keypress error as any other factor.)
84 SJ_SAT_FirstOrder_Question_Presented
SJ_SAT_SecondOrder_Question_Presented
After the explanatory text ("The captain wants to pick up the cargo. She will steer her ship to the planet where she thinks the cargo is. The pirate wants to find the captain. He will steer his ship to wherever he thinks the captain will go."), the first-order question ("Where will you send a ship to meet up with the captain?") or the second-order question ("Where will you send a ship to intercept the pirate?") has just been asked.


Note that the same numeric event code, 84, is used for both SJ_SAT_FirstOrder_Question_Presented and SJ_SAT_SecondOrder_Question_Presented.

85 SJ_SAT_VideoStart_Cruiser_View_Theft
    Planet={TOP, RIGHT, BOTTOM, LEFT}
The video showing the admiral's viewing of the theft has started. The PLANET parameter will always be the same as that of the previous SJ_SAT_VideoStart_CargoDrop.


Note that the numeric event code associated with this event, 85, is incorrect and ought to be 86. This error is the result of a bug in SallyAnneScreen.cs.

85 SJ_SAT_VideoStart_Cruiser_NoScan In the scenario in which the pirate has stolen and re-cached the cargo, and the admiral cannot scan this activity and thus can't report anything to the captain, the video showing only noise and interference on the admiral's viewscreen has started.


Note that the numeric event code associated correctly with this event, 85, also is used by SJ_SAT_VideoStart_Cruiser_View_Theft above. This error is the result of a bug in SallyAnneScreen.cs.

87 SJ_SAT_VideoStart_Cruiser_View_NoTheft
    Planet={TOP, RIGHT, BOTTOM, LEFT}
In the control condition in which no theft has occurred, the video showing the admiral's view of the pirate's scanning but not stealing the cargo has started.


After deletion of codes 255 and greater, and alignment of the behavioural and EEG event sequences just as was done for Maritime Defender, processing should continue by correcting (both in the behavioural log file and in the EEG log file) the numeric event codes that are associated with the following string event descriptors in the behavioural log file (see notes in the table above):
SJ_SAT_FirstOrder_NoTheft_Incorrect
SJ_SAT_FirstOrder_Scan_Correct
SJ_SAT_SecondOrder_NoTheft_Incorrect
SJ_SAT_SecondOrder_Scan_Correct
SJ_SAT_SecondOrder_Question_Presented
SJ_SAT_VideoStart_Cruiser_View_Theft

SJ_EFT_INPUT_Match and SJ_EFT_INPUT_NoMatch events should be re-coded as context-free events based on the success flags (SJ_EFT_HIT, SJ_EFT_REJECT, SJ_EFT_FALSE_ALARM, SJ_EFT_MISS) that follow each of them. The success flags themselves then can be deleted, as they do not mark any actual events distinct from the SJ_EFT_INPUT_Match SJ_EFT_INPUT_NoMatch that they follow.

Similarly, SJ_SAT_INPUT_Response_{Top, Bottom, Left, Right} events should be re-coded as context-free events based on the success flags (SJ_SAT_FirstOrder_{*} and SJ_SAT_SecondOrder_{*}). In the re-coded events, both the four possible locations of the response (Top, Bottom, Left, Right) and the four possible correctness conditions of the response (Correct, IncorrectTrueLocation, IncorrectFakeLocation, IncorrectOtherLocation) should be preserved, thus there will be 4x4 = 16 possible codes. The success flags themselves then can be deleted, as they do not mark any actual events distinct from the SJ_SAT_INPUT_Response_{*} events that they follow.


Behavioural measures related to executive function

As simple behavioural measures of executive function, the software can compute - for each level separately and for all levels cumulatively - the total number of moves, the total time elapsed from start to finish (excluding time spent in EFT and SAT subtests), and the mean and standard deviation of the time between successive moves. Other measures are possible; if you think of one, try it!


Behavioural measures related to perceptual disembedding

d' ("d prime") should be computed from tallies of EFT hits, misses, false alarms and correct rejections. If the tally for any one of these four categories of responses is zero, that tally can be changed to 1 so as to estimate d'.

Mean and standard deviation of EFT reaction time, for correct responses only, should be computed. Also list out the single-trial reaction times.


Behavioural measures related to theory-of-mind

Single-trial data should be extracted and listed out for first-order and second-order questions in all trials. Three types of trials exist: the first-order control condition in which the pirate scans but does not steal the cargo (probability 1/3, and note that in this case it doesn't matter whether or not the admiral observes the pirate; the observed and unobserved cases can be collapsed into a single bin), the first-order theory-of-mind conditions in which the pirate steals the cargo but is not observed by the admiral (probability 1/3), and the second-order theory-of-mind condition in which the pirate steals the cargo but the admiral observes this theft and tells the captain (probability 1/3). The single-trial data should be listed first for the no-theft condition, then separately for the unobserved theft condition, then separately for the observed theft condition. The data for each trial comprise three timings and two correctness values: the time taken for the whole trial (SJ_SAT_END minus SJ_SAT_START), the reaction time for response to the first-order question (SJ_SAT_INPUT_Response_{*} minus SJ_SAT_FirstOrder_Question_Presented) and the correctness condition (Correct, IncorrectTrueLocation, IncorrectFakeLocation, IncorrectOtherLocation) of that response, and the reaction time for response to the second-order question (SJ_SAT_INPUT_Response_{*} minus SJ_SAT_SecondOrder_Question_Presented) and the correctness condition of that response.

Summaries of correctness conditions for each of the two question types (first order, second order) for each of the three theft conditions (no theft, unobserved theft, observed theft) should be given both as tallies and as percentages.

For correct responses only, means and standard deviations (where computable) should be given for the following reaction-time differences:

First-order question:
Unobserved Theft minus No Theft
Unobserved Theft minus Observed Theft
Unobserved Theft minus (No Theft or Observed Theft, pooled)

Second-order question:
Observed Theft minus Unobserved Theft
Observed Theft minus No Theft
Observed Theft minus (Unobserved Theft or No Theft, pooled)


Independent Components Analysis

ICA can be applied separately to epochs from the Embedded Figures Test, epochs from the Sally-Anne test, and perhaps continuous EEG for the maze task.

Remember not to include the skin-conductance channel (the greatest-numbered channel) in the ICA!


Physiological measures related to perceptual disembedding

Standard ERP and ERSP analyses can be constructed round the SJ_EFT_Stimulus_Presented event and corresponding SJ_EFT_INPUT_{*} response events.


Physiological measures related to theory-of-mind

Some possible fMRI-based contrasts:

Gaze perception (blink events or gaze aversion intervals versus direct gaze intervals)
See the timecodes file linked from the SAT_SJ_Video_Stop event in the table above. See also <A HREF="http://dx.doi.org/10.3389/fnhum.2011.00068">Brefczynski-Lewis et al. on ERP response to a social partner's blinks</A>.

Facial expression perception (scowl intervals versus neutral intervals)

Scene perception (egocentric view of person's face, versus allocentric view of person's hands on console, versus external view of spaceships)

Perception of body movements (hands) or gestures (nods)

Also note that, because these stimuli are cinematic, useful information can be gleaned from applying Uri Hasson's approach of correlating individual subjects' voxelwise activation time series against the group mean. This model-free procedure can identidy consistent activations, localised temporally and anatomically, across subjects. The timings of such consistent activities then can be correlated with the video timecodes (see link in the table above).

It may be useful to contrast fMRI time series during the video stimuli analogous to the behavioural (reaction-time) contrasts detailed above - that is, Unobserved Theft minus No Theft, Unobserved Theft minus Observed Theft, and so forth. (For a list of specific contrasts, see the behavioural section above.)

For EEG analyses, given the temporally extended nature of these stimuli spectral and other quantitative-EEG methods may be more appropriate than ERP's.

For background on cognitive electrophysiology of theory-of-mind, see <A HREF="http://dx.doi.org/10.1523/JNEUROSCI.1392-11.2011">McCleery et al., "The Neural and Cognitive Time Course of Theory of Mind", Journal of Neuroscience, 7 September 2011, 31(36):12849-12854</A>.


Physiological measures related to executive function

Here again, quantitative-EEG methods may be most appropriate. In addition, specific time-locked events may generate specific ERP's of interest, for example error-related negativity on SJ_INPUT_Move_{*} events that immediately precede SJ_Caught_TeleportToStart events.