@-------------------------------------------------------------) | seabrook@clark.net _ Anne Arundel Community College | | (410)541-2424 <>:-) Computer Science | @-------------------------------------------------------------) The Brain-Computer Interface: Techniques for Controlling Machines * Richard H.C. Seabrook Abstract Current research in discovering and applying EEG/ERP-based detection and analytical techniques to facilitate a particular branch of human-computer interface (HCI), namely the brain-computer interface (BCI), were reviewed and summarized. Techniques include mu-rhythm conditioning for cursor control, visual evoked potential detection, P300 response detection, EEG frequency mapping and detection of lateral hemisphere differences. New research directions and applications of the techniques for human-computer interface control are suggested. 1. Introduction The last 6 years have witnessed a rapidly-growing body of research and technique development involving detecting human brain responses and putting these techniques to appropriate uses to help people with debilitating diseases or who are disabled in some way - - the so-called "brain-computer interface" (BCI). The chief difference between BCI techniques and those studied in more common human-computer interface (HCI) tasks lies in not relying on any sort of muscular response, but only detectable signals representing responsive or intentional brain activity. This places increased emphasis on detection mechanisms for finding out if and when the desired response occured. Brain-computer interface experiments involve considerable system support. A typical setup includes four areas requiring detailed attention: preparation, presentation, detection and control. Preparation includes all those activities required to adjust the detection apparatus, both generally and for particular subjects, and to prepare the subject to participate. This may require donning special headgear, mounting electrodes (in some cases implanting electrodes surgically) or attaching sensing elements to the subject in other ways, plus extended tuning sessions involving the subject and the detection apparatus to make sure the mechanisms can detect the particular brain response under study when it occurs. Required adjustments may include repositioning electrodes, setting weights and thresholds, instructing the subject what to do or not do, and so on. Presentation includes all those activities involved in stimulating the subject in order to elicit the particular brain response under study. The experimenter controls presentation details to study their relationship with the response, using such parameters as intensity, duration, timing, rate, and so on. Most presentation techniques today involve the visual field in some way, partly due to the site of relevant brain activity (visual cortex) lying closer to the scalp than several other important sites, and partly due to the availability of inexpensive computer display devices. Detection includes all those activities and mechanisms involved in recording and analyzing electrical signals (event- related potentials or ERP's) from the sensing elements attached to the subject quickly and reliably enough to not be a controlling factor in the experimental design. Most techniques involve the electroencephalogram (EEG) in some way and may include wave-form averaging over a number of trials, autocorrelation with a known signal or wave shape, Fourier transformation to detect relative amplitude at different frequencies (power spectrum) and other mathematical processes. Most analysis can be carried out with an inexpensive computer fitted with suitable digital signal processing (DSP) apparatus, software and display components, usually the same machine that presents the stimulus. Feedback to the subject is an important component of most detection setups, either to improve performance or as a reward. Control includes all of those uses of detected subject responses for manipulating the environment in some desired way. Many experiments involve such manipulation as their central component in order to demonstrate the feasibility of controlling computers or other apparatus using the brain alone, as a prosthetic technique for those with limited abilities to manipulate the environment in other ways. Several experimenters identify patients with amyotrophic lateral scleroses (ALS), severe cerebral palsy, head trauma and spinal injuries as the intended beneficiaries of such research, due to their intellectual capabilities remaining largely intact during an extended period of severely reduced physical capability. Experimental uses of the brain responses in this survey include selecting among presented letters and words to facilitate communication, moving displayed targets around on a computer screen to simulate device control, and mapping areas of high or low brain activity following stimulus presentation in order to support prediction. 2. Current Brain-Computer Interface Techniques Technique: P300 detection Farwell [Farwell&Donchin 1988] of the Department of Psychology and Cognitive Psychophysiology Laboratory at the University of Illinois at Urbana-Champaign IL, describes a technique for detecting the P300 component of a subject's event-related brain potential (ERP) and using it to select from an array of 36 screen positions. The P300 component is a positive-going ERP in the EEG with a latency of about 300ms following the onset of a rarely- occuring stimulus the subject has been instructed to detect. The EEG was recorded using electrodes placed at the Pz (parietal) site (10/20 International System), limited with band-pass filters to .02-35Hz and digitized at 50Hz. Electro-oculogram (EOG) data was also recorded from each subject via electrodes placed above and below the right eye. The "odd-ball" paradigm was used to elicit the P300, where a number of stimuli are presented to the experimental subject who is required to pay attention to a particular, rarely-occuring stimulus and respond to it in some non- motor way, such as by counting occurrences. Detecting the P300 response reliably requires averaging the EEG response over many presentations of the stimuli. The purpose of the current experiment was to discover the minimum number of presentations at two different inter-stimulus intervals (ISI) required to detect the P300 response. The experiment presented a 36-position array of letters, plus common typing characters and controls (e.g. space, backspace), made to flash in a random sequence first by rows and then columns. Each trial consisted of a complete set of six column or row flashes. Trials contaminated with muscular or EOG response were rejected and additional trials presented until data were collected from a block of 30 good trials, during which subjects were to fixate on a particular position, and count the number of times it flashed while a control message was elsewhere on the screen. After each block the fixated letter (one of B-R-A-I-N) was added to the screen so that subjects were conscious of slowly spelling out the word "BRAIN" through a succession of five blocks. A set of five blocks was run at each ISI -- 125ms and 500ms. The two presentation rates were chosen to bracket a range of communication rates from a low of 30 averaged trials at 500ms ISI (93.6 seconds of presentation per character) to a high of one trial at 125ms (1.245 seconds of presentation per character), an effective communication rate range of .01 to .8 characters-per- second, respectively. The authors used four techniques to analyze the data for reliable P300 response detection -- stepwise descriminant analysis (SWDA), peak picking, area, and covariance, and identified SWDA as leading to the greatest accuracy at the fastest presentation rate. Results indicated that a character chosen from among 36 items can be detected with 95% accuracy within 26 seconds. Technique: EEG mu-rhythm Conditioning Three papers using this technique were reviewed including Wolpaw [Wolpaw et al 1991], McFarland [McFarland et al 1993], and colleagues at the Wadsworth Center for Laboratories and Research, Albany, NY, and Pfurtscheller [Pfurtscheller et al 1993] and colleagues at the Ludwig Boltzmann Institute of Medical Informatics and Neuroinformatics, Department of Medical Informatics, Institute of Biomedical Engineering, University of Technology Graz, Austria. All three papers describe subjects' abilities to move a cursor toward a target on a computer screen by manipulating their mu- rhythm, a detectable pattern in a great majority of individuals in the EEG 8-12Hz frequency range, centered about 9.1Hz. Work is based on earlier research efforts by Kuhlman [Kuhlman 1978b] who described the mu-rhythm in normal and epileptic subjects. Wolpaw describes detecting subjects' mu-rhythm amplitude, defined as the square-root of the spectral EEG power at 9Hz, using two scalp-mounted electrodes located near location C3 in the International 10/20 System and a digital signal processing board analyzing continuous EEG in 333ms segments, and using it to drive a cursor up or down on a screen toward a target placed randomly at the top or bottom. An experiment operator preset the size of the ranges and number of cursor movement steps assigned to each range for each subject during testing prior to each experimental run. Ranges were set so that the commonest mu-rhythm amplitudes (<4 microvolts) left the cursor in place or moved it downwards moderately while higher amplitudes (>4 microvolts) moved it upwards in increasing jumps. Weights were adjusted as subjects exhibited better control of their mu-rhythm amplitudes for up and down targets in repeated trials. Wolpaw substantiates subjects' learned intentional control over mu-rhythm amplitude in three ways: by performing frequency analysis up to 192Hz on subjects during cursor movement trials and failing to find any relationship between mu- rhythm changes and the higher frequencies associated with muscular (EMG) activity; by subjects statements about not making contralateral movements and observing none; and by failing to find any relationship between mu-rhythm changes and posterior scalp recordings of the visual alpha-rhythm. Four out of five subjects acquired impressive control over their mu-rhythm amplitude during 12 45-minute sessions over a period of two months. Accuracies of 80-95% target hits across experimental subjects were achieved and rates of 10-29 hits per minute. Off-line analysis of two subjects' raw EEG data (see below) provided good support for Wolpaw's experimental results. McFarland used essentially the same experimental setup and introduced greater precision constraints on four subjects' attempts to position a cursor by means of mu-rhythm control. A vertical bar target appeared in one of five different vertical positions on the left side of the screen and crossed the screen from left to right in 8 seconds. Subjects had to move the cursor (initially in the middle of the right edge of the screen) quickly to the correct one of five different vertical screen positions to intercept the target by controlling their mu-rhythm amplitude. Analysis of the average distance between the center of the target and the cursor during succeeding trials indicated that all subjects reduced the distance and three out of four significantly so. Pfurtscheller used contralateral blocking of the mu-rhythm during the 1-second period prior to a motor activity (in this case pressing a microswitch using either the right or the left index finger) to predict which response was to follow. An array of 30 electrodes spaced evenly across the scalp (two were at locations C3 and C4 in the International 10/20 System) was used to record EEG activity. An initial training period for each subject involved using data from all 30 electrodes to train the classification network. During experimental trials, a feature-vector of power values (Hilbert Transform) from electrodes at positions C3 and C4 was constructed at 5 time points and classified using a Learning Vector Quantizer (LVQ) artificial neural network of the type described by Kohonen [Kohonen 1988]. The experimenter achieved the best balance of reliability/speed of classification by using the 1/2-second prior to response and performing a multiple- classification and voting process. EEG data from two subjects in the Wolpaw experiment described above were provided to the Graz Institute for Information Processing for additional analysis described by Flotzinger [Flotzinger et al, 1993] using the Graz LVQ neural net scheme (see above) and a fixed time-segment. Cursor-movement was predicted >from raw data with 90% accuracy. Results also implied that frequency bands other than the mu and beta ranges may contain useful (i.e. target-related) information. Technique: VEP Detection Two papers using using this technique were reviewed including Sutter [Sutter 1992] at the Smith-Kettlewell Eye Research Institute in San Francisco CA, and Cilliers [Cilliers&VanDerKouwe 1993] and colleague at the Department of Electrical and Electronic Engineering, University of Pretoria, South Africa. Sutter describes presenting a 64-position block on a computer screen and detecting which block the subject looks at, while Cillier's work uses a series of four lights. In each case, several simultaneously presented stimuli are made to change rapidly in some controlled way (intensity, pattern, color-shift) and the subject has scalp electrodes placed over the visual cortex (back of the head) in a position to detect changes in the evoked potential (VEP) at that location. Sutter used a lengthy binary sequence to switch 64 screen positions between red and green, and in other trials to reverse a checkerboard pattern. Each screen position was shifted 20ms in the binary control sequence relative to its neighbors, and the entire sequence was autocorrelated with the VEP in overlapping increments (the VEP response components last about 80ms) beginning 20ms apart, with the resultant vector stored in a 64-position array of registers. When a coefficient remains greater than all the others and above a threshold value for a certain amount of time, the corresponding stimulus is considered to have been selected. The 64 positions represent the letters of the alphabet and commonly used words in the English language. The subject can fixate on any word or letter. Whenever the subject fixates on a letter, the commonly used words change to words beginning with that letter, for quick selection of an entire word. Sutter suggests a need to optimize both electrode placement and stimulation mode for each individual subject for good target discrimination. Seventy normal subjects evaluating a prototype system achieved adequate response times ranging from 1 to 3 seconds after an initial tuning process lasting 10-60 minutes. Sutter also tested his techniques on 20 severly disabled persons and describes an experimental version involving an ALS patient using intra-cranial electrodes implanted in the space between the dura and the skull. Cilliers' technique involves varying the intensity of four LED's modulated with a 10Hz sine wave in phase quadrature and detecting the signal in the subject's VEP using a pair of EEG surface electrodes placed on the occipital lobe. The four flashing LED's are arranged around the edge of a computer screen containing an image of a standard four-row keyboard with each row of keys in a different color. Each LED is associated with one of the colors. Fixating on one LED selects a key row, which is redisplayed in four colors for a more detailed selection. The subject can select any particular key in an average of three selections -- about 15 seconds with the current setup. A short initial training period is required where subjects fixate on each LED for 5 seconds. Cilliers' paper describes work with a quadriplegic patient with a C2-level injury. Technique: EEG Pattern Mapping Several experimenters describe techniques for classifying, detecting and mapping EEG patterns. Pfurtscheller's technique (see above) used a neural net featuring learning-vector quantization (LVQ) to map EEG patterns during the 1-second interval before a signal the experimental subject was instructed to wait for. Hiraiwa [Hiraiwa et al 1993?] used a back-propagation artificial neural network to study readiness potentials (RP's) -- patterns in the EEG immediately prior to the subject's uttering one of five different Japanese syllables or moving a joystick in one of four different directions. Twelve channels of EEG data taken >from scalp-mounted electrodes at locations Fp1, Fp2,Fz, C3, C4, Pz, F5, F6, F7, F8, O1 and O2 (International 10/20 system) were used to train and then test two neural networks optimized for averaged data and for single-trial, real-time analysis, respectively. High recognition rates were obtained for the averaged data. Single- trial RP recognition, though less reliable, showed considerable promise in the experimenters' view. Keirn and Aunon [Keirn&Aunon 1990] recorded EEG data from scalp-mounted electrodes at locations P3, P4, C3, C4, O1 and O2 (International 10/20 System) during accomplishment of 5 different tasks during which subjects had their eyes open or closed, for 10 alternative responses. The tasks included (1) relaxing and trying to think of nothing, (2) a non-trivial multiplication problem, (3) a 30-second study of a drawing of a 3-dimensional object after which subjects were to visualize the object being rotated about an axis, (4) mental composition of a letter to a friend, and (5) visualize numbers being written on a blackboard sequentially, with the previous number being erased before the next was written. Feature vectors were constructed from the EEG patterns based on the Wiener-Khinchine method and classified using a Bayes quadratic classifier. The technique discriminated well (>90%) between patterns from all pairs of tasks, and above 70% for the combined tasks. Technique: Detecting lateral hemisphere differences Drake [Drake 1993] studied induced lateral differences in relative brain hemisphere activation after subjects heard arguments through left, right or both earphones which they either strongly agreed with or strongly disagreed with, as determined by prior interviews. Subjects exhibited greater discounting of arguments they disagreed with during left hemisphere activation as measured by ratings of truth. Results supported previous work indicating asymmetries in lateral activation potential during processing pursuasive arguments, however the study did not include measuring directly either activation levels or potentials in the cortex. 3. Discussion The brain-computer interface provides new ways for individuals to interact with their environment. The computer will continue to be a necessary component as long as detecting a brain response reliably remains a complex analytical task. In most cases, the brain response itself is not new, just the means of detecting it and applying it as a control. However, the necessary feedback associated with experimental trials frequently resulted in improved, or at least changed performance. Little is known about the long-term effects of such training either from an individual differences, or from a basic human physiology point of view. All current experiments ally brain responses with one or more of the five senses, most often vision, for stimulating subjects. This emphasis results in experiments that rely on highly directional attention patterns and selection among stimuli presented simultaneously or very close together in time. Absent a detailed understanding of the complex physiological and psychological means of producing the responses, experimenters depend on response traces in the EEG to detect particular occurrences. While the EEG with surface-mounted electrodes provides a minimally invasive way to detect brain activity, many experimenters cite its shortcomings stemming from the extreme reduction of billions of simultaneous electrical events to a few traces, and the attenuation of weak signals by the skull. Some experimenters suggest surgical implantation and single-neuron sensing as supporting more reliable detection. Such techniques were not reviewed in this survey due to their low relevance for BCI applications in normal individuals. However Loeb [Loeb 1989] describes a technique whereby wireless electrodes with transmitters might be implanted using a simple syringe. Reliance on the EEG tends to limit BCI research in several important ways: o Those responses most closely related to stimulus characteristics appear the easiest and most reliable to detect. For example, the VEP experiments described above produce highly reliable results with single trials. o Experimenters favor responses associated with brain areas located closest to the scalp (e.g., P300, VEP) as having the strongest signals over those associated with more remote areas, such as the auditory evoked potential (AEP). o The complex preparation process involving electrode placement and tuning in order to provide for individual differences in brain structure and response strength limits experimenters' abilities to observe gratuitous responses, or brain phenomena other than what they are specifically looking for. Novelty in such experimental settings is most likely to get lost or eliminated as noise by experimental design. 4. Recommendations for Research. Experimenters see BCI techniques as offering much potential for useful applications for individuals with reduced capabilities for muscular response. They cited communicating with others (writing, making their needs known) and manipulating devices (computer, television set, wheelchair) as important targets for brain-computer control. Part of this emphasis stems perhaps from the availability of funds for basic research in prosthetic techniques. BCI techniques might offer all individuals useful ways to interact with their environment. For example, they might play a useful role in delivering information, allowing individuals to peruse vast collections of text, graphic and perhaps auditory material without the requirement for a keyboard, mouse or other hand-operated device. A system based on AEP's might alleviate the need for any sort of display. Electronic games and other forms of entertainment might take advantage of this novel way of responding. Particular vocations that place high demand on one or more current sense modalities, such as piloting aircraft and air-traffic control, might put brain responses to good use. The brain-computer interface has barely emerged as a millieu for useful and important research. Solving even a few of the vast unknowns associated with brain-function and the major difficulties inherent in current apparatus may yield exciting results. As detection techniques and experimental designs improve, the BCI very likely will provide a wealth of alternatives for individuals to interact with their environment. * Prepared for an independent study course at the University of Maryland, Baltimore County (UMBC), under the helpful supervision of Dr. Tony Norcio, January 1994. 5. Bibliography [Cilliers&VanDerKouwe 1993] "A VEP-based Computer Interface for C2-Quadriplegics" Cilliers, P.J., and Van Der Kouwe, A.J.W., presented at 1993 IEEE Conference on Electronic Devices for the Disabled -- Beyond 2000, Fall 1993. [Drake 1993] "Processing Persuasive Arguments: 2. Discounting of Truth and Relevance as a Function of Agreement and Manipulated Activation Asymetry" Drake, R.A., Journal of Research in Personality, 1993, 27:184-196. [Farwell&Donchin 1988] "Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials" Farwell, L.A. and Donchin, E., Electroencephalography and Clinical Neurophysiology, 1988, 70: 510-523. [Flotzinger et al 1993] 'Off-line Classification of EEG from the "New York Brain- Computer Interface (BCI)" Flotzinger, D., Kalcher, J., Wolpaw, J.R., McFarland, J.J., and Pfurtscheller, G., Report #378, IIG-Report Series, IIG: Institutes for Information Processing, Graz University of Technology, Austria 1993. [Hiraiwa et al 1993?] "EEG Topography Recognition by Neural Networks" Hiraiwa, A., Shimohara, K., and Takunaga, Y., journal?, 1993?. [Keirn&Aunon 1990] "Man-Machine Communications Through Brain-Wave Processing" Keirn, Z.A. and Aunon, J.I., IEEE Engineering in Medicine and Biology Magazine, March 1990. [Kohonen 1988] Self-Organization and Associative Memory, 3rd Edition, Berlin: Springer. [Kuhlman 1978] "EEG Feedback Training: Enhancement of Somatosensory Cortical Activity" Kuhlman, W.N., Electroencephalography and Clinical Neurophysiology, 1978a, 45: 290-294. [Kuhlman 1978b] "Functional topography of the human mu rhythm" Kuhlman, W.N., Electroencephalography and Clinical Neurophysiology, 1978a, 44: 83-93. [Loeb 1989] "Neural prosthetic interfaces with the nervous system" Loeb, G.E., TINS, Vol. 12, No. 5, 1989. [McFarland et al 1993] "An EEG-based method for graded cursor control" McFarland, D.J., Neat, G.W., Read, R.F., and Wolpaw, J.R., Psychobiology 1993, 21 (1): 77-81. [Pfurtscheller et al 1993] "Brain-Computer Interface -- a new communication device for handicapped persons" Pfurtscheller, G., Flotzinger, D., and Kalcher, J., Journal of Microcomputer Applications, 1993, 16: 293-299. [Sutter 1992] "The brain response interface: communication through visually- induced electrical brain responses" Sutter, E.E., Journal of Microcomputer Applications, 1992, 15: 31-45. [Wolpaw et al 1991] "An EEG-based brain-computer interface for cursor control" Wolpaw, J.R., McFarland, D.J., Neat, G.W., and Forneris, C.A., Electroencephalography and Clinical Neurophysiology, 1991, 78: 252-259.