P. A. Wussow, R. B. Willey, and J. Steinberg.
Synchronous Visualization of Video-taped Sounds and Motions of Insects.
Psyche 81:209-218, 1974.

Full text (searchable PDF, 1260K)
Durable link: http://psyche.entclub.org/81/81-209.html

The following unprocessed text is extracted from the PDF file, and is likely to be both incomplete and full of errors. Please consult the PDF file for the complete article.

SYNCHRONOUS VISUALIZATION OF VIDEO-TAPED SOUNDS AND MOTIONS OF INSECTS
BY PAUL A. Wussow, ROBERT B. WILLEY.
AND JUNE B. STEINBERG''
The University of Illinois at Chicago Circle, P.O. Box 4348, Chicago, IL 60680;
and the Rocky Mountain Biological Laboratory. Crested Butte, Colorado 8 I 224.
High-speed cinematography with synchronized oscillography has been used to analyze accurately the sound production by a rapidly moving structure during communicative behavior of insects (Walker & Dew, 1972 ; Willey, 1973, 1974). However, this method involves expensive equipment and perhaps kilometers of film containing only a few hundred feet of analvzable sequences. Nevertheless, the ac- curacy and resolution of audio-visual synchrony by this method has been exceeded only by the use of minute magnetometers using the Hall-effect (Elsner, 1970; Elsner & Huber, 1973) and by an in- genious method of flash-photography (Morris & Pipher, 1972). On the other hand, studies needing an average synchrony accuracy of only å 5 msec (essentially equivalent to motion picture framing rates of 60 pictures per sec) can utilize inexpensive or already available video-systems used commonly as educational tools. Video has several advantages over cinematographic methods: I) sound and picture are routinely recorded synchronously, 2) video- tape can be erased and reused, 3) some video-cameras have great sensitivity and can be used even for nocturnal insects at very low light levels, and 4) the system can be used to monitor behavior for hours and then sequences can be selected, duplicated on another tape , -
and then the original tape can be used again. There are disadvan- tages of video for high resolution work and these will be discussed in the Analysis (q.v.) .
, *
'Mailing address for reprints: R. B. Willey, Assoc. Prof., Biological Sciences, University of Illinois at Chicago Circle, P.O. Box 4348, Chicag~.~ IL 60680.
P. A. Wussow is Data-Systems Analyst with NSF Grant GB-35594, and Computer-Aided Instruction Specialist, Office of Instructional Resources Development, University of Illinois; Mrs. J. B. Steinberg is doctoral candi- date and Teaching Assistant in Biological Sciences. Manuscript received by the editor March 1, 1974.



================================================================================

Psyche
[March
MONITOR S
vtr, cam vtra
Fig. 1. Block schematic diagram of the equipment connected for line- by-line synchronization of a video field with a simultaneous full sweep of the cathode ray gun of the oscilloscope. The CRO is laid on its side so that the horizontal sweep passes from top to bottom of the superimposed field in VTRa (see text, sect. 8, Analysis). For ordinary horizontal sweep,
the CRO is in normal position and the external trigger on the vertical sync is disconnected.
It seems strange that, with the exception of publications from our laboratory (Steinberg & Conant, I 974 ; Steinberg & Willey, 1974). we have found only a few studies of insect behavior which have involved the analysis of video-taped sounds and motion (Loher &
Chandrashekaran, I 970, I 972 ; Hoeg-Guldberg, I 972). Moreover, even these studies have neglected to capitalize on the audio-recording and electronic potential of the video-tape system. We have devised a simple method to superimpose the oscillogi-aphic trace of the sound already recorded on any pre-existing video-tape on a duplicate record- ing. Extensions of this method to synchronous oscillography trans- duced from any mode will make video monitoring of insect behavior amenable to more versatile and precise analysis. A sequence was video-recorded with the following sound-related



================================================================================

19741
Wussow et al. - Sounds and Motions of Insects 21 I parameters noted: I) distance from the microphone to the subect, and 2) the average temperature of air between subject and micro- phone. Then we played the recorded sequence on a compatible video-tape reproducer (VTRl) which was linked by a multichannel synchronizing circuit (SWITCHER) to another video-recorder (VTRa) which preferably used at least one-inch tape. Figure i diagrams the set-up schematically. The audio circuit of the VTR1 was linked also to the input of an oscilloscope (CRO), the screen (CRT) of which has an image persistence on the order of micro- seconds ( PI I phosphor coating). A video-camera (CAM ) focused on the CRT image which was positioned so that it would not mask the image of the subject in the superposition. If the trace was to be positioned along the bottom of the frame, the horizontal sweep of the CRO was set at 10 msec/cm so that a single sweep of 12.5 cm will encompass at least eight video fields (q. v., sect. 7 & 8, Analysis and Fig. I for further explanation and an alternative method). We also masked the negative potentials with opaque tape. The video- camera is connected with the VTRa by the vertical interval switcher and the CRT screen is shielded by an improvised hood from the room lights. The results of superposition are simultaneously video- m~nitored.~
The resulting video-tape, minus all the inactive time, can be played on a VTR which has stop-action and single-field forwarding capabil- ity (or the tape can be moved forward by hand). The audio signal can still be heard as well as seen as an oscillo-trace when the VTR is running at normal speed. At low speed the audio circuit auto- matically turns off and, although an adustment can be made in the machine's circuitry to enable the sound to be heard, with the oscillo- graphic trace this is neither necessary nor desirable. With the VTR stopped on a single field, we marked with a wax crayon the positions of the moving parts on the projection screen of the monitor and made measurements of the angle changes with a protractor (a better method will be outlined later in section 9, Analysis). 'Equipment used: "Quiet Room" (Suttle, Inc., Chicago), Audiometric Room, double-walled (Industrial Acoustics, New York), Sony EV-310 25 mm videocorder, Scotch Brand MT 20568 video-tape, General Electric model 4TE33D1 TV camera with a Soligor 10 cm lens, Sony AVC 3200 TV camera with Angenieux 7.5 cm lens with a +1 diopter auxiliary lens, AKG shotgun microphone model D900, vertical interval switcher-sync generator with sync lock capability (Shintron 360), Tektronix 502A os- cilloscope with Pll phosphor, Panasonic video-monitors (3-inch), one SOW electric spot bulb.




================================================================================

Psyche
[March
Analysis of audio-visual synchrony must consider the following factors :
I) A video field is scanned from top of the screen to the bottom once in 16.67 msec at a maximum resolution of 262.5 lines per field; the horizontal sweep time for a line is 0.0635 msec. The vertical blanking interval (retrace time for the electron-gun to move from bottom of the screen to the top) requires up to 21 lines, and sub- tracted from the maximum resolution leaves a net resolution of 241.5 lines per field with a net duration of 15.5 msec, and a vertical blank- ing interval of I to I .2 msec.
2) A video frame consists of two interdigitating video fields with a total net duration of 31 msec and a total net resolution of 483 lines.
3) Transduction of light and electronic impulses can be regarded as instantaneous, however the speed of sound in air is a variable. Near sea level at 37 C and no greater than 10 cm from the micro- phone, one can calculate that the maximum time interval between sound production and electronic transduction is no greater than 0.4 msec and can be neglected as an important factor. 4) A crucial discrepancy in visual-acoustic synchrony and visual analysis of movements is the time taken by the electron-gun of the video camera to trace any action within a field. This factor causes three time-related distortions in the image which must be considered in the analysis, namely (a) vertical loss of resolution, (b) excessive curvature of a rotating image, and (c) inability to record events which occur after a scanning trace has passed until the next field scan. 5) Vertical loss of resolution: the image formed during a single field has a maximum vertical resolution of only 241.5 lines per field whereas a 16 mm optical-film frame has 50-60 lines of resolution per mm (at the center) or about 700 to 800 lines of resolution overall (allowing for loss of about 20 lines of resolution at the front and rear edge of the frame).
On the other hand, the horizontal resolution of a good video system probably approaches that of a comparable optical- film system. However kaluation of information content of a video field and an optical film fxame encompassing 15 msec is obscured by the next considerations.
6) Excessive curvature of rotating image: the time required for scanning a single field naturally produces a distorted image in any movement having lateral components. Plate la-e illustrates this



================================================================================

19741 Wussow et al. -Sounds and Motions of Insects 313 PI. 1. A portion of a stridulation (chirp) by Cftoriophago wirtdijasciuta, showing a pulse of sound in five photographs of consecutive fields (a-e). The male which is chirping is outlined in (a), (b) and (c) show the tight femur blurred as it makes the downward stroke (left femur is missing). The two vertical black lines drawn on the trace in (d) bracket the portion of the oscillotrace which was scanned at the time the leg was moving in this field. To the right of the lines is the base of a trace which really was much higher, but its peaks will not be scanned until the next field (e). The image persistence is due to retention by the video-camera pick-up tube and is about equal to one field
( = 16 msec). The image in if} is the
result of triggering and synchronizing the CRT sweep with the "front porch" of the vertical field (set up as in Fig. I). The field is identical to (e), but only the most intense peaks were registered. (See text, sect. 8)



================================================================================

214 Psyche [March
problem with the oscillographic trace and a barely detectable clock- wise curvature of the femur tip relative to the base in ib and ic. This is caused by the fact that a video field is a metachronic series of 241.5 lines each representing a "shutter speed" of 1/15,750 sec. The scanning time of the femur image in PI. ia is about 45 lines of 2.86 msec. In other words, the tip of the femur, which is moving in an arc to the left, is imaged nearly 3 msec earlier than that of the femur base. Further magnification on the screen would result in greater temporal displacements and greater artifact in the shape of the organ. Any attempt to measure angles of rotation or points of displacement must take into account the video scanning time of the specific image and its vector relative to the vertical trace. 7)Inability to record events after passage of the scanning trace: it is usually desirable to superpose the oscillotrace along the bottom of the video field (PI. ia-e) where the action is less likely to be obscured. However, the lower portion of the field will be scanned milliseconds after the action in the center. Nevertheless, visual- acoustic synchrony can be computed as shown in PI. id. The selec- tion of the horizontal sweep setting of the CRO is determined by the duration of the longest acoustical signal and should be about 40 or 50 msec longer, so that the oscillotrace of the entire action can be visualized in one sweep, and traces in successive frames can be cross- referenced. The additional time allows some time before and after the signal on the screen so that the beginning or the end of the signal and its causative motion aren't lost in the CRT retrace time, the vertical blanking interval, nor at the sometimes distorted edges of the monitor screen. Sometimes traces are badly reproduced or oriented, but constant inspection of the VTRa monitor will allow such mistakes to be corrected immediately by duplicating the sequence again. 8) An alternative method to achieve visualization of motion-sound synchrony is shown in Fig. I and PI. if. Here the CRO is laid on
its side so the oscillotrace will sweep from top to bottom of the camera pickup tube. The external trigger of the CRT electron-gun is triggered on the electronic transition from the vertical sync to the start of the first horizontal scanning line of the video field. The hori- zonal sweep of the CRT is set to equal the time constant of the video field (15 msec per sweep or I cm/msec), thus only the sound re- corded during the video scan will be oscillographed. The oscillotrace peaks appearing at the same horizontal plane as the video-recorded movement will be relatively synchronous with that movement, with



================================================================================

19741
Wussow et at. - Sounds and Motions of Insects 215
allowance for the time lag induced by the speed of the original sound in air (PI. if, arrows).
9) Close examination of CRTs can be a health hazard and the thickness of the protective cover can induce parallax error. There- fore, if exact angle and displacement measurements are desired the fields should be photographed with careful focussing on the video lines. Such photographs will form a permanent working record of key fields of the rather fragile video-tape which can be damaged or destroyed by the constant friction of scanning in the stopped position. This method allows relatively synchronous visualization of motions and the sounds produced by the movements if the actions are rela- tively slow. For example, it is possible to determine whether the stri- dulation by a single up-down motion of a grasshopper's hind femur rubbing against the fore-wing is produced during the up-stroke or down-stroke, provided the entire motion has a duration of >6o msec. In the case of oedipodine grasshoppers, the chirps produced by males during courtship usually are produced by motions which last at least 60 msec and often 100 msec ( Willey, I 974). Since many such sig-
nals are complicated by being segmented into pulses within a single up-down motion, normal-speed cinematographic and audiospectro- graphic investigations of Oedipodinae (Otte, I 970 ; Willey & Willey, 1969) have concluded that the first pulse must be produced on the up-stroke. However, high-speed motion analysis (Willey, 1974) shows that Arphia sulphurea produces the definitive chirp on the downstroke only. A very weak signal was produced on the up-stroke but was of such low amplitude that it had never been audiospectro- graphed. Video-tapes of Chimarocephala (Loher & Chandrashekaran, 1970) and Chortophaga (Steinberg & Willey, 1974) also show that stridulations are produced only on the downstroke of the femur in these species. However, the mechanism for production of two-pulsed chirps must be analyzed by the higher speed methods (Willey, 1974). Further refinements of this system can involve direct synchrony of oscillotrace with the initial video-tape recording. Also the use of a light-emitting diode (LED) and a small high speed digital clock in the background, synchronized with each video field, would simplify identification of particular fields. However, the method we have reported can be used for improving the analysis of already existing tapes.




================================================================================

Psyche
[March
Oscillographic traces of sounds recorded on videotapes can be superimposed easily and synchronously on a duplicate of those tapes. The audio recording remains intact in the duplicate and the oscillo- trace allows rather precise analysis of slower movements and the sounds they produce. This system was tested on pre-existing tapes of courtship in the grasshopper Chorwhaga viridifasciata (De Geer). The method promises to be useful in analysis of tape's used for monitoring behavior and also as a teaching device. We are grateful to Prof. Gershon Berkson for sponsoring our use of the television facilities of the Small Groups Laboratory of the Department of Psychology at the University of Illinois at Chicago Circle. David Mucha helped prepare the photographic figures. Dr. Ruth Willey gave helpful advice during preparation of the manu- script. Technique development and publication were aided by NSF Grant GB-35594 to RBW through the Rocky Mountain Biological Laboratory.
ELSNER, N.
1970. Die Registrierung der Stridulationsbewegungen bei der Feld- heuschrecke Chorthippus mollis mit Hilfe von Hallgeneratoren. Zeitschr. vergl. Physiol., 68 : 417-428
ELSNER, N. AND F. HUBER
1973. Neurale Gundlagen artspezifischer Kommunication bei Orthop- teren. Fortschr. Zool., 22: 1-48.
HOEGH-GULDBERG, 0.
1972. Pupal sound profduction of some Lycaenidae. Jour. Res. Lepid., 10 : 127-147.
LOHER, W. AND M. K. CHANDRASHEKARAN
1970. Acoustical and sexual behaviour in the grasshopper Chimaro- cephala pacifica pacifica (Oedipodinae). Entom. Exp. & Appl., 13: 71-84.
1972. Communicative behavior of the grasshopper Syrbula fuscovittata (Thomas) (Gomphocerinae) with particular consideration of the male courtship. Zeitschr. Tierpsychol., 3 1 : 78-97. MORRIS, G. K. AND R. E. PIPHER
1972. The relation of song structure to tegminal movement in Metnop- tera sphagnorum (Orth., Tettigoniidae). Can. Entomol., 104: 977-985.




================================================================================

19741
Wussow et al. - Sounds and Motions of Insects 21 7
OTTE, D.
1970. A comparative study of communication in grasshoppers. Misc. Publ., Mus. Zool., Univ. Michigan, 149: 1-168. STEINBERG, J. B. AND R. CONANT
1974. An informational analysis of the intermale behaviour of the grasshopper, Chortophaga viridifasciata. Anim. Behav., 22: in Press.
STEINBERG, J. B. AND R. B. WILLEY
1974. Visual and acoustical social displays by the grasshopper Chorto- phaga viridifasciata (Acrididae : Oedipodinae) . In preparation. WALKER, T. J. AND D. DEW
1972. Wing movements of calling katydids: fiddling finesse. Science, 178: 174-178 and cover picture.
WILLEY, R. B.
1974. Slowed motion studies of sound production in the grasshopper genus Arphia (Acrididae: Oedipodinae) . In preparation. WILLEY, R. B. AND R. L. WILLEY
1969. Visual and acoustical social displays by the grasshopper Arphia conspersa (Orthoptera : Acrididae). Psyche, Cambridge, 76 : 280-305.
WILLEY, R. B. AND P. A. WUSSOW
1973. The flight dances of grasshoppers (Acrididae: Oedipodinae) (motion picture). Amer. Zool., 13 : Abstr. no. 2.



================================================================================




================================================================================