Polar bear mother and cup in the Arctic ocean
Automatic Marine Mammal Mitigation by Infrared Imaging
Use of loud seismic airguns and naval sonars in marine surveys is feared to potentially injure whales and seals when nearby these activities. It is therefore commonly requested to shut-down such operations when marine mammals are sighted within predefined exclusion zones. Monitoring such exclusion zone for marine mammals is usually done visually by marine mammal observers. However, night-time, reduced visibility (rough seas, fog, rain, etc.) or observer fatigue and distraction all limit the reliability of visual monitoring. To facilitate monitoring 24/7 with unwavering vigilance, our work aims at developing an automatic whale detection system utilizing a commercial, cooled 360° thermal imaging sensor.
After more than 10 years of research ashore and at sea, we captured thousands of whale cues (mostly blows and bodily displays) on video and analysed the data for dependency on environmental conditions and specificity. We established that IR imaging will work worldwide in any climate zone (rather than, as initially suspected, in polar and subpolar regions only), 24/7 and for months in a row, except for some rare and transient environmental conditions (fog being the major impediment).
Based on this comprehensive operational at-sea experience, we reframed our Tashtego software and are now in the final stages of finishing the most performant automatic detection system ever, Tashtego PRO, with all the bells and whistles you can imagine. It allows for instant replay of any detection, localization of detected events, automatic adjustment to the respective ship's superstructure, auto focus, auto contrast, and on and on…
False color video of two dolphins leading a humpback whale, swimming along the south coast of Kauai, HI, USA. Recorded by the IRCAM GEMINIS 327k ML dual band (LWIR + MWIR) infrared camera at night on 29 Jan 2016 at 22:16:18 Local Time.
Projects
2013 - | Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0479I): MAPS-IR: Development, Validation and First Use of an IR-Imaging based System for Automatic Detection of whales in the environs of ships and platforms. |
2015-2016 | Bundesministerium für Bildung Forschung und Technologie funded project (BMBF FKZ 03F0725A): THETA: Thermography for the benefit of animal and species protection: automatic detections of whales, seals and walruses in warm and cold seas. |
2014-2016 | Office of Naval Research (ONR Award Number: N000141310856 ) funded project: ETAW: Exploring the thermal limits of IR-based automatic whale detection. |
2015-2018 | Canadian Environmental Studies Research Fund (ESRF) funded project: Assessing the quality of marine mammal detections using three complementary methods. |
2019-2020 | AWI funded project: Tashtego PRO: Rerendering Tashtego. |
Milestones
| 2003 | Project idea |
2004 | Proof of concept using a rented hand-held IR camera (uncooled bolometer). |
2005 | Systematic acquisition of IR data with two passively gimballed directional FLIR Thermovision A40 cameras (uncooled Bolometer). Trial version of detection software tests. |
2009 | Acquisition of FIRST Navy cooled and actively gimballed IR sensor. |
2010 | Beta-version of automatic whale detection software “Tashtego” tested. |
2011 | Software "Fedallah" for off-line analysis of IR data released for internal use. |
2013 | 1st peer-reviewed publication demonstrates the efficiency of thermal imaging based automatic whale detection in waters cooler than 10°C. |
2015 | Patent on detection process details granted in US, EU and RUS. |
| 2016 | Software licensed to RDE for use with FIRST Navy sensor sales. |
| 2020 | 2nd peer-reviewed publication compares visual, acoustic and infrared imaging based whale detection methods. |
| 2020 | 3rd peer-reviewed publication explores autodetection in warm climates. |
Peer reviewed publications / patents
2013 | Peer reviewed publication: Zitterbart, D. P. , Kindermann, L. , Burkhardt, E. and Boebel, O. (2013) Automatic Round-the-Clock Detection of Whales for Mitigation from Underwater Noise Impacts , PLoS ONE . doi:https://doi.org/10.1371/journal.pone.0071217 , hdl:10013/epic.41931 |
2015 | United States Patent No. US 8,941,728 B2, January 27, 2015: Method for automated real-time acquisition of marine mammals. |
2020 | Peer reviewed publication: Zitterbart, D. P. , Smith, H. R. , Flau, M. , Richter, S. , Burkhardt, E. , Beland, J. , Bennett, L. , Cammareri, A. , Davis, A. , Holst, M. , Lanfredi, C. , Michel, H. , Noad, M. , Owen, K. , Pacini, A. , Boebel, O. (2020) |
2020 | Peer reviewed publication: Smith, H. R. , Zitterbart, D. P. , Norris, T. F. , Flau, M. , Ferguson, E. L. , Jones, C. G. , Boebel, O. and Moulton, V. D. (2020) |
Field Work
ANT 21.2 Dec 2003 - Jan 2004 ~1h operational 6 blows captured 9MB data saved | Ship based use of a hand-held IR camera (uncooled bolometer, 304x224 pixels, 30 fps) to explore if whale blows can be recognized in IR video. (+) Whale blows are clearly discernable against polar background within a few hundred meters. (-) Handheld system is highly unstable and unsuitable for long-term marine mammal monitoring. |
ANT 22.3 Jan 2005 - Apr 2005 | Deployment of two directional FLIR ThermoVision A40 cameras (uncooled bolometers, 320x240 pixels, horizontal Field of View (FOV) 24° each, 25 fps) mounted in fixed positions on monkey deck of RV Polarstern. First try of custom developed recording and detection software (provided by external company). (+) 41 blows recorded, 8 automatically. (-) Unheated housing with "windshield wipers" which regularly feeze over. Ship's pitch and roll causes unstable videos due to fixed mounting. Software revision too expensive, development discontinued. |
ANT 23.2 Polarstern Nov 2005 - Dec 2005 | Use of same cameras facing strictly forward, but mounted under roof of Polarstern's crow's nest and passively gimballed against pitch. Custom software develped by U of Chemnitz, forming the start to later software developments. (+) Slightly heated housing inhibits freeze-up of lense. (-) Limited FOV causes missing of blows occurring athwartship and aft. Passive gimbal subject to high winds in crows nest, misdirecting camera downwards. Passive gimbal subject to swinging unless damped, which, however, would require complex mechanics subject to freezing. Passive gimballing discontinued. |
ANT 23.6 Jun 2006 - Aug 2006 | Setup as above, but taking the system to the austral winter. (+) Camera setup endured several storms up to Bft 12. (-) Ice build up on protective Germanium lense, to be removed manually ethanol (89%). |
ANT 24.3 Polarstern Feb - Apr 2008 360h operational 80 GB data collected 37 encounters capured | Operation of 2 FLIR ThermoVision A40 cameras with 7° and 12° lense mounted in crows nest. Cameras can be rotated in directions of interest. (+) Minke whales, which tend to keep greater distance to ship and exhibit weaker blows, were captured with high resolution lenses at distance up to 1km. (-) Directional system requires manual pointing of cameras or a ring of 30 cameras to provide full circumferential monitoring. |
ARK 24.1 2009 400h operational 2 TB data collected | Acceptance test of newly acquired FIRST-Navy infrared imager: 7200 x 586 pixels, horizontal FOV 360° at 5 Hz, cooled line sensor. Gimbal and sensor head mounted in crow's nest of RV Polarstern. (+) system operational for entire expedition. Whale blows perceptible in IR images up to 3 km distance. (-) occasional stutter video stream. |
ANT 26.3 Jan 2010 - Apr 2010 436h operational 2.7 TB data | Rewrite of detection kernel software to facilitate real-time processing and modification of GUI to facilitate use by MMOs. Implementation of hot-pluggable storage media. Ship- and geo-referenced mapping of blow positions established. (+) 19 of 20 encounters (each comprising several blows) detected by new kernel. (-) Failure of sensor head due to damaged mechanical slip-ring. |
ANT 27.1 Oct 2010 - Nov 2010 690h operational | Operation of FIRST-Navy during Polarstern's transit from Bremerhaven to Cape Town. Implementation of a picture-in-picture visual camera system. (+) Two ship-whale encounters recorders. (-) Problems with synchronization of rotation of sensor head and image acquisition. Failure of Sterling cooler after 15 days. |
ANT 27.2 2010 2011 1410h operational 145 autodetections | Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea) together with high-resolution visual camera controlled by IR system. Tashtego software developments: Implementation of advanced horizon tracking, replacement of detection algorithm with more efficient system, infrared imager status information, Improved image acquisition, interface to visual detection software "Walog". (+) Sterling cooler of sensor head achieved nearly 3 months lifetime before failure. (-) false alerts triggered by flocks of birds and rapid motion of horizon in image. |
ANT 28.2 Dez 2011 - Jan 2012 751h operational 123 TB data collected | Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea), also during darkness with autodetector. Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts. (+) Continuous operation for entire expedition. First night time autodetections of whale blows. (-) Focus needs occasional (4 times during expedition) adjustment. |
ANT 28.4 Mar - Apr 2012 586h operational 93 TB data collected | Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea). Tashtego software developments: Introduction of dedicated bird classification layer to reduce false alerts. (+) Continuous operation for entire expedition. (-) Focus needs occasional (4 times during expedition) adjustment. |
ARK 27.3 Aug - Oct 2012 1563h operational | Operation of FIRST-Navy infrared imager in the Arctic Ocean. (+) Successful operation of sensor for entire expedtion without any member of the IR team on board. (-) |
ANT 29.2 Nov 2012 - Jan 2013 726h operational 10 TB data collected >887 IR cues logged | Operation of FIRST-Navy infrared imager in the Southern Ocean (Weddell Sea). (+) Sensorhead #1 was operated for 5 months before failure of mechanical slip ring causing drift in image orientation. Tashtego runs quasi-continuously for > 1 month. (-) Sensorhead #2 fails after 11 days of operation. |
Stradbroke Island, AUS Jun-Jul 2014 854h operational
| Land based operation of FIRST-Navy infrared imager at coast of Stradbroke Island, Australia. Water temperature 22°-24°C. (+) (-) Coastal setting introduces surf as part of image. For proper processing, this requires clipping of image. |
Cape Race, CAN Jul-Aug 2015 146h operational
| Land based FIRST-Navy infrared imager operated from a cliff at Cape Race, Newfoundland. Water temperatures around 16°C. (+) 700 cetacean cues logged. (-) Coastal birds introduce large numbers of false positives, a hindrance which is less prevalent in off-shore seismic and renewable energy construction settings. |
Kauai, USA Jan-Mar 2016 926h operational > 6000 cetacean cues autodetected
| Land based operation of FIRST-Navy infrared imager on both, south and north coasts of Kauai, HI, USA. Water temperature 26°C. Parallel operation of directional IRCAM GEMINI 372k dual (MWIR and LWIR) band camera. (+) Porpoising spinner and common dolphins are clearly perceptible in IR footage. (-) Stand-up paddlers and small boats bopping up and down on waves introduce new, long lasting thermal signatures causing false positives, a hindrance which however is unlikely to occurr in off-shore seismic and renewable energy construction settings. |
| Cape Race, CAN Jul-Aug 2016 ca. 400h operational
| Land based operation of FIRST-Navy infrared imager from a cliff at Cape Race, Newfoundland. Water temperatures around 16°C. (+) 2nd stage classifier for birds implemented. (-) Coastal birds introduce large numbers of false positives, a hindrance which is less prevalent in off-shore seismic and renewable energy construction settings. |
Arctic Ocean PS101 Sep-Oct 2016 | Ship based operation of FIRST-Navy infrared imager on RV Polarstern. (+) Operational use to support bridge-based polar bear watch. (-) none. |
RV Leeway Odyssey Jun-Aug 2017
| Ship based operation of FIRST-Navy infrared imager off south coast of Newfoundland. New generation featuring an athermal lense and optical slip ring for data transfer. (+) New sensor generation exhibits even better image sharpness and insensitivty of focus to temperature changes. Continuous operation of IR system during entire survey. (-) The small (38m length) ship exhibited significant roll and pitch, exceeding the gimbals compensation limits and leading to poor (unstable) video data impeding automatic whale detections. |
Polarstern PS115/1 Aug - Sep 2018 ~60 TByte data collected | Ship based operation of FIRST-Navy infrared imager on RV Polarstern for marine mammal mitigation during seismic survey of BGR. (+) Operated by external personell after 1-day training. (-) On occasion, ship's antennas cause frequent false positives.
|
Polarstern MOSAiC | Ship based operation of FIRST-Navy infrared imager on RV Polarstern for polar bear surveillance. |
PS128 765 h operational | FIRST-Navy infrared imager operated productively on board Polarstern from 12 Jan 2022 until 13 Feb 2022, sensor head A040 SN0002 |
PS134 773 h operational | FIRST-Navy infrared imager operated productively on board Polarstern from 18 Jan 2023 until 19 Feb 2023, sensor head A040 SN0002 |
