Two presentations at ASA Victoria Meeting

Applying machine-learning based source separation techniques in the analysis of marine soundscapes

Tzu-Hao Lin1, Tomonari Akamatsu2, Yu Tsao3, Katsunori Fujikura1

1Department of Marine Biodiversity Research, Japan Agency for Marine-Earth Science and Technology
2National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency
3 Research Center for Information Technology Innovation, Academia Sinica

Long-term monitoring of underwater soundscapes provides us a large number of acoustic recordings to study a marine ecosystem. Characteristics of a marine ecosystem, such as the habitat quality, composition of marine fauna, and the level of human interference, may be analyzed using information relevant to environmental sound, biological sound, and anthropogenic sound. Supervised source separation techniques have been widely employed in speech and music separation tasks, but it may not be practical for the analysis of marine soundscapes due to the lack of a database that includes a large mount of paired pure and mixed signals. Even when the paired data is not available, different sound sources with unique spectral or temporal patterns may still be separated by apply semi-supervised or unsupervised learning algorithms. In this presentation, supervised and unsupervised source separation techniques will be demonstrated on long-term spectrograms of a marine soundscape. Separation performances under different levels of simultaneous source influence will also be discussed. In the future, more advanced techniques of source separation are necessary to facilitate the soundscape-based marine ecosystem sensing. An open database of marine soundscape will promote the development of machine learning-based source separation. Therefore, we will open acoustic data tested in this presentation on the Asian Soundscape to encourage the open science of marine soundscape.

Information retrieval from a soundscape by using blind source separation and clustering

Tzu-Hao Lin1, Yu Tsao2, Tomonari Akamatsu3, Mao-Ning Tuanmu4, Katsunori Fujikura1

1Department of Marine Biodiversity Research, Japan Agency for Marine-Earth Science and Technology
2Research Center for Information Technology Innovation, Academia Sinica
3National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency
4Biodiversity Research Center, Academia Sinica

Passive acoustic monitoring represents one of the remote sensing platforms of biodiversity. However, it remains challenging to retrieve meaningful biological information from a large amount of soundscape data when a comprehensive recognition database is not available. To overcome this issue, it is necessary to investigate the basic structure of a soundscape and subsequently retrieve biological information. The recent development of machine learning-based blind source separation techniques allow us to separate biological choruses and non-biological sounds appearing on a long-term spectrogram. After the blind source separation, the temporal-spatial changes of bioacoustic activities can be efficiently investigated by using a clustering algorithm. In this presentation, we will demonstrate the information retrieval in the forest and marine soundscapes. The separation result shows that in addition to biological information, we can also extract information relevant to weather patterns and human activities. Furthermore, the clustering result can be used to establish an audio library of nature soundscapes, which may facilitate the investigation of interactions among wildlife, climate change, and human development. In the future, the soundscape-based ecosystem monitoring will be feasible if we can integrate the soundscape information retrieval in a large-scale soundscape monitoring network.

Advertisements

Two presentations at the 10th International Conference on Ecological Informatics!

Information retrieval from marine soundscape by using machine learning-based source separation

Tzu-Hao Lin 1, Tomonari Akamatsu 2, Yu Tsao 3, Katsunori Fujikura1

1 Department of Marine Biodiversity Research, Japan Agency for Marine-Earth Science and Technology, Japan
2 National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency, Japan
3 Research Center for Information Technology Innovation, Academia Sinica, Taiwan

In remote sensing of the marine ecosystem, visual information retrieval is limited by the low visibility in the ocean environment. Marine soundscape has been considered as an acoustic sensing platform of the marine ecosystem in recent years. By listening to environmental sounds, biological sounds, and human-made noises, it is possible to acoustically identify various geophysical events, soniferous marine animals, and anthropogenic activities. However, the sound detection and classification remain a challenging task due to the lack of underwater audio recognition database and the simultaneous interference of multiple sound sources. To facilitate the analysis of marine soundscape, we have employed information retrieval techniques based on non-negative matrix factorization (NMF) to separate different sound sources with unique spectral-temporal patterns in an unsupervised approach. NMF is a self-learning algorithm which decomposes an input matrix into a spectral feature matrix and a temporal encoding matrix. Therefore, we can stack two or more layers of NMF to learn the spectral-temporal modulation of k sound sources without any learning database [1]. In this presentation, we will demonstrate the application of NMF in the separation of simultaneous sound sources appeared on a long-term spectrogram. In shallow water soundscape, the relative change of fish chorus can be effectively quantified even in periods with strong mooring noise [2]. In deep-sea soundscape, cetacean vocalizations, an unknown biological chorus, environmental sounds, and systematic noises can be efficiently separated [3]. In addition, we can use the features learned in procedures of blind source separation as the prior information for supervised source separation. The self-adaptation mechanism during iterative learning can help search the similar sound source from other acoustic dataset contains unknown noise types. Our results suggest that the NMF-based source separation can facilitate the analysis of the soundscape variability and the establishment of audio recognition database. Therefore, it will be feasible to investigate the acoustic interactions among geophysical events, soniferous marine animals, and anthropogenic activities from long-duration underwater recordings.

Improving acoustic monitoring of biodiversity using deep learning-based source separation algorithms

Mao-Ning Tuanmu1, Tzu-Hao Lin2, Joe Chun-Chia Huang1, Yu Tsao3, Chia-Yun Lee1

1Biodiversity Research Center, Academia Sinica, Taiwan
2Department of Marine Biodiversity Research, Japan Agency for Marine-Earth Science and Technology, Japan
3Research Center for Information Technology Innovation, Academia Sinica, Taiwan

Passive acoustic monitoring of the environment has been suggested as an effective tool for investigating the dynamics of biodiversity across spatial and temporal scales. Recent development in automatic recorders has allowed environmental acoustic data to be collected in an unattended way for a long duration. However, one of the major challenges for acoustic monitoring is to identify sounds of target taxa in recordings which usually contain undesired signals from non-target sources. In addition, high variation in the characteristics of target sounds, co-occurrence of sounds from multiple target taxa, and a lack of reference data make it even more difficult to separate acoustic signals from different sources. To overcome this issue, we developed an unsupervised source separation algorithm based on a multi-layer (deep) non-negative matrix factorization (NMF). Using reference echolocation calls of 13 bat species, we evaluated the performance of the multi-layer NMF in separating species-specific calls. Results showed that the multi-layer NMF, especially when being pre-trained with reference calls, outperformed the conventional supervised single-layer NMF. We also evaluated the performance of the multi-layer NMF in identifying different types of bat calls in recordings collected in the field. We found comparable performance in call types identification between the multi-layer NMF and human observers. These results suggest that the proposed multi-layer NMF approach can be used to effectively separate acoustic signals of different taxa from long-duration field recordings in an unsupervised manner. The approach can thus improve the applicability of passive acoustic monitoring as a tool to investigate the responses of biodiversity to the changing environment.

Information retrieval of marine soundscape via unsupervised source separation

Oral presentation in 2nd Oceanoise Asia at Hakodate, Japan

Information retrieval of marine soundscape via unsupervised source separation

Tzu-Hao Lin1, Tomonari Akamatsu2, Yu Tsao1

1Department of Marine Biodiversity Research, Japan Agency of Marine-Earth Science and Technology, Yokosuka, Japan
2National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency, Yokohama, Japan
3Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan

Soundscape, which is a collection of biophony, geophony, and anthrophony, has been considered as a passive acoustic sensing platform of our marine ecosystem. In recent years, different soundscape indices have been developed to describe the variability of a soundscape. However, the analysis of soundscape remains a challenge because of the influence of simultaneous sound sources. Simultaneous sound sources can be separated by using supervised and unsupervised machine learning approaches. Supervised separation requires reference recordings with good signal-to-noise ratio, which is difficult to collect in marine environments. On the other hand, unsupervised separation may be a feasible solution for the analysis of marine soundscape because no prior information is needed. In this presentation, we will demonstrate the application of nonnegative matrix factorization (NMF), a machine learning algorithm, models spectral features and time encoding information from a spectrogram, for the unsupervised separation of marine soundscape. Periodicity-coded NMF (PC-NMF) is a two-stage NMF that aims to separate biological choruses from other noise sources according to the different periodicity properties of diurnal cycles. By using the PC-NMF, croaker choruses can be isolated from a long-term spectrogram interfered with strong mooring noises. We compared the analysis result of PC-NMF with that of a rule-based fish sound detector. The comparison shows that the PC-NMF and the rule-based detector performed consistently in quiet recording conditions. In noisy situations, the PC-NMF and the rule-based detector tend to report false negatives and false positives, respectively. In addition to biological choruses, other noise sources separated by the PC-NMF also give us insights regarding the variability of environmental and anthropogenic sounds. The results suggest that information retrieval from marine soundscape will be possible by using the NMF-based separation algorithm.

Listening to the deep: Exploring marine soundscape variability by information retrieval techniques

Presentation in the session of Lidar and Passive observing sensors, Oceans’18 Kobe

Listening to the deep: Exploring marine soundscape variability by information retrieval techniques

Tzu-Hao Lin1, Yu Tsao2

1Department of Marine Biodiversity Research, Japan Agency of Marine-Earth Science and Technology, Yokosuka, Japan
2Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan

Information on the dynamics of the deep-sea ecosystem is essential for conservation management. The marine soundscape has been considered as an acoustical sensing
platform to investigate geophysical events, marine biodiversity, and human activities. However, analysis of the marine soundscape remains difficult because of the influence of simultaneous sound sources. In this study, we integrated machine learning-based information retrieval techniques to analyze the variability of the marine soundscape off northeastern Taiwan. A long-term spectral average was employed to visualize the longduration recordings of the Marine Cable Hosted Observatory (MACHO). Biotic and abiotic soundscape components were separated by applying periodicity-coded nonnegative matrix factorization. Finally, various acoustic events were identified
using k-means clustering. Our results show that the MACHO recordings of June 2012 contain multiple sound sources. Cetacean vocalizations, an unidentified biological chorus, environmental noise, and system noise can be accurately separated without an audio recognition database. Cetacean vocalizations were primarily detected at night, which is consistent with the detection results of two rule-based detectors. The unidentified biological chorus, ranging between 2 and 3 kHz, was primarily recorded between 7 p.m. and midnight during the studied period. On the basis of source separation, more acoustic events can be identified in the clustering result. The proposed
information retrieval techniques effectively reduce the difficulty in the analysis of marine soundscape. The unsupervised approach of source separation and clustering can improve the investigation regarding the temporal behavior and spectral characteristics of different sound sources. Based on the findings in the present study, we believe that variability of the deep-sea ecosystem can be efficiently investigated by combining the
soundscape information retrieval techniques and cabled hydrophone networks in the future.

Monitoring of coral reef ecosystem: an integrated approach of marine soundscape and machine learning

Presented in International Symposium on Grids & Clouds 2018

Monitoring of coral reef ecosystem: an integrated approach of marine soundscape and machine learning

 Tzu-Hao Lin1, Tomonari Akamatsu2, Frederic Sinniger3, Saki Harii3, Yu Tsao1

1Research Center for Information Technology Innovation, Academia Sinica, Taipei, Taiwan
2National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency, Yokohama, Japan
3Tropical Biosphere Research Center, University of the Ryukyus, Okinawa, Japan

Coral reefs represent the most biologically diverse marine ecosystem, however, they are vulnerable to environmental changes and impacts. Therefore, information on the variability of environment and biodiversity is essential for the conservation management of coral reefs. In this study, a soundscape monitoring network of shallow and mesophotic coral reefs was established in Okinawa, Japan. Three autonomous sound recorders were deployed in water depths of 1.5 m, 20 m, and 40 m since May 2017. To investigate the soundscape variability, we applied the periodicity-coded nonnegative matrix factorization to separate biological sounds and the other noise sources displayed on long-term spectrograms. The separation results indicate that the coral reef soundscape varied among different locations. At 1.5 m depth, biological sounds were dominated by snapping shrimp sounds and transient fish calls. Although not knowing the specific source, noises were clearly driven by tidal activities. At 20 m and 40 m depths, biological sounds were dominated by nighttime fish choruses and noises were primary related to shipping activities. Furthermore, the clustering result indicates the complexity of biological sounds was higher in mesophotic coral reefs compare to shallow-water coral reefs. Our study demonstrates that the integration of machine learning in the analysis of soundscape is efficient to interpret the variability of biological sounds, environmental and anthropogenic noises. Therefore, the conservation management of coral reefs, especially those rarely studied such as mesophotic coral reefs, can be facilitated by the long-term monitoring of coral reef soundscape.

You can also check the slides of this talk.

Improving biodiversity monitoring through soundscape information retrieval

Presented in International Symposium on Grids & Clouds 2018

Improving biodiversity monitoring through soundscape information retrieval

 Yu Tsao1, Tzu-Hao Lin1, Mao-Ning Tuanmu2, Joe Chun-Chia Huang2, Chia-Yun Lee2, Chiou-Ju Yao3

1Research Center for Information Technology Innovation, Academia Sinica
2Biodiversity Research Center, Academia Sinica
3National Museum of Natural Science

Passive acoustic monitoring has been suggested as an effective tool for investigating the dynamics of biodiversity. For instance, automatic detection and classification of sounds can acquire information of species occurrences and behavioral activities of vocalizing animals. However, current methods of automatic acoustic identification of species remain uncertain for most taxa, which constrains the application of remote acoustic sensing in biodiversity monitoring. One challenge is that most of the training samples more-or-less contains undesired sound signals from non-target sources. To overcome this issue, we developed a source separation algorithm based on a deep version of non-negative matrix factorization (NMF). Using multiple layers of convolutive NMF to learn spectral features and temporal modulation of sound signals from a spectrogram, vocalizations of different species can be effectively separated in an unsupervised manner. Based on the pre-trained features, acoustic activities of target species can be efficiently separated from long-duration field recordings. Besides, spectral features of each vocalizing species can also be archived for further utilizations. In this presentation, we will demonstrate the application of deep NMF on separating sounds from different species for both birds and bats. Our results show that the proposed deep NMF approach can be used to establish recognition database of vocalizing animals for soundscape-based biodiversity monitoring, confirming its promising applicability for the field of soundscape information retrieval.

Improving the evaluation of soundscape variability via blind source separation

Presented in 174th Meeting of the Acoustical Society of America @ New Orleans, USA

Improving the evaluation of soundscape variability via blind source separation

Tzu-Hao Lin, Yu Tsao
Research Center for Information Technology Innovation, Academia Sinica

Tomonari Akamatsu
National Research Institute of Fisheries Science, Japan Fisheries Research and Education Agency

Mao-Ning Tuanmu, Joe Chun-Chia Huang
Biodiversity Research Center, Academia Sinica

Chiou-Ju Yao
National Museum of Natural Science

Shih-Hua Fang
Department of Electrical Engineering, Yuan Ze University

Abstract

Evaluation of soundscape variability is essential for acoustic-based biodiversity monitoring. To study biodiversity change, many researchers tried to quantify the complexity of biological sound. However, the analysis of biological sound remains difficult because the soundscape is made up of multiple sound sources. To facilitate the acoustic analysis, we have applied non-negative matrix factorization (NMF) to separate different sound sources in an unsupervised manner. NMF is a self-learning algorithm which factorizes a non-negative matrix as a basis matrix and an encoding matrix. Based on the periodicity information learned from the encoding matrix, biological chorus and the other noise sources can be efficiently separated. Besides, vocalizations of different species can also be separated by using the encoding information learned from multiple layers of NMF and convolutive NMF. In this presentation, we will demonstrate the application of NMF-based blind source separation in the analysis of long-duration field recordings. Our preliminary results suggest that NMF-based blind source separation can effectively recognize biological and non-biological sounds without any learning database. It can also accurately differentiate different vocalizing animals and improve acoustic-based biodiversity monitoring in a noisy environment.