Computational Audio Analysis for Cultural Heritage Preservation

Christos Plachouras, an undergraduate student and researcher at New York University Abu Dhabi, presented the work “Mapping the Sounds of the Swahili coast and the Arab Mashriq: Music research at the intersection of computational analysis and cultural heritage preservation” at the 2019 Digital Libraries for Musicology (DLfM) Satellite Event of the Conference of the International Society for Music Information Retrieval (ISMIR). Christos shared the importance of the research that the MaSC research group is conducting, his experience in ISMIR, and how it inspires his future research work.

My name is Christos Plachouras, and I attend New York University Abu Dhabi, located in the United Arab Emirates. I am a junior currently majoring in Computer Science and Music, while being especially interested in areas such as Sound and Music Computing and Music Information Retrieval.

At NYU Abu Dhabi, I am part of the Music and Sound Cultures (MaSC) research group, a group of researchers focusing on the multidisciplinary study of music from the Gulf, the Levant, East Africa, and South India. The type of specialization of each researcher varies greatly; there is expertise in artificial intelligence, digital signal processing, computational modeling, library science, ethnomusicology, music composition, and others. This collaboration results in some fascinating insights about how to analyze music from many different perspectives. For example, for a given piece, what can we infer by analyzing its rhythm? What can we conclude about its timbral evolution by analyzing the soundfile itself? What insights can we get by finding its recording location and date? Did some sociopolitical event influence the lyrics of the song?

The work MaSC is carrying out is meaningful and important. It involves musical traditions that are significantly underrepresented in musical research, and especially in computational approaches to musical analysis. By making field recordings, collecting local music, finding correlations between similar songs and their recording location, uncovering musical influence between nations or tracing immigration patterns, MaSC is contributing to cultural and, especially, musical heritage preservation. One of the key pursuits of MaSC is accessible music education, that’s why many of our efforts are concentrated in building an interface with which users can visually and interactively traverse a music collection, and see the differences and similarities between the musical characteristics or associated artist, location, and genre information of songs. All these efforts have resulted in significant contributions to research in fields ranging from computational ethnomusicology and library science to digital signal processing and machine learning.

My main project with MaSC is developing and using computational methods for the analysis of a music compendium consisting of excerpts from the Middle East – North Africa (MENA) region and the Swahili-speaking East Africa region. The main question I am interested in is what can we infer just by using the soundfile of the music pieces themselves, the question underlying the field of Music Information Retrieval (MIR). Research in MIR suggests different ways with which we can try to identify rhythmic structure, melodic and harmonic content, instrumentation, or danceability of a song by computing lower-level features from the audio and mapping them to musical features we understand. The implications of this extend to the ability to visualize the similarity of these features between musical excerpts, and the creation of groups of those excerpts which share similar characteristics.

The work I co-authored with other members of MaSC entitled “Mapping the Sounds of the Swahili coast and the Arab Mashriq: Music research at the intersection of computational analysis and cultural heritage preservation” explores which lower-level audio features we can use to create those interesting groups of similar music pieces. It also provides details into the recording, collection and digitization of the pieces of this music compendium.

In November 2019, I presented a poster based on that paper at the Digital Libraries for Musicology (DLfM) satellite event of the 2019 Conference of the International Society for Music Information Retrieval (ISMIR). I already look back to that conference as a very memorable and significant point in my academic and research career. I got the chance to discuss with experts in my field about my work and their own cutting-edge research, I got familiar with the future directions that MIR is taking, and I was introduced with fascinating ideas for the future development of the project. Most notably, I experienced a passionate and humble community that was extremely accepting, motivating, and willing to help me find and pursue my own research questions and make a meaningful contribution to ethnomusicology, and, hopefully, the world.

Founded in 1978, the Council on Undergraduate Research (CUR) focuses on providing high-quality and collaborative undergraduate research, scholarly, and creative activity. Among the many activities and networking opportunities that CUR provides, the organization also offers support for the professional growth of faculty and administrators through expert-designed institutes, conferences, and a wide-range of volunteer positions. The CUR community, made up of nearly 700 institutions and 13,000 individuals, continues to provide a platform for discussion and other resources related to mentoring, connecting, and creating relationships centered around undergraduate research. CUR’s advocacy efforts are also a large portion of its work as they strive to strengthen support for undergraduate research. Its continued growth in connections with representatives, private foundations, government agencies, and campuses world-wide provides value to its members and gives voice to undergraduate research. CUR is committed to inclusivity and diversity in all of its activities and our community.

CUR focuses on giving a voice to undergraduate research with learning through doing. It provides connections to a multitude of campuses and government agencies, all while promoting networking and professional growth to its community.