The #MusicBricks Toolkit

Gesture Sensors for Music Performance
The R-IoT sensor module embeds a 9 axis sensor with 3 accelerometers, 3 gyroscopes and 3 magnetometers, all 16 bit. It allows for getting 3D acceleration, 3-axis angular velocity and absolute orientation at a framerate of 200 Hz over WiFi. The core of the board is a Texas Instrument WiFi module with a 32 bit Cortex ARM processor that execute the program and deals with the Ethernet / WAN stack. It is compatible with TI’s Code Composer and with Energia, a port of the Arduino environment for TI processors. The sensor module is completed with a series of analysis MaxMSP modules that facilitates its use, based on the MuBu & Co Max library. This collection of analysis tools allows for: filtering and analyzing, computing scalar intensity from accelerometer or gyroscope, kick detection, Mdetection motion patterns such as “freefall”, spinning, shaking, slow motion. Further motion recognition tools are available in the MuBu & Co library.

Melody Extraction
This module includes a number of pitch tracking and melody transcription algorithms implemented in the Essentia library. Applications include visualization of predominant melody, pitch tracking, tuning rating, source separation.

Real-time Onset description
This module allows to detect onsets in real-time and provide a number of audio descriptors. It is part of essentiaRT~, a real-time subset of Essentia (MTG’s open-source C++ library for audio analysis and audio-based music information retrieval) implemented as an external for Pd and Max/MSP. As such, the current version does not yet include all of Essentia’s algorithms, but a number of features to slice and provide on-the-fly descriptors for classification of audio in real-time. A number of extractors analyse instantaneous features like the onset strength, the spectral centroid and the MFCC’s over a fixed-size window of 2048 points, after an onset is reported. Furthermore, essentiaRT~ is able to perform estimations on larger time-frames of user-defined lengths, and to report finer descriptions in terms of noisiness, f0, temporal centroid and loudness.

Sonarflow
A slick UI for browsing music by zooming into a colourful world of bubbles which represent genres, artists or moods which allows to discover new music online from various sources. It is available for iOS and Android, with APIs connecting to 7digital, last.fm, Youtube, Spotify, etc. Demo app in Google Play Store. See sonarflow.com and the source code on github.com/spectralmind.

Synaesthesia
A color detection tool triggering and controlling audio. Based on OpenCV it includes a colour detection engine and object tracking algorithm to control music and sound using plain colours or coloured objects. Using the camera it can trigger musical events. It includes an example app with a dedicated GUI. It is available for OSX and iOS. Find it on github: https://github.com/stromatolite/Synaesthesia

Freesound API
Freesound (www.freesound.org) is a state-of-the-art online collaborative audio database that contains over 200K Creative Commons licensed sound samples. All these sounds are annotated with user-provided free-form tags and textual descriptions that enable text-based retrieval. Content-based audio features are also extracted from sound samples to provide sound similarity search. Users can browse, search, and retrieve information about the sounds, can find similar sounds to a given target (based on content analysis) and retrieve automatically extracted features from audio files; as well as perform advanced queries combining content analysis features and other metadata (tags, etc…). The Freesound API will provide access to a RESTful API with API Clients in Python, Javascript, Objective-C.

Rhythm and Timbre Analysis
This is a library that processes audio data as input and analyzes the spectral rhythmic and timbral information in the audio to describe its acoustic content. It captures rhythmic and timbral features which can be stored or directly processed to compute acoustic similarity between two audio segments, find similar sounding songs (or song segments), create playlists of music of a certain style, detect the genre of a song, make music recommendations and much more. Depending on the needs, a range of audio features is available: Rhythm Patterns, Rhythm Histograms (i.e. a rough BPM peak histogram), Spectrum Descriptors and more. The library is available for Python, Matlab and Java.

MusicBricksTranscriber
(Melody & Bass Transcription + Beat & Key & Tempo Estimation)
The MusicBricksTranscriber provided by Fraunhofer IDMT is an executable that allows to transcribe the main melody and bass line of a given audio file. Also, the beat times, the key, and the average tempo are estimated. The results can be provided as MIDI, MusicXML, or plain XML files. In addition, a Python wrapper is included to further process the analysis results.

POF
POF = Pd + OpenFrameworks : openFrameworks externals for Pure Data, providing openGL mutlithreaded rendering and advanced multitouch events management. A recent addition to #MusicBricks by Antoine Rousseau, in collaboration with Matt Black of Ninja Tune, it makes making PD music apps much easier including cross platform. It also feeds the Mobile Orchestra through SyncJams. Find it on Github.

Musimap
Musimap’s algorithm applies fifty-five weighted variables to each music unit (e.g tracks, genres, labels) so as to model the world’s discography as a multi-layered system of cross-matched influences based on a musicological, lexicological and socio-psychological approach. The granular and proprietary database includes over 3B data points, 2B relations, and soon counting 50M songs. Its neural music network is the result of a unique combination of in-depth human curation and the latest AI technologies to engineer a multi-layered system.

Real-time Pitch Detection
The real-time pitch detection allows to estimate the predominant melody notes (monophonic) or multiple notes (polyphonic) from a consecutive audio sample blocks. This allows to transcribe the currently played / sung note pitches from a recorded instrument / vocal performance. The monophonic version also estimates the exact fundamental frequency values. Typical applications are music games and music learning applications. Fraunhofer IDMT provides a C++ library as well as sample projects that show how to include the functionality.

Search by Sound Music Similarity
The Search by Sound online system is based on the Rhythm and Timbre Analysis (see above) and provides a system which can be used via a REST Web API (called SMINT API) to upload, find and match acoustically similar songs in terms of rhythm and timbre – without the need to install any prerequisite or run the analysis on your own. It can be used with your own custom music dataset or the readily available content from freemusicarchive.org that was already pre-analyzed by rhythm and timbre, to find music matching a particular rhythm or timbre from that archive.

Goatify
The Goatify tool provided by Fraunhofer IDMT is an executable that automatically replaces the main melody in a song with a given sample. Therefore the main melody is extracted and removed from the song. Then the sample is placed and pitched according to the melody notes in the song. For proper pitching of the sample, the pitch of the sample itself is extracted beforehand. The tool is delivered with free sound samples (goat, etc.) from www.freesound.org for direct use.

SyncJams
A recent addition to #MusicBricks, SyncJams is an open source standard to allow wireless inter music app sync and communication of key/scale between players in a ‘mobile orchestra’, authored by Chris McCormick in collaboration with Matt Black of Ninja Tune. Also defined as: “Zero-configuration network-synchronised metronome and state dictionary for music application”. Currently Pure Data and Python are supported. Find it on Github.

Real-time Pitch-Shifting and Time-Stretching
The real-time pitch shifting library allows users to change the pitch of audio material while keeping the tempo. It allows enabled changing the tempo without changing its pitch. Typical applications are music games and music learning applications as well real time performances. Fraunhofer IDMT provides a C++ library as well as sample projects that show how to include the functionality.

Technical Overview

Melody Extraction

Pitch tracking and melody transcription

Platforms:
- Windows
- Mac OS X
- Linux

Programming Environments:
– C++
- Python

Real-Time Usage: NO

Links:
- Overview
- Download
- Tutorial (Python code and Binary executables)

Created by: UPF MTG

Licence: Open source

EssentiaRT~

Onset Description

Platforms:
- Linux
- Mac OS X
- Windows

Programming Environments:
- Pure Data
- Max MSP

Real-Time Usage: YES

Links:
- Documentation and Download

Created by: UPF MTG

Licence: Open Source

RP_extract

Rhythm and Timbre Analysis

Platforms:
- Linux
- Mac OS X
- Windows

Real-Time Usage: YES

Links:
- Overview
- Downloads
- Github Tutorial
- Feature Description

Created by: TU Wien

Licence: Open Source

SMAFE

Rhythm and Timbre

Audio Feature Extraction

Platforms:
- Linux
- Mac OS X
- Windows

Programming Environments:
- C++

Real-Time Usage: YES

Links:
- Github

Created by: Spectralmind

Licence: MIT Licence

SMINT API

Music Similarity API

Platforms:
- Any

Programming Environments:
- Rest API

Real-Time Usage: NO

Links:
- Overview
- API Documentation
- Github

Created by: Spectralmind

Licence: MIT Licence

Search by Sound

Music Similarity Web Frontend

Platforms:
- Any

Programming Environments:
- PHP

Real-Time Usage: NO

Links:
- Frontend Tutorial
- Demo website
- Github

Created by: Spectralmind

Licence: MIT Licence

Sonarflow

Visual Music Discovery App

Platforms:
- iOS
- Android

Programming Environments:
- Objective-C
- Java

Real-Time Usage: NO

Links:
- App Website
- Github
- Google Play

Created by: Spectralmind

Licence: MIT Licence

#MusicBricks Transcriber

Melody & Bass Transcription

Beat & Key & Tempo Estimation

Platforms:
- Linux
- Mac OS X
- Windows

Programming Environments:
- Excecutable
- Python Wrapper

Real-Time Usage: NO

Links:
- Documentation and Download

Created by: Fraunhofer IDMT

Licence: Royalty-free, limited, non-transferable licence

Realtime Pitch Detection

Platforms:
- Linux
- Mac OS X
- Windows
- iOS
- Android

Programming Environments:
- C++ Library

Real-Time Usage: YES

Links:
- Documentation and Download

Created by: Fraunhofer IDMT

Licence: MIT Licence

Real-time Pitch Shifting & Time Stretching

Real-time pitch shifting library

Platforms:
- Windows
- Mac OSX
- Linux
- iOS
- Android

Programming Environments:
- C++ Library

Real-Time Usage: YES

Links:
- Documentation
- Download

Created by: Fraunhofer IDMT

Licence: Royalty-free, limited, non-transferable license

Goatify

Melody extractor and executable

Platforms:
- Windows
- Mac OSX
- Linux

Programming Environments:
– Executable

Real-Time Usage: NO

Links:
- Documentation
- Download

Created by: Fraunhofer IDMT

Licence: Royalty-free, limited, non-transferable license

R-IoT

Gesture Sensors

Gesture Analysis

Platforms:
- Any platform with OSC for receiving raw dataMacOS or Windows for Gesture Analysis with MaxMSP

Programming Environments:
- MaxMSP

Real-Time Usage: YES

Links:
- Documentation

Created by: IRCAM

Licence: Sensor should be returned after the hackathon, Max external and patches free

FreeSound

FreeSound API for Sound Samples

Platforms:
- RESTful Web API

Programming Environments:
- Python
- Javascript
- Objective C clients

Real-Time Usage: NO

Links:
- Documentation

Created by: UPF MTG

Licence: Terms of Use

POF

UI OpenGl components for Pd in openframeworks

Platforms:
- Windows
- Mac
- Linux

Programming Environments:
- PureData
- C++
(openFrameworks)

Real-Time Usage: YES

Links:
- Github

Created by: Antoine Rousseau / Matt Black

Licence: BSD Simplified License

SyncJams

Zero-configuration Music App Sync

Platforms:
- Windows
- Mac
- Linux

Programming Environments:
- PureData
- Python

Real-Time Usage: YES

Links:
- Github

Created by: Chris McCormick / Matt Black

Licence: LGPLv3

Musimap

Musimap API (start-up access) for retrieving musical information about Genres, Artists, Albums and Tracks

Platforms: REST API

Programming Environments:
- Any

Real-Time Usage: YES

Links:
- Musimap

Created by: Musimap

Licence: www.musimap.net/disclaimer

Synaesthesia

A colour detection app triggering and controlling audio

Platforms:
- Mac OS X
- iOS

Programming Environments:
- C++
- Objective-C
- OpenCV

Real-Time Usage: YES

Links:
- Github

Created by: Stromatolite

Licence: MIT Licence