WO2010041147A2 - A music or sound generation system - Google Patents

Abstract

A music system including a processor system; an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; and a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped, wherein the processor system is programmed to generate from the music composition and control data derived from the sensor data a music track, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the presence, order, and grouping of the audio files in the sequence of audio files is determined by both the stored control parameters and the sensor data received through the input signal interface.

Description

A MUSIC OR SOUND GENERATION SYSTEM

[0001] This application claims the benefit of U.S. Provisional Application No. 61/104,041, filed October 9, 2008, the contents of which are incorporated herein by reference.

Technical Field

[0002] This invention relates to a music or entertainment system which employs an adaptive acoustic architecture.

Background of the Invention

[0003] Since the invention of the phonograph, the logic behind the recording and reproduction of music has remained essentially the same. An Artist's music performance is recorded on a physical or digital medium, then replicated and made available to a wider audience. To experience the recorded sound, a listener acquires a copy of the medium and uses a player device suitable for parsing the medium. The reproduced sound is exactly the same each time the piece is played (save for quality variations due to the reproduction equipment).

Summary of the Invention

[0004] The system described herein provides a new medium and method for recording and reproducing music. A new music generation system employing "adaptive acoustic architecture" or "a3" allows an Artist to create a piece of music that has the capability to change every time it is played, following rules and behaviors defined by the Artist. The listener is able to alter several aspects of the music performance, by controlling variables employed in the reproduction process.

[0005] The described system introduces a paradigm shift in how music is experienced. It adds elements of live performance such as unpredictability, improvisation and the ability to respond to the audience to the music experience through the recorded medium, whilst retaining the familiar elements that make a piece of music recognisable. Furthermore, it provides an interactive music experience, allowing the listener to participate in how the music piece evolves, with elements of the music piece changing in response to user or environmental and other triggers. [0006] In the following text the terms user, listener, and audience are used interchangeably.

[0007] The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

Platform

[0008] The system is a music generation system that is programmed by the Artist to produce music changing in real-time according to processing of data that correspond to at least one of sound, location, weather or any other data that can be sensed by the system by employing sensors and/or retrieved by the system by accessing information stored in database or other formats on remote and/or local servers.

Composition Software

[0009] A piece of software running on MAC OS, Windows and/or other platforms allows Artists, composers, and/or users to create a3 content (such as an a3 track) that can be stored in a proprietary a3 file format, preferably with a .a3 file extension.

File Format

[0010] The a3 file format contains a plurality of audio files in compressed (e.g. mp3) or uncompressed (e.g. aiff, wav) format and a file containing the instructions to the a3 player on how the piece is composed in real-time and how it responds to audio and non-audio triggers.

[0011] An a3 file can be modified using the a3 composition software or played using an a3 player.

Player

[0012] The a3 player parses the a3 file format and generates sound in real-time in response to audio and non-audio triggers. The player encompasses a control system that allows the user to change several aspects of the a3 track, to create a user-customisable music experience. Control parameters may include track selection, playback volume, sensitivity, minimum output level, which instruments or voices are on or off, tempo, etc.

Host Devices

[0013] The player can operate on various types of devices, for example mobile phones, personal communication devices, laptop or desktop computers or other standalone electronic devices that can then be embedded in various other products.

[0014] The host devices are preferably connected to microphones and speakers or headphones to sense environmental sound and emit the generated sound into the user's environment or directly to the user. The host devices may also employ other sensors, for example light, temperature or proximity sensors, to monitor other variables or environmental conditions that may be connected to certain aspects of the reproduction of the musical piece. Additionally they may use components for wired or wireless data connections, for example via Bluetooth, WiFi or LAN, or wireless data networks used in mobile telecommunications such as 2G, 2.5G and 3G for accessing information available locally or on the World Wide Web that may be connected to certain other aspects of the reproduction of the piece. For example, weather conditions may be accessed from the host device via a dedicated web address and change the performance of the piece for different weather conditions. Or data on a social networking site or other "Cloud" service may be used to trigger different events in the performance of the piece. Similarly, a particular place of interest, public place or building may emit its identity in a way that can be picked up by the system, so that certain sound characteristics of the piece may be triggered.

Input/Output Devices

[0015] For personal applications, a specially-designed headset comprising microphones mounted on headphones may be used; two microphones may be employed, each mounted on the back of an ear-piece. The headsets may be connected to the host device via wire using custom or standardized audio connector or USB, Firewire or other type of connection, or wirelessly via Bluetooth or other wireless connection protocol. [0016] The headset may incorporate active noise cancellation algorithms, or algorithms that analyze the signals arriving at the two microphones to derive where a particular sound is located relative to the listener.

[0017] For applications in home, office, or other environments, one or more microphones and speaker units may be used, connected to the host units via wire or wirelessly. Depending on the application, the microphones and speakers may be installed in different locations and connected by wire or wirelessly to a host unit, or may be packaged together in a standalone product.

Mobile Phone

[0018] The a3 player may operate on a mobile phone or other personal device connected to the specially designed headset described above or any other commercially-available headset comprising microphone and headphones. The microphone senses ambient sound as the user would hear it and pass the signal to the device. The signal is received through a A/D converter and passed on to the a3 player software running on the device. The software processes the signal and uses data from this analysis to affect the composition of an a3 track selected by the user. The output of the software is passed through a D/A converter to the headphones which are connected to the device by wire or wirelessly as described above.

[0019] The user is able to select through the use of buttons and/or a graphical user interface on the device the active a3 track from a playlist which is preloaded on the device and change basic variables affecting the generated sound, for example the volume, sensitivity (how quickly the generated sound responds to triggers), minimum level, and more advanced variables like tempo, controls that turn instruments or voices on or off or changing their relative volume, etc.

Music with Behavior

[0020] By using a mobile phone or other portable device, the system allows the generation of music that exhibits behavior, in response to certain environmental or other triggers, including but not limited to noise level in various frequency bands, type of noise, GPS location, weather, SMS messages, signature emitted by buildings, location, or systems or devices employed by other users.

[0021] Each music track may contain elements that interact with each other in real-time according to rules defined by the Artist, replicating the improvisation that would occur if the piece was performed live.

[0022] Using the a3 system the user may experience a different version of the track when walking in a big city street, travelling on a train, or relaxing in a quiet place. The track may never be explored fully; for example there may be elements in the track that can only be triggered by a combination of high pitch tones that does not occur in the environments that the listener normally uses the track. Other elements of the track may remain unchanged, giving the music experience a combination of predictability and unpredictability characteristic of live performance.

[0023] Through the a3 system, artists or other users have the capability to influence the music track at playback time, by sending a tagged message to it via SMS, or a social networking or other "Cloud" service. For example, an artist may set up a track that is "listening" for posts with certain tags on Twitter ™ and change the piece at playback time in ways he chooses by posting messages of specific tags and content.

[0024] The a3 system enables a new listening experience. Artists will be able to create pieces that evolve after they have been recorded and their music will remain unexplored even after it's been heard a few times.

Users Creating Their Own Behaviors and Sharing Them with Others

[0025] A listener may interact with the track, by adding other behaviors or editing behaviors that the Artist leaves open for the audience to edit. The Artist programs a set of behaviors using the a3 composition software and creates a base track that is distributed to the audience. The listener may download the track on the mobile or other device and enjoy the track as has been supplied, edit the behaviors, or create new behaviors that can be saved as alternative behaviors for this particular track. For example a user may define a set of behaviors for a certain track that is particularly good for reading in a noisy environment like a public cafe or one that creates a version of the track that the user finds very interesting musically. The user can save these behaviors for re -using later and may share them with others that have the same track. Behaviors can be saved in a very lightweight (for example text) format and be sent to users of other devices via SMS, email, Bluetooth and so forth, or uploaded in web pages for other users to download.

Behavior Auto-Mode

[0026] According to another embodiment of the invention, the a3 player uses an "auto- mode" for the system to recognise different auditory streams in the user's sound environment and create automatic behaviors that group certain components of the a3 track with elements of the user's environment, on the basis of frequency matching (pitch tracking), temporal properties like rhythm, synchronicity, etc. The user can select one of a number of preset behavior types, for example a "social" behavior for the track elements to become louder when the environmental sound becomes louder, or "antisocial" for the track elements to become softer when the environmental sound becomes louder.

Automatic Track Separation

[0027] According to yet another embodiment, the a3 player can process a standard audio file in mp3 or other format and create separate tracks corresponding to individual sound components, for example corresponding to each one of the instruments, vocals, etc. The user can edit the mix (remove and/or change the relative volume of one or more sound components) and set the behaviors for each sound component. Alternatively, the user can select the "auto-mode" described above.

Recording of User Sessions

[0028] The user has the ability to record "user sessions" to share with other users. The sessions are saved as text containing the trigger values that defined the track behavior over a period of time, or as an audio file recording. Audio Commands and Pattern Recognition

[0029] According to another embodiment of the invention, the a3 player software uses an input module that can receive audio commands from the user, for example for pausing and resuming playback, selecting a new track, increasing or decreasing the output level. The module comprises a database of spoken word commands and matches them with input that are received through the microphone of the a3-endabled device.

[0030] In addition one may also use a module for recognising certain types of input and use this categorisation to generate certain types of behaviors. Such inputs are for example traffic noise, male speech, female speech, baby's cry, birdsong, etc.

Location and Position-Based Experiences

[0031] The a3 application may interface with a Global Positioning System (GPS), and evoke certain types of behaviors in response to geographical positions. For example the generated sound experienced by a user in London may change as he's arriving in a tagged location, such as the Tower of London, to emit sound that is allocated to this location. Behaviors or tracks may be emitted at selected locations throughout the city or inside buildings. For example in retail spaces this capability of the system can be used to augment the experience of users while shopping. The users will preferably have the ability to activate an option on their a3-enabled device, to allow or block access of such services to their a3-enabled device.

[0032] Through the a3 music format, users may experience pieces of music as spatial compositions, having their musical elements arranged over a real or virtual territory. As the user moves through this territory, he can physically affect the music experience, for example, approaching one timbre and getting away from another, or moving closer to the lead singer, or the keyboard player in a piece of pop music.

[0033] The a3 track elements may be associated with absolute locations represented by GPS coordinates, or relative positions in relation to where playback of the track was started. In the second case, the user moves through a virtual territory; the elements of the track are positioned around him at certain distances and directions and the music experience changes, as the user is moving. Based on the user position in relation to the a3 track components, the system calculates the dynamic mix using algorithms that simulate the physical propagation of sound in the environment associated with the particular a3 track.

Noise-Related Applications

[0034] Beyond entertainment applications, various embodiments of the invention also have many noise-related applications. Sound is an integral part of our everyday environments and a3- enabled devices can be used in the workplace, the home, hotels, public spaces and public buildings, transport and many other places to create an acoustically-comfortable environment.

[0035] An a3-enabled mobile phone or other portable or fixed device can be used to create for its user a controlled personal sound environment through headphones. The generated sound environment adds a layer of control for the user in public, shared or private spaces, while creating a sound experience that evolves over time, based on the user's location, environmental noise, proximity of other users, etc.

[0036] The device transmits a signature with its state (including information on the activated track and other composition and behavior variables), so that other devices running the a3 software can receive it and synchronise their output to it - so that two or more devices can form an extended connected sound environment between them.

[0037] The software allows users to communicate via Bluetooth when occupying the same virtual sound environment, such as described above.

[0038] The software uses the input module described earlier to selectively respond to the user's name. For example, when the user's name is heard, the sound emitted by the a3 application gradually fades out, in order to allow communication to take place with persons outside the user's sound environment.

[0039] In a similar fashion the a3 player can operate on a laptop or desktop computer to create an immersive sound environment for the user, that can be experienced as the extension of the user's desktop environment and be customised through the operating system. This sound environment can in addition respond to events from other elements of the operating system or applications running on it. For example the generated sound environment may change to signify that a new email message has arrived for the user. Such an application of the invention can positively affect the work performance of knowledge workers, by creating a sound environment that is a part of their workplace and they can quickly return to when they wish to resume a computer-based task.

Adaptive Ring Tones for Mobile Phones

[0040] According to another embodiment, a ring tone is created that is automatically selected according to the ambient sound environment surrounding the user and/or adjusted to the appropriate volume to become louder in loud environments and softer in quiet environments. The ring tone is preferably selected from a library of available ring tones, so that its sound is not masked by ambient noise, for example by having some of its frequency components in parts of the audio spectrum where there's no environmental noise.

Adaptive/Responsive Sound Environments for Games

[0041] According to another embodiment, a sound environment or soundtrack for a game is generated in response to game variables, such as user/character actions, game events, game levels, etc.

Services

[0042] Various services are developed to support the users of the various host devices, including a web-based service for purchasing and downloading a3 audio tracks, a service for users to store and/or distribute to other users altered audio tracks and/or behaviors, based on purchased base tracks and/or tracks created by the users and a service that allows a client machine to receive through wireless or cellular networks a3 audio tracks and/or behaviors.

[0043] In the case of the wireless hosts, a wireless service supplies real-time a3 sound to a local site through access to a remote centralised station.

[0044] In the case of location-based content, a service is used for distributing and updating content of various physical locations and for distributing to users geo-tagged a3 content. Business Models

[0045] Suitable business models are employed for the process of creating a3 tracks and distributing them for a fee via online or other channels, user editing of a base track or the track behaviors or creating new behaviors and selling them for a fee and for the supply and maintenance of location-emitted sound content in places of interest, buildings, retail spaces, etc.

Synopsis

[0046] In general, in one aspect, the invention features a music system including: a processor system; an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; and a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped, wherein the processor system is programmed to generate from the music composition and control data derived from the sensor data a music track, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the presence, order, and grouping of the audio files in the sequence of audio files is determined by both the stored control parameters and the sensor data received through the input signal interface.

[0047] In general, in another aspect, the invention features a music system including: an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; and a composition engine configured to generate from the music composition and control data derived from the sensor data a music track, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the presence, order, and grouping of the audio files in the sequence of audio files is determined by both the stored control parameters and the sensor data received through the input signal interface.

[0048] In general, in still another aspect, the invention features a method of generating a music track for playing on a music system, the method involving: obtaining a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; receiving sensor data carrying information about the environment in which the music system is being operated; and generating the music track from the music composition and control data derived from the sensor data, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein generating involves using the control parameters and the sensor data received through the input signal interface to determine the presence, order, and grouping of the audio files in the sequence of audio files.

[0049] In general, in still yet another aspect, the invention features a music system including: a processor system; an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; and a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped, wherein the processor system is programmed to generate from the music composition a music track, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the order and grouping of the audio files in the sequence of audio files is determined at least in part by the stored control parameters and wherein a volume for each of at least some of the audio files in the music track is determined at least in part by control data derived from the received sensor data.

[0050] In yet another aspect, the invention features a method of generating a music track for playing on a music system, the method involving: obtaining a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; receiving sensor data carrying information about the environment in which the music system is being operated; and generating the music track from the music composition and control data derived from said sensor data, the music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein generating involves using the control parameters to determine the order and grouping of the audio files in the sequence of audio files and also involves using the sensor data to determine at least in part a volume for each of at least some of the audio files in the music track. [0051] In still yet another aspect, the invention features a music system including: a first input interface for receiving a sound signal from the environment in which the music system is being operated; an analyzer module which during operation derives a plurality of separate auditory streams from the sound signal; a second input interface for receiving a plurality of music streams; and a matching module configured to generate a music track by combining at least some of the plurality of auditory streams with at least some of the plurality of music streams based on user-selectable criteria.

[0052] In another aspect, the invention features a method of generating a music track for playing on a music system, the method involving: receiving a sound signal from the environment in which the music system is being operated; deriving a plurality of separate auditory streams from the sound signal; receiving a plurality of music streams; and generating the music track by combining at least some of the plurality of auditory streams with at least some of the plurality of music streams based on user-selectable criteria.

[0053] In another aspect, the invention features a storage medium storing a music composition for playing on a music system that receives sensor data carrying information about the environment in which the music system is being operated, the stored music composition including: a plurality of audio files; control parameters specifying how the audio files are to be ordered and grouped by the music system when the music system constructs a music track from the music composition; and a plurality of rules specifying how control data that is derived from the sensor data controls selections by the music system of audio files from among the plurality of audio files for inclusion in the music track.

Brief Description of the Drawings

[0054] Fig. 1 is a schematic of a host device.

[0055] Fig. 2 is a schematic of the general architecture of the music system using a microphone as a sensor.

[0056] Fig. 3 is a schematic of the architecture of the Analyzer of sound input.

[0057] Fig. 4 is a schematic of the Soundfile Player Logic. [0058] Fig. 5 is a schematic of the architecture of the Adaptive Control for the sound input Analyzer.

[0059] Fig. 6 is a graph demonstrating the effect of the curve parameter to generate nonlinear volume control for the Soundfile Player Adaptive Control.

[0060] Fig. 7 is a schematic representation of a Digital Audio Track in a3 file format.

[0061] Fig. 8 shows views of the main user interface form of the a3 Composition Software.

[0062] Fig. 9 shows a view of the main user interface form for each of the Soundfile Players.

[0063] Fig. 10 shows a view of the user interface form for each of the Adaptive Controls.

[0064] Fig. 11 is a schematic of a host device employing multiple sensors.

[0065] Fig. 12 is a schematic of the general architecture of the music system for receiving control data from other sensors.

[0066] Fig. 13 is a schematic of a composition method according to one embodiment of the invention.

[0067] Fig. 14 is a schematic of a composition method according to another embodiment of the invention.

[0068] Fig. 15 is a schematic of a composition method according to another embodiment of the invention.

[0069] Fig. 16 is a schematic of the general system architecture for behavior auto-mode.

[0070] Fig. 17 is a schematic of the general system architecture for automatic track separation.

[0071] Fig. 18 is a schematic of the general system architecture for receiving audio commands.

[0072] Fig. 19 is a schematic of a client - remote host system. [0073] Fig. 20 shows the music system running on an Apple™ mobile phone device.

[0074] Fig. 21 shows views of the user interface of the music system running on an Apple™ mobile phone device.

[0075] Fig. 22 shows a specially designed headset for use with the music system.

[0076] Fig. 23 shows the a3 music system running on a mobile phone with display and buttons for user instructions.

[0077] Fig. 24 is schematic of a derivative product in the form of a sound bulb, with associated components.

[0078] Fig. 25 shows a derivative product in the form of a curtain.

[0079] Fig. 26 shows a derivative product in the form of a device attached on a glass window.

[0080] Fig. 27 shows a derivative product in the form of a portable device.

[0081] Fig. 28 shows a derivative product in the form of a head- wearable device.

[0082] Fig. 29 shows a derivative product in the form of a sound dress.

[0083] Fig. 30 shows a derivative product in the form of a sound pillow.

[0084] Fig. 31 shows a derivative product embedded in a lounge chair.

[0085] Fig. 32 shows a derivative product embedded in a train seat.

[0086] Fig. 33 shows a derivative product in the form of a sound booth

[0087] Fig. 34 is a representation of derivative products embedded in a building structure.

Detailed Description

[0088] Fig. 1 is a schematic of a host device 100, such as a mobile phone or portable music player, running the a3 music system 102. The music system is connected to a memory 104, which supplies a digital audio track 106 containing sound data and composition logic, created by the Artist. The system parses the audio track and uses the sound and logic data to re-compose the music piece in real-time and output it to the headphone or speaker 108.

[0089] The composition is preferably modified in real-time by data supplied by a sensor 110, such as a microphone sampling sound from the user's environment at the time of the playback.

[0090] A display 112 shows information on the state of the music system, such as for example the name of the active audio track, the playback volume, a graphical representation of the data received by the sensor, etc.

[0091] A user interface 114 allows the user to change various parameters that affect the composition process, allowing him to activate or de-activate parts of the composition, control how the track responds to sensor data, load different tracks and so on.

System Architecture

[0092] The general architecture of one embodiment of the music system 102 using a microphone as sensor 110 is shown in Fig. 2. The system comprises an Audio Track Parser 116, a Composition Engine 118 and Analyzer 130.

[0093] The Audio Track Parser 116 parses the Digital Audio Track (called "a3 Track") and loads all the sound data and composition parameters into the Composition Engine 118. The sound data and composition parameters are set by the Artist and saved in an a3 Track file format. More details on the structure of the a3 file format are given below.

[0094] The Composition Engine 118 comprises a number of Soundfile Player (SFP) modules 122 (e.g. 12), each of which is capable of playing pre-recorded audio files. Each SFP maintains a list of sound files and plays them in a sequence according to parameters set by the active a3 Track.

[0095] For this embodiment receiving Audio input from the Sensor 110, the Analyzer 130 is designed to receive audio input and perform a spectral analysis of it. The signal analysis takes place in the spectral domain, dividing the signal into a number of bands, reflecting human auditory perception.

[0096] The strength of the signals in these bands are passed, as Band Levels 150, to the Adaptive Control 120 that facilitates communication between the Analyzer and Soundfile Players. The Adaptive Control in this embodiment allows the Soundfile Players to react to the input signal, for instance adjusting their volume as ambient noise increases and decreases. Soundfile Players may respond in other ways to changes in the input signal, for instance they may start or stop playback in response to the ambient levels, or adjust their playback volume.

Analyzer

[0097] Fig. 3 is a schematic of the architecture of the Analyzer 130. When the system is in active ("Play") state, the audio signal from the user's environment is fed into a Buffer & Window Input module 136. This buffer is sampled regularly and a spectral-domain representation of the signal is calculated in step 140. A windowed FFT implementation is used, typically with 1024 points.

[0098] The spectral representation is reduced to a vector of absolute amplitude Band levels 150 in Equivalent Rectangular Bandwidths (ERBs), in the Band Analysis Module 134. The default setting is a division of the spectral domain into 42 bands with frequency boundaries as shown below.

[0099] The ERBs are optionally calculated using a more advanced, but more computationally expensive, Masked Threshold mode in Excitation Calculation module 148. This calculation takes into account the effect of the spread of masking between different bands and temporal masking.

[0100] It will be appreciated that the Analyzer arrangement described above is suitable for processing sound signals when a microphone is employed as sensor 110. Different Analyzer modules will be employed for processing signals from other sensors, such as proximity or location sensors, accelerometers, light sensors and so on.

Sound File Player

[0101] Fig. 4 is a schematic of the logic of each of the Soundfϊle Players. Each Soundfile Player can be programmed to accept up to 64 different sound files that can be played back in deterministic or non-deterministic sequences. Each sound file has an index number ranging from 0 to one less than the number of sound files in use. When a Soundfile Player starts to play, it will by default choose one of the sound files at random from the listed sound files. Optionally, it may be specified that play should always commence with the first sound file listed.

[0102] The Soundfile Player of this embodiment uses a simple state machine model in which a sound file is played from beginning to end; on reaching the end of file (EOF), the sound file can be followed by a period of silence lasting from zero seconds to one hour. When the period of silence ends, a sound file is selected randomly from a list of "follower sound files". If only one sound file is specified, the state transition is necessarily deterministic. If there is no valid follower index for a given sound file, the Soundfile Player will begin as at the beginning of playback, either playing the first sound file or selecting one of the sound files at random.

[0103] As said above, the playback Volume of each Soundfile Player can be controlled by an associated Adaptive Control module (also see below for more details). The A/C can also start, stop, or mute the Soundfile Player playback.

Adaptive Control

[0104] The architecture of the Adaptive Control for this embodiment is shown in the schematic of Fig. 5. [0105] All Soundfile Players have an associated Adaptive Control (AIC) module. Each Soundfile Player has an associated AJC unit that may be enabled to control the Soundfile Player's Volume, or to control whether the Soundfile Player itself is active or inactive. These controls may be enabled in any combination. If any of the control functions is enabled the AJC is said to be activated; if all are disabled, the AJC is inactive.

[0106] For each AJC module, a contiguous Region of ERBs is specified for the AJC to "listen" to, between a lowest and highest band.

[0107] The latency with which the Band Processor responds to changes in signal level can also be specified. This latency is specified in terms of Attack Time and Decay Time. Attack and Decay are defined as the amount of time for the region signal to increase or decrease by 6OdB. If the actual audio input signal is changing faster than the specified Attack or Decay Time, the Band Processor limits its calculated level to the rate of change specified. This limitation is not applied to signals below the threshold of auditory perception. The software uses a simplified definition of this threshold; signal levels measuring below -96dB are not subjected to Attack Time limitations.

[0108] Attack and Decay times at zero seconds allow the system to respond to ambient noise as fast as the underlying hardware and drivers are able to (subject also to the latency of the FFT/IFFT transform performed in the Audio Analysis and Synthesis stages).

[0109] The actual Attack and Decay times to be used are calculated based on two interdependent sets of parameters. First, each Band Processor maintains a minimum and maximum value for both Attack and Decay time. These parameters are typically set by a3 track supplied by the Artist. Additionally, the global Sensitivity setting (typically set by the User) is read by each Band Processor, to determine an interpolation between the minima and maxima of the Attack and Decay times.

[0110] The formula used for calculating the interpolation is non-linear, reflecting the exponential nature of human perception of time. Also note the negative logic: lower sensitivity implies longer Attack and Decay times.

[0111] The formula implemented is the following:

where

δ_eff = Effective Attack or Decay time δ_mm,δ_mRX = Minimum, Maximum Attack/Decay time s = sensitivity factor

[0112] To conserve CPU usage, only those Band Processors that are active actually have their levels calculated. This can lead to small inaccuracies in the response to input signals if a Band Processor is activated or deactivated during playback. However, the User will typically only be able to activate or deactivate Band Processors by changing to a new a3 Track; in this case so many other parameters will change that any inaccuracy in the level calculations of Band Processors is negligible.

[0113] The Adaptive Control can map the average level in the specified region to control the volume of the associated Soundfile Player; the AJC can stop, start, or mute playback of the associated Soundfile Player in response to input levels. All these function can be activated independently.

AJC Volume Control

[0114] When active, volume control maps the input levels calculated by the AJC to a playback volume for the associated Soundfile Player. By convention, the output volume is specified as a MIDI Volume level. This value is mapped to a gain using the formula g = (127 - 0.5v) [dB], where g represents gain and v represents the MIDI Volume.

[0115] The mapping is linear by default, but a polynomial curve mapping (for instance, parabolic) can be specified by a Curve parameter. The graph of Fig. 6 demonstrates the effect of the curve parameter to generate non-linear volume control. The software linearly maps the input level to the unit range (relative to in input min/max values), then applies the polynomial formula curve y - ^χ , where k = 2 [0116] The result of this calculation is then linearly mapped to the output range. Mapped values are clipped to the user-specified output range by default. Alternately, the mapping can extrapolate beyond the output range specified.

[0117] The Volume control can, in some circumstances, reduce output levels below the threshold of audibility. This is similar in effect to the Playback Start/Stop/Mute function described below. However, there are subtle differences in the effect. With Volume control the Soundfile Player keeps on playing, progressing through whatever sequence of events it traverses unperturbed by the artifact of inaudibility.

A/C Playback Start/Stop/Mute Control

[0118] The Playback Control is used to start, stop, or mute a Soundfile Player in response to measured input levels.

[0119] User-specified thresholds specify that Region levels above a given threshold turn the associated Soundfile Player "On", and that levels below a (possibly different) threshold turn the Soundfile Player "Off. The Off threshold must be less than or equal to the On threshold.

[0120] A Soundfile Player can be turned Off by controlling its Mute status. In this case, whenever Region levels sink below the Off threshold, the Soundfile Player is muted. This is the equivalent of the User setting the Soundfile Player's state to muted. The Soundfile Player remains muted until the measured Region levels exceed the On threshold. At this point the mute state is turned back on. While muted the Soundfile Player halts all event processing; when the mute is disabled, the Soundfile Player resumes processing from wherever it was when it was muted. Concretely: if a Soundfile Soundfile Player is muted while playing, for instance, its third Soundfile, it will resume playback from the position within the Soundfile it had reached when the mute was set. This is termed the "mute model" of playback control.

[0121] Alternately, Playback control can be affected via switching between a given Layer setting and the "Off" layer. The On and Off thresholds behave the same, however, turning a Soundfile Player Off and then On with this method causes the Soundfile Player to restart from its notion of "beginning". Concretely: if a Soundfile Player has its layer set to off while playing any sound file, when Region levels subsequently exceed the On threshold and the Layer is reset to On, the playback starts with the beginning of either the first file, or the beginning of a randomly chosen file (depending on the Soundfile Player's "Start With" parameter).

a3 File Format

[0122] The music system described above is an interactive music system that can generate music composed in real-time, responding to sensor data reflecting conditions in the user's environment during playback.

[0123] Artists create a3 Tracks by programming the Composition Engine using specially designed Composition Software, which allows them to program compositions using pre-recorded sound material. The sounds they use and all the composition parameters they set are saved in a specially designed a3 file format that has the structure shown in Fig. 7. The a3 file comprises an a3 Header section 201, an a3 Composition Settings section 202 and an a3 Audio Files section 203.

[0124] The a3 Header section 201 contains information about the a3 Track, including Track title, Artist, Album, Track artwork, short description, a3 version, frequency rate, etc.

[0125] The a3 Composition Settings section 202 contains all the composition settings in a formatted text file, preferably in JSON or XML format. This contains the names of all the Audio Files used in the a3 Track and all the programming parameters that inform the system how they are grouped and sequenced, how the Adaptive Controls are programmed, etc.

[0126] The a3 Audio Files section contains all the audio files used in the a3 Track, preferably in compressed (e.g. mp3) or uncompressed (e.g. aiff or wav) format. The different a3 Tracks may be of different durations (from a few milliseconds to several minutes long) and they may be of different sample or bit rates, according to the type of sound.

[0127] The following is part of an example formatted text file in JSON format contained in section 202 of the a3 Track file. The file contains names of composition parameters in quotation marks (e.g. "Soundfile Player 1", "state," etc.) followed by their corresponding values, as set by the Artist. The Audio Track Parser 116 parses this text file, and assigns each value to the corresponding parameter field of the Composition Engine 118.

{

"soundfile player": [

{

"name (do not edit)": "Soundfile Player 1",

"state": {

"mute": false,

"layer": [1, 1]

},

"adaptive control": { "audio region": { "band range": [0, 12], "attack time": [1, 10], "decay time": [5, 20]

},

"volume parameters": {

"active": true,

"trigger levels": [-83, -54],

"output range": [60, 127],

"scaling curvature": -20,

"clip output": true

},

"onVoff parameters": {

"active": false,

"trigger levels": [-50, -70],

"mute control": true,

"layer control parameters": [1, 1]

}

"sound files": [ {

"name": "lst.wav",

"active": true,

"silence after": 5,

"can be followed by": "1"

},

{

"name": "2nd.wav",

"active": true,

"silence after": 0,

"can be followed by": "0 1 2 3"

},

{

"name": "3rd.wav",

"active": true,

"silence after": 0,

"can be followed by": "6"

},

{

"name": "4th.wav",

"active": true,

"silence after": 5,

"can be followed by": "0"

},

{

"name": "5th.wav",

"active": true,

"silence after": 5,

"can be followed by": "0"

},

{ "name": "6th.wav", "active": true, "silence after": 5, "can be followed by": "All" }, {

"name": "7th.wav", "active": true, "silence after": 10, "can be followed by": "2" }, ], "can start with": "0",

"volume": 45, "pan": 64

{

"name (do not edit)": "Soundfile Player 2",

[0128] In the example copied above the Composition parameters for Soundfile Player 1 will be active and set on Layer 1,1.

[0129] The Volume output of the Soundfile Player will be actively controlled, in response to Frequency Region between lower frequency limit of Band 0 (20 Hz) and the upper limit of Band 13 (775 Hz). Energies in that frequency range between -83 dBFS and -54 dBFS are mapped to Soundfile player volume between 60 and 127 in the MIDI range.

[0130] The Soundfile Player's state (on/off/mute) is not actively controlled, so the Player will remain active all the time. [0131] The Attack time is set between 1 sec and 10 sec and Decay time between 5 sec and 20 sec (the active Attack and Decay times will depend on the User Sensitivity Setting as said above).

[0132] The playback logic will follow the setting of the table below:

These settings will result in sound file with IDO to be played first followed by 5 sec silence. IDl will be played next and then immediately, one of the IDO, IDl, ID2 or ID3 will be randomly selected and played. If ID2 is selected, the Soundfile Player will enter a loop playing only ID2 and ID6, with 10 sees silence after each time that ID6 is played. If any of the ID3 or ID4 is selected, the Soundfile Player will go back to IDl and so forth. If ID5 is played, any of the available sound file will be selected and played at random (incl. ID5).

[0133] For this a3 Track to be reproduced on an a3-enabled device, all the sound files named above need to be contained in the a3 Audio File section 203 of the a3 Track.

[0134] A summary of the sound variables set for each Soundfile Player in this embodiment is shown in the Table below:

Layer State Assigns the player to a Layer. Layer [0,0] is off, layers [1,1], [1,2], [1,3], [2,1], [2,2], [2,3], [3,1], [3,2] are on

List of soundfϊles, Defines which soundfiles silence intervals will be played back and and following how they are played back soundfiles Starting sound file Sets which soundfile should be played first ("All" is set for one to be chosen randomly)

Audio region band Adaptive Defines the low and high range control band of the audio region for the adaptive control of the soundfile player

Audio region Adaptive Defines the minimum minimum attack control attack time within region time for adaptive control

Audio region Adaptive Defines the maximum maximum attack control attack time within region time for adaptive control

Audio region Adaptive Defines the minimum minimum decay control decay time within region time for adaptive control

Audio region Adaptive Defines the maximum maximum attack control dacay time within region time for adaptive control

a3 Composition Software

[0135] A piece of software running on MAC OS, Windows and/or other platforms allows the Artist to create a3 tracks that can be stored in a proprietary a3 file format. Such a piece of software will be described here, by referring to Figs. 8, 9 and 10, which show views of its graphical user interface.

[0136] Fig. 8 shows views of the main user interface form containing the global software controls and the Tracks and Analysis sub-menus.

[0137] Processing can be started or stopped via the "Stop" and "Play" buttons.

[0138] The In/Out Mix slider affects the mix between the signal sensed by the microphone which is connected to the computer and the sound output generated by the composition engine in real-time.

[0139] The Volume Slider affects the global maximum volume of the generated output, whereas the Minimum Level slider changes the minimum volume of the generated output. Raising the Minimum Level creates a virtual input to the composition engine that makes all adaptive components behave as if the ambient noise level were always at least at the specified dB level.

[0140] The Sensitivity slider allows a certain amount of global control of how quickly or slowly adaptive elements of the system will react to changes in ambient noise. As described below, the Artist specifies minimum and maximum values for latency in increases and decreases in input signal levels (Attack and Decay). The sensitivity setting causes each such adaptive element to calculate a value within the range specified by the minimum and maximum values. Empirical testing has indicated that a slightly non-linear interpolation between minimum and maximum gives a better feeling of response. [0141] The Analysis menu allows the control of size of the FFT analysis (FFT Frame Length) as well as the number of in Equivalent Rectangular Bandwidths used in the analysis of the system input signal. The default values are 1024 bins/FFT and 42 ERBs. Additionally, there are two analysis modes known as "Critical Band" (default) and "Masked Threshold" analysis that can be selected.

[0142] Fig. 9 is a screenshot of the main UI form of each of the Soundfϊle Players.

[0143] Via the Layer Menu on the top left, the Artist can assign the Soundfile Player to one of the available Layers. Layers are structural units used by the Artist to organize how the different Soundfϊle Players function in his concept of the composition. However, the selection of a particular layer has no effect on the functionality of the Soundfile Players, beyond the fact that setting the Layer to "Off deactivates the Soundfϊle Players. Setting a Soundfϊle Players to any other Layer activates the Soundfϊle Player. The Layers available are grouped into three Static Layers (labeled Al, A2, A3); three Responsive Layers (labeled Bl, B2, B3); and two Incidental Layers (labeled Cl, C2).

[0144] Static layers contain Soundfϊle Players that do not respond to input dynamically. They are scripted soundfϊle sequences determined by playback logic such as the one described in the example above. Dynamic layers contain Soundfϊle Players that are always on, but their playback is controlled dynamically by an input signal. For example, energy of sensed noise within a specific frequency band can be mapped to the output level of a Player. Incidental layers contain Soundfile Players that are not always on, but are triggered by the sensed inputs. For example, when energy of sensed noise within a specific frequency band goes beyond a set threshold, a Player on an incidental layer will be turned on.

[0145] All Soundfile Players also have a Mute checkbox, allowing the Artist to manually set the Soundfile Players to mute state.

[0146] The table in the center of the form allows the Artist to set all the main playback parameters. The sound files to use are selected from a Combo Box/Popup Menu. The Silence before Next column will accept non-negative integers less than or equal to the maximum time between sound files (currently set at one hour). The Artist can type in arbitrary lists of sound file indices into the Next Soundfile[s] column. Commas and/or spaces may be used as delimiters; on instantiation the Form formats the list using unspaced commas as a delimiter. As a convenience, the word "All" can be used as shorthand for all sound file indices from 1 to 64.

[0147] Soundfile playback level can be controlled by the Gain trackbar.

[0148] Through the Adaptive Control button on the bottom right, the A/C parameters can be set, as shown in Fig. 10.

[0149] The controls in the Group Box labelled Region Parameters function allows the definition of the Frequency Region the A/C will follow by setting the Low and High Band number boxes. The number boxes on the right set attack and Decay times.

[0150] While the a3 engine is running, the level measured by this A/C unit is displayed, updated approximately every 100ms.

[0151] In the Volume Tab, the "Map Input to Volume" checkbox controls whether the A/C should control Volume (Gain) in the parent Soundfile Player. If checked, the A/C unit overrides the manual setting made in the parent Soundfile Player Form. The Artist specifies the minimum and maximum input levels (in dBFS) with the two Input Range trackbars and the output levels (measured in MIDI Volume units) with the two Output Range numeric controls. The equivalent dBFS values for the output range are displayed in the form. Although the Low and High values of the input range must be strictly increasing, it is valid for the "Low" value to be greater than or equal to the "High" value of the output range. Equality creates a degenerate situation in which the A/C sets the Soundfile gain to a constant. In situations where the "Low" value is greater that the "High", the Artist is specifying that the Soundfile Player gain reacts inversely to the measured audio levels: when the Band Levels measured in the specified Region increase, the Soundfile Player gain decreases.

[0152] By default, input levels between the low and high extremes of the user-specified Input Range are scaled linearly to output levels in the output range specified. However, if a Curve value other than zero is specified, a polynomial interpolation, as described above will be applied. [0153] If the "Clip Scaled Values to Output Range" checkbox is checked, the Back End will clip the output values to specified Output Range. Otherwise the Back End will extrapolate Band Levels outside the input range to values outside the specified range using the same formula as used for interpolating values inside the Input Range. However, output values are always clipped to the valid MIDI range of [0 .. 127], ie a gain from -∞ to 0 dB.

[0154] Independently of whether Volume control is selected or not, the Artist can specify that the Band Levels in the A/C Region control the playback of the parent Soundfile Player, starting, stopping, or pausing playback.

[0155] If the "Input Triggers On/Off checkbox is checked, the A/C will control playback according to the other parameters in the On/Off tab. The Artist specifies levels for triggering the On and Off states. When the input level is greater than or equal to the value set in the Levels Above trackbar (which is annotated with the label "Turn Player On"), Soundfile Playback will be on. When the input level is less than or equal to the value set in the Levels Below trackbar (annotated with the label "Turn Player Off), Soundfile Playback will be off. Input levels between the two specified values will not cause any change in the playback state; the Soundfile Player will simply remain in its current state.

[0156] There are two modes of playback control: Mute Control and Layer Control. These are selected by the Radio Buttons On/Off by Mute Control and On/Off by Layer Control. In the first case, when the input band levels drop below the "Turn Player Off threshold, playback is stopped by setting the Mute state of the parent Soundfile Player to true. In the case of the Soundfile Player this is analogous to pressing "Pause" on a CD or Media Player. When the input levels later rise above the "Turn Player On" threshold, playback resumes from exactly the place where it was paused.

[0157] On the other hand, if Layer Control is selected, when playback resumes because the input levels have risen about the Turn Player On threshold, the Soundfile Player will restart with the beginning of the first soundfile (or by choosing a soundfile at random, if the Always Start with First Soundfile checkbox in the parent Form is unchecked). [0158] When Layer Control is selected, the A/C will, by default, use the same Layer as that selected in the Layer menu of the parent Soundfile Player's Form. However, if the manual selection in the parent Soundfile Player Form is "Off, the default Layer used by the A/C is Layer Al . The user can override the default selection by choosing from the items listed in the Layer Control Combo Box. The Combo Box offers exactly the same selection of layers as the standard Layer menu.

Additional Sensors

[0159] Fig. 11 is a schematic of a host device employing additional sensors, which may be embedded in the host device, or be wirelessly connected to it.

[0160] Such sensors track other variables from the user's environment, such as light, temperature, etc., or variables representing the user's state, such as heart rate, galvanic skin resistance, etc., or the user's interaction with the device, via accelerometer or proximity sensor. The host device may additionally be equipped with Global Positioning System and data networking capability to allow it to access information available locally or on the World Wide Web.

[0161] Fig. 12 is a schematic of an alternative embodiment of the Music System 102, employing Analyzer and Adaptive Controls components that receive and process data from the sensors described above, or from data available locally or on the World Wide Web.

[0162] Such data may be programmed to affect the playback of an a3 track in similar ways to the microphone data described earlier. For example, time of the day or weather conditions may be retrieved by the host device from a dedicated web address and change the performance of the piece for different times of the day or weather conditions. Or input from the host device accelerometer may be used to synchronise the rhythm of a particular a3 track to the pace of a user running or dancing.

[0163] It should be appreciated that the architecture of the system shown in Fig. 12 may contain multiple instances of the Analyzer 130, each instance corresponding to a different type of sensor data, which could be used in combination. [0164] Using such a system, the Artist is able to establishing complex rules to trigger sounds within the composition. The rules are conditional statements using one or more types of sensor data, thus programming complex behaviors into the system. For example, a behavior rule may be programmed as follows:

IF time is between 10 am and lpm AND location is user's office, THEN trigger soundfile XYZ.

[0165] This type of system can also be used to create music experiences that can be influenced by the Artist at the time of playback, or are connected to virtual environments that the user may be interested in.

[0166] For example, an a3 track may be released with such control parameters that certain parts of the composition are hidden, ie not recalled by the system. At some arbitrary time after the track is released and distributed to a wide audience, the artist may unveil these hidden parts by sending a message through a network service (for example by sending an appropriately tagged Twitter™ message). The message changes the control parameters so that these parts of the composition are recalled and a seemingly new part of the track is revealed to the listener.

[0167] Or, an a3 track may be connected to the user's Facebook™ page through control rules that change the music experience when certain events happen in the user's social group. For example, SoundFile players or sound files within them may be activated or de-activated, depending on "friends" activity; one of the background layers in an ambient a3 track may disappear if a chosen friend doesn't post an update in a specified time period; or foreground sounds may be triggered every time any member of the user's group posts a message on the user's Facebook™ page.

[0168] Fig. 13 shows another embodiment of the invention, allowing the Artist to graphically position the sound files on a timeline and assign behavior rules using a graphical user interface. [0169] The Artist assigns a finite playback duration to each sound file, and also defines beginning and end times. The Artist is able to set more than one sound files as available options at chosen times.

[0170] For example, sound file A may be active for the first 1 :30 minutes, and respond to sensor data using behavior rule i. Rule i could be, for example, varying the output volume of sound file A between 60 and 127 in midi values, matching the energy contained within critical bands 0 to 12. At 1:30 minutes the algorithm is set to choose one of four sound files 1, 2, 3, 4 using rule j. Rule j could be, for example, a time-based rule, such as: activate branch 1 if time is 7- 10am, or branch 2 if 10-am-lpm, or branch 3 if lpm-8pm, or branch 4 at any other time. If time is between 10am and 1 pm, branch 2 is activated and sound file 2 is chosen for playback using behavior rule m. Rule m could be, for example, allowing playback of soundfile 2 only if a twitter entry containing a specific tag is logged within a specific preceding timeframe. If such an entry is not logged, soundfile 2 is not played, even though branch 2 is selected, resulting in a silent interval for the duration of soundfile 2, between 1 :30 minutes and 2:20 minutes. At 2:20 minutes, the algorithm plays sound file B with rule o and at 2:40 minutes it chooses again one of five sound files a, b, c, d, e based on rule p. For the last 50 seconds the algorithm plays sound file C until the track ends at 4 minutes duration. Using this method, the Artist can create a track with segments that are the same each time the track is played and segments that are variable. In this example, the listener would need to listen to the track several times (at least 20) to hear all different possible variations, and different environmental triggers will make even same variations behave different depending on the associated rules.

[0171] More complex rules can be constructed by combining simple rules, like the ones mentioned above, using logic programming.

[0172] Rules are stored in the a3 Text File segment 202 of the a3 File format and affect the music experience at playback time.

[0173] Fig. 14 shows another embodiment of the invention, which allows the Artist to position the sound files on a timeline, but define trigger points along one sound file that forms the base of the track. [0174] Each trigger point has an associated rule, for example 1 , 2, 3, etc. When the timeline arrives at the trigger point, the rule is examined and if the associated conditions are met, a new sound file is played in parallel to the main track. Once triggered each sound file may respond to triggers following a defined behavior rule as in the embodiment described above.

[0175] Using this method the Artist can create a track that has always one main sound component and is thus always recognisable, but with additional sound components, for example new instruments or vocals added when the track is played back in different environments.

[0176] Fig. 15 shows another embodiment, which is similar to the method described above but allows sound files to be "listening" for certain patterns to occur for them to be triggered.

[0177] Such patterns may comprise variables that are "internal" to the track, for example rhythm combinations, tonal sequences, etc, and/or "external" to the track representing a combination of microphone and other sensor values. The "listening window" for each sound file is also defined with start and end times. For example, sound file D2 is triggered if pattern 3 occurs within the 1 :30-2:30 time window. Once triggered each sound file may respond to triggers following a defined behavior rule as in the embodiment described above.

[0178] Using this method the Artist can create a track that contains sound elements that represent different instruments and are programmed so that they replicate rules derived from live improvisation, evident in jazz and other music genres.

Behavior Auto-Mode

[0179] Fig. 16 is a schematic of an additional embodiment of the invention, that allows the system to use an "auto-mode" to recognise different auditory streams in the user's sound environment and create automatic behaviors that group certain components of the track with elements of the user's environment, on the basis of frequency matching (pitch tracking), temporal properties like rhythm, synchronicity or other properties.

[0180] The microphone input is passed to an ASA Analyzer module that performs Auditory Scene Analysis on the incoming sound to derive separate auditory streams in the user's environment. For example, a stream corresponding to speech and a stream corresponding to engine noise may be derived from a microphone input signal when the system is used in an airplane.

[0181] The system then uses a Matching Module to group the Auditory Streams with the Music Streams, by using user-selected criteria, such as masking, or rhythm coherence. The grouping is then received by the Composition engine, which uses information from the Auditory Streams, such as Energy Level, to control the playback of the Music Streams, such as playback level. For example, a music stream with high-energy concentration in low frequencies may be controlled by the energy level of the aircraft's engine.

[0182] The user may be able select the global behavior type, for example a "social" behavior which would result in the track elements becoming softer when the corresponding auditory stream become louder, or "antisocial" which would make the track elements louder when the corresponding auditory stream becomes louder.

[0183] For example, a pop piece written as a3 track may comprise 4 music streams:

- 1 vocal

- 1 keyboard

- 1 guitar

- 1 drums

Each one of those streams is generated by a different Soundfile Player and has programmed behaviors defined by the artist.

[0184] By enabling "behavior auto-mode" the user asks the algorithm to potentially override these behaviors and create behaviors that are associated with the environment the user happens to be in at the time of playback.

[0185] For example, the user may be travelling on an airplane. The user activates the behavior auto-mode setting the matching criterion to masking. The ASA analyser module analyses the environment and derives 2 different streams:

- 1 speech stream, associated with voices in the cabin

- 1 low- frequency steady-state noise stream, associated with engine noise The matching module checks these auditory streams against the music streams and finds that the speech auditory stream is in the same frequency domain as the vocal music stream and the low- end noise stream has 80% overlap with the frequency spectrum of the keyboard music stream.

[0186] The system asks the user via the UI if these streams should be matched and presents the available suitable behaviors, in this case "social" and "antisocial." The user selects the desired behavior and these two streams are grouped with the new automatic behaviors, whereas the other two music streams preserve their original behaviors.

Automatic Track Separation

[0187] Fig. 17 is a schematic of another embodiment, allowing standard music files in mp3 or other format to be used with the system.

[0188] The mp3 track is passed to a Track Analyzer module which uses a Blind Source Separation algorithm to derive the individual sound components, for example each one of the instruments, vocals, etc. from the mp3 track. The resulting Music Streams are then passed to a Matching Module with functionality as described above. Alternatively the user can set the behavior of each Music Stream manually.

[0189] The user can also edit the mix of the Music Streams, for example remove and/or change the relative volume of one or more sound components.

Audio Commands and Pattern Recognition

[0190] Fig. 18 is a schematic of an additional feature that uses an input module that can receive audio commands from the user, for example for pausing and resuming playback, selecting a new track, increasing or decreasing the output level. The module uses a database of spoken- word commands and matches them with input received through the microphone of the device.

[0191] In addition, various embodiments may also use a module for recognising certain types of input and use this characterisation for generating certain types of behaviors. Such inputs are for example traffic noise, male speech, female speech, baby's cry, birdsong, etc. Wireless Host (a3 Radio Service)

[0192] Fig. 19 is a schematic of another embodiment of the invention, allowing a client device to be connected wirelessly to a remote host.

[0193] The client device is similar to the system of Fig 1 but has additionally a wireless transmitter and receiver to communicate with the remote host.

[0194] The remote host contains a library of a3 tracks and/or a3 streams, that can be served wirelessly to the client device at the client's request. An a3 stream is similar to an a3 track, but contains a3 audio streams instead of the a3 audio files 203.

[0195] The user can browse the content on the remote host and select an a3 track or a3 stream for playback. The host sends the a3 track or a3 stream, which is stored or buffered locally at the client. The client device processes the track or stream using the

[0196] This system can be seen as a next generation digital radio service, where listeners are able to tune in to audio broadcasted by a remote a3 music station, but have the capability to influence the playback of the music they listen, along the control dimensions afforded by the a3 system.

Mobile Implementation

[0197] Fig. 20 shows the a3 player running on an Apple™ mobile phone device.

[0198] The application is preferably used with a specially designed headphone unit with microphones fitted on each ear piece (similar to a stereo hands-free kit, but with different positioning of the mic).

[0199] Fig. 21 shows screenshots of the main User Interface views on Apple™ mobile phone device. The user is able to control the following settings:

• Basic settings: o a3 track selection (selected in the same fashion as mp3 tracks) o Previous Track - Play/Pause - Next track o Output Volume, selected via slider as with mp3 player volume (range -inf to 0 dBFS)

• Advanced settings: o Sensitivity (range: minimum to maximum) o Minimum level of a3 output, selected via slider (range -inf to 0 dBFS) o Microphone selection (typically from built-in mic, headset mic, or wirelessly connected mics)

[0200] Fig. 22 shows a specially designed headset which can be used with the system comprising two microphones may be used, mounted on the back of ear-pieces. The headsets may be connected to the host device via wire or wirelessly, for example via Bluetooth protocol.

[0201] Fig. 23 shows the a3 player running on a mobile phone device connected to a headset. User input is allowed via the buttons on the bottom part of the device.

Derivative Products

[0202] Further embodiments of the invention will now be described, arising out of embedding one of the host devices in products suitable for different applications.

Sound Bulb

[0203] Fig. 24 is a schematic of a "sound bulb."

[0204] A "sound bulb" is a device used for conditioning the sound environment in a room, similar to how a light bulb is used to condition a light environment.

[0205] The sound bulb contains the a3-enabled electronics connected to microphones that pick up noise from inside and/or the outside the room and speaker units to emit sound into the room. The microphones are wired to the main electronic unit or are connected to it wirelessly, for example via Bluetooth connection. The sound bulb may be connected to a number of microphones, including microphones attached on windows to pick up noise from the outside. Speaker elements are connected to the main unit via wire or wirelessly.

[0206] The user controls the sound bulb by flicking a switch next to the light switch or via a remote control. The user may condition his sound environment for reducing the effect of noise, if any, or for creating a more pleasant or entertaining space imbued with music that exhibits behavior as described above.

[0207] A sound bulb may be employed in each room in a domestic environment. Sound bulbs may be synchronised or work independently from each other. Sound bulbs create spaces that are acoustically intelligent and provide the right type of sound to their users for the activities they perform during different times of the day. A sound bulb may sense occupancy and emit sound only when a room is occupied. It may be integrated with other entertainment devices, for example hi-fi, TV and home entertainment systems, so that it doesn't emit sound when the user is watching a film or listening to the radio.

[0208] Sound bulbs may also be used in non-domestic environments, like hotels, hospitals, offices and public building of small or large scale. Eventually the sound bubble may become a standardised component for controlling the sound in architectural environment and variations will occur that are optimised for different types of installations.

Window Treatment Systems

[0209] The host device may be embedded in window treatment systems, like curtains or blinds, to provide additionally sound attenuation of noise from inside or outside the room as described in our previous patent applications.

[0210] A window treatment system in the form of a curtain is shown in Fig. 25.

Building Materials

[0211] The host device may also be attached directly on glass, or embedded in walls or other building materials. [0212] Fig. 26 shows such a system attached directly on glass.

[0213] The device is preferably attached on the window by means of suction. It uses a microphone that picks noise preferentially from the outside and a directional speaker that emits sound toward the inside. The device may be power by rechargeable battery and house a solar panel component, using solar energy to recharge the battery.

Electronic Equipment

[0214] The host device may also be embedded in other electronic devices, like radios, Hi-Fis, TVs, etc.

[0215] Fig. 27 shows such a handheld electronic device.

Sound Garment

[0216] The host device may also be embedded in "sound garments," which are wearables with the capability to listen to, process and generate sound. A sound garment may be in the form of a dress, scarf, hat, coat or other garment, or in the form of jewellery like earrings, necklace, etc. The host device is preferably miniaturised and connects to miniaturised microphones and speakers that are either weaved or attached to a fabric layer in the garment. A sound garment may employ sensors to monitor the wearer's heart beat, stress levels, etc. and use other sensors similar to the one referenced above to respond to place, noise, time, weather, proximity, other sound garments, etc.

[0217] Fig. 28 shows such a host device embedded in a hat, and Fig. 29 in a dress.

Furniture

[0218] The host device may be embedded in furniture, for example beds, chairs, desks, freestanding partitions, office cubicles, etc.

[0219] Fig. 30 shows a host device embedded in a pillow. [0220] Fig. 31 shows a host device embedded in a chair. A microphone is preferably located on the back of the chair, whereas speakers are embedded in the headrest and emit towards the front of the chair.

Aircraft, Train, Car Seat

[0221] The host device may be embedded in aircraft, train or car seats to create a personalised sound environment for the passenger responding to noise from the cabin and other passengers.

[0222] Fig. 32 shows a device embedded in a train seat. A microphone is preferably located on the top of the chair to pick general cabin noise, whereas speakers are embedded in the headrest to emit sound towards the person seating in the chair. A simple control using is integrated in the armrest.

Sound Booth

[0223] Fig. 33 shows a sound booth with integrated a3 music system. The booth uses microphones to pick sound from the inside of the booth and speakers that emit sound toward the surrounding space. The booth responds to the voice of someone occupied the booth and emits sound towards the outside to provide privacy.

Sound Architecture

[0224] Fig 34 is a representation of derivative products embedded in a building structure.

[0225] For further details about a related system which can benefit from or implement some of the ideas presented herein refer to U.S. Patent Application Publication No. 2005/0254663 (U.S.S.N. 10/996,330 filed November 3, 2004), which is incorporated herein by reference.

[0226] Also, note that various modules and functionality that is described herein can be implemented by a processor system appropriately programmed. This includes the audio track parser, the composition engine, the different analyzers, and the matching module, to name a few examples. [0227] Other embodiments are within the following claims. For example, the a3 files or parts thereof could be made available on a CD or some other data storage medium or they could be received electronically over a network from a remotely located transmitting source.

Claims

CLAIMS:

1. A music system comprising: a processor system; an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; and a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped, wherein the processor system is programmed to generate from the music composition and control data derived from said sensor data a music track, said music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the presence, order, and grouping of the audio files in the sequence of audio files is determined by both the stored control parameters and the sensor data received through the input signal interface.

2. The music system of claim 1, wherein the information about the environment in which the music system is being operated relates to a condition selected from the group consisting of location, relative location, change in location, weather, temperature, proximity, light, and a state of a user of the music system.

3. The music system of claim 1, wherein the environment is a virtual environment.

4. The music system of claim 1, wherein the information about the environment in which the music system is being operated relates to receipt or availability of specific information over a network.

5. The music system of claim 1, wherein the memory system is also for storing rules associated with the music composition, and wherein the processor system is further programmed to use the stored rules along with the control data that is derived from the received sensor date to determine which of the plurality of audio files are included in the music track.

6. The music system of claim 1, wherein the memory system is also for storing rules associated with the music composition, and wherein the processor system is further programmed to use the stored rules along with the control data that is derived from the received sensor date to determine which of the plurality of audio files are included in the music track and the order of the audio files included in the music track.

7. The music system of claim 1, wherein the information about the environment in which the music system is being operated relates to receipt of a twitter message.

8. The music system of claim 7, wherein the memory system is also for storing rules associated with the music composition, and wherein the processor system is further programmed to use the stored rules along with the control data that is derived from the received sensor date to determine which of the plurality of audio files are included in the music track and the order of the audio files included in the music track and wherein one of the stored rules instructs the processor to include a particular one of the plurality of audio files in the music track only if receipt of a particular kind of twitter message is detected.

9. The music system of claim 1, further comprising an audio output device for outputting the music track as audible music.

10. The music system of claim 1, further comprising a sensor selected from the group consisting of temperature sensors, location sensors, movement sensors, change in position sensors, proximity sensors, and light sensors and wherein the sensor provides the sensor data to the input signal interface.

11. The music system of claim 1 , further comprising a sensor selected from the group consisting a heart rate monitor and a device for measuring galvanic skin resistance of the user and wherein the sensor provides the sensor data to the input signal interface.

12. The music system of claim 1, further comprising a microphone and wherein the microphone provides the sensor data to the input signal interface.

13. The music system of claim 1, further comprising a plurality of different sensors and wherein the plurality of different sensors provides the sensor data to the input signal interface.

14. A music system comprising: an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; and a composition engine configured to generate from the music composition and control data derived from said sensor data a music track, said music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the presence, order, and grouping of the audio files in the sequence of audio files is determined by both the stored control parameters and the sensor data received through the input signal interface.

15. The music system of claim 14, further comprising an analyzer for analyzing the sensor data to generate the control data.

16. The music system of claim 15, wherein said analyzer is a spectral analyzer and the control data are band levels within a plurality of spectral bands.

17. A method of generating a music track for playing on a music system, said method comprising: obtaining a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; receiving sensor data carrying information about the environment in which the music system is being operated; and generating said music track from the music composition and control data derived from said sensor data, said music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein generating involves using the control parameters and the sensor data received through the input signal interface to determine the presence, order, and grouping of the audio files in the sequence of audio files.

18. A music system comprising: a processor system; an input signal interface for receiving sensor data carrying information about the environment in which the music system is being operated; and a memory system for storing a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped, wherein the processor system is programmed to generate from the music composition a music track, said music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein the order and grouping of the audio files in the sequence of audio files is determined at least in part by the stored control parameters and wherein a volume for each of at least some of the audio files in the music track is determined at least in part by control data derived from the received sensor data.

19. The music system of claim 18, wherein the sensor data is derived from sound in the environment in which the music system is being operated.

20. The music system of claim 18, further comprising a microphone which during operation generates a signal from which the sensor data is derived.

21. The music system of claim 18, further comprising an analyzer for performing spectral analysis of sensor data.

22. The music system of claim 18, wherein the memory system is also for stored rules which indicate how the sensor data is to affect the volume control parameters and wherein the processor system is further programmed to employ the stored rules to set the volume for each of at least some of the audio files in the music track.

23. A method of generating a music track for playing on a music system, said method comprising: obtaining a music composition comprising a plurality of audio files and control parameters, wherein the control parameters indicate how the audio files are to be ordered and grouped; receiving sensor data carrying information about the environment in which the music system is being operated; and generating said music track from the music composition and control data derived from said sensor data, said music track being a sequence of audio files assembled from audio files selected from the plurality of audio files, wherein generating involves using the control parameters to determine the order and grouping of the audio files in the sequence of audio files and also involves using the sensor data to determine at least in part a volume for each of at least some of the audio files in the music track.

24. A music system comprising: a first input interface for receiving a sound signal from the environment in which the music system is being operated; an analyzer module which during operation derives a plurality of separate auditory streams from the sound signal; a second input interface for receiving a plurality of music streams; and a matching module configured to generate a music track by combining at least some of the plurality of auditory streams with at least some of the plurality of music streams based on user-selectable criteria.

25. The music system of claim 24, wherein the matching module is configured to combine at least some of the plurality of auditory streams with at least some of the plurality of music streams also based on matching frequency domains of at least some of the plurality of music streams with the frequency domains of at least some of the auditory streams.

26. The music system of claim 24, further comprising a memory system for storing a music composition comprising a plurality of audio files, wherein the plurality of music streams are derived from the plurality of audio files stored in the memory system.

27. The music system of claim 24, further comprising a track analyzer for receiving an mp3 track and deriving the plurality of music streams from the mp3 track.

28. A method of generating a music track for playing on a music system, said method comprising: receiving a sound signal from the environment in which the music system is being operated; deriving a plurality of separate auditory streams from the sound signal; receiving a plurality of music streams; and generating the music track by combining at least some of the plurality of auditory streams with at least some of the plurality of music streams based on user-selectable criteria.

29. A storage medium storing a music composition for playing on a music system that receives sensor data carrying information about the environment in which the music system is being operated, said stored music composition comprising: a plurality of audio files; control parameters specifying how the audio files are to be ordered and grouped by the music system when the music system constructs a music track from the music composition; and a plurality of rules specifying how control data that is derived from the sensor data controls selections by the music system of audio files from among the plurality of audio files for inclusion in the music track.