Constraint Research at SONY Computer Science Laboratory - Paris

François Pachet

The SONY CSL laboratory (Computer Science Laboratory) was created in 1996 by Luc Steels, and is devoted to basic research in computer science and closely-related fields. It is an offshot of the successful Sony Computer Science Laboratory in Tokyo. The CSL-Paris lab consists of two main teams: a language team, headed by Luc Steels, which is interested in the origin and evolution of language, and a music team, headed by François Pachet, which is interested in the modeling of musical perception, and the development of innovative software environments for enriching the musical experience.

The goals of the music team is to identify and develop the software technology needed to build better, more adaptive listening environments for the future. To do so, we address three basic issues: listening environments which provide semantic control on music, computational models of individual music perception, and models of interactions and evolution in music. Now, what constraints have to do with all that ? One of our claim is that constraints - with various meanings that can be attached to this word - provide an excellent technology for developing musical environments with meaningful user controls. This is what we call active listening (Pachet, 1999).

The team has so far developped two projects using constraints and music. The first one is the MusicSpace project, which concerns the issue of controlling the localization of sound sources in a virtual 3D space. The main idea underlying MusicSpace is to use constraints as relations between sound sources that should always be maintained automatically by the system. This allows the user to modify the configuration of a set of related sound sources by moving a single object. The constraint system completes the user action by moving other sound sources, so that the resulting configuration is always "coherent" from a sound engineering viewpoint. We developped a library of constraints useful for specifying properties of sound source configuration. These constraints include a variety of geometrical constraints such as source balancing constraints (several sources are mutually opposed), source grouping constraints (sources should remain with constant distance ratio to listener). Additionally, some constraints concern non geometrical parameters of sound sources, such as the directivity. The current design allows to handle most of these constraints in real time, using a backtrak-free perturbation algorithm. Some properties, however are impossible to represent using this algorithm. For instance, ensuring that the global configuration conforms to a given spectrum function require a search in the space of configurations. This search is for the moment incompatible for the real time "constraint" of the system.
This project has led to the design of a real time constraint solver, based on an extension of the DeltaBlue algorithm, with specific global constraints designed especially for sound mixing (Pachet and Delerue, 1998a, 1998b). A MIDI version of the system is now being made available. A fully-fledged audio version is in progress.

The other main project involving constraint is the music scheduling project. Here, the issue raised is the exploration of large music catalogues (for instance, SONY Music's catalogue contains about 500.000 music titles). Our idea is to specify music programs, that is, sequences of music titles, with global properties, themselves represented as global constraints (Pachet et al., 1999). These properties are expressed as predicates holding on content attributes of music titles. These attributes are stored in a content-based catalogue, and represent information such as the musical genre (or style) or the title, the type of voice, the main instruments, etc. The team has developped a constraint satisfaction solver in Java (BackJava), based on the specifications of the BackTalk system (Roy and Pachet, 1997). Additionally, specific global constraints have been designed and implemented in BackJava to specify music sequences: constraints of cardinality, continuity and discontinuity (Pachet and Roy, 1999). These global constraints are used to build various music scheduling systems. For instance, PathBuilder builds paths between two music titles chosen by the user. The path ensures that each title is "close" to the next title, using a distance function.

The team develops other projects involving constraints and music. A rhythm specification system for instance, uses constraints to relate percussive instruments in time. This allows to move several percussions simultaneously using musical relations, instead of using the low-level traditional matrix-based rhythm machine approach, in which the temporal location of all the instruments have to be specified one by one by the user. The team has long been working on the issue of automatic harmonization, i.e. producing four-voice arrangements of given melodies or bass lines (Pachet, 1999; Pachet and Roy, 1999, see the four-voice harmonization system implemented in BackTalk). A workshop on the team of constraints and musical/artistic applications was organized by the team at the last ECAI conference.

Members of the team include François Pachet, Olivier Delerue, and Pierre Roy (from the LIP6 laboratory). The SONY web site refers to publications of the team in this domain.
 

  • Pachet, F. "Constraints and Multimedia", PACLP invited paper, pp. 3-13, 1999. A synthesis can be found in the ALP newsletter, Vol 12/3, 1999.
  • Pachet, F. "Active Listening", in "Música y nuevas tecnologías: Perspectivas para el siglo XXI", in Spanish, Miranda, Eurado Ed., L'Angelot, 1999.
  • Pachet, F. Codognet, P. Special issue of the Constraints Journal based on the ECAI 98 workshop on "Constraints for Artistic Applications", ECAI, Brighton, to appear, 1999.
  • Pachet, F. Delerue, O. "A Constraint-based Temporal Music Spatializer", ACM Multimedia Conference, Bristol, 1998a.
  • Pachet, F. Delerue, O. "A Mixed 2D/3D Interface for Music Spatialization", First International Conference on Virtual Worlds (VW98), Springer Verlag, Lecture Notes in Computer Science, n. 1434, pp. 298-307, 1998b.
  • Pachet, F. Roy, P. "Automatic Generation of Music Programs", Proceedings of CP'99, Alexandria, Virginia, 1999.
  • Pachet, F. Roy, P. Cazaly. D. "A Combinatorial approach to content-based music selection". Proceedings of the IEEE Multimedia Computing and Systems Int. Conf., Firenze, (Italy), pp. 457-462, 1999.
  • Roy, P. Liret, A. Pachet, F. "The Framework Approach for Constraint Satisfaction", chapter in Object Oriented Application Frameworks, Wiley Eds., Vol. 2, 1999.
  • Roy, P. Liret, A. Pachet, F. "The Framework Approach for Constraint Satisfaction", ACM Computing Surveys on Frameworks, M. Fayad Ed, December 1999.
  • Roy, P. Pachet, F. Reifying Constraint Satisfaction in Smalltalk. Journal of Object-Oriented Programming (JOOP), 10 (4), pp. 43-51, July/August 1997.

François Pachet,
 SONY CSL-Paris,
pachet@csl.sony.fr