Download User Guide

Flexible Audio Source Separation Toolbox (FASST)
Version 1.0
User Guide
Alexey Ozerov 1 , Emmanuel Vincent
1
and Fr´ed´eric Bimbot
2
1
INRIA, Centre de Rennes - Bretagne Atlantique
2
IRISA, CNRS - UMR 6074
Campus de Beaulieu, 35042 Rennes cedex, France
{alexey.ozerov, emmanuel.vincent}@inria.fr, [email protected]
April 7, 2011
1
Introduction
This user guide describes how to use FASST, an implementation of the general flexible source
separation framework presented in [1]. Before reading the user guide you are strongly encouraged
to read [1], at least the two first sections.
This guide is organized as follows. Some notations and abbreviations used throughout this
document are listed in section 2. Section 3 gives a detailed specification of the mixture structure
(a Matlab structure), used to define the available prior information. The main functions the user
should know about are listed in section 4 and an example of usage is given in section 5.
2
Some abbreviations and notations
2.1
Abbreviations
GMM
GSMM
HMM
Gaussian mixture model
Gaussian Scaled Mixture Model
Hidden Markov Model
NMF
PSD
Nonnegative matrix factorization
Power Spectral Density
QERB
S-HMM
Quadratic Equivalent Rectangular Bandwidth transform
Scaled Hidden Markov Model
STFT
Short-Time Fourier Transform
1
2.2
Notations
F
N
Number of frequency bins in the corresponding time-frequency representation
Number of time frames in the corresponding time-frequency representation
I
Jspat
Number of channels (this version is only implemented for I = 1 or I = 2)
Number of spatial components (see Section 3)
Jspec
Rj
Number of spectral components (see Section 3)
Rank of the covariance matrix of the j-th spatial component
Cj
Number of factors in the j-th spectral component
− Cj = 1: direct model
− Cj = 2: factored excitation-filter model
Number of narrowband excitation spectral patterns (see [1]) in the j-th spec. comp.,
Number of characteristic excitation spectral patterns (see [1]) in the j-th spec. comp.,
Lex
j
Kjex
Mjex
Lft
j
Kjft
Mjft
Number of characteristic filter spectral patterns (see [1]) in the j-th spec. comp.,
Number of time-localized filter patterns (see [1]) in the j-th spec. comp.,
Aj
Mixing parameters (∈ CI×Rj ×F ×N ) in the j-th spatial comp. (see [1]),
Wjex
Narrowband excitation spectral patterns (∈ R+
Number of time-localized excitation patterns (see [1]) in the j-th spec. comp.,
Number of narrowband filter spectral patterns (see [1]) in the j-th spec. comp.,
F ×Lex
j
) in the j-th spec. comp. (see [1]),
Hft
j
Lex ×Kjex
Excitation spectral pattern weights (∈ R+j
) in the j-th spec. comp. (see [1]),
Kjex ×Mjex
Excitation time pattern weights (∈ R+
) in the j-th spec. comp. (see [1]),
Mjex ×N
Time-localized excitation patterns (∈ R+
) in the j-th spec. comp. (see [1]),
F ×Lft
j
Narrowband filter spectral patterns (∈ R+
) in the j-th spec. comp. (see [1]),
ft
Lft
j ×Kj
Filter spectral pattern weights (∈ R+
) in the j-th spec. comp. (see [1]),
Kjft ×Mjft
Filter time pattern weights (∈ R+
) in the j-th spec. comp. (see [1]),
Mjft ×N
Time-localized filter patterns (∈ R+
) in the j-th spec. comp. (see [1]),
R
R+
Set of real numbers
Set of nonnegative real numbers
C
Set of complex numbers
Uex
j
Gex
j
ex
Hj
Wjft
Uft
j
Gft
j
3
Mixture structure
The mixture structure is a Matlab structure that is used to incorporate prior information into the
framework. The structure has a hierarchical organization that can be seen from the example in figure 1. Global parameters (e.g., signal representation) are defined on the first level of the hierarchy.
The second level consists of Jspat spatial components and Jspec spectral components. Each source
is typically modeled by one spectral component, although some sources (e.g., drums) might be
modeled by several spectral components (e.g., bass drum, snare, etc.). Furthermore, each spectral
component must be associated with one spatial component, and each spatial component must have
at least one spectral component associated to it. 1 Compared to the description of the framework
in [1], this implementation is more general in the sense that the number of spectral components is
1 This extension makes it possible to model the fact that several sources have the same direction, which is
very often the case for professionally produced music recordings. It is implemented by simply adding the power
spectrograms of the spectral components corresponding to the same spatial component.
2
not necessarily equal to that of spatial components, and more precisely Jspec ≥ Jspat . The third
level of the hierarchy consists in factorizing each spectral component into one or more factors
representing for instance excitation and filter structures (see [1]) 2 . Finally, on the fourth level of
the hierarchy, each factor is represented as the product of three or four matrices (see Table 4),
which are not represented in Figure 1. For instance, the factor representing excitation structure
ex ex
is either represented as the product of four matrices Wjex Uex
j Gj Hj representing, respectively,
narrowband spectral patterns, spectral pattern weights, time pattern weights and time-localized
ex
ex
patterns (see [1]) or as the product of threes matrices Wjex Uex
j Gj when Hj is marked by the
empty matrix [] 3 . Almost all the fields of the mixture structure must be filled as specified in
Tables 1, 2, 3 and 4, except those marked by the empty matrix [].
Figure 1: Visualization of a mixture structure example.
4
Main functions
The user should know about three main functions comp transf Cx, estim param a post model
and separate spec comps, allowing, respectively, to compute the input time-frequency transform,
estimate the model parameters and separate the spectral components. The headers of these functions are listed in Figures 2, 3 and 4.
2 Note that in [1] the usage of two factors (excitation and filter) is described. The implementation presented here
is more flexible, since one can use any number of factors Cj , and it reduces to [1] when Cj = 2. This is done for
convenience of usage. For example if one needs to implement an excitation model only or a filter model only (direct
model), one simply needs to choose Cj = 1 without bothering to specify and to process an additional dummy factor.
3 In [1] only the case of four matrices is considered, and the case of three matrices Wex Uex Gex is just equivalent
j
j
j
ex
to fixing Hex
j to the N × N identity matrix. Since N may be quite big, we fix Hj to [] by convention in the latter
case in order to avoid storing a big identity matrix in memory.
3
5
Examples of usage
The user should also know how to fill and browse the mixture structure and how to use the abovementioned three functions. An example of mixture structure filling and browsing is given in Figures
5 and 6. An example script for the separation of an instantaneous mixture of music signals is given
in Figure 7.
Function EXAMPLE prof rec sep drums bass melody.m contains a more sophisticated example
allowing the separation of the following four sources:
ˆ drums,
ˆ bass,
ˆ melody (singing voice or leading melodic instrument),
ˆ remaining sounds,
from a stereo music recording. Due to memory limits in Matlab this function cannot process sound
excerpts longer than 30 seconds. For full length music recording the function
EXAMPLE prof rec sep drums bass melody FULL.m
should be used. This function simply cuts the full recording into small parts, and applies
EXAMPLE prof rec sep drums bass melody.m to each of them.
References
[1] A. Ozerov, E. Vincent, and F. Bimbot, “A general flexible framework for the handling of
prior information in audio source separation,” IEEE Transactions on Audio, Speech and Signal
Processing, vol. 20, no. 4, pp. 1118–1133, 2012.
[2] A. Ozerov and C. F´evotte, “Multichannel nonnegative matrix factorization in convolutive mixtures for audio source separation,” IEEE Transactions on Audio, Speech and Language Processing, vol. 18, no. 3, pp. 550–563, March 2010.
4
Field
Cx
transf
fs
wlen
Noise PSD
spat comps
spec comps
Description
F × N × I × I complex-valued tensor of local
mixture covariances
Input time-frequency transform
Sampling frequency in Hz
Analysis window length
(used to compute STFT or QERB) in samples
F × 1 real-valued nonnegative vector of additive
noise PSD, e.g., for annealing
1 × Jspat cell array of spatial component structures
1 × Jspec cell array of spectral component structures
Value
∈ CF ×N ×I×I
’stft’ for STFT
’qerb’ for QERB
∈ {16000, 44100, . . .}
∈ {512, 1024, . . .}
∈ R1×F or []
see Table 2
see Table 3
Table 1: Specification of the mixture structure (mix str).
Field
time dep
Description
Stationarity of mixing
mix type
Mixing type
frdm prior
Degree of adaptability
params
Tensor of mixing parameters
(corresponding to Aj from [1])
Value
’indep’ for time-invariant mixing
’dep’ for time-varying mixing
’inst’ for instantaneous (freq.-indep.)
’conv’ for convolutive (freq.-dep.)
’free’ for adaptive
’fixed’ for fixed
for mix type = ’inst’
∈ RI×Rj
for mix type = ’conv’
∈ CI×Rj ×F
Table 2: Specification of the spatial component structure (spat comps{j}, j = 1, . . . , Jspat ).
Field
spat comp ind
factors
Description
Index of the corresponding spatial component
1 × Lj cell array of factor structures
Value
∈ {1, . . . , Jspat }
Table 3: Specification of the spectral component structure (spec comps{j}, j = 1, . . . , Jspec ).
5
Field
FB frdm prior
FW frdm prior
TW frdm prior
TB frdm prior
FB
FW
TW
TB
TW constr
Description
Degree of adaptability
for narrowband spectral patterns
Degree of adaptability
for spectral pattern weights
Degree of adaptability
for time pattern weights
Degree of adaptability
for time-localized patterns
Narrowband spectral patterns (Frequency Blobs)
(corresponding to Wjex or Wjft )
Spectral pattern weights (Frequency Weights)
ft
(corresponding to Uex
j or Uj )
Time pattern weights (Time Weights)
ft
(corresponding to Gex
j or Gj )
Time-localized patterns (Time Blobs)
ft
(corresponding to Hex
j or Hj )
Constraint on the time pattern weights
(note that nontrivial constraints, i.e.,
different from ’NMF’ are not
compatible with nonempty time patterns TB)
TW DP params
Discrete probability (DP) parameters
for the time pattern weights
(needed only when TW constr 6= ’NMF’)
TW DP frdm prior
Degree of adaptability for DP parameters
(needed only when TW constr 6= ’NMF’)
Matrix of all time weights
˜ ex or G
˜ ft from [1])
(corresponding to G
j
j
(needed only when TW constr 6= ’NMF’)
TW all
Value
’free’ for adaptive
’fixed’ for fixed
’free’ for adaptive
’fixed’ for fixed
’free’ for adaptive
’fixed’ for fixed
’free’ for adaptive
’fixed’ for fixed
F ×Lex
j
∈ R+
F ×Lft
j
or ∈ R+
Lex ×Kjex
∈ R+j
K ex ×Mjex
∈ R+ j
M ex ×N
∈ R+ j
Lft ×Kjft
or ∈ R+j
K ft ×Mjft
or ∈ R+ j
M ft ×N
, ∈ R+ j
’NMF’ no constraint
’GMM’ for GMM
’HMM’ for HMM
’GSMM’ for GSMM
’SHMM’ for S-HMM
1 × Kjex (1 × Kjft ) vector
of Gaussian weights
for GMM or GSMM
Kjex × Kjex (Kjft × Kjft )
matrix of transition
probabilities
for HMM or S-HMM
’free’ for adaptive
’fixed’ for fixed
Nonnegative real-valued
matrix of the same
size as TW
Table 4: Specification of the spectral component factor structure (factors{l}, l = 1, . . . , Lj ).
6
or []
f u n c t i o n C x = c o m p _ t r a n s f _ C x ( x , t r a n s f , w i n _ l e n , fs , q e r b _ n b i n )
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
Cx = c o m p t r a n s f C x ( x ,
compute s p a t i a l
transf , win len ,
fs , qerb nbin ) ;
covariance matrices f o r the corresponding transform
input
−−−−−
x
:
transf
:
win len
fs
qerb nbin
:
:
:
[ I x nsampl ] m a t r i x c o n t a i n i n g I time−domain m i x t u r e s i g n a l s
w i t h nsampl s a m p l e s
transform
' stft '
' qerb '
window l e n g t h
( o p t ) s a m p l i n g f r e q u e n c y ( Hz )
( o p t ) number o f b i n s f o r q e r b t r a n s f o r m
output
−−−−−−
Cx
:
[ F x N x I x I ] matrix c o n t a i n i n g the s p a t i a l c o v a r i a n c e
m a t r i c e s o f t h e i n p u t s i g n a l i n a l l time−f r e q u e n c y b i n s
Figure 2: comp transf Cx : FASST function for the computation of the input time-frequency
transform.
function [ mix_str , log_like_arr ] = estim_param_a_post_model ( mix_str_inp ,
iter_num , sim_ann_opt , Ann_PSD_beg , Ann_PSD_end )
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
[ mix str , l o g l i k e a r r ] = estim param a post model ( mix str inp ,
i t e r n u m , s i m a n n o p t , Ann PSD beg , Ann PSD end ) ;
...
...
e s t i m a t e a p o s t e r i o r i m i x t u r e model p a r a m e t e r s
input
−−−−−
mix str inp
iter num
sim ann opt
Ann PSD beg
Ann PSD end
: input mixture s t r u c t u r e
: ( o p t ) number o f EM i t e r a t i o n s ( d e f = 1 0 0 )
: ( o p t ) s i m u l a t e d a n n e a l i n g o p t i o n ( d e f = ' ann ' )
' no ann '
: no a n n e a l i n g ( z e r o n o i s e )
' ann '
: annealing
' ann ns inj '
: a n n e a l i n g with n o i s e i n j e c t i o n
' upd ns prm '
: update n o i s e p a r a m e t e r s
( Noise PSD i s updated t h r o u g h EM)
: ( o p t ) [ F x 1 ] b e g i n n i n g v e c t o r o f a n n e a l i n g n o i s e PSD
( d e f = X power / 1 0 0 )
: ( o p t ) [ F x 1 ] end v e c t o r o f a n n e a l i n g n o i s e PSD
( d e f = X power / 1 0 0 0 0 )
output
−−−−−−
mix str
log like arr
: estimated output mixture s t r u c t u r e
: a r r a y o f l o g −l i k e l i h o o d s
Figure 3: estim param a post model : FASST function for the estimation of the model parameters.
7
f u n c t i o n ie = s e p a r a t e _ s p e c _ c o m p s ( x , mix_str ,
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
sep_cmp_inds )
i e = separate spec comps (x , mix str , sep cmp inds ) ;
separate
s p e c t r a l components
input
−−−−−
x
mix str
sep cmp inds
: [ nchan x nsampl ] m i x t u r e s i g n a l
: i n p u t mix s t r u c t u r e
: ( o p t ) a r r a y o f i n d i c e s f o r components t o s e p a r a t e
( d e f = {1 , 2 , . . . , K spec })
output
−−−−−−
ie
:
[ K sep x nsampl x nchan ] e s t i m a t e d s p e c t r a l components images ,
where K sep = l e n g t h ( s e p c m p i n d s ) i s t h e number o f
components t o s e p a r a t e
Figure 4: separate spec comps : FASST function for the separation of the spectral component
signals.
8
f u n c t i o n m i x _ s t r = i n i t _ m i x _ s t r u c t _ M u l t _ N M F _ i n s t ( Cx , J , K , t r a n s f , fs , w l e n )
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
%
m i x s t r = i n i t m i x s t r u c t M u l t N M F i n s t ( Cx , J , K,
An example o f m i x t u r e s t r u c t u r e i n i t i a l i z a t i o n ,
m u l t i c h a n n e l NMF model ( i n s t a n t a n e o u s c a s e )
Most o f p a r a m e t e r s a r e i n i t i a l i z e d randomly
transf ,
f s , wlen ) ;
corresponding to
input
−−−−−
Cx
:
J
K
transf
fs
wlen
:
:
:
:
:
[ F x N x I x I ] matrix c o n t a i n i n g the s p a t i a l c o v a r i a n c e
m a t r i c e s o f t h e i n p u t s i g n a l i n a l l time−f r e q u e n c y b i n s
or [ F x N] s i n g l e channel v a r i a n c e matrix
number o f components ( h e r e J s p a t = J s p e c )
number o f NMF components p e r s o u r c e
t r a n s f o r m ( ' s t f t ' o r ' qerb ' )
s a m p l i n g f r e q u e n c y i n Hz
l e n g t h o f t h e t i m e i n t e g r a t i o n window ( must be a power o f 2 )
:
initialized
output
−−−−−−
mix str
mixture s t r u c t u r e
rank = 1 ;
[ F , N , I , I ] = s i z e ( Cx ) ;
mix_str
mix_str
mix_str
mix_str
mix_str
mix_str
. Cx
. transf
. fs
. wlen
. spat_comps
. spec_comps
=
=
=
=
=
=
Cx ;
transf ;
fs ;
wlen ;
cell (1 , J ) ;
cell (1 , J ) ;
for j = 1: J
% i n i t i a l i z e s p a t i a l component
mix_str . spat_comps {j }. time_dep
mix_str . spat_comps {j }. mix_type
mix_str . spat_comps {j }. frdm_prior
mix_str . spat_comps {j }. params
= ' indep ' ;
= ' inst ' ;
= ' free ' ;
= randn ( I , rank ) ;
% i n i t i a l i z e s i n g l e f a c t o r s p e c t r a l component
mix_str . spec_comps {j }. spat_comp_ind = j ;
mix_str . spec_comps {j }. factors
= cell (1 , 1) ;
factor1
factor1
factor1
factor1
factor1
factor1
factor1
factor1
factor1
. FB
. FW
. TW
. TB
. FB_frdm_prior
. FW_frdm_prior
. TW_frdm_prior
. TB_frdm_prior
. TW_constr
=
=
=
=
=
=
=
=
=
0 . 7 5 * abs ( randn ( F , K ) ) + 0 . 2 5
diag ( ones (1 , K ) ) ;
0 . 7 5 * abs ( randn ( K , N ) ) + 0 . 2 5
[];
' free ' ;
' fixed ' ;
' free ' ;
[];
'NMF ' ;
*
ones ( F , K ) ;
*
ones ( K , N ) ;
m i x _ s t r . s p e c _ c o m p s { j } . f a c t o r s {1} = f a c t o r 1 ;
end ;
Figure 5: Example of filling of the mixture structure corresponding to the multichannel NMF
method [2] (instantaneous case).
9
>> m i x _ s t r
mix_str =
C x : [4 − D d o u b l e ]
transf : ' s t f t '
f s : 16000
w l e n : 1024
s p a t _ c o m p s : { [ 1 x1 struct ]
s p e c _ c o m p s : { [ 1 x1 struct ]
N o i s e _ P S D : [ 5 1 3 x1 double ]
[ 1 x1 struct ]
[ 1 x1 struct ]
[ 1 x1 struct ] }
[ 1 x1 struct ] }
>> m i x _ s t r . s p a t _ c o m p s {2}
ans =
time_dep
mix_type
frdm_prior
params
:
:
:
:
indep '
inst '
' free '
[ 2 x1 double ]
'
'
>> m i x _ s t r . s p e c _ c o m p s {3}
ans =
spat_comp_ind : 3
f a c t o r s : { [ 1 x1 struct ] }
>> m i x _ s t r . s p e c _ c o m p s { 3 } . f a c t o r s {1}
ans =
FB :
FW :
TW :
TB :
FB_frdm_prior :
FW_frdm_prior :
TW_frdm_prior :
TB_frdm_prior :
TW_constr :
[ 5 1 3 x4 double ]
[ 4 x4 double ]
[ 4 x98 double ]
[]
' free '
' fixed '
' free '
[]
'NMF '
Figure 6: Browsing in Matlab of the example mixture structure in Table 5.
10
function EXAMPLE_ssep_Mult_NMF_inst ()
data_dir
=
result_dir =
file_prefix =
transf
wlen
nsrc
NMF_ncomp
iter_num
=
=
=
=
=
example data / ' ;
example data / ' ;
' Shannon Hurley
'
'
' stft
1024;
3;
4;
200;
'
Sunrise
inst
'
;
;
% number o f s o u r c e s
% number o f NMF components
% load mixture
f p r i n t f ( ' I n p u t time−f r e q u e n c y r e p r e s e n t a t i o n \n ' ) ;
[ x , fs , n b i n s ]= wavread ( [ d a t a _ d i r f i l e _ p r e f i x ' mix . wav ' ] ) ;
x = x . ';
mix_nsamp = s i z e (x , 2) ;
% compute time−f r e q u e n c y r e p r e s e n t a t i o n
Cx = c o m p _ t r a n s f _ C x ( x , transf , wlen , fs ) ;
% f i l l in mixture s t r u c t u r e
m i x _ s t r = i n i t _ m i x _ s t r u c t _ M u l t _ N M F _ i n s t ( Cx , n s r c , N M F _ n c o m p , t r a n s f , fs , w l e n ) ;
% r e i n i t i a l i z e mixing parameters
A = [ s i n ( pi /8) , s i n ( pi /4) , s i n (3* pi /8) ; cos ( pi /8) , cos ( pi /4) , cos (3* pi /8) ] ;
f o r j = 1: nsrc
mix_str . spat_comps {j }. params = A ( : , j ) ;
end ;
% run p a r a m e t e r s e s t i m a t i o n ( w i t h s i m u l a t e d a n n e a l i n g )
m i x _ s t r = e s t i m _ p a r a m _ a _ p o s t _ m o d e l ( m i x _ s t r , i t e r _ n u m , ' ann ' ) ;
% source separation
ie_EM = separate_spat_comps ( x , mix_str ) ;
% Computation o f t h e s p a t i a l s o u r c e i m a g e s
f p r i n t f ( ' Computation o f t h e s p a t i a l s o u r c e i m a g e s \n ' ) ;
f o r j =1: n s r c ,
w a v w r i t e ( r e s h a p e ( i e _ E M ( j , : , : ) , m i x _ n s a m p , 2 ) , fs , n b i n s , . . .
[ r e s u l t _ d i r f i l e _ p r e f i x ' s i m ' i n t 2 s t r ( j ) ' . wav ' ] ) ;
end
Figure 7: Example of usage involving all three main functions (runs the multichannel NMF method
[2] in the instantaneous case).
11