Skip to content

End to End Example

Jump to bottom
Alba Solsona edited this page Oct 29, 2019 · 109 revisions

In addition to the DetEdit examples in Getting Started, the following is an additional example showing a complete end-to-end application of DetEdit, from audio file to time series plots of detections.

The following steps will guide you through an example of acoustic data with a beaked whale encounter, and how to detect beaked whale echolocation clicks applying a generic energy detector, create files required for the DetEdit interface, annotate the data and create summary plots.

Download the EndToEnd_Example_Test_Set folder from Test datasets Drive folder, and learn how to download and set up DetEdit repository in the quick setup section.

Content

  1. mkLtsa: Create LTSA file from a collection of acoustic recordings
  2. Create TPWS files
    1. Edetect: Apply generic energy detector to detect signals from acoustic recordings
    2. make_TPWS: Provide detection times to create TPWS file
  3. mkLTSAsessions: Create LTSA sessions files
  4. detEdit: Launch DetEdit interface to annotate detections
  5. modDet: Clean up of TPWS files
  6. summaryParams: Create summary plots of annotated data

1. mkLtsa: Create LTSA file from a collection of acoustic recordings

An example audio file ( DCPP01C_DL67_130205_054500.x.wav) is provided in EndToEnd_Example_Test_Set folder. Use the Command Window in MATLAB to create the .ltsa file of the example audio file as follows:

>> mkLtsa

and specify the audio file type: press 1 for WAV files or 2 for XWAV files.

filetype.

It will prompt you to specify a folder containing a collection of audio files, in this case, select the EndToEnd_Example_Test_Set folder.

After, set up the LTSA parameters:

  • Time Average: length of time for each spectral average
  • Frequency Bin Size for LTSA

LTSAparams

Since these parameters are data sample rate dependent, setup with caution. The default parameters for broadband data sampled at 200 kHz are usually 5s averages and 200 Hz frequency bins. For other sample rates, such as 48 kHz or 96 kHz, 5s averages and 48 Hz frequency bins are recommended.

Then, select the folder to save the .ltsa file.

It is recommended to choose the same folder as the audio files so that the LTSA can provide a link and index to these fine-scale data files. The amount of time required to create the LTSA is over several hours for multiple audio files.

While running, the user will be presented with progress information.

Click here to visualize printed information in Command Window 
>> mkLtsa
1 data files for LTSA
Total number of raw files: 30
LTSA version 4
Number of samples for fft: 1000
XWAV to LTSA Compression Factor: 1000
  
Opened File: 
G:\EndToEnd_Example_Test_Set\DCPPC01.ltsa
Completed Processing audio file 1
Time to calculate 450 spectra is 119.0585

2. Create TPWS files

i. Edetect: Apply generic energy detector to detect signals from acoustic recordings

Edetect applies a simple energy detector on audio files. User is required to select a folder or multiple folders containing audio files, provide a spreadsheet containing detector parameters, transfer function, and a spreadsheet containing the times to search for signals.

An example of each file required to execute the generic energy detector are provided in the EndToEnd_Example_Test_Set folder:

Detector parameter files:

The detector parameters to specify:

Parameter Value Description
Sample rate (#/sec) 200000 Your data sample rate
Bpf low freq (Hz) 10000 Band-pass filter low frequency
Bpf high freq (Hz) 95000 Band-pass filter high frequency
Threshold (dBpp re uPa) 121 Minimum received level threshold
Frequency for transfer function (Hz) 40200 Calibrate data corresponding to peak frequency of signal on-axis
Time lockout btw detections (ms) 30 Minimum time for next signal to be detected
Pulse length (ms) 1 Maximum signal length to be detected
Un-filtered data 0 Store unfiltered spectra of detections

Event time file:

A spreadsheet containing start and end times (in serial date number format) to search for signals. DCPP01C_event_times.xlsx contains the start and end times of the audio data. Or multiple start and end times can be given to localized the search.

Transfer function:

The transfer function file (682_120919_invSensit.tf) containing preamp gain at different frequencies based on the individual hydrophone calibration.

Execute generic energy detector:

You can execute Edetect providing the files and folders directly in the function call as follows:

>> Edetect('paramFile','G:\EndToEnd_Example_Test_Set\BeakedWhale_parameters_file.xlsx',...
    'tfFile','G:\EndToEnd_Example_Test_Set\682_120919_invSensit.tf',...
    'timeFile','G:\EndToEnd_Example_Test_Set\DCPP01C_event_times.xlsx',...
    'audioDir','G:\EndToEnd_Example_Test_Set',...
    'outputFileName','DCPP01C_Cuviers',...
    'fileType',2,... 
    'channel',1)

Multiple folders can be specified to search for audio files by giving audioDir the multiple paths as 'audioDir',{'G:\EndToEnd_Example_Test_Set';'G:\Other_folder'},...

or through GUI tools that will prompt the user to select the files and folders:

>> Edetect

It would assume that audio files provided would be of extended wave file format (.x.wav). If you still want to execute it through GUI tools for .wav file format, call it as follows:

>> Edetect('fileType',1)

Click here to visualize printed information in Command Window 
>> Edetect

Parameter File: G:\EndToEnd_Example_Test_Set\BeakedWhale_parameters_file.xlsx
Loading Parameter File
Transfer Function: G:\EndToEnd_Example_Test_Set\682_120919_invSensit.tf
Loading Transfer Function File

Event Time File: G:\EndToEnd_Example_Test_Set\DCPP01C_event_times.xlsx
Load event time file

No file type specified. Reading pseudo-wav (x.wav) files.

G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_Edetect_output.txt

Number of Events to run Edetector : 1

 
1  audio data files in directory G:\EndToEnd_Example_Test_Set
 
Number of data chunks to evaluate 28
Detecting in chunk 1 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 2 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 3 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 31
Detecting in chunk 4 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 263
Detecting in chunk 5 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 218
Detecting in chunk 6 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 215
Detecting in chunk 7 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 246
Detecting in chunk 8 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 264
Detecting in chunk 9 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 268
Detecting in chunk 10 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 270
Detecting in chunk 11 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 248
Detecting in chunk 12 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 109
Detecting in chunk 13 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 42
Detecting in chunk 14 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 14
Detecting in chunk 15 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 102
Detecting in chunk 16 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 87
Detecting in chunk 17 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 2
Detecting in chunk 18 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 19 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 20 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 21 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 22 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 23 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 24 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 25 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 26 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 27 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
Detecting in chunk 28 in G:\EndToEnd_Example_Test_Set\DCPP01C_DL67_130205_054500.x.wav
Number of Detections in file segment = 0
 
Number of Detections in Event = 2379

TPWS file saved in G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_TPWS1.mat
Total Number of Detections for All Events = 2379
 
Runtime : 20.4387 seconds

The summary printed information in the Command Window will be stored automatically in a text file (e.g. DCPP01C_Cuviers_Edetect_output.txt).

ii. make_TPWS: Provide detection times to create TPWS file

The input to the DetEdit GUI is a Matlab binary file (*TPWS1.mat), which contains matrices of detection start times, peak-to-peak received levels, waveform, and spectra parameters from acoustic detections. These matrices can be created manually by the user or with make_TPWS. Two example detection files with detections are provided in EndToEnd_Example_Test_Set folder to test and learn how to use make_TPWS.

Example file containing start times and waveform of each detection

The user is required to modify make_TPWS to match the variable names provided from a detection file (.mat) and the directory paths. The minimum required variables in the detection file to create a TPWS file are:

  • detection start times
  • waveform for each detection
  • transfer function: preamp gain per frequency

The detection file (DCPP01C_beakedwhale_detections_example1.mat) contains 3 variables:

  • detTimes vector of detection start times
  • detWaveform is matrix of detection time series.
  • transferFunction matrix with frequency and gain values.

Spectra matrices and received levels (dBpp re 1uPa) for each detection would be computed. The user is required to specify fftLength and sampleRate. It is recommended to specify bandPassEdges as well.

Modify make_TPWS as follows:

% Define input/output locations
inDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing detection files.
myFileFlag = 'DCPP01C_beaked_whale_detections_example1.mat'; % include a string to match for identifying the files to be processed.
iterationNum = 1; % Iteration number, usually 1 when creating TPWS files from scratch.
saveDir = 'G:\EndToEnd_Example_Test_Set\TPWS';
outputFileName = 'DCPP01C_Cuviers';

%% Begin calculations
% Find all the files to be processed
detectionFileList = dir(fullfile(inDir,myFileFlag));

% Initialize variables
fnew = [];
MSP = [];
MSN = [];
MTT = [];
MPP = [];

% Iterate over each file, collecting and calculating the various
% parameters.
for iFile = 1:length(detectionFileList)
    % load each detection file
    thisDetFile = fullfile(detectionFileList(iFile).folder,...
        detectionFileList(iFile).name);
    load(thisDetFile)
    
    [MPPnew,MTTnew,MSPnew,MSNnew,fnew] = make_TPWS_vars('timeSeries',detWaveform,...
        'detectionTimes',detTimes,'fftLength',200,'sampleRate',200000,'bandPassEdges',[5 95],...
        'tfFunFrequency',transferFunction(:,1),'tfFunValues',transferFunction(:,2));
    
    MPP = [MPP;MPPnew];
    MTT = [MTT;MTTnew];
    MSN = [MSN;MSNnew];
    MSP = [MSP;MSPnew];
    f   = fnew;
    
    % Optionally can add logic in here for rolling into a new output file
    % if things get too large (>1.5 million detections is a rough estimate 
    % of max desirable file size). Preallocating space for MSP and MSN will
    % improve performance.
end

% Save output
fullOutputFileName = [outputFileName,'_TPWS',num2str(iterationNum),'.mat'];
save(fullfile(saveDir,fullOutputFileName),'MPP','MSP','MSN','MTT','f','-v7.3')

Example file containing start times, waveform, and spectra of each detection

If the user provides spectra of detections, a vector with the corresponding frequencies must be given as well. In addition, the user is required to specify a minimum received level (dBpp re 1uPa).

The detection file (DCPP01C_beakedwhale_detections_example2.mat) contains 4 variables:

  • detTimes vector of detection start times.
  • detWaveform is matrix of detection time series.
  • detSpectra matrix of spectra.
  • freqVector vector of frequencies.

Modify make_TPWS as follows:

% make_TPWS.m

% Define input/output locations
inDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing detection files.
myFileFlag = 'DCPP01C_beaked_whale_detections_example2.mat'; % include a string to match for identifying the files to be processed.
iterationNum = 1; % Iteration number, usually 1 when creating TPWS files from scratch.
saveDir = 'G:\EndToEnd_Example_Test_Set\TPWS';
outputFileName = 'DCPP01C_Cuviers';

%% Begin calculations
% Find all the files to be processed
detectionFileList = dir(fullfile(inDir,myFileFlag));

% Initialize variables
fnew = [];
MSP = [];
MSN = [];
MTT = [];
MPP = [];

% Iterate over each file, collecting and calculating the various
% parameters.
for iFile = 1:length(detectionFileList)
    % load each detection file
    thisDetFile = fullfile(detectionFileList(iFile).folder,...
        detectionFileList(iFile).name);
    load(thisDetFile)

    [MPPnew,MTTnew,MSPnew,MSNnew,fnew] = make_TPWS_vars('timeSeries',detWaveform,...
        'detectionTimes',detTimes,'signalSpectra',detSpectra,'f',freqVector,...
        'peak2peakAmp',121);
    
    MPP = [MPP;MPPnew];
    MTT = [MTT;MTTnew];
    MSN = [MSN;MSNnew];
    MSP = [MSP;MSPnew];
    f   = fnew;
end

% Save output
fullOutputFileName = [outputFileName,'_TPWS',num2str(iterationNum),'.mat'];
save(fullfile(saveDir,fullOutputFileName),'MPP','MSP','MSN','MTT','f','-v7.3')

After executing the modified make_TPWS, a TPWS file (e.g. DCPP01C_Cuviers_TPWS1.mat) will be created in the specified folder, in this case, the TPWS folder.

3. mkLTSAsessions: Create LTSA sessions files

After DCPP01C_Cuviers_TPWS1.mat has been created, the following step is making the snippets of LTSAs corresponding to bouts of detections.

Create data Settings Script - EndToEnd_Example_Settings

A data settings script is required to create LTSAs snippets and annotate data. An example script (EndToEnd_Example_Settings.m) based on the beakedwhaleSettings file is created for this example and also provided in the EndToEnd_Example_Test_Set folder:

% EndToEnd_Example_Settings.m
% Based of beakedwhaleSettings

% Example script to define directories and parameter preference for the
% interface. You can make different versions of this, with different names
% for different species or sites.

% Define input/output locations.REQUIRED
filePrefix = 'DCPP01C'; % TPWS file name to match. 
% Optional, replace file prefix to a more generic name to specify settings 
% for mkLTSAsessions or modDet, it will run in multiple files.
% Example: GofMX_DT03 (will run modDet to all files matching the generic name) 
iterationNum = '1'; % iteration number
sampleRate = 200; % replace with your sample rate
sp = 'Zc'; % species code 
% Example:  '' (Unknown), 'De' (Dolphin), 'Pm' (sperm whale)
% (See comments in initSpParams.m for more species codes)
tpwsDir = 'G:\EndToEnd_Example_Test_Set\TPWS'; % identify folder containing TPWS files

% Specific input/output locations (comment them if not in use)
% tfName = 'E:\MyTFfolder'; % identify folder containing transfer function 
% files (.tf). Required if spectra has not been calculated peak to received levels 

ltsaDir = 'G:\EndToEnd_Example_Test_Set'; % identify folder containing ltsa files (.ltsa)
% REQUIRED to run mkLTSAsessions.m


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%% Setting preferences to override defaults parameters for the interface
% (Uncomment these in as needed to override detEdit defaults defined at
% initDefaultParams.m and initSpParams.m)

%% Bout preferences
% paramsUser.threshRL = 130; % minimum RL threshold in dB peak-to-peak
paramsUser.tfSelect = 0; % freq used for transfer function, leave at 0 if no adjustment
paramsUser.c4fd = 100; % Detections step size to estimate false detection rate
paramsUser.threshRMS = 58; % default for < command, RMS threshold cutoff

Using EndToEnd_Example_Settings.m, the LTSA snippets for each detection bout is performed as follows:

>> mkLTSAsessions(@EndToEnd_Example_Settings)

This will work if EndToEnd_Example_Test_Set folder is added in the Matlab set path

or you can invoke the function directly

>> mkLTSAsessions

and it will prompt you with a window to select the data settings script.

Click here to visualize printed information in Command Window 
>> mkLTSAsessions
No TF Applied 
 Removed too low:148
reading ltsa headers, please be patient ...
done reading ltsa headers
Number Bouts : 1
Session: 1  Start: 05-Feb-2013 05:45:00  End:05-Feb-2013 06:22:30   Duration: 975.4317 sec
Done with file G:\EndToEnd_Example_Test_Set\DCPP01C_Cuviers_TPWS1.mat
Elasped Time : 13.6579 s

4. detEdit: Launch DetEdit interface to annotate detections

5. modDet: Clean up of TPWS files

6. summaryParams: Create summary plots of annotated data

🚧 Under construction!

This software is Copyright © 2019 The Regents of the University of California. All Rights Reserved.

Clone this wiki locally