diff --git a/.gitignore b/.gitignore index 599aae6af..5ff81869b 100644 --- a/.gitignore +++ b/.gitignore @@ -15,3 +15,9 @@ matcherr_* build dist EQcorrscan.egg-info +*.xml +.cache +.eggs +.tox +eqcorrscan/tutorials/*.jpg +pytest_runner* diff --git a/.travis.yml b/.travis.yml index 3b6621adb..9040eb410 100644 --- a/.travis.yml +++ b/.travis.yml @@ -1,20 +1,28 @@ language: python -os: linux +os: + - linux + # - osx # Note, travis doesn't support python builds on osx python: "2.7" - +virtualenv: + system_site_packages: true sudo: false +# Test with multiple obspy versions, allow some backwards compatability +env: + - OBSPY_VERSION=0.10.2 + - OBSPY_VERSION=1.0.0 + addons: apt: source: - lucid packages: - - python-opencv + # - python-opencv + - pandoc -# before_install: -# - sudo apt-get update -# - sudo apt-get install python-opencv -# - sudo dpkg -L python-opencv +# before install: +# - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew update ; fi +# - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then brew install pandoc; fi install: - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then @@ -59,19 +67,23 @@ install: lxml sqlalchemy mock - nose gdal pyproj docopt coverage requests + pytest + pytest-pep8 + opencv - source activate test-environment # install packages not available via conda # - pip install -r requirements.txt - pip install coveralls - pip install geographiclib + - pip install pypandoc # current pyimgur stable release has a py2.6 incompatibility - pip install https://github.com/megies/PyImgur/archive/py26.zip + - pip install obspy==$OBSPY_VERSION - pip freeze - conda list # done installing dependencies @@ -79,13 +91,15 @@ install: - pip install . script: # Run the tests available, more should be added - - python eqcorrscan/tests/runtests.py + # - python eqcorrscan/tests/install_test.py + # py.test + python setup.py test notifications: email: false webhooks: urls: - https://webhooks.gitter.im/e/10122f10ed5043c58ae7 - on_success: never # options: [always|never|change] default: always + on_success: change # options: [always|never|change] default: always on_failure: always # options: [always|never|change] default: always on_start: never # options: [always|never|change] default: always diff --git a/CHANGES.md b/CHANGES.md index 58df77756..537f07be6 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -1,3 +1,15 @@ +## 0.1.0 +* Implimented tests for synthetic generation and match-filter functions +* Developed new tutorials that download from GeoNet and are much clearer +* Changed from PICK and EVENTINFO classes to obspy.core.event classes, note +this will break some previous scripts, however wrappers are included for this, +this ended up being the biggy, and is the reason for ceasing the 0.0.x line. +* Added synthetic seismogram generation, very basic seismograms, but they work +as templates for the detection of *some* seismicity. +* Added compatibility with Obspy v.1.0.0. +* All files now follow pep8. +* Removed parameter files completely, and redundant scripts. + ## 0.0.9 * Working towards following pep8 * Change how match_filter handles missing data - now remove template traces (which have been copied) diff --git a/MANIFEST.in b/MANIFEST.in index 352fec84a..74b64a4cf 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -3,19 +3,20 @@ include README.md # include eqcorrscan/test_data/tutorial_data.tgz -include eqcorrscan/tutorial.py +# include eqcorrscan/tutorial.py recursive-include eqcorrscan/utils *.py -recursive-include eqcorrscan/par *.py -recursive-include eqcorrscan/scripts *.py +# recursive-include eqcorrscan/par *.py +# recursive-include eqcorrscan/scripts *.py +recursive-include eqcorrscan/tutorials *.py recursive-include eqcorrscan/doc * recursive-include eqcorrscan/core *.py recursive-include eqcorrscan/tests *.py # exclude rules -global-exclude *.pyc *.ms *.tgz +# global-exclude *.pyc *.ms *.tgz prune eqcorrscan/doc/_build/latex prune eqcorrscan/plot prune eqcorrscan/detections prune eqcorrscan/grid -prune eqcorrscan/stack_templates -prune eqcorrscan/templates +# prune eqcorrscan/stack_templates +# prune eqcorrscan/templates diff --git a/README.md b/README.md index 55e4d516b..4455fbe25 100644 --- a/README.md +++ b/README.md @@ -7,36 +7,47 @@ [![DocumentationStatus](http://readthedocs.org/projects/eqcorrscan/badge/?version=latest)](http://eqcorrscan.readthedocs.org/en/latest/?badge=latest) # Installation -We are now listed on pypi! Installation has been tested on both OSX and Linux (Ubuntu) and -is as simple as: +Installation has been tested on both OSX and Linux (Ubuntu), we currently do not support +Windows systems, but plan to in the future. Installation for Linux and OS X should be as simple as: -*pip install EQcorrscan* +```pip install EQcorrscan``` If upgrading from a previous version, rather than running install --upgrade, I recommend the following: -*pip install -U --no-deps EQcorrscan* +```pip install -U --no-deps EQcorrscan``` -This will not try to upgrade your dependencies, which is not needed for 0.0.8. +This will not try to upgrade your dependencies, which is not needed. You may wish +to update your obspy version to 1.0.0 which was recently released. We have tested +this and support it, nevertheless, if you find any issues then let us know. -If you want to be kept informed about releases, bug-tracking and enhancements -without having to keep looking on github, subscribe to our [google group](https://groups.google.com/forum/#!forum/eqcorrscan-users). - -You will likely need sudo permissions to run this command. This installation -method is quite new to the package (as of v0.0.4), so there are some things that -are changing in the install process at the moment. This should be smoothed out -by v0.1.0 (maybe mid 2016). +*You will likely need sudo permissions to run this command.* If you have any issues installing please let me know. You will need to install openCV -seperately using (on Linux): +separately using (on Linux): -*sudo apt-get install python-opencv* +```apt-get install python-opencv``` Or, for Mac users, this is available on Macports or other similar package managers. +## Updates + +If you want to be kept informed about releases, bug-tracking and enhancements +without having to keep looking on github, subscribe to our [google group](https://groups.google.com/forum/#!forum/eqcorrscan-users). + +# Documentation + The full documentation for this package can be found here: -[Docs](http://calum-chamberlain.github.io/EQcorrscan/) - you are also -welcome to suggest documentation updates or update the doc in the develop branch, please -do not work on the documentation in the gh-pages branch! +[Docs](http://eqcorrscan.readthedocs.org/en/latest/?badge=latest). +Any errors including typos and just missing bits can either be fixed by you, +or flagged in the issues tab here. We host our docs on readthedocs, which +uses sphinx to scrape the docstrings in the codes, so it is simple to +match the docs to the codes and change the docstrings. + +We also have a github-pages site [EQcorrscan](http://calum-chamberlain.github.io/EQcorrscan/), +which uses jekyll to build the site. Changes or additions to this site can be made on +the gh-pages branch. + +# Functionality This package contains routines to enable the user to conduct match-filter earthquake detections using [obspy](https://github.com/obspy/obspy/wiki) bindings when reading @@ -51,37 +62,40 @@ Also within this package are: * Peak finding algorithm (basic); * Automatic amplitude picker for local magnitude scale; * [Seisan](http://seisan.info/) S-file integration for database management and routine earthquake location; +* Obspy.core.event integration, which opens up lots of other functions (Seishub, hypoDDpy etc.); * Stacking routines including phase-weighted stacking based on Thurber at al. (2014); -* Brightness based template creation based on the work of Frank et al. (2014) +* Brightness based template creation based on the work of Frank et al. (2014); +* Singular Value Decomposition derived magnitude calculations based on Rubinstein & Ellsworth (2010). -This package is written by Calum Chamberlain of Victoria University of Wellington, and -is distributed under the LGPL GNU License, Copyright Calum Chamberlain 2015. +We are currently hovering around 9,000 lines of code (including doc-strings) - it is probably worth +having a look at the docs to check what functions we have. We plan to write a series of tutorials to be +included on the EQcorrscan API to highlight key functions, currently our tutorial only shows +how to do the core matched-filter detection. + +# Licence + +This package is written by Calum Chamberlain and Chet Hopp of Victoria University of Wellington, and +is distributed under the LGPL GNU License, Copyright Calum Chamberlain 2015, 2016. -# Parameter files -To use this package you will need to set up default parameters in the parameter -file. Currently these are located in the source directory, this will change in -the future, hopefully. It is recommended that you copy these default parameter -files before adding your own to allow you to easily transfer back to other -parameter set ups. **These are being migrated to a script directory along with all scripts -for version 0.0.5*** # Contributing + Please fork this project and work on it there then create a pull request to merge back into develop. When you make changes please run the tests in the test directory to ensure -everything merges with minimum effort. +everything merges with minimum effort. If there is not yet a test to cope +with your changes then please write one. Please document your functions following the other documentation within the functions, these doc-scripts will then be built into the main documentation using Sphinx. -We are trying to implement a better branching model, following that found here: -http://nvie.com/posts/a-successful-git-branching-model/ +We are trying to implement a better branching model, following that found [here](http://nvie.com/posts/a-successful-git-branching-model/). To this end, please fork the development branch if you want to develop things, and flag issues in the master for us to bugfix. If you have a feature you want to develop please create a new branch -of the development branch for this and work in there, we can then merge +from the development branch and work on it there, we can then merge it back in to the development branch when it is stable enough. This branching model (git-flow) is pretty well established, and I would recommend @@ -93,3 +107,4 @@ will keep us all branching in the same way. * CJ Chamberlain, DR Shelly, J Townend, TA Stern (2014) [Low‐frequency earthquakes reveal punctuated slow slip on the deep extent of the Alpine Fault, New Zealand](http://onlinelibrary.wiley.com/doi/10.1002/2014GC005436/full), __G-cubed__,doi:10.1002/2014GC005436 * Thurber, C. H., Zeng, X., Thomas, A. M., & Audet, P. (2014). [Phase‐Weighted Stacking Applied to Low‐Frequency Earthquakes](http://www.bssaonline.org/content/early/2014/08/12/0120140077.abstract), __BSSA__, doi:10.1785/0120140077. * Frank, W. B., & Shapiro, N. M. (2014). [Automatic detection of low-frequency earthquakes (LFEs) based on a beamformed network response](http://gji.oxfordjournals.org/content/197/2/1215.short), __Geophysical Journal International__, 197(2), 1215-1223, doi:10.1093/gji/ggu058. +* Rubinstein, J. L., & Ellsworth, W. L. (2010). [Precise estimation of repeating earthquake moment: Example from Parkfield, California](http://www.bssaonline.org/content/100/5A/1952.short), __BSSA__, doi:10.1785/0120100007 diff --git a/eqcorrscan/LFE_brightness_search.py b/eqcorrscan/LFE_brightness_search.py deleted file mode 100755 index dcbf3b4af..000000000 --- a/eqcorrscan/LFE_brightness_search.py +++ /dev/null @@ -1,456 +0,0 @@ -#!/usr/bin/env python - -#------------------------------------------------------------------------------ -# Purpose: Script to call all elements of EQcorrscan module to search -# continuous data for likely LFE repeats -# Author: Calum John Chamberlain -#------------------------------------------------------------------------------ - -""" -LFE_brightness_search - Script to generate templates using the birghtness function -then seaach for repeats of them in contnuous data. - -Copyright 2015 Calum Chamberlain - -This file is part of EQcorrscan. - - EQcorrscan is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - EQcorrscan is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with EQcorrscan. If not, see . - -""" - -import sys, glob -import datetime as dt -instance=0 -Split=False -startdate=False -sys.path.insert(0,"/home/processor/Desktop/EQcorrscan") -parallel=True -oldnodes=False -if len(sys.argv) == 2: - flag=str(sys.argv[1]) - if flag == '--debug': - Test=True - Prep=False - elif flag == '--debug-prep': - Test=False - Prep=True - elif flag == '--old-nodes': - Test=False - Prep=False - oldnodes=True - else: - raise IOError("I don't recognise your arguments I know --debug and --debug-prep, and --instance, --splits, --startdate, --enddate") -elif len(sys.argv) == 5: - args=sys.argv[1:len(sys.argv)] - if args[0] == '--instance' or args[2]=='--instance': - # Arguments to allow the code to be run in multiple instances - Split=True - Test=False - Prep=False - for i in xrange(len(args)): - if args[i] == '--instance': - instance=int(args[i+1]) - print 'I will run this for instance '+str(instance) - elif args[i] == '--splits': - splits=int(args[i+1]) - print 'I will divide the days into '+str(splits)+' chunks' - elif args[0] == '--startdate' or args[2] == '--startdate': - Split=False - Test=False - Prep=False - for i in xrange(len(args)): - if args[i]== '--startdate': - startdate=dt.datetime.strptime(str(args[i+1]),'%Y/%m/%d') - print 'Start date is: '+dt.datetime.strftime(startdate, '%Y/%m/%d') - elif args[i] == '--enddate': - enddate=dt.datetime.strptime(str(args[i+1]), '%Y/%m/%d') - print 'End date is: '+dt.datetime.strftime(enddate, '%Y/%m/%d') - else: - raise IOError("I don't recognise your arguments, I know --debug and"+\ - "--debug-prep, and --instance, --splits, --startdate, --enddate") -elif len(sys.argv) == 6: - args=sys.argv[1:len(sys.argv)] - if args[0] == '--instance' or args[1]=='--instance' or args[2]=='--instance': - # Arguments to allow the code to be run in multiple instances - Split=True - Test=False - Prep=False - for i in xrange(len(args)): - if args[i] == '--instance': - instance=int(args[i+1]) - print 'I will run this for instance '+str(instance) - elif args[i] == '--splits': - splits=int(args[i+1]) - print 'I will divide the days into '+str(splits)+' chunks' - elif args[i] == '--old-nodes': - oldnodes=True - elif args[0] == '--startdate' or argc[1] == '--startdate' or args[2] == '--startdate': - Split=False - Test=False - Prep=False - for i in xrange(len(args)): - if args[i]== '--startdate': - startdate=dt.datetime.strptime(str(args[i+1]),'%Y/%m/%d') - print 'Start date is: '+dt.datetime.strftime(startdate, '%Y/%m/%d') - elif args[i] == '--enddate': - enddate=dt.datetime.strptime(str(args[i+1]), '%Y/%m/%d') - print 'End date is: '+dt.datetime.strftime(enddate, '%Y/%m/%d') - elif args[i] == '--old-nodes': - oldnodes=True - else: - raise IOError("I don't recognise your arguments, I know --debug and"+\ - "--debug-prep, and --instance, --splits, --startdate, --enddate") - -elif len(sys.argv) == 7: - args=sys.argv[1:len(sys.argv)] - Split=False - Test=False - Prep=False - for i in xrange(len(args)): - if args[i]== '--startdate': - startdate=dt.datetime.strptime(str(args[i+1]),'%Y/%m/%d') - print 'Start date is: '+dt.datetime.strftime(startdate, '%Y/%m/%d') - elif args[i] == '--enddate': - enddate=dt.datetime.strptime(str(args[i+1]), '%Y/%m/%d') - print 'End date is: '+dt.datetime.strftime(enddate, '%Y/%m/%d') - elif args[i] == '--instance': - instance=int(args[i+1]) - print 'This instance is given the flag '+str(instance) -elif not len(sys.argv) == 1: - raise ValueError("I only take one argument, no arguments, or two flags with arguments") -else: - Test=False - Prep=False - Split=False - -from par import LFE_template_gen_par as templatedef -from par import match_filter_par_LFEs as matchdef -from par import bright_lights_par as brightdef -from utils import seismo_logs -if brightdef.plotsave: - import matplotlib - matplotlib.use('Agg') - import matplotlib.pyplot as plt - plt.ioff() -#from par import lagcalc as lagdef -from obspy import UTCDateTime, Stream, read as obsread -# First generate the templates -from core import bright_lights, match_filter -from utils import pre_processing -from utils import EQcorrscan_plotting as plotting -from obspy.signal.filter import bandpass -from joblib import Parallel, delayed -import warnings, pickle - -templates=[] -delays=[] -stations=[] -print 'Template generation parameters are:' -print ' sfilebase: '+templatedef.sfilebase -print ' samp_rate: '+str(templatedef.samp_rate)+' Hz' -print ' lowcut: '+str(templatedef.lowcut)+' Hz' -print ' highcut: '+str(templatedef.highcut)+' Hz' -print ' length: '+str(templatedef.length)+' s' -print ' swin: '+templatedef.swin+'\n' - -if not oldnodes: - # Use the brightness function to search for possible templates - # First read in the travel times - print 'Reading in the original grids' - stations, allnodes, alllags = \ - bright_lights._read_tt(brightdef.nllpath,brightdef.stations,\ - brightdef.phase, phaseout='S', \ - ps_ratio=brightdef.ps_ratio) - print 'I have read in '+str(len(allnodes))+' nodes' - # Resample the grid to allow us to run it quickly! - print 'Cutting the grid' - stations, nodes, lags = bright_lights._resample_grid(stations, allnodes, - alllags, - brightdef.mindepth, - brightdef.maxdepth, - brightdef.corners, - brightdef.resolution) - del allnodes, alllags - # Check that we still have a grid! - if len(nodes) == 0: - raise IOError("You have no nodes left") - # Remove lags that have a similar network moveout, e.g. the sum of the - # differences in moveouts is small. - print "Removing simlar lags" - stations, nodes, lags = bright_lights._rm_similarlags(stations, nodes, lags, - brightdef.nodesimthresh) - # print "Plotting new grid" - # plotting.threeD_gridplot(nodes, save=brightdef.plotsave, savefile='Nodes_in.png') - # Call the main function! - - fnodesin=open('Nodes_in.csv','w') - for node in nodes: - fnodesin.write(node[0]+','+node[1]+','+node[2]+'\n') - fnodesin.close() - with open('Nodes_in.pkl','wb') as pickle_file: - pickle.dump(nodes, pickle_file, protocol=pickle.HIGHEST_PROTOCOL) - with open('Lags_in.pkl','wb') as pickle_file: - pickle.dump(lags, pickle_file, protocol=pickle.HIGHEST_PROTOCOL) - with open('Stations_in.pkl','wb') as pickle_file: - pickle.dump(stations, pickle_file, protocol=pickle.HIGHEST_PROTOCOL) -else: - with open('Nodes_in.pkl','rb') as pickle_load: - nodes = pickle.load(pickle_load) - with open('Lags_in.pkl','rb') as pickle_load: - lags = pickle.load(pickle_load) - with open('Stations_in.pkl','rb') as pickle_load: - stations = pickle.load(pickle_load) - print 'Read in '+str(len(nodes))+' nodes' - -templates=[] -nodesout=[] - -if startdate: - dates=[UTCDateTime(startdate)+i for i in xrange(0, int(UTCDateTime(enddate) -\ - UTCDateTime(startdate)),\ - 86400)] -else: - dates=brightdef.dates - -ndays=len(dates) -print 'Will loop through '+str(ndays)+' days' -if Split: - if instance==splits-1: - ndays=ndays-(ndays/splits)*(splits-1) - dates=dates[-ndays:] - else: - ndays=ndays/splits - dates=dates[ndays*instance:(ndays*instance)+ndays] - print 'This instance will run for '+str(ndays)+' days' - print 'This instance will run from '+str(min(dates)) - - -for day in dates: #Loop through dates - # Set up where data are going to be read in from - if 'stream' in locals(): - del stream - print 'Working on day '+str(day) - if not Test: - # Read in the data - # if 'stream' in locals(): - # del stream - for station in stations: - # Check logs for this day - if station == 'WHAT2': - sta='WHAT' - useful_chans=['2'] - elif station == 'POCR2': - sta='POCR' - useful_chans=['2'] - elif station == 'FRAN': - sta=station - useful_chans=['2'] - else: - sta=station - useful_chans=['N','2'] - rawdir='/projects/nesi00219/logfiles/Volumes/Taranaki_01/data/boseca/SAMBA_mar09/'+sta+'/'+\ - str(day.year)+str(day.julday).zfill(3) - errors, full = seismo_logs.check_all_logs(rawdir, \ - 1.0/templatedef.samp_rate) - if len(errors) > 1: - warnings.warn('Found GPS errors exceeding one sample for station '+station) - continue - else: - print 'No GPS errors found, continuing' - if len(station) == 3: - netcode='NZ' - else: - netcode='AF' - for base in matchdef.contbase: - if base[2]==netcode: - contbase=base - if not 'contbase' in locals(): - raise NameError('contbase not defined for netcode '+netcode) - if contbase[1]=='yyyymmdd': - daydir=str(day.year)+str(day.month).zfill(2)+\ - str(day.day).zfill(2) - elif contbase[1]=='Yyyyy/Rjjj.01': - daydir='Y'+str(day.year)+'/R'+str(day.julday).zfill(3)+'.01' - print ' Reading data from: ' - for chan in useful_chans: # only take N horizontal components - if glob.glob(contbase[0]+'/'+daydir+'/*'+station+'.*'+chan+'.*'): - print contbase[0]+'/'+daydir+'/*'+station+'.*'+chan+'.*' - if not 'stream' in locals(): - stream=obsread(contbase[0]+'/'+daydir+'/*'+station+'.*'+chan+'.*') - else: - stream+=obsread(contbase[0]+'/'+daydir+'/*'+station+'.*'+chan+'.*') - else: - for station in stations: - fname='test_data/'+station+'-*-'+str(day.year)+\ - '-'+str(day.month).zfill(2)+\ - '-'+str(day.day).zfill(2)+'-processed.ms' - if glob.glob(fname): - if not 'stream' in locals(): - stream=obsread(fname) - else: - stream+=obsread(fname) - # Process the stream - if not Test: - print 'Processing the data' - stream=stream.merge(fill_value='interpolate') - # Merge stream so that each trace is a single channel to - # send to pre-processing - if not parallel: - for tr in stream: - # tr.plot() - tr=pre_processing.dayproc(tr, brightedef.lowcut, brightdef.highcut,\ - brightdef.filter_order, brightdef.samp_rate,\ - templatedef.debug, day) - else: - stream=Parallel(n_jobs=10)(delayed(pre_processing.dayproc)(tr, brightdef.lowcut,\ - brightdef.highcut,\ - brightdef.filter_order,\ - brightdef.samp_rate,\ - templatedef.debug, day)\ - for tr in stream) - stream=Stream(stream) - print stream - if not Prep: - #stream_copy=stream.copy() # Keep the stream safe - print "Running the detection routine" - # Check that the data are okay - detect_templates, detect_nodes=bright_lights.brightness(stations, \ - nodes, lags, stream, - brightdef.threshold, brightdef.thresh_type,\ - brightdef.coherance, instance, matchdef, templatedef) - del detect_templates#, stream # Delete templates from memory to conserve RAM! - #stream=stream_copy - nodesout+=detect_nodes - if Split: - plotting.threeD_gridplot(nodesout, save=brightdef.plotsave,\ - savefile='Detected_nodes_'+str(instance)+'.png') - else: - plotting.threeD_gridplot(nodesout, save=brightdef.plotsave,\ - savefile='Detected_nodes.png') - - else: - for tr in stream: - print "Writing data as: test_data/"+tr.stats.station+'-'+tr.stats.channel+\ - '-'+str(tr.stats.starttime.year)+\ - '-'+str(tr.stats.starttime.month).zfill(2)+\ - '-'+str(tr.stats.starttime.day).zfill(2)+\ - '-processed.ms' - tr.write('test_data/'+tr.stats.station+'-'+tr.stats.channel+\ - '-'+str(tr.stats.starttime.year)+\ - '-'+str(tr.stats.starttime.month).zfill(2)+\ - '-'+str(tr.stats.starttime.day).zfill(2)+\ - '-processed.ms', format='MSEED') - sys.exit() - -import numpy as np -# Now do the detections with these templates! -# station=[] -# print np.shape(templates) -# for template in templates: - # # Calculate the delays for each template, do this only once so that we - # # don't have to do it heaps! - # # Get minimum start time - # mintime=UTCDateTime(3000,1,1,0,0) - # for tr in template: - # if tr.stats.starttime < mintime: - # mintime=tr.stats.starttime - # delay=[] - # # Generate list of delays - # for tr in template: - # delay.append(tr.stats.starttime-mintime) - # delays.append(delay) - # # Generate list of stations in templates - # for tr in template: - # station.append(tr.stats.station+'.'+tr.stats.channel[0]+\ - # '*'+tr.stats.channel[1]) - # stations.append(tr.stats.station+'.'+tr.stats.channel[0]+\ - # '*'+tr.stats.channel[1]) - # Save a miniseed copy of the templates - #print 'Writing template as: '+str(template[0].stats.starttime)+'.ms' - #template.write(templatedef.saveloc+'/'+str(template[0].stats.starttime)+'.ms',format="MSEED") - -# Sort stations into a unique list - this list will ensure we only read in data -# we need, which is VERY important as I/O is very costly and will eat memory -stations=list(set(stations)) - -if Split: - node_file=open('Nodes_detected_'+str(instance)+'.csv','w') -else: - node_file=open('Nodes_detected.csv','w') -for node in nodesout: - node_file.write(node[0]+','+node[1]+','+node[2]+'\n') -node_file.close() - -# print len(stations) -# for station in stations: - # print station - -# Now run the match filter routine - -# # Loop over days -# ndays=int(matchdef.enddate-matchdef.startdate+1) -# newsfiles=[] -# for i in range(0,ndays): - # # Set up where data are going to be read in from - # day=matchdef.startdate+(i*86400) - # if matchdef.baseformat=='yyyy/mm/dd': - # daydir=str(day.year)+'/'+str(day.month).zfill(2)+'/'+\ - # str(day.day).zfill(2) - # elif matchdef.baseformat=='Yyyyy/Rjjj.01': - # daydir='Y'+str(day.year)+'/R'+str(day.julday).zfill(3)+'.01' - # # Read in data using obspy's reading routines, data format will be worked - # # out by the obspy module - # # Note you might have to change this bit to match your naming structure - # for stachan in stations: - # # station is of the form STA.CHAN, to allow these to be in an odd - # # arrangements we can seperate them - # try: - # station=stachan.split('.')[0] - # channel=stachan.split('.')[1] - # except: - # print 'Issues with this station name: '+stachan - # sys.exit() - # if glob.glob(matchdef.contbase+'/'+daydir+'/'+station+'*.'+channel+'.*'): - # #print 'Reading data from: '+\ - # # glob.glob(matchdef.contbase+'/'+daydir+'/'+station+'*.'+channel+'.*')[0] - # if not 'st' in locals(): - # st=obsread(matchdef.contbase+'/'+daydir+'/'+station+'*.'+channel+'.*') - # else: - # st+=obsread(matchdef.contbase+'/'+daydir+'/'+station+'*.'+channel+'.*') - # else: - # print 'No data for '+stachan+' for day '+daydir+' in '+matchdef.contbase - # if not 'st' in locals(): - # print 'No data found for day: '+str(day) - # break - # # Process data - # print 'Processing the data' - # for tr in st: - # tr=pre_processing.dayproc(tr, templatedef.lowcut, templatedef.highcut,\ - # templatedef.filter_order, templatedef.samp_rate,\ - # matchdef.debug, day) - - # # Call the match_filter module - returns detections, a list of detections - # # containted within the detection class with elements, time, template, - # # number of channels used and cross-channel correlation sum. - # print 'Running the detection routine' - # detections=match_filter.match_filter(templatedef.sfiles, templates, delays, st, - # matchdef.threshold, matchdef.threshtype, - # matchdef.trig_int*st[0].stats.sampling_rate, - # matchdef.plot) - - # for detection in detections: - # print 'template: '+detection.template_name+' detection at: '\ - # +str(detection.detect_time)+' with a cccsum of: '+str(detection.detect_val) - # # Call the lag generation routine - returns list of s-files generated - # #newsfiles+=lagcalc(detections, st) diff --git a/eqcorrscan/WHATVsearch.py b/eqcorrscan/WHATVsearch.py deleted file mode 100755 index 1c5e8aec9..000000000 --- a/eqcorrscan/WHATVsearch.py +++ /dev/null @@ -1,352 +0,0 @@ -#!/usr/bin/env python - -#------------------------------------------------------------------------------ -# Purpose: Script to call all elements of EQcorrscan module to search -# continuous data for likely LFE repeats -# Author: Calum John Chamberlain -#------------------------------------------------------------------------------ - -""" -LFEsearch - Script to generate templates from previously picked EQs and then -search for repeats of them in contnuous data. - -Copyright 2015 Calum Chamberlain - -This file is part of EQcorrscan. - - EQcorrscan is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - EQcorrscan is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with EQcorrscan. If not, see . - -""" - -import sys -import os -import glob -bob=os.path.realpath(__file__) -bob=bob.split('/') -path='/' -for i in xrange(len(bob)): - path+=bob[i]+'/' -print path -sys.path.insert(0,path) -startdate=False - -from par import template_gen_par as templatedef -from par import match_filter_par as matchdef -#from par import lagcalc as lagdef -from obspy import UTCDateTime, Stream, read as obsread -# First generate the templates -from eqcorrscan.core import template_gen -from eqcorrscan.utils import seismo_logs -import datetime as dt - -Split=False -instance=False -if len(sys.argv) == 2: - flag=str(sys.argv[1]) - if flag == '--debug': - Test=True - Prep=False - elif flag == '--debug-prep': - Test=False - Prep=True - else: - raise ValueError("I don't recognise the argument, I only know --debug and --debug-prep") -elif len(sys.argv) == 5: - args=sys.argv[1:len(sys.argv)] - if args[0] == '--instance' or args[1]=='--instance' or args[2]=='--instance': - # Arguments to allow the code to be run in multiple instances - Split=True - Test=False - Prep=False - args=sys.argv[1:len(sys.argv)] - for i in xrange(len(args)): - if args[i] == '--instance': - instance=int(args[i+1]) - print 'I will run this for instance '+str(instance) - elif args[i] == '--splits': - splits=int(args[i+1]) - print 'I will divide the days into '+str(splits)+' chunks' - elif args[0] == '--startdate' or argc[1] == '--startdate' or args[2] == '--startdate': - Split=False - Test=False - Prep=False - for i in xrange(len(args)): - if args[i]== '--startdate': - startdate=dt.datetime.strptime(str(args[i+1]),'%Y/%m/%d') - print 'Start date is: '+dt.datetime.strftime(startdate, '%Y/%m/%d') - elif args[i] == '--enddate': - enddate=dt.datetime.strptime(str(args[i+1]), '%Y/%m/%d') - print 'End date is: '+dt.datetime.strftime(enddate, '%Y/%m/%d') - -elif not len(sys.argv) == 1: - raise ValueError("I only take one argument, no arguments, or two flags with arguments") -else: - Test=False - Prep=False - Split=False - -templates=[] -delays=[] -stations=[] -print 'Template generation parameters are:' -print 'sfilebase: '+templatedef.sfilebase -print 'samp_rate: '+str(templatedef.samp_rate)+' Hz' -print 'lowcut: '+str(templatedef.lowcut)+' Hz' -print 'highcut: '+str(templatedef.highcut)+' Hz' -print 'length: '+str(templatedef.length)+' s' -print 'swin: '+templatedef.swin+'\n' -# for sfile in templatedef.sfiles: -# print 'Working on: '+sfile+'\r' -# if not os.path.isfile(templatedef.saveloc+'/'+sfile+'_template.ms'): -# print sfile -# template=template_gen.from_contbase(sfile, prepick=0.1) -# -# print 'saving template as: '+templatedef.saveloc+'/'+\ -# sfile.split('/')[-1]+'_template.ms' -# template.write(templatedef.saveloc+'/'+\ -# sfile.split('/')[-1]+'_template.ms',format="MSEED") -# del template -# else: -# template=obsread(templatedef.saveloc+'/'+sfile+'_template.ms') - # Will read in seisan s-file and generate a template from this, - # returned name will be the template name, used for parsing to the later - # functions - -# for tfile in templatedef.tfiles: - # # Loop through pre-existing template files - # sys.stdout.write("\rReading in pre-existing template: "+tfile+"\r") - # sys.stdout.flush() - # templates.append(obsread(tfile)) - -templates=[obsread(tfile) for tfile in templatedef.tfiles] - -print 'Read in '+str(len(templates))+' templates' - -for template in templates: - # Calculate the delays for each template, do this only once so that we - # don't have to do it heaps! - # Check that all templates are the correct length - for tr in template: - if not templatedef.samp_rate*templatedef.length == tr.stats.npts: - raise ValueError('Template for '+tr.stats.station+'.'+\ - tr.stats.channel+' is not the correct length, recut.'+\ - ' It is: '+str(tr.stats.npts)+' and should be '+ - str(templatedef.samp_rate*templatedef.length)) - # Generate list of stations in templates - for tr in template: - # Correct FOZ channels - if tr.stats.station=='FOZ' and len(tr.stats.channel)==3: - tr.stats.channel='HH'+tr.stats.channel[2] - if len(tr.stats.channel)==3: - stations.append(tr.stats.station+'.'+tr.stats.channel[0]+\ - '*'+tr.stats.channel[2]+'.'+tr.stats.network) - tr.stats.channel=tr.stats.channel[0]+tr.stats.channel[2] - elif len(tr.stats.channel)==2: - stations.append(tr.stats.station+'.'+tr.stats.channel[0]+\ - '*'+tr.stats.channel[1]+'.'+tr.stats.network) - else: - raise ValueError('Chanels are not named with either three or two charectars') - - -# Template generation and processing is over, now to the match-filtering - -# Sort stations into a unique list - this list will ensure we only read in data -# we need, which is VERY important as I/O is very costly and will eat memory -stations=list(set(stations)) - -# Now run the match filter routine -from core import match_filter -from obspy import read as obsread -# from obspy.signal.filter import bandpass -# from obspy import Stream, Trace -# import numpy as np -from utils import pre_processing -from joblib import Parallel, delayed - -if startdate: - dates=[UTCDateTime(startdate)+i for i in xrange(0, int(UTCDateTime(enddate) -\ - UTCDateTime(startdate)),\ - 86400)] -else: - dates=matchdef.dates - - -# Loop over days -ndays=len(dates) -newsfiles=[] -# f=open('detections/run_start_'+str(UTCDateTime().year)+\ - #str(UTCDateTime().month).zfill(2)+\ - #str(UTCDateTime().day).zfill(2)+'T'+\ - #str(UTCDateTime().hour).zfill(2)+str(UTCDateTime().minute).zfill(2),'w') -print 'Will loop through '+str(ndays)+' days' -if Split: - if instance==splits-1: - ndays=ndays-(ndays/splits)*(splits-1) - dates=dates[-ndays:] - else: - ndays=ndays/splits - dates=dates[ndays*instance:(ndays*instance)+ndays] - print 'This instance will run for '+str(ndays)+' days' - print 'This instance will run from '+str(min(dates)) -else: - dates=dates - -f=open('detections/'+str(min(dates).year)+\ - str(min(dates).month).zfill(2)+\ - str(min(dates).day).zfill(2)+'_'+\ - str(max(dates).year)+str(max(dates).month).zfill(2)+\ - str(max(dates).day).zfill(2),'w') -f.write('template, detect-time, cccsum, threshold, number of channels\n') - -for day in dates: - if 'st' in locals(): - del st - # Read in data using obspy's reading routines, data format will be worked - # out by the obspy module - # Note you might have to change this bit to match your naming structure - actual_stations=[] # List of the actual stations used - for stachan in stations: - # station is of the form STA.CHAN, to allow these to be in an odd - # arrangements we can seperate them - station=stachan.split('.')[0] - channel=stachan.split('.')[1] - netcode=stachan.split('.')[2] - rawdir='/Volumes/Taranaki_01/data/boseca/SAMBA_mar09/'+station+'/'+\ - str(day.year)+str(day.julday).zfill(3) - errors, full = seismo_logs.check_all_logs(rawdir, \ - 1.0/templatedef.samp_rate) - if len(errors) > 1: - continue - if not Test: - # Set up the base directory format - for base in matchdef.contbase: - if base[2]==netcode: - contbase=base - if not 'contbase' in locals(): - raise NameError('contbase is not defined for '+netcode) - baseformat=contbase[1] - if baseformat=='yyyy/mm/dd': - daydir=str(day.year)+'/'+str(day.month).zfill(2)+'/'+\ - str(day.day).zfill(2) - elif baseformat=='Yyyyy/Rjjj.01': - daydir='Y'+str(day.year)+'/R'+str(day.julday).zfill(3)+'.01' - elif baseformat=='yyyymmdd': - daydir=str(day.year)+str(day.month).zfill(2)+str(day.day).zfill(2) - - # Try and find the appropriate files - if baseformat=='Yyyyy/Rjjj.01': - if glob.glob(contbase[0]+'/'+daydir+'/'+station+'.*.'+channel+\ - '.'+str(day.year)+'.'+str(day.julday).zfill(3)): - chan_available=True - else: - chan_available=False - else: - if glob.glob(contbase[0]+'/'+daydir+'/*'+station+'.'+channel+'.*'): - chan_available=True - else: - chan_available=False - if chan_available: - if not 'st' in locals(): - if baseformat=='Yyyyy/Rjjj.01': - st=obsread(contbase[0]+'/'+daydir+'/*'+station+'.*.'+\ - channel+'.'+str(day.year)+'.'+\ - str(day.julday).zfill(3)) - else: - st=obsread(contbase[0]+'/'+daydir+'/*'+station+'.'+\ - channel+'.*') - else: - if baseformat=='Yyyyy/Rjjj.01': - st+=obsread(contbase[0]+'/'+daydir+'/*'+station+'.*.'+\ - channel+'.'+str(day.year)+'.'+\ - str(day.julday).zfill(3)) - else: - st+=obsread(contbase[0]+'/'+daydir+'/*'+station+'.'+\ - channel+'.*') - actual_stations.append(station) # Add to this list only if we have the data - else: - print 'No data for '+stachan+' for day '+daydir+' in '\ - +contbase[0] - else: - fname='test_data/'+station+'-'+channel+'-'+str(day.year)+\ - '-'+str(day.month).zfill(2)+\ - '-'+str(day.day).zfill(2)+'-processed.ms' - if glob.glob(fname): - if not 'st' in locals(): - st=obsread(fname) - else: - st+=obsread(fname) - actual_stations.append(station) - actual_stations=list(set(actual_stations)) - - st=st.merge(fill_value='interpolate') # Enforce trace continuity - if not 'st' in locals(): - print 'No data found for day: '+str(day) - elif len(actual_stations) < matchdef.minsta: - print 'Data from fewer than '+str(matchdef.minsta)+' stations found, will not detect' - else: - if not Test: - # Process data - print 'Processing the data for day '+daydir - if matchdef.debug >= 4: - for tr in st: - tr=pre_processing.dayproc(tr, templatedef.lowcut, templatedef.highcut,\ - templatedef.filter_order, templatedef.samp_rate,\ - matchdef.debug, day) - else: - st=Parallel(n_jobs=10)(delayed(pre_processing.dayproc)(tr, templatedef.lowcut,\ - templatedef.highcut,\ - templatedef.filter_order,\ - templatedef.samp_rate,\ - matchdef.debug, day)\ - for tr in st) - if not Prep: - # For some reason st is now a list rather than a stream - if 'stream_st' in locals(): - del stream_st - for tr in st: - if 'stream_st' in locals(): - stream_st+=tr - else: - stream_st=Stream(tr) - st=stream_st - # Call the match_filter module - returns detections, a list of detections - # containted within the detection class with elements, time, template, - # number of channels used and cross-channel correlation sum. - print 'Running the detection routine' - template_names=[tfile.split('/')[-1] for tfile in templatedef.tfiles] - template_names.append(templatedef.sfiles) - if not os.path.isdir('temp_'+str(instance)): - os.makedirs('temp_'+str(instance)) - detections=match_filter.match_filter(template_names, templates, st, - matchdef.threshold, matchdef.threshtype, - matchdef.trig_int, matchdef.plot, - 'temp_'+str(instance)) - - for detection in detections: - # output detections to file - f.write(detection.template_name+', '+str(detection.detect_time)+\ - ', '+str(detection.detect_val)+', '+str(detection.threshold)+\ - ', '+str(detection.no_chans)+'\n') - print 'template: '+detection.template_name+' detection at: '\ - +str(detection.detect_time)+' with a cccsum of: '+str(detection.detect_val) - if detections: - f.write('\n') - else: - for tr in st: - tr.write('test_data/'+tr.stats.station+'-'+tr.stats.channel+\ - '-'+str(tr.stats.starttime.year)+\ - '-'+str(tr.stats.starttime.month).zfill(2)+\ - '-'+str(tr.stats.starttime.day).zfill(2)+\ - '-processed.ms', format='MSEED') -f.close() diff --git a/eqcorrscan/__init__.py b/eqcorrscan/__init__.py index 8d92c83d8..3d6f75ae9 100755 --- a/eqcorrscan/__init__.py +++ b/eqcorrscan/__init__.py @@ -33,7 +33,7 @@ __all__ = ['core', 'utils', 'par'] -__version__ = '0.0.9' +__version__ = '0.1.0' if __name__ == '__main__': import doctest doctest.testmod(exclude_empty=True) diff --git a/eqcorrscan/core/__init__.py b/eqcorrscan/core/__init__.py index fd2ef3304..f3c1528fc 100644 --- a/eqcorrscan/core/__init__.py +++ b/eqcorrscan/core/__init__.py @@ -1,12 +1,4 @@ -#!/usr/bin/python - -#------------------------------------------------------------------------------ -# Purpose: Convenience imports for EQcorrscan.core module -# Author: Calum John Chamberlain -#------------------------------------------------------------------------------ - -""" -EQcorrscan is a python module designed to run match filter routines for +"""EQcorrscan is a python module designed to run match filter routines for \ seismology, within it are routines for integration to seisan and obspy. With obspy integration (which is necessary) all main waveform formats can be read in and output. diff --git a/eqcorrscan/core/bright_lights.py b/eqcorrscan/core/bright_lights.py index b5e1df8e6..2d2098566 100644 --- a/eqcorrscan/core/bright_lights.py +++ b/eqcorrscan/core/bright_lights.py @@ -1,30 +1,17 @@ #!/usr/bin/python -r"""Code to determine the brightness function of seismic data according to a\ -three-dimensional travel-time grid. This travel-time grid should be generated\ -using the grid2time function of the NonLinLoc package by Anthony Lomax which\ -can be found here: http://alomax.free.fr/nlloc/ and is not distributed within\ -this package as this is a very useful stand-alone library for seismic event\ -location. +r"""Code to determine the brightness function of seismic data according to a \ +three-dimensional travel-time grid. This travel-time grid should be \ +generated using the grid2time function of the NonLinLoc package by Anthony \ +Lomax which can be found here: http://alomax.free.fr/nlloc/ and is not \ +distributed within this package as this is a very useful stand-alone library \ +for seismic event location. This code is based on the method of Frank & Shapiro 2014.\ -Code generated by Calum John Chamberlain of Victoria University of Wellington,\ +Code written by Calum John Chamberlain of Victoria University of Wellington, \ 2015. - -.. rubric:: Note -Pre-requisites: - - gcc - for the installation of the openCV correlation routine - - python-cv2 - Python bindings for the openCV routines - - python-joblib - used for parallel processing - - python-obspy - used for lots of common seismological processing - - requires: - - numpy - - scipy - - matplotlib - - NonLinLoc - used outside of all codes for travel-time generation - -Copyright 2015 Calum Chamberlain +Copyright 2015, 2016 Calum Chamberlain This file is part of EQcorrscan. @@ -42,14 +29,18 @@ along with EQcorrscan. If not, see . """ +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals import numpy as np import warnings def _read_tt(path, stations, phase, phaseout='S', ps_ratio=1.68, lags_switch=True): - r"""Function to read in .csv files of slowness generated from Grid2Time\ - (part of NonLinLoc by Anthony Lomax) and convert this to a useful format\ + r"""Function to read in .csv files of slowness generated from Grid2Time \ + (part of NonLinLoc by Anthony Lomax) and convert this to a useful format \ here. It should be noted that this can read either P or S travel-time grids, not @@ -63,14 +54,20 @@ def _read_tt(path, stations, phase, phaseout='S', ps_ratio=1.68, :param phaseout: What phase to return the lagtimes in :type ps_ratio: float :param ps_ratio: p to s ratio for coversion - :type lags_switch: Bool - :param lags_switch: Return lags or raw travel-times, if set to true will\ - return lags. - - :return: list stations, list of lists of tuples nodes, \ - :class: 'numpy.array' lags station[1] refers to nodes[1] and \ - lags[1] nodes[1][1] refers to station[1] and lags[1][1]\ - nodes[n][n] is a tuple of latitude, longitude and depth + :type lags_switch: bool + :param lags_switch: Return lags or raw travel-times, if set to true will \ + return lags. + + :return: list stations, list of lists of tuples nodes, np.ndarray of \ + lags. station[1] refers to nodes[1] and lags[1] nodes[1][1] refers to \ + station[1] and lags[1][1] nodes[n][n] is a tuple of latitude, \ + longitude and depth. + + .. note:: This function currently needs comma seperated grid files in \ + NonLinLoc format. Only certain versions of NonLinLoc write these csv \ + files, however it should be possible to read the binary files \ + directly. If you find you need this capability let us know and we \ + can try and impliment it. """ import csv @@ -89,7 +86,7 @@ def _read_tt(path, stations, phase, phaseout='S', ps_ratio=1.68, # Read the files allnodes = [] for gridfile in gridfiles: - print ' Reading slowness from: ' + gridfile + print(' Reading slowness from: ' + gridfile) f = open(gridfile, 'r') grid = csv.reader(f, delimiter=' ') traveltime = [] @@ -121,33 +118,36 @@ def _read_tt(path, stations, phase, phaseout='S', ps_ratio=1.68, def _resample_grid(stations, nodes, lags, mindepth, maxdepth, corners, resolution): - r"""Function to resample the lagtime grid to a given volume. For use if the - grid from Grid2Time is too large or you want to run a faster, downsampled - scan. + r"""Function to resample the lagtime grid to a given volume. For use if \ + the grid from Grid2Time is too large or you want to run a faster, \ + downsampled scan. :type stations: list - :param stations: List of station names from in the form where stations[i]\ - refers to nodes[i][:] and lags[i][:] + :param stations: List of station names from in the form where stations[i] \ + refers to nodes[i][:] and lags[i][:] :type nodes: list, tuple - :param nodes: List of node points where nodes[i] referes to stations[i]\ - and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is longitude in\ - degrees, nodes[:][:][2] is depth in km. + :param nodes: List of node points where nodes[i] referes to stations[i] \ + and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is \ + lonitude in degrees, nodes[:][:][2] is depth in km. :type lags: :class: 'numpy.array' - :param lags: Array of arrays where lags[i][:] refers to stations[i].\ - lags[i][j] should be the delay to the nodes[i][j] for stations[i] in\ - seconds. + :param lags: Array of arrays where lags[i][:] refers to stations[i]. \ + lags[i][j] should be the delay to the nodes[i][j] for stations[i] in \ + seconds. :type mindepth: float :param mindepth: Upper limit of volume :type maxdepth: float :param maxdepth: Lower limit of volume :type corners: matplotlib.Path - :param corners: matplotlib path of the corners for the 2D polygon to cut\ - to in lat and long + :param corners: matplotlib path of the corners for the 2D polygon to cut \ + to in lat and long :return: list stations, list of lists of tuples nodes, :class: \ - 'numpy.array' lags station[1] refers to nodes[1] and lags[1]\ - nodes[1][1] refers to station[1] and lags[1][1]\ - nodes[n][n] is a tuple of latitude, longitude and depth. + 'numpy.array' lags station[1] refers to nodes[1] and lags[1] \ + nodes[1][1] refers to station[1] and lags[1][1] \ + nodes[n][n] is a tuple of latitude, longitude and depth. + + .. note:: This is an internal function and \ + should not be called directly. """ import numpy as np @@ -161,41 +161,44 @@ def _resample_grid(stations, nodes, lags, mindepth, maxdepth, corners, resamp_nodes.append(node) resamp_lags.append([lags[:, i]]) # Reshape the lags - print np.shape(resamp_lags) + print(np.shape(resamp_lags)) resamp_lags = np.reshape(resamp_lags, (len(resamp_lags), len(stations))).T # Resample the nodes - they are sorted in order of size with largest long # then largest lat, then depth. - print ' '.join(['Grid now has ', str(len(resamp_nodes)), 'nodes']) + print(' '.join(['Grid now has ', str(len(resamp_nodes)), 'nodes'])) return stations, resamp_nodes, resamp_lags def _rm_similarlags(stations, nodes, lags, threshold): - r"""Function to remove those nodes that have a very similar network moveout - to another lag. + r"""Function to remove those nodes that have a very similar network \ + moveout to another lag. - Will, for each node, calculate the difference in lagtime at each station - at every node, then sum these for each node to get a cumulative difference - in network moveout. This will result in an array of arrays with zeros on - the diagonal. + Will, for each node, calculate the difference in lagtime at each \ + station at every node, then sum these for each node to get a \ + cumulative difference in network moveout. This will result in an \ + array of arrays with zeros on the diagonal. :type stations: list - :param stations: List of station names from in the form where stations[i]\ - refers to nodes[i][:] and lags[i][:] + :param stations: List of station names from in the form where stations[i] \ + refers to nodes[i][:] and lags[i][:] :type nodes: list, tuple - :param nodes: List of node points where nodes[i] referes to stations[i]\ - and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is longitude in\ - degrees, nodes[:][:][2] is depth in km. + :param nodes: List of node points where nodes[i] referes to stations[i] \ + and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is \ + longitude in degrees, nodes[:][:][2] is depth in km. :type lags: :class: 'numpy.array' - :param lags: Array of arrays where lags[i][:] refers to stations[i].\ - lags[i][j] should be the delay to the nodes[i][j] for stations[i] in\ - seconds + :param lags: Array of arrays where lags[i][:] refers to stations[i]. \ + lags[i][j] should be the delay to the nodes[i][j] for stations[i] in \ + seconds :type threhsold: float :param threshold: Threshold for removal in seconds :returns: list stations, list of lists of tuples nodes, :class: \ - 'numpy.array' lags station[1] refers to nodes[1] and lags[1]\ - nodes[1][1] refers to station[1] and lags[1][1]\ - nodes[n][n] is a tuple of latitude, longitude and depth. + 'numpy.array' lags station[1] refers to nodes[1] and lags[1] \ + nodes[1][1] refers to station[1] and lags[1][1] \ + nodes[n][n] is a tuple of latitude, longitude and depth. + + .. note:: This is an internal function and \ + should not be called directly. """ import sys @@ -206,22 +209,25 @@ def _rm_similarlags(stations, nodes, lags, threshold): lags.T[i]).sum(axis=1).reshape(1, len(nodes)))\ > threshold netdif = np.concatenate((netdif, _netdif), axis=0) - sys.stdout.write("\r" + str(float(i) / len(nodes) * 100) + "% \r") + sys.stdout.write("\r" + str(float(i) // len(nodes) * 100) + "% \r") sys.stdout.flush() nodes_out = [nodes[0]] node_indeces = [0] - print "\n" - print len(nodes) + print("\n") + print(len(nodes)) for i in xrange(1, len(nodes)): if np.all(netdif[i][node_indeces]): node_indeces.append(i) nodes_out.append(nodes[i]) lags_out = lags.T[node_indeces].T - print "Removed " + str(len(nodes) - len(nodes_out)) + " duplicate nodes" + print("Removed " + str(len(nodes) - len(nodes_out)) + " duplicate nodes") return stations, nodes_out, lags_out + def _rms(array): """Calculate RMS of array + + .. note:: Just a lazy function using numpy functions. """ from numpy import sqrt, mean, square return sqrt(mean(square(array))) @@ -238,20 +244,23 @@ def _node_loop(stations, lags, stream, clip_level, :type stream: :class: `obspy.Stream` :param stream: Data stream to find the brightness for. :type clip_level: float - :param clip_level: Upper limit for energy as a multiplier to the mean\ - energy. + :param clip_level: Upper limit for energy as a multiplier to the mean \ + energy. :type i: int :param i: Index of loop for parallelisation. :type mem_issue: bool - :param mem_issue: If True will write to disk rather than storing data in\ - RAM. + :param mem_issue: If True will write to disk rather than storing data in \ + RAM. :type instance: int - :param instance: instance for bulk parallelisation, only used if\ - mem_issue=true. + :param instance: instance for bulk parallelisation, only used if \ + mem_issue=true. :type plot: bool :param plot: Turn plotting on or off, defaults to False. :return: (i, energy (np.ndarray)) + + .. note:: This is an internal function to ease parallel processing and \ + should not be called directly. """ import warnings if plot: @@ -335,15 +344,18 @@ def _node_loop(stations, lags, stream, clip_level, def _cum_net_resp(node_lis, instance=0): - r"""Function to compute the cumulative network response by reading the\ + r"""Function to compute the cumulative network response by reading \ saved energy .npy files. :type node_lis: np.ndarray :param node_lis: List of nodes (ints) to read from - :type instance: Int + :type instance: int :param instance: Instance flag for parallelisation, defaults to 0. :returns: np.ndarray cum_net_resp, list of indeces used + + .. note:: This is an internal function to ease parallel processing and \ + should not be called directly. """ import os @@ -367,28 +379,31 @@ def _cum_net_resp(node_lis, instance=0): def _find_detections(cum_net_resp, nodes, threshold, thresh_type, samp_rate, realstations, length): - r"""Function to find detections within the cumulative network response\ + r"""Function to find detections within the cumulative network response \ according to Frank et al. (2014). :type cum_net_resp: np.ndarray :param cum_net_resp: Array of cumulative network response for nodes :type nodes: list of tuples - :param nodes: Nodes associated with the source of energy in the\ - cum_net_resp + :param nodes: Nodes associated with the source of energy in the \ + cum_net_resp :type threshold: float :param threshold: Threshold value :type thresh_type: str - :param thresh_type: Either MAD (Median Absolute Deviation) or abs\ - (absolute) or RMS (Root Mean Squared) + :param thresh_type: Either MAD (Median Absolute Deviation) or abs \ + (absolute) or RMS (Root Mean Squared) :type samp_rate: float :param samp_rate: Sampling rate in Hz :type realstations: list of str - :param realstations: List of stations used to make the cumulative network\ - response, will be reported in the DETECTION + :param realstations: List of stations used to make the cumulative network \ + response, will be reported in the DETECTION :type length: float :param length: Maximum length of peak to look for in seconds :return: detections as :class: DETECTION + + .. note:: This is an internal function to ease parallel processing and \ + should not be called directly. """ from eqcorrscan.core.match_filter import DETECTION from eqcorrscan.utils import findpeaks @@ -396,17 +411,17 @@ def _find_detections(cum_net_resp, nodes, threshold, thresh_type, cum_net_resp = np.nan_to_num(cum_net_resp) # Force no NaNs if np.isnan(cum_net_resp).any(): raise ValueError("Nans present") - print 'Mean of data is: ' + str(np.median(cum_net_resp)) - print 'RMS of data is: ' + str(np.sqrt(np.mean(np.square(cum_net_resp)))) - print 'MAD of data is: ' + str(np.median(np.abs(cum_net_resp))) + print('Mean of data is: ' + str(np.median(cum_net_resp))) + print('RMS of data is: ' + str(np.sqrt(np.mean(np.square(cum_net_resp))))) + print('MAD of data is: ' + str(np.median(np.abs(cum_net_resp)))) if thresh_type == 'MAD': thresh = (np.median(np.abs(cum_net_resp)) * threshold) elif thresh_type == 'abs': thresh = threshold elif thresh_type == 'RMS': thresh = _rms(cum_net_resp) * threshold - print 'Threshold is set to: ' + str(thresh) - print 'Max of data is: ' + str(max(cum_net_resp)) + print('Threshold is set to: ' + str(thresh)) + print('Max of data is: ' + str(max(cum_net_resp))) peaks = findpeaks.find_peaks2(cum_net_resp, thresh, length * samp_rate, debug=0, maxwidth=10) detections = [] @@ -419,22 +434,22 @@ def _find_detections(cum_net_resp, nodes, threshold, thresh_type, 'brightness', realstations)) else: detections = [] - print 'I have found ' + str(len(peaks)) + ' possible detections' + print('I have found ' + str(len(peaks)) + ' possible detections') return detections def coherence(stream_in, stations=['all'], clip=False): - r"""Function to determine the average network coherence of a given\ - template or detection. You will want your stream to contain only\ - signal as noise will reduce the coherence (assuming it is incoherant\ + r"""Function to determine the average network coherence of a given \ + template or detection. You will want your stream to contain only \ + signal as noise will reduce the coherence (assuming it is incoherant \ random noise). :type stream: obspy.Stream - :param stream: The stream of seismic data you want to calculate the\ + :param stream: The stream of seismic data you want to calculate the \ coherence for. :type stations: List of String :param stations: List of stations to use for coherence, default is all - :type clip: Tuple of Float + :type clip: tuple of Float :param clip: Default is to use all the data given - \ tuple of start and end in seconds from start of trace @@ -483,60 +498,60 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, coherence_stations=['all'], coherence_clip=False, gap=2.0, clip_level=100, instance=0, pre_pick=0.2, plotsave=True, cores=1): - r"""Function to calculate the brightness function in terms of energy for\ + r"""Function to calculate the brightness function in terms of energy for \ a day of data over the entire network for a given grid of nodes. Note data in stream must be all of the same length and have the same sampling rates. :type stations: list - :param stations: List of station names from in the form where stations[i]\ - refers to nodes[i][:] and lags[i][:] + :param stations: List of station names from in the form where stations[i] \ + refers to nodes[i][:] and lags[i][:] :type nodes: list, tuple - :param nodes: List of node points where nodes[i] referes to stations[i]\ - and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is longitude in\ - degrees, nodes[:][:][2] is depth in km. + :param nodes: List of node points where nodes[i] referes to stations[i] \ + and nodes[:][:][0] is latitude in degrees, nodes[:][:][1] is \ + longitude in degrees, nodes[:][:][2] is depth in km. :type lags: :class: 'numpy.array' - :param lags: Array of arrays where lags[i][:] refers to stations[i].\ - lags[i][j] should be the delay to the nodes[i][j] for stations[i] in\ - seconds. + :param lags: Array of arrays where lags[i][:] refers to stations[i]. \ + lags[i][j] should be the delay to the nodes[i][j] for stations[i] in \ + seconds. :type stream: :class: `obspy.Stream` :param data: Data through which to look for detections. :type threshold: float - :param threshold: Threshold value for detection of template within the\ - brightness function + :param threshold: Threshold value for detection of template within the \ + brightness function :type thresh_type: str - :param thresh_type: Either MAD or abs where MAD is the Median Absolute\ - Deviation and abs is an absoulte brightness. + :param thresh_type: Either MAD or abs where MAD is the Median Absolute \ + Deviation and abs is an absoulte brightness. :type template_length: float :param template_length: Length of template to extract in seconds :type template_saveloc: str :param template_saveloc: Path of where to save the templates. :type coherence_thresh: tuple of floats - :param coherence_thresh: Threshold for removing incoherant peaks in the\ - network response, those below this will not be used as templates.\ - Must be in the form of (a,b) where the coherence is given by:\ - a-kchan/b where kchan is the number of channels used to compute\ + :param coherence_thresh: Threshold for removing incoherant peaks in the \ + network response, those below this will not be used as templates. \ + Must be in the form of (a,b) where the coherence is given by: \ + a-kchan/b where kchan is the number of channels used to compute \ the coherence :type coherence_stations: list - :param coherence_stations: List of stations to use in the coherance\ + :param coherence_stations: List of stations to use in the coherance \ thresholding - defaults to 'all' which uses all the stations. :type coherence_clip: float :param coherence_clip: tuple - :type coherence_clip: Start and end in seconds of data to window around,\ + :type coherence_clip: Start and end in seconds of data to window around, \ defaults to False, which uses all the data given. :type pre_pick: float :param pre_pick: Seconds before the detection time to include in template :type plotsave: bool - :param plotsave: Save or show plots, if False will try and show the plots\ - on screen - as this is designed for bulk use this is set to\ - True to save any plots rather than show them if you create\ - them - changes the backend of matplotlib, so if is set to\ + :param plotsave: Save or show plots, if False will try and show the plots \ + on screen - as this is designed for bulk use this is set to \ + True to save any plots rather than show them if you create \ + them - changes the backend of matplotlib, so if is set to \ False you will see NO PLOTS! :type cores: int :param core: Number of cores to use, defaults to 1. :type clip_level: float - :param clip_level: Multiplier applied to the mean deviation of the energy\ + :param clip_level: Multiplier applied to the mean deviation of the energy \ as an upper limit, used to remove spikes (earthquakes, \ lightning, electircal spikes) from the energy stack. :type gap: float @@ -552,9 +567,10 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, plt.ioff() # from joblib import Parallel, delayed from multiprocessing import Pool, cpu_count - from eqcorrscan.utils.Sfile_util import PICK from copy import deepcopy from obspy import read as obsread + from obspy.core.event import Catalog, Event, Pick, WaveformStreamID, Origin + from obspy.core.event import EventDescription, CreationInfo, Comment import obspy.Stream import matplotlib.pyplot as plt from eqcorrscan.utils import EQcorrscan_plotting as plotting @@ -589,22 +605,22 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, mem_issue = False # Loop through each node in the input # Linear run - print 'Computing the energy stacks' + print('Computing the energy stacks') if not parallel: for i in range(0, len(nodes)): - print i + print(i) if not mem_issue: j, a = _node_loop(stations, lags[:, i], stream, plot=True) if 'energy' not in locals(): energy = a else: energy = np.concatenate((energy, a), axis=0) - print 'energy: ' + str(np.shape(energy)) + print('energy: ' + str(np.shape(energy))) else: j, filename = _node_loop(stations, lags[:, i], stream, i, mem_issue) energy = np.array(energy) - print np.shape(energy) + print(np.shape(energy)) else: # Parallel run num_cores = cores @@ -612,28 +628,28 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, num_cores = len(nodes) if num_cores > cpu_count(): num_cores = cpu_count() - pool = Pool(processes=num_cores, maxtasksperchild=None) + pool = Pool(processes=num_cores) results = [pool.apply_async(_node_loop, args=(stations, lags[:, i], stream, i, clip_level, mem_issue, instance)) for i in range(len(nodes))] pool.close() if not mem_issue: - print 'Computing the cumulative network response from memory' + print('Computing the cumulative network response from memory') energy = [p.get() for p in results] pool.join() energy.sort(key=lambda tup: tup[0]) energy = [node[1] for node in energy] energy = np.concatenate(energy, axis=0) - print energy.shape + print(energy.shape) else: pool.join() # Now compute the cumulative network response and then detect possible # events if not mem_issue: - print energy.shape + print(energy.shape) indeces = np.argmax(energy, axis=0) # Indeces of maximum energy - print indeces.shape + print(indeces.shape) cum_net_resp = np.array([np.nan] * len(indeces)) cum_net_resp[0] = energy[indeces[0]][0] peak_nodes = [nodes[indeces[0]]] @@ -642,25 +658,25 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, peak_nodes.append(nodes[indeces[i]]) del energy, indeces else: - print 'Reading the temp files and computing network response' - node_splits = len(nodes) / num_cores + print('Reading the temp files and computing network response') + node_splits = len(nodes) // num_cores indeces = [range(node_splits)] for i in range(1, num_cores - 1): indeces.append(range(node_splits * i, node_splits * (i + 1))) indeces.append(range(node_splits * (i + 1), len(nodes))) - pool = Pool(processes=num_cores, maxtasksperchild=None) + pool = Pool(processes=num_cores) results = [pool.apply_async(_cum_net_resp, args=(indeces[i], instance)) for i in range(num_cores)] pool.close() results = [p.get() for p in results] pool.join() responses = [result[0] for result in results] - print np.shape(responses) + print(np.shape(responses)) node_indeces = [result[1] for result in results] cum_net_resp = np.array(responses) indeces = np.argmax(cum_net_resp, axis=0) - print indeces.shape - print cum_net_resp.shape + print(indeces.shape) + print(cum_net_resp.shape) cum_net_resp = np.array([cum_net_resp[indeces[i]][i] for i in range(len(indeces))]) peak_nodes = [nodes[node_indeces[indeces[i]][i]] @@ -683,7 +699,7 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, # outfile='NR_timeseries.eps') # Find detection within this network response - print 'Finding detections in the cumulatve network response' + print('Finding detections in the cumulatve network response') detections = _find_detections(cum_net_resp, peak_nodes, threshold, thresh_type, stream[0].stats.sampling_rate, realstations, gap) @@ -692,41 +708,56 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, nodesout = [] good_detections = [] if detections: - print 'Converting detections in to templates' + print('Converting detections in to templates') + # Generate a catalog of detections + detections_cat = Catalog() for j, detection in enumerate(detections): - print 'Converting for detection ' + str(j) + ' of ' +\ - str(len(detections)) + print('Converting for detection ' + str(j) + ' of ' + + str(len(detections))) + # Create an event for each detection + event = Event() + # Set up some header info for the event + event.event_descriptions.append(EventDescription()) + event.event_descriptions[0].text = 'Brightness detection' + event.creation_info = CreationInfo(agency_id='EQcorrscan') copy_of_stream = deepcopy(stream_copy) - # Convert detections to PICK type - name of detection template - # is the node. + # Convert detections to obspy.core.event type - + # name of detection template is the node. node = (detection.template_name.split('_')[0], detection.template_name.split('_')[1], detection.template_name.split('_')[2]) - print node + print(node) # Look up node in nodes and find the associated lags index = nodes.index(node) detect_lags = lags[:, index] - picks = [] + ksta = Comment(text='Number of stations=' + len(detect_lags)) + event.origins.append(Origin()) + event.origins[0].comments.append(ksta) + event.origins[0].time = copy_of_stream[0].stats.starttime +\ + detect_lags[0] + detection.detect_time + event.origins[0].latitude = node[0] + event.origins[0].longitude = node[1] + event.origins[0].depth = node[2] for i, detect_lag in enumerate(detect_lags): station = stations[i] st = copy_of_stream.select(station=station) if len(st) != 0: for tr in st: - picks.append(PICK(station=station, - channel=tr.stats.channel, - impulsivity='E', phase='S', - weight='3', polarity='', - time=tr.stats.starttime + - detect_lag + detection.detect_time - - pre_pick, - coda='', amplitude='', peri='', - azimuth='', velocity='', AIN='', - SNR='', azimuthres='', timeres='', - finalweight='', distance='', - CAZ='')) - print 'Generating template for detection: ' + str(j) - template = (_template_gen(picks, copy_of_stream, template_length, - 'all')) + _waveform_id = WaveformStreamID(station_code=tr.stats. + station, + channel_code=tr.stats. + channel, + network_code='NA') + event.picks.append(Pick(waveform_id=_waveform_id, + time=tr.stats.starttime + + detect_lag + + detection.detect_time + + pre_pick, + onset='emergent', + evalutation_mode='automatic')) + print('Generating template for detection: ' + str(j)) + template = (_template_gen(event.picks, copy_of_stream, + template_length, 'all')) template_name = template_saveloc + '/' +\ str(template[0].stats.starttime) + '.ms' # In the interests of RAM conservation we write then read @@ -737,13 +768,13 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, float(coherence_thresh[1]) if temp_coher > coh_thresh: template.write(template_name, format="MSEED") - print 'Written template as: ' + template_name - print '---------------------------------coherence LEVEL: ' +\ - str(temp_coher) + print('Written template as: ' + template_name) + print('---------------------------------coherence LEVEL: ' + + str(temp_coher)) coherant = True else: - print 'Template was incoherant, coherence level: ' +\ - str(temp_coher) + print('Template was incoherant, coherence level: ' + + str(temp_coher)) coherant = False del copy_of_stream, tr, template if coherant: @@ -751,7 +782,7 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, nodesout += [node] good_detections.append(detection) else: - print 'No template for you' + print('No template for you') if plotvar: all_detections = [(cum_net_trace[-1].stats.starttime + detection.detect_time).datetime @@ -777,3 +808,8 @@ def brightness(stations, nodes, lags, stream, threshold, thresh_type, title='Network response') nodesout = list(set(nodesout)) return templates, nodesout + + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/eqcorrscan/core/lag_calc.py b/eqcorrscan/core/lag_calc.py deleted file mode 100644 index d0d0d2d91..000000000 --- a/eqcorrscan/core/lag_calc.py +++ /dev/null @@ -1,197 +0,0 @@ -#!/usr/bin/python -""" -Functions to generate lag-times for events detected by correlation. - -Part of the EQcorrscan module to integrate seisan nordic files into a full -cross-channel correlation for detection routine. -EQcorrscan is a python module designed to run match filter routines for -seismology, within it are routines for integration to seisan and obspy. -With obspy integration (which is necessary) all main waveform formats can be -read in and output. - -This main section contains a script, LFE_search.py which demonstrates the usage -of the built in functions from template generation from picked waveforms -through detection by match filter of continuous data to the generation of lag -times to be used for relative locations. - -The match-filter routine described here was used a previous Matlab code for the -Chamberlain et al. 2014 G-cubed publication. The basis for the lag-time -generation section is outlined in Hardebeck & Shelly 2011, GRL. - -Code generated by Calum John Chamberlain of Victoria University of Wellington, -2015. - - -.. rubric:: Note -Pre-requisites: - - gcc - for the installation of the openCV correlation routine - - python-cv2 - Python bindings for the openCV routines - - python-joblib - used for parallel processing - - python-obspy - used for lots of common seismological processing - - requires: - - numpy - - scipy - - matplotlib - - NonLinLoc - used outside of all codes for travel-time generation - -Copyright 2015 Calum Chamberlain - -This file is part of EQcorrscan. - - EQcorrscan is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. - - EQcorrscan is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with EQcorrscan. If not, see . - -""" -import numpy as np -from match_filter import normxcorr2 - - -def _channel_loop(detection, template, i=0): - """ - Utility function to take a stream of data for the detected event and parse - the correct data to lag_gen - - :type detection: obspy.Stream - :param detection: Stream of data for the slave event detected using \ - template - :type template: obspy.Stream - :param template: Stream of data as the template for the detection. - :type i: int, optional - :param i: Used to track which process has occured when running in parallel - - :returns: picks, a tuple of (lag in s, cross-correlation value, station,\ - chan) - """ - from eqcorrscan.utils.Sfile_util import PICK - picks = [] - for tr in template: - image = detection.select(station=tr.stats.station, - channel=tr.stats.channel) - if image: # Ideally this if statement would be removed. - ccc = normxcorr2(tr.data, image[0].data) - shiftlen = len(ccc)*image[0].stats.sample_rate - # Convert the maximum cross-correlation time to an actual time - picktime = image[0].stats.starttime + (np.argmax(ccc) * - image[0].stats.delta) - # Note, this function is incomplete! - # picks.append(PICK()) - # ((lag, np.max(ccc), tr.stats.station, - # tr.stats.channel)) - return (i, picks) - - -def day_loop(detection_streams, template): - """ - Function to loop through multiple detections for one template - ostensibly - designed to run for the same day of data for I/O simplicity, but as you - are passing stream objects it could run for all the detections ever, as long - as you have the RAM! - - :type detection_streams: List of obspy.Stream - :param detection_streams: List of all the detections for this template that you - want to compute the optimum pick for. - :type template: obspy.Stream - :param template: The original template used to detect the detections passed - - :returns: lags - List of List of tuple: lags[i] corresponds to detection[i], - lags[i][j] corresponds to a channel of detection[i], within - this tuple is the lag (in seconds), normalised correlation, - station and channel. - """ - from multiprocessing import Pool, cpu_count # Used to run detections in parallel - lags=[] - num_cores=cpu_count() - if num_cores > len(detection_streams): - num_cores=len(detection_streams) - pool=Pool(processes=num_cores, maxtasksperchild=None) - results=[pool.apply_async(_channel_loop, args=(detection_streams[i], template, i))\ - for i in xrange(len(detection_streams))] - pool.close() - lags=[p.get() for p in results] - lags.sort(key=lambda tup: tup[0]) # Sort based on i - lags=[lag[1] for lag in lags] - # Convert lag time to moveout time - mintime=template.sort(['starttime'])[0].stats.starttime - for lag in lags: - delay=template.select(station=lag[2], channel=lag[3])[0].stats.starttime-\ - mintime - lag[0]+=delay - return lags - -def lag_calc(detections, detect_data, templates, shift_len=0.2, min_cc=0.4): - """ - Overseer function to take a list of detection objects, cut the data for - them to lengths of the same length of the template + shift_len on - either side. This will then write out SEISAN s-file for the detections - with pick times based on the lag-times found at the maximum correlation, - providing that correlation is above the min_cc. - - :type detections: List of DETECTION - :param detections: List of DETECTION objects - :type detect_data: obspy.Stream - :param detect_data: All the data needed to cut from - can be a gappy Stream - :type templates: List of tuple of String, obspy.Stream - :param templates: List of the templates used as tuples of template name, template - :type shift_len: float - :param shift_len: Shift length allowed for the pick in seconds, will be - plus/minus this amount - default=0.2 - :type min_cc: float - :param min_cc: Minimum cross-correlation value to be considered a pick, - default=0.4 - """ - from eqcorrscan.utils import Sfile_util - from obspy import Stream - # First work out the delays for each template - delays=[] # List of tuples - for template in templates: - temp_delays=[] - for tr in tempate[1]: - temp_delays.append((tr.stats.station, tr.stats.channel,\ - tr.stats.starttime-template.sort['starttime'][0].stats.starttime)) - delays.append((template[0], temp_delays)) - detect_streams=[] - for detection in detections: - detect_stream=[] - for tr in detect_data: - tr_copy=tr.copy() - template=[t for t in templates if t[0]==detection.template_name][0] - template=template.select(station=tr.stats.station, - channel=tr.stats.channel) - if template: - template_len=len(template[0]) - else: - continue # If there is no template-data match then skip the rest - # of the trace loop. - delay=[delay for delay in delays if delay[0]==detection.template_name][0] - delay=[d for d in delay if d[0]==tr.stats.station and \ - d[1]==tr.stats.channel][0] - detect_stream.append(tr_copy.trim(starttime=detection.detect_time-\ - shift_len+delay, endtime=detection.detect_time+delay+\ - shift_len+template_len)) - detect_streams.append((detection.template_name, Stream(detect_stream))) - # Tuple of template name and data stream - # Segregate detections by template - lags=[] - for template in templates: - template_detections=[detect[1] for detect in detect_streams\ - if detect[0]==template[0]] - lags.append(day_loop(template_detections, template[1])) - - # Write out the lags! - for event in lags: - # I think I have an old version of Sfile_util here - sfilename=Sfile_util.blanksfile(wavefile, 'L', 'PYTH', 'out', True) - picks=[] - for pick in event: - picks.append(Sfile_util.PICK()) - Sfile_util.populateSfile(sfilename, picks) diff --git a/eqcorrscan/core/match_filter.py b/eqcorrscan/core/match_filter.py index a46ac77c9..c217e8a90 100644 --- a/eqcorrscan/core/match_filter.py +++ b/eqcorrscan/core/match_filter.py @@ -1,47 +1,18 @@ #!/usr/bin/python -r"""Function to cross-correlate templates generated by template_gen function\ -with data and output the detecitons. The main component of this script is the\ -normxcorr2 function from the openCV image processing package. This is a\ -highly optimized and accurate normalized cross-correlation routine. The\ -details of this code can be found here:\ -* http://www.cs.ubc.ca/research/deaton/remarks_ncc.html\ -The cpp code was first tested using the Matlab mex wrapper, and has since been\ -ported to a python callable dynamic library. - -Part of the EQcorrscan module to integrate seisan nordic files into a full\ -cross-channel correlation for detection routine.\ -EQcorrscan is a python module designed to run match filter routines for\ -seismology, within it are routines for integration to seisan and obspy.\ -With obspy integration (which is necessary) all main waveform formats can be\ -read in and output. - -This main section contains a script, LFE_search.py which demonstrates the\ -usage of the built in functions from template generation from picked waveforms\ -through detection by match filter of continuous data to the generation of lag\ -times to be used for relative locations. - -The match-filter routine described here was used a previous Matlab code for\ -the Chamberlain et al. 2014 G-cubed publication. The basis for the lag-time\ -generation section is outlined in Hardebeck & Shelly 2011, GRL. - -Code generated by Calum John Chamberlain of Victoria University of Wellington,\ -2015. - - -.. rubric:: Note -Pre-requisites: - - gcc - for the installation of the openCV correlation routine - - python-cv2 - Python bindings for the openCV routines - - python-joblib - used for parallel processing - - python-obspy - used for lots of common seismological processing - - requires: - - numpy - - scipy - - matplotlib - - NonLinLoc - used outside of all codes for travel-time generation - - -Copyright 2015 Calum Chamberlain +r"""Function to cross-correlate templates generated by template_gen function \ +with data and output the detecitons. The central component of this is \ +the match_template function from the openCV image processing package. This \ +is a highly optimized and accurate normalized cross-correlation routine. \ +The details of this code can be found here: \ +http://www.cs.ubc.ca/research/deaton/remarks_ncc.html \ + +The matched-filter routine described here was used a previous Matlab code for \ +the Chamberlain et al. 2014 G-cubed publication. + +Code written by Calum John Chamberlain of Victoria University of \ +Wellington, 2015. + +Copyright 2015, 2016 Calum Chamberlain This file is part of EQcorrscan. @@ -59,31 +30,35 @@ along with EQcorrscan. If not, see . """ +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals import numpy as np import warnings class DETECTION(object): - r"""Information required for a full detection based on cross-channel\ + r"""Information required for a full detection based on cross-channel \ correlation sums. Attributes: :type template_name: str - :param template_name: The name of the template for which this\ - detection was made. + :param template_name: The name of the template for which this \ + detection was made. :type detect_time: :class: 'obspy.UTCDateTime' :param detect_time: Time of detection as an obspy UTCDateTime object :type no_chans: int - :param no_chans: The number of channels for which the cross-channel\ - correlation sum was calculated over. + :param no_chans: The number of channels for which the cross-channel \ + correlation sum was calculated over. :type chans: list of str :param chans: List of stations for the detection :type cccsum_val: float - :param cccsum_val: The raw value of the cross-channel correlation sum\ - for this detection. + :param cccsum_val: The raw value of the cross-channel correlation sum \ + for this detection. :type threshold: float - :param threshold: The value of the threshold used for this detection,\ - will be the raw threshold value related to the cccsum. + :param threshold: The value of the threshold used for this detection, \ + will be the raw threshold value related to the cccsum. :type typeofdet: str :param typeofdet: Type of detection, STA, corr, bright """ @@ -109,20 +84,20 @@ def __repr__(self): return "DETECTION()" def __str__(self): - """Full print in the style of a pick in an S-file.""" + """Full print.""" print_str = "Detection on " + self.template_name + " at " + \ str(self.detect_time) return print_str def normxcorr2(template, image): - r"""Base function to call the c++ correlation routine from the openCV image\ - processing suite. Requires you to have installed the openCV python\ - bindings, which can be downloaded on Linux machines using:\ + r"""Base function to call the c++ correlation routine from the openCV \ + image processing suite. Requires you to have installed the openCV python \ + bindings, which can be downloaded on Linux machines using: \ **sudo apt-get install python-openCV**. - Here we use the cv2.TM_CCOEFF_NORMED method within openCV to give the\ - normalized cross-correaltion. Documentation on this function can be\ + Here we use the cv2.TM_CCOEFF_NORMED method within openCV to give the \ + normalized cross-correaltion. Documentation on this function can be \ found here: **http://docs.opencv.org/modules/imgproc/doc/object_detection.html?highlight=matchtemplate#cv2.matchTemplate** @@ -130,17 +105,17 @@ def normxcorr2(template, image): :type template: :class: 'numpy.array' :param template: Template array :type image: :class: 'numpy.array' - :param image: image to scan\ - the template through. The order of these matters, if you put the template\ - after the image you will get a reversed correaltion matrix + :param image: image to scan the template through. The order of these \ + matters, if you put the template after the image you will get a \ + reversed correaltion matrix - :return: New :class: 'numpy.array' object of the correlation values for\ - the correlation of the image with the template. + :return: New :class: 'numpy.array' object of the correlation values for \ + the correlation of the image with the template. """ import cv2 # Check that we have been passed numpy arrays if type(template) != np.ndarray or type(image) != np.ndarray: - print 'You have not provided numpy arrays, I will not convert them' + print('You have not provided numpy arrays, I will not convert them') return 'NaN' # Convert numpy arrays to float 32 cv_template = template.astype(np.float32) @@ -158,27 +133,27 @@ def normxcorr2(template, image): def _template_loop(template, chan, station, channel, debug=0, i=0): - r"""Sister loop to handle the correlation of a single template (of multiple\ - channels) with a single channel of data. + r"""Sister loop to handle the correlation of a single template (of \ + multiple channels) with a single channel of data. :type template: obspy.Stream :type chan: np.array :type station: string :type channel: string :type i: int - :param i: Optional argument, used to keep track of which process is being\ - run. - - :returns: tuple of (i,ccc) with ccc as an ndarray - - .. rubric:: Note - ..This function currently assumes only one template-channel per\ - data-channel, while this is normal for a standard matched-filter routine,\ - if we wanted to impliment a subspace detector, this would be the function\ - to change, I think. E.g. where I currently take only the first matching\ - channel, we could loop through all the matching channels and then sum the\ - correlation sums - however I don't really understand how you detect based\ - on that. More reading of the Harris document required. + :param i: Optional argument, used to keep track of which process is being \ + run. + + :returns: tuple of (i, ccc) with ccc as an ndarray + + .. note:: This function currently assumes only one template-channel per \ + data-channel, while this is normal for a standard matched-filter \ + routine, if we wanted to impliment a subspace detector, this would be \ + the function to change, I think. E.g. where I currently take only \ + the first matching channel, we could loop through all the matching \ + channels and then sum the correlation sums - however I haven't yet + implimented detection based on that. More reading of the Harris \ + document required. """ from eqcorrscan.utils.timer import Timer @@ -206,17 +181,17 @@ def _template_loop(template, chan, station, channel, debug=0, i=0): # Convert to float16 to save memory for large problems - lose some # accuracy which will affect detections very close to threshold if debug >= 2 and t.secs > 4: - print "Single if statement took %s s" % t.secs + print("Single if statement took %s s" % t.secs) if not template_data: - print "Didn't even correlate!" - print station + ' ' + channel + print("Didn't even correlate!") + print(station + ' ' + channel) elif debug >= 2: - print "If statement without correlation took %s s" % t.secs + print("If statement without correlation took %s s" % t.secs) if debug >= 3: - print '********* DEBUG: ' + station + '.' +\ - channel + ' ccc MAX: ' + str(np.max(ccc[0])) - print '********* DEBUG: ' + station + '.' +\ - channel + ' ccc MEAN: ' + str(np.mean(ccc[0])) + print('********* DEBUG: ' + station + '.' + + channel + ' ccc MAX: ' + str(np.max(ccc[0]))) + print('********* DEBUG: ' + station + '.' + + channel + ' ccc MEAN: ' + str(np.mean(ccc[0]))) if np.isinf(np.mean(ccc[0])): warnings.warn('Mean of ccc is infinite, check!') if debug >= 3: @@ -224,35 +199,35 @@ def _template_loop(template, chan, station, channel, debug=0, i=0): np.save('inf_cccmean_template.npy', template_data.data) np.save('inf_cccmean_image.npy', image) if debug >= 3: - print 'shape of ccc: ' + str(np.shape(ccc)) - print 'A single ccc is using: ' + str(ccc.nbytes / 1000000) + 'MB' - print 'ccc type is: ' + str(type(ccc)) + print('shape of ccc: ' + str(np.shape(ccc))) + print('A single ccc is using: ' + str(ccc.nbytes / 1000000) + 'MB') + print('ccc type is: ' + str(type(ccc))) if debug >= 3: - print 'shape of ccc: ' + str(np.shape(ccc)) - print "Parallel worker " + str(i) + " complete" + print('shape of ccc: ' + str(np.shape(ccc))) + print("Parallel worker " + str(i) + " complete") return (i, ccc) def _channel_loop(templates, stream, cores=1, debug=0): r""" - Loop to generate cross channel correaltion sums for a series of templates\ - hands off the actual correlations to a sister function which can be run in\ - parallel. + Loop to generate cross channel correaltion sums for a series of templates \ + hands off the actual correlations to a sister function which can be run \ + in parallel. :type templates: :class: 'obspy.Stream' - :param templates: A list of templates, where each one should be an\ - obspy.Stream object containing multiple traces of seismic data and the\ - relevant header information. - :param stream: A single obspy.Stream object containing daylong seismic\ - data to be correlated through using the templates. This is in effect the\ - image. + :param templates: A list of templates, where each one should be an \ + obspy.Stream object containing multiple traces of seismic data and \ + the relevant header information. + :param stream: A single obspy.Stream object containing daylong seismic \ + data to be correlated through using the templates. This is in effect \ + the image. :type core: int :param core: Number of cores to loop over :type debug: int :param debug: Debug level. - :return: New list of :class: 'numpy.array' objects. These will contain\ - the correlation sums for each template for this day of data. + :return: New list of :class: 'numpy.array' objects. These will contain \ + the correlation sums for each template for this day of data. :return: list of ints as number of channels used for each cross-correlation """ import time @@ -280,53 +255,53 @@ def _channel_loop(templates, stream, cores=1, debug=0): station = tr.stats.station channel = tr.stats.channel if debug >= 1: - print "Starting parallel run for station " + station + " channel " +\ - channel + print("Starting parallel run for station " + station + + " channel " + channel) tic = time.clock() with Timer() as t: # Send off to sister function - pool = Pool(processes=num_cores, maxtasksperchild=None) + pool = Pool(processes=num_cores) results = [pool.apply_async(_template_loop, args=(templates[i], tr_data, station, channel, debug, i)) for i in range(len(templates))] pool.close() if debug >= 1: - print "--------- TIMER: Correlation loop took: %s s" % t.secs - print " I have " + str(len(results)) + " results" + print("--------- TIMER: Correlation loop took: %s s" % t.secs) + print(" I have " + str(len(results)) + " results") with Timer() as t: cccs_list = [p.get() for p in results] pool.join() if debug >= 1: - print "--------- TIMER: Getting results took: %s s" % t.secs + print("--------- TIMER: Getting results took: %s s" % t.secs) with Timer() as t: # Sort by placeholder returned from _template_loop cccs_list.sort(key=lambda tup: tup[0]) if debug >= 1: - print "--------- TIMER: Sorting took: %s s" % t.secs + print("--------- TIMER: Sorting took: %s s" % t.secs) with Timer() as t: cccs_list = [ccc[1] for ccc in cccs_list] if debug >= 1: - print "--------- TIMER: Extracting arrays took: %s s" % t.secs + print("--------- TIMER: Extracting arrays took: %s s" % t.secs) if debug >= 3: - print 'cccs_list is shaped: ' + str(np.shape(cccs_list)) + print('cccs_list is shaped: ' + str(np.shape(cccs_list))) with Timer() as t: cccs = np.concatenate(cccs_list, axis=0) if debug >= 1: - print "--------- TIMER: cccs_list conversion: %s s" % t.secs + print("--------- TIMER: cccs_list conversion: %s s" % t.secs) del cccs_list if debug >= 2: - print 'After looping through templates the cccs is shaped: ' +\ - str(np.shape(cccs)) - print 'cccs is using: ' + str(cccs.nbytes / 1000000) +\ - ' MB of memory' + print('After looping through templates the cccs is shaped: ' + + str(np.shape(cccs))) + print('cccs is using: ' + str(cccs.nbytes / 1000000) + + ' MB of memory') cccs_matrix[1] = np.reshape(cccs, (1, len(templates), max(np.shape(cccs)))) del cccs if debug >= 2: - print 'cccs_matrix shaped: ' + str(np.shape(cccs_matrix)) - print 'cccs_matrix is using ' + str(cccs_matrix.nbytes / 1000000) +\ - ' MB of memory' + print('cccs_matrix shaped: ' + str(np.shape(cccs_matrix))) + print('cccs_matrix is using ' + str(cccs_matrix.nbytes / 1000000) + + ' MB of memory') # Now we have an array of arrays with the first dimensional index # giving the channel, the second dimensional index giving the # template and the third dimensional index giving the position @@ -334,8 +309,8 @@ def _channel_loop(templates, stream, cores=1, debug=0): # np.shape(cccsums)=(len(stream), len(templates), len(ccc)) if debug >= 2: - print 'cccs_matrix as a np.array is shaped: ' +\ - str(np.shape(cccs_matrix)) + print('cccs_matrix as a np.array is shaped: ' + + str(np.shape(cccs_matrix))) # First work out how many channels were used for i in range(0, len(templates)): if not np.all(cccs_matrix[1][i] == 0): @@ -348,17 +323,17 @@ def _channel_loop(templates, stream, cores=1, debug=0): with Timer() as t: cccsums = cccs_matrix.sum(axis=0).astype(np.float32) if debug >= 1: - print "--------- TIMER: Summing took %s s" % t.secs + print("--------- TIMER: Summing took %s s" % t.secs) if debug >= 2: - print 'cccsums is shaped thus: ' + str(np.shape(cccsums)) + print('cccsums is shaped thus: ' + str(np.shape(cccsums))) cccs_matrix[0] = cccsums del cccsums toc = time.clock() if debug >= 1: - print "--------- TIMER: Trace loop took " + str(toc - tic) +\ - " s" + print("--------- TIMER: Trace loop took " + str(toc - tic) + + " s") if debug >= 2: - print 'cccs_matrix is shaped: ' + str(np.shape(cccs_matrix)) + print('cccs_matrix is shaped: ' + str(np.shape(cccs_matrix))) cccsums = cccs_matrix[0] return cccsums, no_chans @@ -366,61 +341,60 @@ def _channel_loop(templates, stream, cores=1, debug=0): def match_filter(template_names, template_list, st, threshold, threshold_type, trig_int, plotvar, plotdir='.', cores=1, tempdir=False, debug=0, plot_format='jpg'): - r"""Over-arching code to run the correlations of given templates with a\ + r"""Over-arching code to run the correlations of given templates with a \ day of seismic data and output the detections based on a given threshold. :type template_names: list - :param template_names: List of template names in the same order as\ - template_list + :param template_names: List of template names in the same order as \ + template_list :type template_list: list :class: 'obspy.Stream' - :param template_list: A list of templates of which each template is a\ + :param template_list: A list of templates of which each template is a \ Stream of obspy traces containing seismic data and header information. :type st: :class: 'obspy.Stream' - :param st: An obspy.Stream object containing all the data available and\ - required for the correlations with templates given. For efficiency\ - this should contain no excess traces which are not in one or more of\ - the templates. This will now remove excess traces internally, but\ - will copy the stream and work on the copy, leaving your input stream\ + :param st: An obspy.Stream object containing all the data available and \ + required for the correlations with templates given. For efficiency \ + this should contain no excess traces which are not in one or more of \ + the templates. This will now remove excess traces internally, but \ + will copy the stream and work on the copy, leaving your input stream \ untouched. :type threshold: float :param threshold: A threshold value set based on the threshold_type :type threshold_type: str - :param threshold_type: The type of threshold to be used, can be MAD,\ - absolute or av_chan_corr. MAD threshold is calculated as the\ - threshold*(median(abs(cccsum))) where cccsum is the cross-correlation\ - sum for a given template. absolute threhsold is a true absolute\ - threshold based on the cccsum value av_chan_corr is based on the mean\ - values of single-channel cross-correlations assuming all data are\ + :param threshold_type: The type of threshold to be used, can be MAD, \ + absolute or av_chan_corr. MAD threshold is calculated as the \ + threshold*(median(abs(cccsum))) where cccsum is the cross-correlation \ + sum for a given template. absolute threhsold is a true absolute \ + threshold based on the cccsum value av_chan_corr is based on the mean \ + values of single-channel cross-correlations assuming all data are \ present as required for the template, \ - e.g. av_chan_corr_thresh=threshold*(cccsum/len(template)) where\ - template is a single template from the input and the length is the\ + e.g. av_chan_corr_thresh=threshold*(cccsum/len(template)) where \ + template is a single template from the input and the length is the \ number of channels within this template. :type trig_int: float :param trig_int: Minimum gap between detections in seconds. :type plotvar: bool :param plotvar: Turn plotting on or off :type plotdir: str - :param plotdir: Path to plotting folder, plots will be output here,\ + :param plotdir: Path to plotting folder, plots will be output here, \ defaults to run location. :type tempdir: String or False :param tempdir: Directory to put temporary files, or False :type cores: int :param cores: Number of cores to use :type debug: int - :param debug: Debug output level, the bigger the number, the more the\ + :param debug: Debug output level, the bigger the number, the more the \ output. - :return: :class: 'DETECTIONS' detections for each channel formatted as\ - :class: 'obspy.UTCDateTime' objects. - - .. rubric:: Note - Plotting within the match-filter routine uses the Agg backend with\ - interactive plotting turned off. This is because the function is\ - designed to work in bulk. If you wish to turn interactive plotting on\ - you must import matplotlib in your script first, when you them import\ - match_filter you will get the warning that this call to matplotlib has\ - no effect, which will mean that match_filter has not changed the\ - plotting behaviour. + :return: :class: 'DETECTIONS' detections for each channel formatted as \ + :class: 'obspy.UTCDateTime' objects. + + .. note:: Plotting within the match-filter routine uses the Agg backend \ + with interactive plotting turned off. This is because the function \ + is designed to work in bulk. If you wish to turn interactive \ + plotting on you must import matplotlib in your script first, when you \ + them import match_filter you will get the warning that this call to \ + matplotlib has no effect, which will mean that match_filter has not \ + changed the plotting behaviour. """ import matplotlib matplotlib.use('Agg') @@ -449,10 +423,10 @@ def match_filter(template_names, template_list, st, threshold, template_stachan = list(set(template_stachan)) data_stachan = list(set(data_stachan)) if debug >= 3: - print 'I have template info for these stations:' - print template_stachan - print 'I have daylong data for these stations:' - print data_stachan + print('I have template info for these stations:') + print(template_stachan) + print('I have daylong data for these stations:') + print(data_stachan) # Perform a check that the daylong vectors are daylong for tr in stream: if not tr.stats.sampling_rate * 86400 == tr.stats.npts: @@ -467,7 +441,7 @@ def match_filter(template_names, template_list, st, threshold, # data make the data NaN to return NaN ccc_sum # Note: this works if debug >= 2: - print 'Ensuring all template channels have matches in daylong data' + print('Ensuring all template channels have matches in daylong data') template_stachan = [] for template in templates: for tr in template: @@ -499,19 +473,19 @@ def match_filter(template_names, template_list, st, threshold, dtype=np.float32) template += nulltrace if debug >= 2: - print 'Starting the correlation run for this day' + print('Starting the correlation run for this day') [cccsums, no_chans] = _channel_loop(templates, stream, cores, debug) if len(cccsums[0]) == 0: raise ValueError('Correlation has not run, zero length cccsum') outtoc = time.clock() - print ' '.join(['Looping over templates and streams took:', - str(outtoc - outtic), 's']) + print(' '.join(['Looping over templates and streams took:', + str(outtoc - outtic), 's'])) if debug >= 2: - print ' '.join(['The shape of the returned cccsums is:', - str(np.shape(cccsums))]) - print ' '.join(['This is from', str(len(templates)), 'templates']) - print ' '.join(['Correlated with', str(len(stream)), - 'channels of data']) + print(' '.join(['The shape of the returned cccsums is:', + str(np.shape(cccsums))])) + print(' '.join(['This is from', str(len(templates)), 'templates'])) + print(' '.join(['Correlated with', str(len(stream)), + 'channels of data'])) detections = [] for i, cccsum in enumerate(cccsums): template = templates[i] @@ -522,13 +496,13 @@ def match_filter(template_names, template_list, st, threshold, elif threshold == 'av_chan_corr': rawthresh = threshold * (cccsum / len(template)) else: - print 'You have not selected the correct threshold type, I will' +\ - 'use MAD as I like it' + print('You have not selected the correct threshold type, I will' + + 'use MAD as I like it') rawthresh = threshold * np.mean(np.abs(cccsum)) # Findpeaks returns a list of tuples in the form [(cccsum, sample)] - print ' '.join(['Threshold is set at:', str(rawthresh)]) - print ' '.join(['Max of data is:', str(max(cccsum))]) - print ' '.join(['Mean of data is:', str(np.mean(cccsum))]) + print(' '.join(['Threshold is set at:', str(rawthresh)])) + print(' '.join(['Max of data is:', str(max(cccsum))])) + print(' '.join(['Mean of data is:', str(np.mean(cccsum))])) if np.abs(np.mean(cccsum)) > 0.05: warnings.warn('Mean is not zero! Check this!') # Set up a trace object for the cccsum as this is easier to plot and @@ -541,8 +515,8 @@ def match_filter(template_names, template_list, st, threshold, cccsum_plot.stats.sampling_rate = stream[0].stats.sampling_rate # Resample here to maintain shape better cccsum_hist = cccsum_plot.copy() - cccsum_hist = cccsum_hist.decimate(int(stream[0].stats.sampling_rate / - 10)).data + cccsum_hist = cccsum_hist.decimate(int(stream[0].stats. + sampling_rate / 10)).data cccsum_plot = EQcorrscan_plotting.chunk_data(cccsum_plot, 10, 'Maxabs').data # Enforce same length @@ -553,11 +527,13 @@ def match_filter(template_names, template_list, st, threshold, stream_plot, rawthresh, True, plotdir + '/cccsum_plot_' + template_names[i] + '_' + - stream[0].stats.starttime.datetime.strftime('%Y-%m-%d') + + stream[0].stats.starttime. + datetime.strftime('%Y-%m-%d') + '.' + plot_format) if debug >= 4: - print ' '.join(['Saved the cccsum to:', template_names[i], - stream[0].stats.starttime.datetime.strftime('%Y%j')]) + print(' '.join(['Saved the cccsum to:', template_names[i], + stream[0].stats.starttime.datetime. + strftime('%Y%j')])) np.save(template_names[i] + stream[0].stats.starttime.datetime.strftime('%Y%j'), cccsum) @@ -566,20 +542,20 @@ def match_filter(template_names, template_list, st, threshold, np.save('cccsum_' + str(i) + '.npy', cccsum) if debug >= 3 and max(cccsum) > rawthresh: peaks = findpeaks.find_peaks2_short(cccsum, rawthresh, - trig_int * stream[0].stats.sampling_rate, - debug, + trig_int * stream[0].stats. + sampling_rate, debug, stream[0].stats.starttime, stream[0].stats.sampling_rate) elif max(cccsum) > rawthresh: peaks = findpeaks.find_peaks2_short(cccsum, rawthresh, - trig_int * stream[0].stats.sampling_rate, - debug) + trig_int * stream[0].stats. + sampling_rate, debug) else: - print 'No peaks found above threshold' + print('No peaks found above threshold') peaks = False toc = time.clock() if debug >= 1: - print ' '.join(['Finding peaks took:', str(toc - tic), 's']) + print(' '.join(['Finding peaks took:', str(toc - tic), 's'])) if peaks: for peak in peaks: detecttime = stream[0].stats.starttime +\ @@ -590,3 +566,8 @@ def match_filter(template_names, template_list, st, threshold, 'corr')) del stream, templates return detections + + +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/eqcorrscan/core/match_filter_internal.py b/eqcorrscan/core/match_filter_internal.py index 3640a2732..5b731f911 100644 --- a/eqcorrscan/core/match_filter_internal.py +++ b/eqcorrscan/core/match_filter_internal.py @@ -1,55 +1,59 @@ -#!/usr/bin/python """ Functions written to be compilled by Cython as the inner loops of the match_filter.py routine """ +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals +import numpy as np -import numpy as np -# def _channel_loop(np.ndarray templates, np.ndarray stream, \ - #np.ndarray delays, int ktemplates): -def _channel_loop( templates, stream, delays, ktemplates, savedir=False, cores=10): +def _channel_loop(templates, stream, delays, ktemplates, savedir=False, + cores=10): """ Loop to generate cross channel correaltion sums for a series of templates hands off the actual correlations to a sister function which can be run in parallel. :type templates: .ndarray - :param templates: A series of templates, organized into one np.ndarray to the\ - extent that the returned ccc[i][:] will correspond to template[i][:] + :param templates: A series of templates, organized into one np.ndarray \ + to the extent that the returned ccc[i][:] will correspond to \ + template[i][:] :type stream: np.ndarray :param stream: Image stream - np.ndarray, should be daylong :type delays: np.ndarray - :param delays: Delays for each template in the templates array - must be\ - in samples, these will be the length of the pads applied to the\ + :param delays: Delays for each template in the templates array - must be \ + in samples, these will be the length of the pads applied to the \ stream arrays :type ktemplates: int - :param ktemplates: The number of templates given, should be the actual\ - number of templates, with each template potentially containing\ + :param ktemplates: The number of templates given, should be the actual \ + number of templates, with each template potentially containing \ multiple traces. :type savedir: Str or bool - :param savedir: If false, data will be kept in memory, otherwise, data will\ - be stored on disk if memory is tight. + :param savedir: If false, data will be kept in memory, otherwise, data \ + will be stored on disk if memory is tight. :type cores: int :param cores: Number of cores to use. - :return: :class: 'numpy.ndarray' objects. These will contain the\ - correlation sums for each template for this day of data. + :return: :class: 'numpy.ndarray' objects. These will contain the \ + correlation sums for each template for this day of data. :return: list of ints as number of channels used for each cross-correlation .. rubric: Note - templates must be arranged into a numpy array of numpy arrays. The inner\ - numpy arrays must be shaped (len(template_trace),) - the outer np.ndarray\ - must be shaped (number of channels in stream*number of templates,), such\ - that if there are 5 traces in the stream (e.g. stream is shaped (5,n))\ - and there are 10 templates each of length 20, the templates ndarray is shaped\ - (50,20). + templates must be arranged into a numpy array of numpy arrays. The inner \ + numpy arrays must be shaped (len(template_trace),) - the outer np.ndarray \ + must be shaped (number of channels in stream*number of templates,), such \ + that if there are 5 traces in the stream (e.g. stream is shaped (5,n)) \ + and there are 10 templates each of length 20, the templates ndarray is \ + shaped (50,20). """ # cimport numpy as np - import os, multiprocessing as mp + import os + import multiprocessing as mp DTYPE = np.float32 # ctypedef np.float32_t DTYPE_t - num_core=cores + num_cores = cores if len(templates) < num_cores: num_cores = len(templates) @@ -63,73 +67,80 @@ def _channel_loop( templates, stream, delays, ktemplates, savedir=False, cores=1 # each of the ktemplates, with each cccsum as long as the # correlation overlap window. # cdef np.ndarray cccsums=np.array([np.array([0.0]*len(stream[0])-\ - cccsums=np.array([np.array([0.0]*(len(stream[0])-len(templates[0])+1),\ - dtype=DTYPE)]*ktemplates) + cccsums = np.array([np.array([0.0]*(len(stream[0])-len(templates[0])+1), + dtype=DTYPE)]*ktemplates) # Initialize an empty array for the return of the number of channels # cdef np.ndarray nchans=np.array([0]*ktemplates, dtype=int) - nchans=np.array([0]*ktemplates, dtype=int) + nchans = np.array([0] * ktemplates, dtype=int) # Loop through the templates, using some kind of clever indexing # Note, this is where we could parallelise this! - image_ind=0 - template_ind=0 - j_ind=np.concatenate([np.arange(0,len(stream))]*ktemplates) # Array of indexes for stream! - pool=mp.Pool(processes=num_cores, maxtasksperchild=None) + image_ind = 0 + template_ind = 0 + j_ind = np.concatenate([np.arange(0, len(stream))] * ktemplates) + # Array of indexes for stream! + pool = mp.Pool(processes=num_cores) if savedir: - results=[pool.apply_async(_template_loop, args=(templates[i], stream[j_ind[i]], \ - delays[i],\ - savedir+'/'+str(i), i)) - for i in range(len(templates))] + results = [pool.apply_async(_template_loop, args=(templates[i], + stream[j_ind[i]], + delays[i], + savedir+'/'+str(i), + i)) + for i in range(len(templates))] else: - results=[pool.apply_async(_template_loop, args=(templates[i], stream[j_ind[i]], \ - delays[i], False, i)) - for i in range(len(templates))] + results = [pool.apply_async(_template_loop, args=(templates[i], + stream[j_ind[i]], + delays[i], False, i)) + for i in range(len(templates))] pool.close() if not savedir: ccc_list = [p.get() for p in results] ccc_list.sort(key=lambda tup: tup[0]) - ccc_list=[ccc[1] for ccc in ccc_list] + ccc_list = [ccc[1] for ccc in ccc_list] else: # order_list = [p.get() for p in results] del order_list pool.join() - print "Finished parallel run" + print("Finished parallel run") for i in range(len(templates)): # if i in range(0,len(templates),len(templates)/100): - # print str(i/len(templates))+' % read back in' + # print(str(i/len(templates))+' % read back in') # Check if there was data for that station for both the - if not (np.all(np.isnan(stream[image_ind])) or\ - np.all(np.isnan(templates[i]))): - nchans[template_ind]+=1 + if not (np.all(np.isnan(stream[image_ind])) or + np.all(np.isnan(templates[i]))): + nchans[template_ind] += 1 if not savedir: - cccsums[template_ind]=np.sum([cccsums[template_ind],\ - ccc_list[i]], axis=0) + cccsums[template_ind] = np.sum([cccsums[template_ind], + ccc_list[i]], axis=0) else: - cccsums[template_ind]=np.sum([cccsums[template_ind],\ - np.load(savedir+'/'+str(i)+'.npy')],\ - axis=0) + cccsums[template_ind] = np.sum([cccsums[template_ind], + np.load(savedir+'/'+str(i) + + '.npy')], + axis=0) os.remove(savedir+'/'+str(i)+'.npy') - if image_ind < len(stream)-1: - image_ind+=1 + if image_ind < len(stream) - 1: + image_ind += 1 else: # Move on to the next template - image_ind=0 - template_ind+=1 + image_ind = 0 + template_ind += 1 # Reshape the array to give what we want for i in range(len(cccsums)): - cccsums[i]=cccsums[i].reshape(len(cccsums[i],)) + cccsums[i] = cccsums[i].reshape(len(cccsums[i],)) return cccsums, nchans + def _template_loop(template, stream, delay, savefile=False, i=0): """ Helper loop for parallelisation """ import cv2 - image=np.append(stream,np.array([0]*int(round(delay))))[int(round(delay)):] + image = np.append(stream, + np.array([0] * int(round(delay))))[int(round(delay)):] # Compute the cross correlation - ccc=cv2.matchTemplate(image.astype(np.float32), \ - template.astype(np.float32),\ - cv2.TM_CCOEFF_NORMED) - ccc=ccc.T.reshape(len(ccc),) + ccc = cv2.matchTemplate(image.astype(np.float32), + template.astype(np.float32), + cv2.TM_CCOEFF_NORMED) + ccc = ccc.T.reshape(len(ccc),) if savefile: np.save(savefile, ccc) del ccc diff --git a/eqcorrscan/core/resolution_test.py b/eqcorrscan/core/resolution_test.py deleted file mode 100644 index 7369da888..000000000 --- a/eqcorrscan/core/resolution_test.py +++ /dev/null @@ -1,151 +0,0 @@ -#!/usr/bin/python -""" -Part of the EQcorrscan package written by Calum Chamberlain of Victoria -University of Wellington. - -Functions designed to test the reosultion of templat matching cross-correlation -function inspired by Emily Warren-Smith. -""" - -import sys, os -bob=os.path.realpath(__file__) -bob=bob.split('/') -path='/' -for i in xrange(len(bob)-2): - path+=bob[i]+'/' -print path -sys.path.insert(0,path) -import numpy as np -from match_filter import normxcorr2 -from utils import findpeaks - -def ccc_gen(image, template): - """ - Function to test if a detection is possible at this moveout - - :type image: obspy.Stream - :type template: obspy.Stream - :type delays: list - :type threhsold: float - - :returns: ccc, a matrix of correlation values - """ - ccc=np.array([np.array([0]*(len(image[0].data)-len(template[0].data)+1))]*len(image)) - print 'Correlation matrix is shaped: '+str(ccc.shape) - for i in xrange(len(image)): - templatetr=template.select(station=image[i].stats.station,\ - channel=image[i].stats.channel) - ccc[i]=normxcorr2(templatetr[0].data, image[i].data)[0] - return ccc - -def _node_loop(freq, node, ccc, lags, threshold, thresh_type): - """ - """ - for j in xrange(len(ccc)): - pad=np.array([0]*int(round(lags[j,0]*freq))) - if not 'cccsum' in locals(): - cccsum=np.append(ccc[j],pad)[len(pad):] - else: - bob=np.append(ccc[j],pad)[len(pad):] - cccsum=np.sum([cccsum, bob], axis=0) - if thresh_type=='MAD': - rawthresh=threshold*np.median(np.abs(cccsum)) - elif thresh_type=='absolute': - rawthresh=threshold - elif thresh_type=='av_chan_corr': - rawthresh=threshold*(cccsum/len(image)) - else: - rawthresh=threshold*np.median(np.abs(cccsum)) - peaks = findpeaks.find_peaks2(cccsum, rawthresh,\ - freq, 0) - if not len(peaks) == 0: - return node - else: - return - -def moveout_check(template, nodes, lags, threshold, thresh_type, lowcut,\ - highcut, filt_order): - """ - Function to check different moveouts for detectability - """ - # Generate random noise and seed with the template - from copy import deepcopy - from obspy.signal.filter import bandpass - from joblib import Parallel, delayed - parallel=True - image=deepcopy(template) - for i in xrange(len(image)): - image[i].data=np.random.randn(len(image[i].data)+\ - (86400*image[i].stats.sampling_rate)\ - -len(image[i].data)) - image[i].data[(len(image[i].data)-len(template[i].data))/2:\ - (len(image[i].data)-len(template[i].data))/2+len(template[i].data)]=\ - image[i].data[(len(image[i].data)-len(template[i].data))/2:\ - (len(image[i].data)-len(template[i].data))/2+len(template[i].data)]+\ - template[i].data/np.mean(template[i].data**2)**0.5 - image[i].data=bandpass(image[i].data, lowcut, highcut,\ - image[i].stats.sampling_rate, filt_order) - ccc=ccc_gen(image, template) - # Lags - possible_locations=[] - freq=image[0].stats.sampling_rate - if not parallel: - for i in xrange(len(nodes)): - possible_locations+=_node_loop(freq, nodes[i], ccc, lags[:,[i]],\ - threshold, thresh_type) - else: - possible_locations = Parallel(n_jobs=10, verbose=5)(delayed(_node_loop)\ - (freq, nodes[i], ccc,\ - lags[:,[i]], threshold,\ - thresh_type)\ - for i in xrange(len(nodes))) - return possible_locations - -if __name__ == '__main__': - import sys - if len(sys.argv) == 1: - raise IOError("Needs one argument, the template to check") - templatename=str(sys.argv[1]) - from par import match_filter_par as defaults - from par import template_gen_par as tempdef - from par import bright_lights_par as brightdef - from obspy import read - from core.bright_lights import _read_tt, _resample_grid - template=read(templatename) - stations_in=[] - for tr in template: - stations_in+=[tr.stats.station] - stations_in=list(set(stations_in)) - stations, nodes, lags = _read_tt(brightdef.nllpath,stations_in,\ - brightdef.phase, phaseout='S', \ - ps_ratio=brightdef.ps_ratio) - stations, nodes, lags = _resample_grid(stations, nodes, lags, brightdef.volume,\ - brightdef.resolution) - # print np.shape(lags) - for station in stations: - if not 'template_dummy' in locals(): - template_dummy=template.select(station=station) - else: - template_dummy+=template.select(station=station) - template=template_dummy - for tr in template: - for i in xrange(len(stations)): - if tr.stats.station == stations[i]: - if not 'alllags' in locals(): - alllags=[lags[i]] - else: - # print stations[i] - alllags=np.concatenate((alllags, [lags[i]]), axis=0) - # print np.shape(alllags) - lags=alllags - print 'Lags is shaped: '+str(np.shape(lags)) - print 'I have '+str(len(template))+' channels of data' -# Indexing will be an issue, currently don't check that stations match between data and lags - possible_locations = moveout_check(template, nodes, lags, defaults.threshold,\ - defaults.threshtype, tempdef.lowcut,\ - tempdef.highcut, tempdef.filter_order) - from utils import EQcorrscan_plotting as plotting - if not len(possible_locations) == 0: - plotting.threeD_gridplot(possible_locations) - else: - raise ValueError("No possible location found") diff --git a/eqcorrscan/core/template_gen.py b/eqcorrscan/core/template_gen.py index 4ba4bc564..4e0d9f8c2 100644 --- a/eqcorrscan/core/template_gen.py +++ b/eqcorrscan/core/template_gen.py @@ -1,40 +1,12 @@ #!/usr/bin/python -r"""Function to generate template waveforms and information to go with them\ -for the application of cross-correlation of seismic data for the detection of\ - repeating events. - -Part of the EQcorrscan module to read nordic format s-files\ -EQcorrscan is a python module designed to run match filter routines for\ -seismology, within it are routines for integration to seisan and obspy.\ -With obspy integration (which is necessary) all main waveform formats can be\ -read in and output. - -This main section contains a script, LFE_search.py which demonstrates the\ -usage of the built in functions from template generation from picked waveforms\ -through detection by match filter of continuous data to the generation of lag\ -times to be used for relative locations. - -The match-filter routine described here was used a previous Matlab code for\ -the Chamberlain et al. 2014 G-cubed publication. The basis for the lag-time\ -generation section is outlined in Hardebeck & Shelly 2011, GRL. - -Code generated by Calum John Chamberlain of Victoria University of Wellington,\ -2015. - - -.. rubric:: Note -Pre-requisites: - - gcc - for the installation of the openCV correlation routine - - python-cv2 - Python bindings for the openCV routines - - python-joblib - used for parallel processing - - python-obspy - used for lots of common seismological processing - - requires: - - numpy - - scipy - - matplotlib - - NonLinLoc - used outside of all codes for travel-time generation - -Copyright 2015 Calum Chamberlain +r"""Functions to generate template waveforms and information to go with them \ +for the application of cross-correlation of seismic data for the detection of \ +repeating events. + +Code written by Calum John Chamberlain & Chet Hopp of \ +Victoria University of Wellington, 2015. + +Copyright 2015, 2016 Calum Chamberlain, Chet Hopp. This file is part of EQcorrscan. @@ -52,36 +24,44 @@ along with EQcorrscan. If not, see . """ +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals def from_sfile(sfile, lowcut, highcut, samp_rate, filt_order, length, swin, - debug=0): - r"""Function to read in picks from sfile then generate the template from the - picks within this and the wavefile found in the pick file. + debug=0, plot=False): + r"""Function to read in picks from sfile then generate the template from \ + the picks within this and the wavefile found in the pick file. :type sfile: string - :param sfile: sfilename must be the\ - path to a seisan nordic type s-file containing waveform and pick\ - information. + :param sfile: sfilename must be the \ + path to a seisan nordic type s-file containing waveform and pick \ + information. :type lowcut: float - :param lowcut: Low cut (Hz), if set to None will look in template\ + :param lowcut: Low cut (Hz), if set to None will look in template \ defaults file :type highcut: float - :param lowcut: High cut (Hz), if set to None will look in template\ + :param highcut: High cut (Hz), if set to None will look in template \ defaults file :type samp_rate: float - :param samp_rate: New sampling rate in Hz, if set to None will look in\ + :param samp_rate: New sampling rate in Hz, if set to None will look in \ template defaults file :type filt_order: int - :param filt_order: Filter level, if set to None will look in\ + :param filt_order: Filter level, if set to None will look in \ template defaults file :type swin: str :param swin: Either 'all', 'P' or 'S', to select which phases to output. :type length: float - :param length: Extract length in seconds, if None will look in template\ + :param length: Extract length in seconds, if None will look in template \ defaults file. :type debug: int :param debug: Debug level, higher number=more output. + :type plot: bool + :param plot: Turns template plotting on or off. + + :returns: obspy.Stream Newly cut template """ # Perform some checks first import os @@ -94,45 +74,56 @@ def from_sfile(sfile, lowcut, highcut, samp_rate, filt_order, length, swin, # Read in the header of the sfile wavefiles = Sfile_util.readwavename(sfile) pathparts = sfile.split('/')[0:-1] + new_path_parts = [] for part in pathparts: if part == 'REA': part = 'WAV' - wavpath = os.path.join(pathparts) + new_path_parts.append(part) + # * argument to allow .join() to accept a list + wavpath = os.path.join(*new_path_parts) + '/' + # In case of absolute paths (not handled with .split() --> .join()) + if sfile[0] == '/': + wavpath = '/' + wavpath # Read in waveform file for wavefile in wavefiles: - print ''.join(["I am going to read waveform data from: ", wavpath, - wavefile]) + print(''.join(["I am going to read waveform data from: ", wavpath, + wavefile])) if 'st' not in locals(): st = obsread(wavpath + wavefile) else: st += obsread(wavpath + wavefile) for tr in st: if tr.stats.sampling_rate < samp_rate: - print 'Sampling rate of data is lower than sampling rate asked for' - print 'Not good practice for correlations: I will not do this' + print('Sampling rate of data is lower than sampling rate asked ' + + 'for') + print('Not good practice for correlations: I will not do this') raise ValueError("Trace: " + tr.stats.station + " sampling rate: " + str(tr.stats.sampling_rate)) # Read in pick info - picks = Sfile_util.readpicks(sfile) - print "I have found the following picks" + catalog = Sfile_util.readpicks(sfile) + # Read the list of Picks for this event + picks = catalog[0].picks + print("I have found the following picks") for pick in picks: - print ' '.join([pick.station, pick.channel, pick.phase, - str(pick.time)]) + print(' '.join([pick.waveform_id.station_code, + pick.waveform_id.channel_code, pick.phase_hint, + str(pick.time)])) # Process waveform data + st.merge(fill_value='interpolate') st = pre_processing.shortproc(st, lowcut, highcut, filt_order, samp_rate, debug) - st1 = _template_gen(picks, st, length, swin) + st1 = _template_gen(picks, st, length, swin, plot=plot) return st1 def from_contbase(sfile, contbase_list, lowcut, highcut, samp_rate, filt_order, - length, prepick, swin, debug=0): - r"""Function to read in picks from sfile then generate the template from\ - the picks within this and the wavefiles from the continous database of\ - day-long files. Included is a section to sanity check that the files are\ - daylong and that they start at the start of the day. You should ensure\ - this is the case otherwise this may alter your data if your data are\ + length, prepick, swin, debug=0, plot=False): + r"""Function to read in picks from sfile then generate the template from \ + the picks within this and the wavefiles from the continous database of \ + day-long files. Included is a section to sanity check that the files are \ + daylong and that they start at the start of the day. You should ensure \ + this is the case otherwise this may alter your data if your data are \ daylong but the headers are incorrectly set. :type sfile: string @@ -141,25 +132,25 @@ def from_contbase(sfile, contbase_list, lowcut, highcut, samp_rate, filt_order, be numbers save for swin which must be either P, S or all \ (case-sensitive). :type contbase_list: List of tuple of string - :param contbase_list: List of tuples of the form\ - ['path', 'type', 'network']. Where path is the path to the continuous\ - database, type is the directory structure, which can be either\ - Yyyyy/Rjjj.01, which is the standard IRIS Year, julian day structure, or,\ - yyyymmdd which is a single directory for every day. + :param contbase_list: List of tuples of the form \ + ['path', 'type', 'network']. Where path is the path to the \ + continuous database, type is the directory structure, which can be \ + either Yyyyy/Rjjj.01, which is the standard IRIS Year, julian day \ + structure, or, yyyymmdd which is a single directory for every day. :type lowcut: float - :param lowcut: Low cut (Hz), if set to None will look in template\ + :param lowcut: Low cut (Hz), if set to None will look in template \ defaults file :type highcut: float - :param lowcut: High cut (Hz), if set to None will look in template\ + :param lowcut: High cut (Hz), if set to None will look in template \ defaults file :type samp_rate: float - :param samp_rate: New sampling rate in Hz, if set to None will look in\ + :param samp_rate: New sampling rate in Hz, if set to None will look in \ template defaults file :type filt_order: int - :param filt_order: Filter level, if set to None will look in\ + :param filt_order: Filter level, if set to None will look in \ template defaults file :type length: float - :param length: Extract length in seconds, if None will look in template\ + :param length: Extract length in seconds, if None will look in template \ defaults file. :type prepick: float :param prepick: Pre-pick time in seconds @@ -167,6 +158,10 @@ def from_contbase(sfile, contbase_list, lowcut, highcut, samp_rate, filt_order, :param swin: Either 'all', 'P' or 'S', to select which phases to output. :type debug: int :param debug: Level of debugging output, higher=more + :type plot: bool + :param plot: Turns template plotting on or off. + + :returns: obspy.Stream Newly cut template """ # Perform some checks first import os @@ -177,26 +172,28 @@ def from_contbase(sfile, contbase_list, lowcut, highcut, samp_rate, filt_order, from eqcorrscan.utils import pre_processing from eqcorrscan.utils import Sfile_util import glob - from obspy import UTCDateTime from obspy import read as obsread # Read in the header of the sfile - header = Sfile_util.readheader(sfile) - day = UTCDateTime('-'.join([str(header.time.year), - str(header.time.month).zfill(2), - str(header.time.day).zfill(2)])) + event = Sfile_util.readheader(sfile) + day = event.origins[0].time # Read in pick info - picks = Sfile_util.readpicks(sfile) - print "I have found the following picks" + catalog = Sfile_util.readpicks(sfile) + picks = catalog[0].picks + print("I have found the following picks") pick_chans = [] used_picks = [] for pick in picks: - if pick.station + pick.channel not in pick_chans\ - and pick.phase in ['P', 'S']: - pick_chans.append(pick.station + pick.channel) + station = pick.waveform_id.station_code + channel = pick.waveform_id.channel_code + phase = pick.phase_hint + pcktime = pick.time + if station + channel not in pick_chans and phase in ['P', 'S']: + pick_chans.append(station + channel) used_picks.append(pick) - print pick + print(pick) + # #########Left off here for contbase in contbase_list: if contbase[1] == 'yyyy/mm/dd': daydir = os.path.join([str(day.year), @@ -210,93 +207,387 @@ def from_contbase(sfile, contbase_list, lowcut, highcut, samp_rate, filt_order, daydir = day.datetime.strftime('%Y%m%d') if 'wavefiles' not in locals(): wavefiles = (glob.glob(os.path.join([contbase[0], daydir, - '*' + pick.station + + '*' + station + '.*']))) else: wavefiles += glob.glob(os.path.join([contbase[0], daydir, - '*' + pick.station + + '*' + station + '.*'])) - elif pick.phase in ['P', 'S']: - print ' '.join(['Duplicate pick', pick.station, pick.channel, - pick.phase, str(pick.time)]) - elif pick.phase == 'IAML': - print ' '.join(['Amplitude pick', pick.station, pick.channel, - pick.phase, str(pick.time)]) + elif phase in ['P', 'S']: + print(' '.join(['Duplicate pick', station, channel, + phase, str(pcktime)])) + elif phase == 'IAML': + print(' '.join(['Amplitude pick', station, channel, + phase, str(pcktime)])) picks = used_picks wavefiles = list(set(wavefiles)) # Read in waveform file wavefiles.sort() for wavefile in wavefiles: - print "I am going to read waveform data from: " + wavefile + print("I am going to read waveform data from: " + wavefile) if 'st' not in locals(): st = obsread(wavefile) else: st += obsread(wavefile) - # Porcess waveform data + # Process waveform data st.merge(fill_value='interpolate') for tr in st: tr = pre_processing.dayproc(tr, lowcut, highcut, filt_order, samp_rate, debug, day) # Cut and extract the templates - st1 = _template_gen(picks, st, length, swin, prepick=prepick) + st1 = _template_gen(picks, st, length, swin, prepick=prepick, plot=plot) return st1 -def _template_gen(picks, st, length, swin, prepick=0.05, plot=False): - r"""Function to generate a cut template in the obspy\ - Stream class from a given set of picks and data, also in an obspy stream\ - class. Should be given pre-processed data (downsampled and filtered) +def from_quakeml(quakeml, st, lowcut, highcut, samp_rate, filt_order, + length, prepick, swin, debug=0, plot=False): + r"""Function to generate a template from a local quakeml file \ + and an obspy.Stream object. - :type picks: :class: 'makesfile.pick' + :type quakeml: string + :param quakeml: QuakeML file containing pick information, can contain \ + multiple events. + :type st: class: obspy.Stream + :param st: Stream containing waveform data for template (hopefully) + :type lowcut: float + :param lowcut: Low cut (Hz), if set to None will look in template \ + defaults file + :type highcut: float + :param lowcut: High cut (Hz), if set to None will look in template \ + defaults file + :type samp_rate: float + :param samp_rate: New sampling rate in Hz, if set to None will look in \ + template defaults file + :type filt_order: int + :param filt_order: Filter level, if set to None will look in \ + template defaults file + :type length: float + :param length: Extract length in seconds, if None will look in template \ + defaults file. + :type prepick: float + :param prepick: Pre-pick time in seconds + :type swin: str + :param swin: Either 'all', 'P' or 'S', to select which phases to output. + :type debug: int + :param debug: Level of debugging output, higher=more + :type plot: bool + :param plot: Display template plots or not + + :returns: list of obspy.Stream Newly cut templates + """ + # Perform some checks first + import os + import warnings + if not os.path.isfile(quakeml): + raise IOError('QuakeML file does not exist') + import obspy + if int(obspy.__version__.split('.')[0]) >= 1: + from obspy import read_events + else: + from obspy import readEvents as read_events + from obspy import UTCDateTime + from eqcorrscan.utils import pre_processing + stations = [] + channels = [] + st_stachans = [] + # Process waveform data + st.merge(fill_value='interpolate') + for tr in st: + tr = pre_processing.dayproc(tr, lowcut, highcut, filt_order, + samp_rate, debug, + UTCDateTime(tr.stats.starttime.date)) + # Read QuakeML file into Catalog class + catalog = read_events(quakeml) + templates = [] + for event in catalog: + # Read in pick info + print("I have found the following picks") + for pick in event.picks: + print(' '.join([pick.waveform_id.station_code, + pick.waveform_id.channel_code, + pick.phase_hint, str(pick.time)])) + stations.append(pick.waveform_id.station_code) + channels.append(pick.waveform_id.channel_code) + # Check to see if all picks have a corresponding waveform + for tr in st: + st_stachans.append('.'.join([tr.stats.station, tr.stats.channel])) + for i in xrange(len(stations)): + if not '.'.join([stations[i], channels[i]]) in st_stachans: + warnings.warn('No data provided for ' + stations[i] + '.' + + channels[i]) + st1 = st.copy() + # Cut and extract the templates + template = _template_gen(event.picks, st1, length, swin, + prepick=prepick, plot=plot) + templates.append(template) + return templates + + +def from_seishub(catalog, url, lowcut, highcut, samp_rate, filt_order, + length, prepick, swin, debug=0, plot=False): + r"""Function to generate templates from a SeisHub database.Must be given \ + an obspy.Catalog class and the SeisHub url as input. The function returns \ + a list of obspy.Stream classes containting steams for each desired \ + template. + + :type catalog: obspy.Catalog + :param catalog: Catalog class containing desired template events + :type url: string + :param url: url of SeisHub database instance + :type lowcut: float + :param lowcut: Low cut (Hz), if set to None will look in template \ + defaults file + :type highcut: float + :param lowcut: High cut (Hz), if set to None will look in template \ + defaults file + :type samp_rate: float + :param samp_rate: New sampling rate in Hz, if set to None will look in \ + template defaults file + :type filt_order: int + :param filt_order: Filter level, if set to None will look in \ + template defaults file + :type length: float + :param length: Extract length in seconds, if None will look in template \ + defaults file. + :type prepick: float + :param prepick: Pre-pick time in seconds + :type swin: str + :param swin: Either 'all', 'P' or 'S', to select which phases to output. + :type debug: int + :param debug: Level of debugging output, higher=more + :type plot: bool + :param plot: Plot templates or not. + + :returns: obspy.Stream Newly cut template + """ + # This import section copes with namespace changes between obspy versions + import obspy + if int(obspy.__version__.split('.')[0]) >= 1: + from obspy.clients.seishub import Client + else: + from obspy.seishub import Client + from eqcorrscan.utils import pre_processing + client = Client(url) + temp_list = [] + for event in catalog: + # Figure out which picks we have + day = event.origins[0].time + picks = event.picks + print("Fetching the following traces from SeisHub") + for pick in picks: + net = pick.waveform_id.network_code + sta = pick.waveform_id.station_code + chan = pick.waveform_id.channel_code + loc = pick.waveform_id.location_code + starttime = pick.time - (prepick + 600) + # Enforce some pad, 10min either side, to reduce filter effects + endtime = pick.time + length + 600 - prepick + if debug > 0: + print('start-time: ' + str(starttime)) + print('end-time: ' + str(endtime)) + print('pick-time: ' + str(pick.time)) + print('.'.join([net, sta, loc, chan])) + if sta in client.waveform.getStationIds(network=net): + if 'st' not in locals(): + st = client.waveform.getWaveform(net, sta, loc, chan, + starttime, endtime) + else: + st += client.waveform.getWaveform(net, sta, loc, chan, + starttime, endtime) + else: + print('Station not found in SeisHub DB') + if debug > 0: + st.plot() + print('Preprocessing data for event: '+str(event.resource_id)) + st.merge(fill_value='interpolate') + st1 = pre_processing.shortproc(st, lowcut, highcut, filt_order, + samp_rate, debug) + template = _template_gen(event.picks, st1, length, swin, prepick, + plot=plot) + del st, st1 + temp_list.append(template) + return temp_list + + +def from_client(catalog, client_id, lowcut, highcut, samp_rate, filt_order, + length, prepick, swin, debug=0, plot=False): + r"""Function to generate templates from a SeisHub database.Must be given \ + an obspy.Catalog class and the SeisHub url as input. The function returns \ + a list of obspy.Stream classes containting steams for each desired \ + template. + + :type catalog: obspy.Catalog + :param catalog: Catalog class containing desired template events + :type url: string + :param url: url of SeisHub database instance + :type lowcut: float + :param lowcut: Low cut (Hz), if set to None will look in template\ + defaults file + :type highcut: float + :param lowcut: High cut (Hz), if set to None will look in template\ + defaults file + :type samp_rate: float + :param samp_rate: New sampling rate in Hz, if set to None will look in\ + template defaults file + :type filt_order: int + :param filt_order: Filter level, if set to None will look in\ + template defaults file + :type length: float + :param length: Extract length in seconds, if None will look in template\ + defaults file. + :type prepick: float + :param prepick: Pre-pick time in seconds + :type swin: str + :param swin: Either 'all', 'P' or 'S', to select which phases to output. + :type debug: int + :param debug: Level of debugging output, higher=more + :type plot: bool + :param plot: Plot templates or not. + + :returns: obspy.Stream Newly cut template + """ + # This import section copes with namespace changes between obspy versions + import obspy + if int(obspy.__version__.split('.')[0]) >= 1: + from obspy.clients.fdsn import Client + from obspy.clients.fdsn.header import FDSNException + else: + from obspy.fdsn import Client + from obspy.fdsn.header import FDSNException + from eqcorrscan.utils import pre_processing + import warnings + + client = Client(client_id) + temp_list = [] + for event in catalog: + # Figure out which picks we have + day = event.origins[0].time + print("Fetching the following traces from " + client_id) + for pick in event.picks: + net = pick.waveform_id.network_code + sta = pick.waveform_id.station_code + chan = pick.waveform_id.channel_code + loc = pick.waveform_id.location_code + starttime = pick.time - (prepick + 600) + # Enforce some pad, 10min either side, to reduce filter effects + endtime = pick.time + length + 600 - prepick + if debug > 0: + print('start-time: ' + str(starttime)) + print('end-time: ' + str(endtime)) + print('pick-time: ' + str(pick.time)) + print('.'.join([net, sta, loc, chan])) + if 'st' not in locals(): + try: + st = client.get_waveforms(net, sta, loc, chan, + starttime, endtime) + except FDSNException: + warnings.warn('Found no data for this station') + else: + try: + st += client.get_waveforms(net, sta, loc, chan, + starttime, endtime) + except FDSNException: + warnings.warn('Found no data for this station') + if debug > 0: + st.plot() + print('Pre-processing data for event: '+str(event.resource_id)) + st.merge(fill_value='interpolate') + st1 = pre_processing.shortproc(st, lowcut, highcut, filt_order, + samp_rate, debug) + if debug > 0: + st1.plot() + template = _template_gen(event.picks, st1, length, swin, prepick, + plot) + del st, st1 + temp_list.append(template) + return temp_list + + +def _template_gen(picks, st, length, swin='all', prepick=0.05, plot=False): + r"""Function to generate a cut template in the obspy \ + Stream class from a given set of picks and data, also in an obspy stream \ + class. Should be given pre-processed data (downsampled and filtered). + + :type picks: List of obspy.core.event.Pick :param picks: Picks to extract data around :type st: :class: 'obspy.Stream' :param st: Stream to etract templates from :type length: float :param length: Length of template in seconds :type swin: string - :param swin: P, S or all + :param swin: P, S or all, defaults to all :type prepick: float - :param prepick: Length in seconds to extract before the pick time\ + :param prepick: Length in seconds to extract before the pick time \ default is 0.05 seconds :type plot: bool :param plot: To plot the template or not, default is True + + :returns: obspy.Stream Newly cut template. + + .. note:: By convention templates are generated with P-phases on the \ + vertical channel and S-phases on the horizontal channels, normal \ + seismograph naming conventions are assumed, where Z denotes vertical \ + and N, E, R, T, 1 and 2 denote horizontal channels, either oriented \ + or not. To this end we will **only** use Z channels if they have a \ + P-pick, and will use one or other horizontal channels **only** if \ + there is an S-pick on it. + + .. warning:: If there is no phase_hint included in picks, and swin=all, \ + all channels with picks will be used. """ import copy - from eqcorrscan.utils.EQcorrscan_plotting import pretty_template_plot as tplot + from eqcorrscan.utils.EQcorrscan_plotting import pretty_template_plot as\ + tplot from obspy import Stream import warnings stations = [] channels = [] st_stachans = [] for pick in picks: - stations.append(pick.station) - channels.append(pick.channel) + # Check to see that we are only taking the appropriate picks + if swin == 'all': + # Annoying compatability with seisan two channel codes + stations.append(pick.waveform_id.station_code) + channels.append(pick.waveform_id.channel_code[0] + + pick.waveform_id.channel_code[-1]) + elif swin == 'P' and 'P' in pick.phase_hint.upper(): + # Use the 'in' statement to cope with phase names like 'PN' etc. + stations.append(pick.waveform_id.station_code) + channels.append(pick.waveform_id.channel_code[0] + + pick.waveform_id.channel_code[-1]) + elif swin == 'S' and 'S' in pick.phase_hint.upper(): + stations.append(pick.waveform_id.station_code) + channels.append(pick.waveform_id.channel_code[0] + + pick.waveform_id.channel_code[-1]) + else: + raise IOError('Phase type is not in [all, P, S]') for tr in st: st_stachans.append('.'.join([tr.stats.station, tr.stats.channel])) - for i in xrange(len(stations)): - if not '.'.join([stations[i], channels[i]]) in st_stachans: - warnings.warn('No data provided for ' + stations[i] + '.' + + for i, station in enumerate(stations): + if not '.'.join([station, channels[i]]) in st_stachans: + warnings.warn('No data provided for ' + station + '.' + channels[i]) # Select which channels we actually have picks for for tr in st: if tr.stats.station in stations: - if swin == 'all': - if len(tr.stats.channel) == 3: - temp_channel = tr.stats.channel[0] + tr.stats.channel[2] - elif len(tr.stats.channel) == 2: - temp_channel = tr.stats.channel - # if temp_channel in channels: - tr.stats.channel = temp_channel - if 'st1' not in locals(): - st1 = Stream(tr) - else: - st1 += tr + # This is used to cope with seisan handling channel codes as + # two charectar codes, internally we will do the same. + if len(tr.stats.channel) == 3: + temp_channel = tr.stats.channel[0] + tr.stats.channel[2] + elif len(tr.stats.channel) == 2: + temp_channel = tr.stats.channel + # Take all channels + tr.stats.channel = temp_channel + if 'st1' not in locals(): + st1 = Stream(tr) else: - if 'st1' not in locals(): - st1 = Stream(tr) - else: - st1 += tr + st1 += tr + if 'st1' not in locals(): + msg = ('No data available for these picks or no picks match ' + + 'these data! Will not error, but you should check yo self') + warnings.warn(msg) + return st = copy.deepcopy(st1) del st1 if plot: @@ -307,30 +598,48 @@ def _template_gen(picks, st, length, swin, prepick=0.05, plot=False): del starttime if swin == 'all': for pick in picks: - if pick.station == tr.stats.station and \ - pick.channel == tr.stats.channel and\ - pick.phase == 'P': - starttime = pick.time - prepick - elif pick.station == tr.stats.station and\ - tr.stats.channel[-1] in ['1', '2', 'N', 'E'] and\ - pick.phase == 'S': - starttime = pick.time - prepick + if not pick.phase_hint: + msg = 'Pick for ' + pick.waveform_id.station_code + '.' +\ + pick.waveform_id.channel_code + ' has no phase ' +\ + 'hint given, you should not use this template for ' +\ + 'cross-correlation re-picking!' + warnings.warn(msg) + if pick.waveform_id.station_code == tr.stats.station and \ + pick.waveform_id.channel_code[0] + \ + pick.waveform_id.channel_code[-1] == \ + tr.stats.channel: + starttime = pick.time - prepick + else: + # If there is phase information then we should use our + # convention. + if pick.waveform_id.station_code == tr.stats.station and \ + pick.waveform_id.channel_code[0] + \ + pick.waveform_id.channel_code[-1] ==\ + tr.stats.channel \ + and 'P' in pick.phase_hint.upper(): + starttime = pick.time - prepick + elif pick.waveform_id.station_code == tr.stats.station and\ + tr.stats.channel[-1] in ['1', '2', 'N', + 'E', 'R', 'T'] and\ + 'S' in pick.phase_hint.upper(): + starttime = pick.time - prepick else: for pick in picks: - if pick.station == tr.stats.station and pick.phase == swin: + if pick.waveform_id.station_code == tr.stats.station and\ + swin in pick.phase_hint.upper(): starttime = pick.time - prepick if 'starttime' in locals(): - print "Cutting " + tr.stats.station + '.' + tr.stats.channel + print("Cutting " + tr.stats.station + '.' + tr.stats.channel) tr.trim(starttime=starttime, endtime=starttime + length, nearest_sample=False) - print tr.stats.starttime - print tr.stats.endtime + print(tr.stats.starttime) + print(tr.stats.endtime) if 'st1' not in locals(): st1 = Stream(tr) else: st1 += tr else: - print 'No pick for ' + tr.stats.station + '.' + tr.stats.channel + print('No pick for ' + tr.stats.station + '.' + tr.stats.channel) # Ensure that the template is the correct length if len(tr.data) == (tr.stats.sampling_rate * length) + 1: tr.data = tr.data[0:-1] @@ -350,47 +659,47 @@ def extract_from_stack(stack, template, length, pre_pick, pre_pad, Z_include=False, pre_processed=True, samp_rate=False, lowcut=False, highcut=False, filt_order=False): r"""Function to extract a new template from a stack of previous detections. - Requires the stack, the template used to make the detections for the stack, - and we need to know if the stack has been pre-processed. + Requires the stack, the template used to make the detections for the \ + stack, and we need to know if the stack has been pre-processed. :type stack: :class:obspy.Stream - :param stack: Waveform stack from detections. Can be of any length and\ - can have delays already included, or not. + :param stack: Waveform stack from detections. Can be of any length and \ + can have delays already included, or not. :type template: :class:obspy.Stream - :param template: Template used to make the detections in the stack. Will\ - use the delays of this for the new template. + :param template: Template used to make the detections in the stack. Will \ + use the delays of this for the new template. :type length: float :param length: Length of new template in seconds :type pre_pick: float :param pre_pick: Extract additional data before the detection, seconds :type pre_pad: float - :param pre_pad: Pad used in seconds when extracting the data, e.g. the\ - time before the detection extracted. If using\ - clustering.extract_detections this half the length of the extracted\ - waveform. + :param pre_pad: Pad used in seconds when extracting the data, e.g. the \ + time before the detection extracted. If using \ + clustering.extract_detections this half the length of the extracted \ + waveform. :type Z_include: bool - :param Z_include: If True will include any Z-channels even if there is\ - no template for this channel, as long as there is a template for this\ - station at a different channel. If this is False and Z channels are\ - included in the template Z channels will be included in the new_template\ - anyway. + :param Z_include: If True will include any Z-channels even if there is \ + no template for this channel, as long as there is a template for this \ + station at a different channel. If this is False and Z channels are \ + included in the template Z channels will be included in the \ + new_template anyway. :type pre_processed: bool - :param pre_processed: Have the data been pre-processed, if True (default)\ + :param pre_processed: Have the data been pre-processed, if True (default) \ then we will only cut the data here. :type samp_rate: float - :param samp_rate: If pre_processed=False then this is required, desired\ + :param samp_rate: If pre_processed=False then this is required, desired \ sampling rate in Hz, defaults to False. :type lowcut: float - :param lowcut: If pre_processed=False then this is required, lowcut in Hz,\ - defaults to False + :param lowcut: If pre_processed=False then this is required, lowcut in \ + Hz, defaults to False. :type highcut: float - :param highcut: If pre_processed=False then this is required, highcut in\ - Hz, defaults to False + :param highcut: If pre_processed=False then this is required, highcut in \ + Hz, defaults to False :type filt_order: int - :param filt_order: If pre_processed=False then this is required, filter\ + :param filt_order: If pre_processed=False then this is required, filter \ order, defaults to False - :returns: :class:obspy.Stream Newly cut template + :returns: obspy.Stream Newly cut template """ from eqcorrscan.utils import pre_processing import warnings @@ -402,7 +711,7 @@ def extract_from_stack(stack, template, length, pre_pick, pre_pad, # Generate a list of tuples of (station, channel, delay) with delay in # seconds delays = [(tr.stats.station, tr.stats.channel[-1], - mintime - tr.stats.starttime) for tr in template] + tr.stats.starttime - mintime) for tr in template] # Loop through the stack and trim! for tr in new_template: # Process the data if necessary @@ -432,10 +741,6 @@ def extract_from_stack(stack, template, length, pre_pick, pre_pad, return new_template -if __name__ == '__main__': - import sys - if len(sys.argv) != 2: - raise IOError('Requires 1 arguments: sfilename') - sfile = str(sys.argv[1]) - template = from_sfile(sfile) - template.write(sfile + '_template.ms', format="MSEED") +if __name__ == "__main__": + import doctest + doctest.testmod() diff --git a/eqcorrscan/core/thresholding.py b/eqcorrscan/core/thresholding.py index 30559ded8..6ae2e0590 100755 --- a/eqcorrscan/core/thresholding.py +++ b/eqcorrscan/core/thresholding.py @@ -1,38 +1,23 @@ -#!/usr/bin/python -""" -Part of the EQcorrscan package written by Calum Chamberlain of Victoria -University of Wellington in 2015. The main goal of this package is to generate -waveform templates from continuous data and search for repeats of this waveform -in continuous data using a match-filter cross-correlation routine. - -The prupose of this module specifically is to determine thresholds for the use -in various detection routines within the core modules. - -Copyright 2015 Calum Chamberlain - -This file is part of EQcorrscan. - - EQcorrscan is free software: you can redistribute it and/or modify - it under the terms of the GNU General Public License as published by - the Free Software Foundation, either version 3 of the License, or - (at your option) any later version. +r""" +The prupose of this module specifically is to determine thresholds for the \ +use in various detection routines within the core modules. - EQcorrscan is distributed in the hope that it will be useful, - but WITHOUT ANY WARRANTY; without even the implied warranty of - MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - GNU General Public License for more details. - - You should have received a copy of the GNU General Public License - along with EQcorrscan. If not, see . +** Unfinished ** """ +from __future__ import absolute_import +from __future__ import division +from __future__ import print_function +from __future__ import unicode_literals import numpy as np -from obspy import read + def coherence_test(stream, stations, nodes, lags, wlen): """ - Function to determine how the coherence of a day of contious multi-channel, - multi-station seismic data varies with varying grid nodes. It would be + Function to determine how the coherence of a day of contious \ + multi-channel, multi-station seismic data varies with varying grid nodes. + + It would be wise to apply this to a day of seismic data thought to be quiet from within the grid given, and to a day thought to be loud with energy sources within the grid to see how this function varies. @@ -44,34 +29,34 @@ def coherence_test(stream, stations, nodes, lags, wlen): :type nodes: list of tuple :param nodes: A list of the node points as (lat, lon, depth) :type lags: np.array - :param lags: Array of arrays where lags[i][:] refers to station[i] for all - nodes and lags[i][j] refers to the lag for station[i] at node[j] - should be in seconds. + :param lags: Array of arrays where lags[i][:] refers to station[i] for \ + all nodes and lags[i][j] refers to the lag for station[i] at node[j] \ + should be in seconds. :type wlen: int - :param wlen: Length of window to determine coherence for - should be the - same as your determined template length - in seconds + :param wlen: Length of window to determine coherence for - should be the \ + same as your determined template length - in seconds. - :return: coherence, array of arrays where coherence[i][:] refers to the - running daily coherence at node[i] + :return: coherence, array of arrays where coherence[i][:] refers to the \ + running daily coherence at node[i]. """ from core.bright_lights import coherence from copy import deepcopy import sys # Convert wlen to samples wlen = int(wlen * stream[0].stats.sampling_rate) - #Set up coherence array - dailycoherence=np.array([np.array([np.nan]*(len(stream[0].data)-wlen))]\ - *len(nodes), dtype=np.float32) # Use half precision as we don't need - # true doubles + # Set up coherence array + dailycoherence = np.array([np.array([np.nan]*(len(stream[0].data)-wlen))] + * len(nodes), dtype=np.float32) + # Use half precision as we don't need true doubles # Keep a sacred copy of the stream, as we will be applying the lags # directly to a copy of this - copyof_stream=deepcopy(stream) + copyof_stream = deepcopy(stream) for i in xrange(len(nodes)): sys.stdout.write("\r"+str(float(i)/len(nodes)*100)+"% \r") sys.stdout.flush() for j in xrange(len(stations)): - #Apply delays - st = copyof_stream.select(station = stations[j]) + # Apply delays + st = copyof_stream.select(station=stations[j]) for tr in st: tr.data = tr.data[int(lags[j][i]*tr.stats.sampling_rate):] pad = np.zeros(int(lags[j][i]*tr.stats.sampling_rate)) @@ -81,52 +66,56 @@ def coherence_test(stream, stations, nodes, lags, wlen): window = deepcopy(copyof_stream) for tr in window: tr.data = tr.data[j:j+wlen] - sys.stdout.write("\r"+str((float(i+1)*float(j+1))/(len(nodes)*len(dailycoherence[i]))*100)+"% \r") + sys.stdout.write("\r"+str((float(i + 1) * float(j + 1)) / + (len(nodes) * len(dailycoherence[i])) * + 100)+"% \r") sys.stdout.flush() dailycoherence[i][j] = coherence(window) return dailycoherence -if __name__ == '__main__': - import sys - if len(sys.argv) == 1: - raise IOError("Needs arguments, e.g. --coherence 2009/07/14") - else: - if sys.argv[1] == "--coherence": - date = sys.argv[2] - import glob - from obspy import read - stream = read('test_data/*'+date.split('/')[0]+'-'+\ - date.split('/')[1]+'-'+date.split('/')[2]+'-processed.ms') - if not len(stream) == 0: - import os - path = os.path.dirname(os.path.abspath(__file__)) - sys.path.insert(0,path[0:len(path)-5]) - from par import template_gen_par as templatedef - from par import bright_lights_par as brightdef - from core import bright_lights - from utils import EQcorrscan_plotting as plotting - # Use the brightness function to search for possible templates - # First read in the travel times - print 'Reading in the original grids' - stations, allnodes, alllags = \ - bright_lights._read_tt(brightdef.nllpath,brightdef.stations,\ - brightdef.phase) - # Resample the grid to allow us to run it quickly! - print 'Cutting the grid' - stations, nodes, lags = bright_lights._resample_grid(stations, allnodes, - alllags, - brightdef.volume, - brightdef.resolution) - # Remove lags that have a similar network moveout, e.g. the sum of the - # differences in moveouts is small. - print "Removing simlar lags" - stations, nodes, lags = bright_lights._rm_similarlags(stations, nodes, lags, - brightdef.nodesimthresh) - print "Plotting new grid" - plotting.threeD_gridplot(nodes) - dailycoherence = coherence_test(stream, stations, nodes, lags, \ - templatedef.length) - else: - raise IOError("No traces read in for this day, are they processed?") - else: - raise IOError("I only know --coherence at the moment") +# if __name__ == '__main__': +# import sys +# if len(sys.argv) == 1: +# raise IOError("Needs arguments, e.g. --coherence 2009/07/14") +# else: +# if sys.argv[1] == "--coherence": +# date = sys.argv[2] +# stream = read('test_data/*'+date.split('/')[0]+'-' + +# date.split('/')[1]+'-'+date.split('/')[2] + +# '-processed.ms') +# if not len(stream) == 0: +# import os +# path = os.path.dirname(os.path.abspath(__file__)) +# sys.path.insert(0, path[0:len(path)-5]) +# from core import bright_lights +# from utils import EQcorrscan_plotting as plotting +# # Use the brightness function to search for possible +# # templates +# # First read in the travel times +# print('Reading in the original grids') +# stations, allnodes, alllags = \ +# bright_lights._read_tt(nllpath, stations, +# brightdef.phase) +# # Resample the grid to allow us to run it quickly! +# print('Cutting the grid') +# stations, nodes, lags = \ +# bright_lights._resample_grid(stations, allnodes, +# alllags, +# brightdef.volume, +# brightdef.resolution) +# # Remove lags that have a similar network moveout, e.g. +# # the sum of the differences in moveouts is small. +# print("Removing simlar lags") +# stations, nodes, lags = \ +# bright_lights._rm_similarlags(stations, nodes, lags, +# brightdef.nodesimthresh) +# print("Plotting new grid") +# plotting.threeD_gridplot(nodes) +# dailycoherence = coherence_test(stream, stations, nodes, +# lags, +# templatedef.length) +# else: +# raise IOError("No traces read in for this day, are they " + +# "processed?") +# else: +# raise IOError("I only know --coherence at the moment") diff --git a/eqcorrscan/doc/EQcorrscan_logo.png b/eqcorrscan/doc/EQcorrscan_logo.png index 37f2903e4..98bb717c3 100644 Binary files a/eqcorrscan/doc/EQcorrscan_logo.png and b/eqcorrscan/doc/EQcorrscan_logo.png differ diff --git a/eqcorrscan/doc/EQcorrscan_logo.svg b/eqcorrscan/doc/EQcorrscan_logo.svg index 5857d317e..d7394f29b 100644 --- a/eqcorrscan/doc/EQcorrscan_logo.svg +++ b/eqcorrscan/doc/EQcorrscan_logo.svg @@ -14,7 +14,7 @@ id="svg2" version="1.1" inkscape:version="0.48.3.1 r9886" - sodipodi:docname="EQcorrscan_logo.png"> + sodipodi:docname="EQcorrscan_logo.svg"> image/svg+xml - + @@ -58,7 +58,7 @@ transform="translate(-20.714286,-70.615406)"> diff --git a/eqcorrscan/doc/_build/doctrees/core.doctree b/eqcorrscan/doc/_build/doctrees/core.doctree index 2dd930ee6..9091bae25 100644 Binary files a/eqcorrscan/doc/_build/doctrees/core.doctree and b/eqcorrscan/doc/_build/doctrees/core.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/environment.pickle b/eqcorrscan/doc/_build/doctrees/environment.pickle index 0735a1dfb..6c8ae4f9d 100644 Binary files a/eqcorrscan/doc/_build/doctrees/environment.pickle and b/eqcorrscan/doc/_build/doctrees/environment.pickle differ diff --git a/eqcorrscan/doc/_build/doctrees/index.doctree b/eqcorrscan/doc/_build/doctrees/index.doctree index dd46aa633..37e1085f7 100644 Binary files a/eqcorrscan/doc/_build/doctrees/index.doctree and b/eqcorrscan/doc/_build/doctrees/index.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/intro.doctree b/eqcorrscan/doc/_build/doctrees/intro.doctree index 64ec44377..9b99ffdd7 100644 Binary files a/eqcorrscan/doc/_build/doctrees/intro.doctree and b/eqcorrscan/doc/_build/doctrees/intro.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/core.bright_lights.doctree b/eqcorrscan/doc/_build/doctrees/submodules/core.bright_lights.doctree index 26e52aeec..7c782fa0d 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/core.bright_lights.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/core.bright_lights.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/core.lag_calc.doctree b/eqcorrscan/doc/_build/doctrees/submodules/core.lag_calc.doctree index 80312dda5..24dd944eb 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/core.lag_calc.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/core.lag_calc.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/core.match_filter.doctree b/eqcorrscan/doc/_build/doctrees/submodules/core.match_filter.doctree index aa3b0b7c8..67e1a5595 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/core.match_filter.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/core.match_filter.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/core.template_gen.doctree b/eqcorrscan/doc/_build/doctrees/submodules/core.template_gen.doctree index 941c6e500..5b49aa846 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/core.template_gen.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/core.template_gen.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.EQcorrscan_plotting.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.EQcorrscan_plotting.doctree index 57f8f2bd8..80f842f00 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.EQcorrscan_plotting.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.EQcorrscan_plotting.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.Sfile_util.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.Sfile_util.doctree index 0788365aa..b5abec22a 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.Sfile_util.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.Sfile_util.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.catalogue2DD.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.catalogue2DD.doctree index 83890099b..04f5a17d5 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.catalogue2DD.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.catalogue2DD.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.clustering.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.clustering.doctree index 1888732cf..bc3864425 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.clustering.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.clustering.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.findpeaks.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.findpeaks.doctree index 39ccb7895..351a3080a 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.findpeaks.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.findpeaks.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.locate.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.locate.doctree new file mode 100644 index 000000000..415a74bb5 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/submodules/utils.locate.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.mag_calc.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.mag_calc.doctree index cb081d1df..e771dbb83 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.mag_calc.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.mag_calc.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.pre_processing.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.pre_processing.doctree index 889da939b..ea0087291 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.pre_processing.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.pre_processing.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.seismo_logs.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.seismo_logs.doctree new file mode 100644 index 000000000..0d0a7bef2 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/submodules/utils.seismo_logs.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.stacking.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.stacking.doctree index 500a76476..02b21b0dd 100644 Binary files a/eqcorrscan/doc/_build/doctrees/submodules/utils.stacking.doctree and b/eqcorrscan/doc/_build/doctrees/submodules/utils.stacking.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/submodules/utils.synth_seis.doctree b/eqcorrscan/doc/_build/doctrees/submodules/utils.synth_seis.doctree new file mode 100644 index 000000000..d480d8398 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/submodules/utils.synth_seis.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorial.doctree b/eqcorrscan/doc/_build/doctrees/tutorial.doctree index 0ad295274..8cc83e22a 100644 Binary files a/eqcorrscan/doc/_build/doctrees/tutorial.doctree and b/eqcorrscan/doc/_build/doctrees/tutorial.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorials/clustering.doctree b/eqcorrscan/doc/_build/doctrees/tutorials/clustering.doctree new file mode 100644 index 000000000..13895543a Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/tutorials/clustering.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorials/lag-calc.doctree b/eqcorrscan/doc/_build/doctrees/tutorials/lag-calc.doctree new file mode 100644 index 000000000..0cfb9d7eb Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/tutorials/lag-calc.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorials/mag-calc.doctree b/eqcorrscan/doc/_build/doctrees/tutorials/mag-calc.doctree new file mode 100644 index 000000000..1803d4652 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/tutorials/mag-calc.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorials/matched-filter.doctree b/eqcorrscan/doc/_build/doctrees/tutorials/matched-filter.doctree new file mode 100644 index 000000000..f85d8f7b8 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/tutorials/matched-filter.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/tutorials/template-creation.doctree b/eqcorrscan/doc/_build/doctrees/tutorials/template-creation.doctree new file mode 100644 index 000000000..b627d5934 Binary files /dev/null and b/eqcorrscan/doc/_build/doctrees/tutorials/template-creation.doctree differ diff --git a/eqcorrscan/doc/_build/doctrees/utils.doctree b/eqcorrscan/doc/_build/doctrees/utils.doctree index 7bf2c2d35..898ca372f 100644 Binary files a/eqcorrscan/doc/_build/doctrees/utils.doctree and b/eqcorrscan/doc/_build/doctrees/utils.doctree differ diff --git a/eqcorrscan/doc/_build/html/.buildinfo b/eqcorrscan/doc/_build/html/.buildinfo index f3684e47e..396a78b21 100644 --- a/eqcorrscan/doc/_build/html/.buildinfo +++ b/eqcorrscan/doc/_build/html/.buildinfo @@ -1,4 +1,4 @@ # Sphinx build info version 1 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done. -config: 1c6d36bf1d570ea38e9663891769649a +config: 66c486cd85f32812f89c64141c06ce4c tags: 645f666f9bcd5a90fca523b33c5a78b7 diff --git a/eqcorrscan/doc/_build/html/_images/EQcorrscan_logo.png b/eqcorrscan/doc/_build/html/_images/EQcorrscan_logo.png index 37f2903e4..98bb717c3 100644 Binary files a/eqcorrscan/doc/_build/html/_images/EQcorrscan_logo.png and b/eqcorrscan/doc/_build/html/_images/EQcorrscan_logo.png differ diff --git a/eqcorrscan/doc/_build/html/_modules/EQcorrscan_plotting.html b/eqcorrscan/doc/_build/html/_modules/EQcorrscan_plotting.html index dd9901452..d005d9076 100644 --- a/eqcorrscan/doc/_build/html/_modules/EQcorrscan_plotting.html +++ b/eqcorrscan/doc/_build/html/_modules/EQcorrscan_plotting.html @@ -7,7 +7,7 @@ - EQcorrscan_plotting — EQcorrscan 0.0.8 documentation + EQcorrscan_plotting — EQcorrscan 0.1.0 documentation @@ -32,7 +32,7 @@ - + @@ -65,33 +65,38 @@