Changes

LIST OF CHANGES

release 51.1.1 (2025-01-16)
 - Fixed perlbrew installation by installing libdevel-patchperl-perl in
   GitHub runner.

release 51.1.0 (2024-10-25)
 - Added --drop_irods_locations argument for run deletion script

release 51.0.1 (2024-10-04)
 - Added .github/dependabot.yml file to auto-update GitHub actions
 - GitHub CI - updated deprecated v2 runner action to v3
 - Following a release on 07/09/2024, see https://metacpan.org/dist/App-perlbrew/changes,
   the checksum of the script served by https://install.perlbrew.pl had changed.
   https://install.perlbrew.pl is a redirect to raw
   https://github.com/gugod/App-perlbrew/blob/master/perlbrew-install, so
   the change originates from GitHub and can be trusted. Our CI flow compares
   the checksum of the downloaded script to the expected value. We now store
   an updated expected checksum value, which corresponds to the latest release.

release 51.0.0 (2024-07-25)
 - PacBio loader - include loading from 3 fields with changed names in
   SMRT Link output files
 - Remove a plugin for npg_tracking::daemon, which was used to demonise
   the npg_mlwarehouse_fkrepair script.

release 50.5.0 (2024-05-28)
 - Removing Tidyp dependency from CI
 - Minor change npg_warehouse::loader::pacbio::product to incorporate
   loading 1 new field.

release 50.4.0
 - PacBio - minor tweak to hifi yield values due to SMRT Link v13
 - Extend npg_warehouse::loader::pacbio::product to load metrics per (usually
   deplexed) product.

release 50.3.0
 - Minor change to Illumina product metrics loader to include extra metrics
   for BGE libraries.
 - Ensure that pull requests to the master branch source dependencies from
   master branches (devel branch was hardcoded).

release 50.2.0
 - Minor PacBio run loader tweak to avoid trying to parse garbage XML

release 50.1.0
 - Switch to Perlbrew to obtain multiple Perl versions

release 50.0.0
 - Change PacBio loader to support plate number for Revio runs.

release 49.0.0
 - Minor change to PacBio loader to find movie name from alternate source
   on ccs dataset if not in standard location on well.
 - Removed samplesheet daemon class

release 48.0.0
 - Removed all ml_warehouse lims drivers. Drivers moved to npg_tracking with
   associated tests and file history

release 47.1.0
 - PacBio loader is extended to load deplex stats where deplexing
   has been run on board the instrument

release 47.0.0
 - move get_tags method to ml_warehouse repo

release 46.10.0
 - add sl_ccs_uuid to the pacbio warehouse loader

release 46.9.2
 - Relaxed the 'required' constraint for two attributes in
   npg_warehouse::loader::pacbio::base. This is a short term solution to a
   problem elsewhere. The change allows for using classes that consume this
   role in the context of other packages.

release 46.9.1
 - Bug fix: resolved the discrepancy in the attribute name between the
   npg_pacbio_runs2mlwarehouse script and the npg_warehouse::loader::pacbio::run
   class.

release 46.9.0
 - PacBio loader is extended to load the outcomes from PacBio manual QC,
   see https://github.com/wtsi-npg/npg_langqc for information on the system
   that supports PacBio manual QC.

release 45.8.0
 - PacBio loader:
   - move tag sequence parsing into separate function,
   - generate well's product IDs only for new records,
   - prior to creating run product records, drop product records for runs
     where at least one product record is not linked to the pac_bio_run
     table; this ensures that product records are properly relinked.
 - drop the CI build dependency on npg_irods - the only package that was
   sourced from that repo - WTSI::NPG::HTS::PacBio::Sequel::APIClient -
   has been moved to https://github.com/wtsi-npg/perl-dnap-utilities

release 45.7.1
 - small fix for PacBio SMRT Link v11 num control reads
 - update action versions in CI

release 45.7.0
 - add product id generation and loading to the pac bio loader
 - change CI runner from Ubuntu 18.04 to ubuntu-latest

release 45.6.0
 - add run transfer date to the pacbio warehouse loader.

release 45.5.1
 - add 1 missing field from the substitution metrics table to the autoqc loader
 - force pulling tags to overwrite github local

release 45.5.0
 - add extra SMRT Link v11 related fields to the pac_bio_run_well_metrics
   table
 - add autoqc substitution metrics to the iseq_product_metrics table

release 45.4.1
 - patched the loader for Illumina run parameters XML files so that
   it does not fail when the glob expression contains only the bracket
   expansion (this is what the pipeline passes to the script)
 - tweak to pacbio loader for SMRT Link v11 and small bug fix

release 45.4.0
 - tweak to pacbio loader to deal with different SMRT Link versions
 - add a loader for Illumina run parameters XML files

release 45.3.0
 - extend fields loaded by the npg_pacbio_runs2mlwarehouse script
 - bug fixes and improvements for the npg_irods_locations2ml_warehouse script

release 45.2.0
 - update the npg_irods_locations2ml_warehouse script to make it able to read
   input data from multiple files residing in a common directory

release 45.1.0
 - add npg_irods_locations2ml_warehouse - a generic script to load iRODS location
   for sequencing products from different pipelines to ml warehouse
 - extend npg_external_report2ml_warehouse  - add loading data for file status columns
 
release 45.0.0
 - remove provisions for loading the old Sequencescape warehouse
   since the data is not longer loaded to that warehouse and the loader
   code has been removed, see
   https://github.com/wtsi-npg/npg_sequencescape_warehouse/pull/30

release 44.3.1
 - tag_index column is dropped from the tag_metrics table of the
   QC database - update test fixtures accordingly.

release 44.3.0
 - add npg_check_study_staging_location for use in cron to warn of
   studies not in expected staging areas

release 44.2.0
 - update regex to discriminate crams to consider in manifest4study bin.
   regex will look for filename patterns including:
    - xxxxx#z.cram
    - xxxxx_y#z.cram
    - xxxxx_y-y#z.cram
 - add control metrics to pacbio loader
 - load data to new columns in the iseq_heron_metrics_table
   (pp_repo_url, ivar_md, ivar_amd, pct_n_bases_amd, longest_no_n_run_amd) 

release 44.1.0
 - inital ml warehouse run data loader for pacbio data
 - move from Travis CI to GitHub Actions

release 44.0.0
 - change table loading to include the new iseq_run table in ml_warehouse
 - remove provisions to copy run statuses to the old warehouse

release 43.3.0
 - a new flag --[no-]require_qc_complete for study manifest generation
   to suppress (default) or not the manifest generation for runs that
   do not have 'qc complete' as their current status

release 43.2.0
 - samplesheet generation for MiSeq instruments (both a script and a class)
   is moved to this package from npg_tracking, the class is renamed
   (npg::samplesheet::auto -> npg_tracking::daemon::samplesheet::auto); it is
   changed to use the ml_warehouse driver for LIMs data access

release 43.1.0
 - retrieval and loading of heron artic autoqc data is refactored
   to make the code more generic and flexible, in particular:
   1. when retrieving autoqc data, the data from all portable
      pipelines are stored under a single top-level key, thus
      making it easy to skip this data when loading non-pp tables
   2. if the same column names are used in pp and non-pp tables,
      preference is given to data from the generic result objects
   3. since in-memory data structure for the generic autoqc results
      is now more flexible, it becomes possible to accomodate
      multiple data sets, for example, storing ampliconstats data
      per entity per amplicon, rather than flattenning ampliconstats
      data 
 - retrieval of ampiconstats autoqc data and loading the data to the
   new iseq_product_ampliconstats table

release 43.0.0
 - ml warehouse run loader extended to load autoqc data to the
   iseq_heron_product_metrics table

release 42.12.0
 -  ml warehouse run loader: use the Log::Log4perl logger

release 42.11.0
 - new options in manifest for study generation:
     --files_info_from_stdin - to read file listing from STDIN;
     --manifest_name_suffix - to customise manifest name
 - wh loader: when linking to LIMS data, for whole lane libraries
   disregard superfluous tag_index value in the iseq_flowcell table

release 42.10.0
 - load uqc outcomes to iseq_product_metrics table and use
   this value, if defined, to set the overall qc outcome
   for a product
 - load interop autoqc check results to iseq_run_lane_metrics table

release 42.9.0
 - manifest generation:
     flag-driven upload of an empty manifest for external products
     flag-driven skipping of validation against staging md5 
     a bug in counting the number of remote files is fixed

release 42.8.0
 - warehouse loader - add loading manufacturer's instrument
   name

release 42.7.1
 - manifest generation: exclude annulled products

release 42.7
 - external product report loader:
     column mapping update,
     bug fixes for merged top-up data

release 41.6.1
 - recognise 'NA' string as a valid value for Fractional_NRD and
   NRD_pass report columns; meaning - the array data for that sample
   is not available
 - report what is being parsed

release 41.6.0
 - run data deletion from ml warehouse - bug fix
   (product rows for merged data were not deleted)
 - loader for an external report:
    deal with externally merged data and external QC data

release 41.5.0
 - script to generate and upload a manifest for a study
 - script to load external report to ml warehouse

release 41.4.0
 - test data enhancement to ensure forward compatibility

release 41.3
 - run folders for test data restructured to reflect new-style
product hierarchy
 - bug fix for setting (or not) foreign keys on update

release 41.2.1
 - change lims_fk_repair flag default from false to true to
   mitigate code regression in release 41.2, which erases value of
   the foreign key into iseq_flowcelt table on subsequent runs of
   the loader for a particular run (is the foreign key is set
   correctly only if no data exist in the product table)

release 41.2
 - a new role - npg_warehouse::loader::product - consolidates
   methods for retrieving autoqc data for compositions and
   loading this data to the iseq_product_metrics table
 - a new script - npg_products2mlwarehouse - for loading product
   autoqc data to the iseq_product_metrics table

release 41.1
 - add target autosome stats to autoqc loader
 - correct retrieval of split and non-split data for bam_flagstats
     autoqc results

release 41.0
 - autoqc retriever:
     use special function for retrieval of ref_match result,
     do not process old contamination autoqc results for ml warehouse
 - wh loader changes following ml warehouse schema update in
   https://github.com/wtsi-npg/ml_warehouse/pull/10:
     update existing row instead of deleting and re-inserting them,
     on insert to product table create rows in the product components table,
     on product table row update do not assign LIMs fk values unless
     lims_fk_repair flag is true
 - a new script for back-populating product components table
 - a new script for deleting run data from ml warehouse

release 40.1
 - wh loader:
     add ability to load lanes that do not exists in tracking database,
     load autoqc data for all statuses (ie including 'data discarded'),
     push defined library outcomes from merged entities to single plexes
     if the library outcome for teh latter is not set

release 40.0
 - wh loader: refactor to accommodate loading results for
   multi-component entities
 - autoqc data retriever - stop retrieving data for the old split stats
   check which is only needed for the old warehouse
 - if multiple runs are loaded, load in the decreasing run id order
 - wh loader script - add a new option, num_runs, to load a limited
   number of latest runs
 - load instrument_side and workflow_type columns of the
   iseq_run_lane_metrics table
 - loadnrd_percent column of the iseq_product_metrics table

release 39.11
 - auto QC loader query costruction: do not use invalid option
 - wh loader: skip loading results for multi-component entities
 - to retrieve lane-level results for multi-component entities,
   ask for lanes and plexes separately
 - stop loading Illumina qc data that came from chached_query table
   of the npg_qc database (pf_cluster_count, raw_cluster_count, pf_bases)
 - load q30 and q40 yields from qX_yield autoqc check results     
 - load bam_flagstats target metrics into iseq_product_metrics table
 - a script to launch warehouse loader script for certain runs

release 39.10
 - two gbs metrics to be added to the iseq_product_metrics table

release 39.9
 - multiple RNA-related metrics to be loaded into iseq_product_metrics
   when updating the data warehouse
 - added support for NPG SeQC metric tables that only have composition
   and not id_run, position and tag_index columns such as rna_seqc
 - added tag_hops_percent and tag_hops_power

release 39.8
 - code and tests changed to enable finding QC outcomes via a linked composition
 - Travis builds might fail under Perl interpreters without support for threads,
   see wtsi-npg/npg_tracking#428 for details. Production code does not run
   under Perl 5.16, so removed Travis build for this Perl version

release 39.7
 - test data changes to ensure compatibility with npg_qc release 64.1

release 39.6
 - qc database fixtures additions to enable finding autoqc
     results via a linked composition

release 39.5
 - to enforce business logic consistency, common parts of lims drivers
   factored out; a driver parent object is created
 - early failure with a clear error message when essential parts of
   a flowcell database record are missing
 - correct computation of expected abs path in test
 
release 39.4
 - correct computation of expected abs path in test

release 39.3
 - added iseq_run_lane_metrics.run_priority to warehouse update (from npg_tracking run.priority)

release 39.2
 - code changes to eliminate warnings when running under Perl 5.22.2
 - PDL return value is now dealt with withing autoqc result objects,
   the warehouse loader can rely on them being either a number
   or undefined
 - Travis CI testing added

release 39.1
 - compensate for loss of split_stats role in npg_qc
  (see https://github.com/wtsi-npg/npg_qc/pull/342)

release 39.0
 - added tests for 'purpose'
 - ml_warehouse lims driver refinments to ensure db connection is
   propagated
 - add run/flowcell level caching st:api:lims driver

release 38.0
 - autonomous ml_warehouse driver which takes query by run id
 - extended ml warehouse loader to load final qc values

release 37.10
 - Corrected calculation of unexpected_tags_percent

release 37.9
 - Try to figure out id_run from IseqProductMetrics if a flowcell
   identifier is provided.
 - daemon to continiously repair foreign keys from iseq_product_metrics
   table in ml warehouse to iseq_flowcell table

release 37.8
 - Load unexpected tag and chimeric fields in iseq_product_metric and 
   iseq_run_lane_metric tables

release 37.7
 - wh loader bug fix to prevent dropping existing product data if
   no new data is available (GCLP staging data not visible from seq farm
   and other way around)  

release 37.6.1
 - update/fix tests to comply with ml_warehouse v1.5 and npg_tracking v84.7 

release 37.6
 - delay wh loader till the run is finished so that we have time to
   update the number of lanes in the npg_tracking db

release 37.5.1
 - DBIx batch insert does not work correctly, leaves some columns empty.
   Reverted to per-row insert.

release 37.5
 - a utility for repairing unset foreign keys from npg product table to
   the flowcell table in the ml warehouse
 - minimise time tables are locked for when loading data to the warehouse
 - use fast batch insert mode

release 37.4
 - fixed a bug in recognising an allowed error in lims data retrieval

release 37.3
 - ml_warehouse driver for st::api::lims
 - ml_warehouse loader - use common code for retrieving flowcell LIMs data from the database

release 37.2
 - loader changes resulting from dropping most of autoqc columns from the run lane metrics
   and removing from this table a foreign key into the flowcell table
 - give preference to batch_id when linking to the flowcell table
 - add explain flag for logging problems linking to tle flowcell table
 - link as much data as possible for a flowcell with duplicate entries
 - retrieve and load data for verify_bam_id check and for normal mode distribution
   in the insert size check

release 37.1
 - add run_is_indexed method to npg tracking data source

release 37.0
 - the following modules
     npg_warehouse::loader::autoqc
     npg_warehouse::loader::npg
     npg_warehouse::loader::qc
     npg_warehouse::loader::run_status
   were moved to this package from the SVN data-handling package (release 36.7).
 - npg_warehouse::loader::run_status module was changed to accomodate
     loading run statuses to the multi-lims (ml) schema defined in WTSI::DNAP::Warehouse::Schema
 - initial version of the module (npg_warehous::loade::run) for loading the analysis data to
     the ml warehouse
 - initial version of scripts for loading run statuses and analysis data to the ml warehouse