forked from wtsi-npg/npg_tracking
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Changes
2009 lines (1752 loc) · 98 KB
/
Changes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
LIST OF CHANGES
release 92.5.0
- Fixed instrument name check.
- Fixed the problem in the script that generates the Illumina instruments
uptime report. A few month ago the lab stopped recording the washing
cycle. As a consequence, statuses of some instruments do not get updated
for many months. The instruments with no status within the report period
were excluded from the report.
- Simplify directory view for Tileviz data.
release 92.4.0
- Staging area daemon - when the folder name is not in the database,
prior to updating the database check that for the given run id:
1. the current run status is "run pending",
2. the instrument name is consistent between the database record
and the RunInfo.xml file.
- Switched the default st::api::lims driver from xml to samplesheet.
Fixed tests, which implicitly relied on the xml driver being the default.
Some tests used the default driver type. For those tests either test
samplesheets were generated and used as a source of LIMS data or the
driver type was explicitly set as 'xml'.
- In samplesheet generation (npg::samplesheet) explicitly set the default
st::api::lims driver to xml. This driver was previously used implicitly.
- Apache https config:
redirected all non-SSL requests to a port with enabled SSL
removed provisions for self-compiled httpd server.
release 92.3.1
- A WHILE loop in run_list.tt2 iterates over a number of pages (40 runs
per page) in a visual listing of runs. The number of pages is now
(January 2022) sufficiently high to trigger an error in Template Toolkit,
which, by default, allows 1000 iterations only. The value of
$Template::Directive::WHILE_MAX variable is, therefore, set to 2000.
release 92.3.0
- New link in sigularity image to pass npg_tracking configuration file.
- Removed ability of the tracking server to provide directory views of
visible to the server /nfs partitions since nfs staging servers are
no longer used.
- Removed co-location functionality in npg_tracking views. Co-location checks
will not make sense once npg_tracking runs in containers in head nodes.
Staging areas are not local to head node/reachable from within the
container.
- Removed old team names from run add template.
- Removed unused daemon runners for staging daemons and the Jenkins server.
- Made NovaSeq instruments skip wash statuses after completing a run, this
does not affect statuses set manually through the UI.
- Added lab location to instrument pages. Instruments are also now
colour-coded on the graphical instrument list page.
release 92.2.0
- Added singularity definition for the tracking web server.
- Tracking web pages:
added folder name column to run listing pages,
added external instrument name to the instrument page.
- Patched and refined a script for adding roles to users.
release 92.1.0
- Change LDAP URL to read-only URL
- Added a SOP for creating/decommitioning instruments.
- Removed the samplesheet daemon definition npg_tracking::daemon::samplesheet
and the auto-generator for samplesheets npg::daemon::samplesheet::auto.
Both are no longer in use, the functionality was moved to
https://github.com/wtsi-npg/npg_ml_warehouse, see
https://github.com/wtsi-npg/npg_ml_warehouse/commit/0b98fc4defb8338914f8d11448df4b30cd75528c
release 92.0.0
- Create 'primer_panel' alias for the 'gbs_plex_name' methods in
st::api::lims. The new accessor will be used in contexts that are outside
of processing GBS runs.
- Remove the API wrapping around tracking XML API.
This API has not been used in production for a long time. The part of it
that links run id and batch id (npg::api::run) was used in tests. Tests
are amended either to use samplesheets is a source of LIMS data or, where
they are still using XML LIMS driver for st::api::lims, batch id is used
explicitly.
st::api::lims::xml previously had dependency on npg::api::run to allow
for retrieval of batch_id from run XML API in cases where it was not
supplied explicitly. This dependency is removed without changing the
public interface of the class. Formally speaking, the batch_id
attribute remains optional for creating an instance of the class.
However, failure to supply it will result in an error when trying to
access batch XML API of the LIMS system.
- Remove code for sending run status events to LIMS, this functionality is
no longer required.
- Remove st::api::event class, which is no longer used.
- Stop generating XML output from the tracking server. Remove all templates
for generating XML output, all associated code in views, tests and test
data.
release 91.15.0
- Remove status daemon. This daemon is no longer run since the pipeline
updates run and lane statuses by writing directly to the database.
Remove the script, npg_status_watcher, for detecting run status JSON
files in run folders and updating database run and lane statuses,
which was run by the status daemon.
- Removed live tests for XML and warehouse drivers. Live data can change
at any time without notice, causing tests to fail.
release 91.14.0
- Remove live tests for st::api::lims XML driver and other XML-based APIs.
The CI tests are mostly run on hosts with no access to either production
or development hosts.
- NPG_TEST_TRACKING_CONFIGPATH environment variable can be used to set a
custom location of the Clearpress configuration file for unit tests.
The default location of the file, data/config.ini, is used if the
variable is unset.
- Move README file to README.md, improve Markdown format. Add sections
describing environment variables and tags.
- Move db schema dump, which is only used for tests, from the data to t/data
directory.
- Block access to tsv files via httpd_sfweb.conf
release 91.13.0
- Extend npg_status2file script to optionally save statuses to the tracking
database and renamed the script to npg_status_save.
- Move from Travis CI to GitHub Actions.
- metaref_repository attribute is added to the reference repository API
release 91.12.0
- Change instrument and instrument format pages to only show current
instruments and instrument formats
- Remove superfluous columns in display tables for lists of instruments
and instrument formats
- Remove unused views (templates) for an instrument manufacturer
- Remove manufacturer model and view classes, create manufacturer_name
method in the instrument and instrument_format model classes
release 91.11.0
- a partial reimplementation of the serialization framework for components
and compositions in order to speed-up frequently used methods
- add geno files to disabled access list in http_sfweb.conf
- cookie decryption - use an up-to-date Crypt::CBC API and algorithm
- reduce the sleep type of the status daemon from 60 to 1 sec in order to
reduce the time it takes to loop over the run folders on lustre
release 91.10.0
- script npg_samplesheet4MiSeq is removed this utility has been moved
to a different git package; the samplesheet daemon stays in this
package till all MiSeq instruments are redirected off old staging servers
release 91.9.0
- Change label 'New Instrument Mod' to 'New Instrument Upgrade' on the Admin page
- Remove 'New Instrument Status' and 'New Run Status' options from Admin page
- Sort the dropdown menu of instruments on the 'New Run' page
- Staging servers usage page retired
release 91.8.0
- BotSeq renamed Duplex-Seq
- added new functions to long_info for BotSeq analysis
- added support for NovaSeq v1.5 reagents, i5 index sequenced in opposite direction
release 91.7.0
- remove the template for an unused tracking server url - run/summary -
and all underlying code to remove obsolete SQL queries used by that code
release 91.6.0
- enforce custom type NpgTrackingReadableFile to check that the
given path is not only readable, but is actually a file
- discontinue 'npgdonotmove' and 'npg_do_not_move' file and
directory names being used as flags to stop moving the run folder
from analysis to outgoing
release 91.5.0
- ignore new revision field optionally added to primer panel name in LIMs.
release 91.4.0
- disable downloading *.fa (consensus) files from tracking server
views of staging folders
- add executable for npg samplesheet4MiSeq wrapper
- st::api::lims - two new accessors, 'sample_is_control'
and 'sample_control_type'
release 91.3.0
- pp_archive_path runfolder accessor - a path to an archive directory
for third party portable pipelines
release 91.2.0
- add support for finding primer_panel bed files
release 91.1.0
- staging monitor changes for SP flowcells
- Update URL for production Sequencescape LIMS
- method for creating a lane-level st::api::lims object from
any other st::api::lims object where the resulting object has
the same driver as the original one
- add support for finding primer_panel bed files
release 91.0.0
- modified is_i5opposite when there is no reverse read
- removed scripts for monitoring instruments via ftp; related
modules and methods, their tests and test data are removed
- code for staging monitor reorganised so that the Monitor::Runfolder
class has methods for updating run, run_lane records and run tags
and the Monitor::Runfolder::Staging class mainly deals with staging
folder inspection and changes; some methods renamed to remove
ambiguity about their functionality
- some of the functionality of the instrument monitor is incorporated
into the staging monitor script code, namely, (1) setting 'multiplex',
'paired_read' and 'single_read' tags and (2) deletion of superfluous
lanes; this functionality is now available for NovaSeq instruments,
while previously it was missing for this instrument type
- redundant 'rta' run tag is not longer set
- redundant Mirror.completed file is no longer created in the run
folder
- functionality for checking instruments' wash status and setting
'was required' instrument tag is lost; this shoud be done outside
of staging monitor
- npg_tracking::util::pipeline_config:
move bqsr and haplotype caller to a tertiary section which can be
set specifically by reference or by using a default study config;
restrict access to attributes which should not be set by the caller
(local_bin, product_config);
enable setting product release configuration file path via the
new product_conf_file_path attribute
- genome reference hierarchy in the current samplesheet is not compatible with
the latest version of MiSeq Illumina software; setting Workflow value to
GenerateFastQ prevents the instrument's software from using this value,
hence we can stop setting genome reference path altogether
- make impossible to download cbcl file via tracking web pages
- runs_on_batch method of npg::run::model is refactored not to cache
the return value; proper input checking is added
- error on run creation if the batch id associated with the run is already
associated with one of active runs, i.e. a run that has no current status
or its current status is not either 'run cancelled' or 'run stopped early
release 90.3.0
- npg_tracking::glossary::moniker -
added a method for file name parsing to retrieve id_run,
position, tag_index, suffix and extension;
private attributes are prevented from being serialized under
MooseX::Storage framework
- tracking server instrument visualization - remove visual cue
for connection delay since ftp-ing to instruments is discontinued
release 90.2.1
- staging area daemon to save the flowcell barcode to the
database if it does not have this value already available
release 90.2.0
- reference finder for compositions not to return
a PhiX reference when at least one other reference
is found
- staging area daemon will validate database flowcell barcode
against the value in files in the run folder; in case of mismatch
the run will not track
release 90.1.0
- no_archive attribute for run folder API
release 90.0.0
- cleanup and simplification of runfolder subpaths logic
- add uses_patterned_flowcell attribute to long_info
release 89.1.0
- Where semantic names are not possible, standard file and directory
names are based on sha256 digest of the composition JSON
(previously the md5 digest was used). This makes possible
conversion from the name back to the composition using the ml
warehouse records, which might be useful in case of clients'
queries.
- Unused run folder path accessors removed (reports_path,
lane_archive_path, lane_archive_paths, lane_qc_path,
lane_qc_paths).
release 89.0.1
- live test patch to reflect a change in spiked PhiX tag index
release 89.0.0
- the samplesheet driver's parser extended to cope with data from
multiple runs
- samplesheet generator extended to cache data for multiple runs
- status monitor will now work only with RFC 3339 compliant
timestamps defined in WTSI::DNAP::Utilities::Timestamp
- pipeline configuration module is moved to this package from
npg_seq_pipeline package in order for the product configuration
be accessible from other packages
- staging area daemon modified so processing for each folder is wrapped as
try/catch block
release 88.2.1
- patch to accommodate different URL info passed to the CGI
module by different Apache httpd versions
release 88.2
- scripts to list run folders that are candidates for deletion in
the Data Recovery area
- status monitor: improved logging, which now shows timestamps
- alleviate differences in cloud hosts time zone configuration by
storing UTC offset as a part of the timestamp in serialized
run and lane status objects
release 88.1.1
- test data fixes following ml warehouse schema changes, see
https://github.com/wtsi-npg/ml_warehouse/pull/104 for details
release 88.1
- status monitor running with inotify disabled should be able to
recognise when a cached file is changed
- 2 sec sleep introduced in the beginning of the job that saves
statuses to a file to ensure no two statuses of the same run/lane
have the same timestamp so that the iscurrent db flag could be
assigned inambigiously when the statuses are saved to a database
release 88.0
- tracking database:
add column `incompatible_tag` to the tag table;
populate `incompatible_tag` column with values previously
hardcoded in a DBIx class;
add tags for instrument workflow;
simplify code in DBIx classes for setting and unsetting tags;
drop requirement for a valid user id when unsetting a tag;
drop a check for tag existence when unsetting a tag
- staging monitor to set/fix tracking db record about instrument side and
workflow type by setting/resetting appropriate run tags
- Change staging area monitor so for NovaSeq runs, both CopyComplete + RTAComplete
or RTAComplete + max waiting time must be true before a run can be considered
copy complete.
release 87.4
- changes to support geno_refset repository
- moved method for creating full file names from npg_tracking::product
to the moniker role
- extension of the composition/component serialization role to ensure that the
serialised objects can be deserialized
- tracking server run page links to run folder locations:
'Latest Analysis' and 'Latest IVC' links removed since the target files
are no longer generated;
'Tileviz' link is updated to point to a location in the new-style runfolder
- 'locate_runfolder' cgi script:
code to handle redundant links and paths is removed;
if the target tileviz directory is not available, fall back to the one
expected for the old-style run folder
- redundant filename method of the illumina component object removed
- status monitor:
two modes - with and without inotify;
option to set the prefix from command line so that multiple monitors
could potentially run on the same host, each monitor dealing with
its own staging area;
avoid repeatedly saving the same statuses
- when searching snv_path use bait_name=Exome for (cD|R)NA library types
- tracking database:
add a unique db constraint for all types of status descriptions;
make temporal_index column in run_status_dict table non-nullable and
add a unique constraint on it;
simplify DBIx binding for statuses;
add a method for temporal comparison of run statuses
release 87.3
- for a run on ESA, redirect to an ESA url
release 87.2
- added method to find file with MT genes inside transcriptome repository
- change attributes in transcriptome-find to use lazy and builder rather than
lazy_build to conform with Moose's best practices
- sequencing instrument checker:
treat error reading a recipe file as non-fatal;
adjust warning message generation to use methods supported by the poller
object instance
- a new factory method, create_tag_zero_object, to create st::api::lims object
for tag zero
- a new factory method, aggregate_xlanes, to create a list of st::api::lims
objects for merges across lanes
release 87.1
- bug fix in JavaScript for run creation - the reflector cannot
be used any longer, allow client side JS to request XML feeds
directly from LIMs server
release 87.0
- new role - file and directory naming factory
- added bin/npg_move_runfolder script
- make NovaSeq instruments represented on web pages
- removing reflector allow client side JS to request XML feeds directly from LIMs server
- use lims_url config file entry to set LIMs server location
- gseq cluster hoop jumping for test 10-gbs_plex.t
- use 'fasta' as default aligner when searching for references
- long_info role:
when parsing RunInfo.xml look for FlowcellLayout rather than version,
remove unused methods and attributes,
use run parameters file to retrieve additional information
release 86.10.1
- following changes to npg_tracking::illumina::run::short_info role,
patch staging monitor, which started to fail if run id could
not be retrieved
- other changes to staging monitor:
improve reporting
change the order when validating run folders, will try
to retrieve run id first, then validate the name
propagate db schema handle
use db schema accessor that is used by npg_tracking::illumina::runfolder
and its roles
remove custom code where existing APIs can be used
release 86.10
- the bug in paged run views where a particular instrument was selected
is fixed
- the frame height for displaying runs on an instrument page is increased
- displayed instrument statuses in the instrument view are limited
to most recent (created within a year from the current date)
- npg_tracking::illumina::runfolder is simplified at a cost of losing
ability to pass to the constructor an arbitrary path
- npg_tracking::illumina::run::short_info role - added database
look-up aided ability to deal with arbitrary run folder names
- use icons from the latest set of brand images
- display institute's logo and favicon only if pages are served
from institute's servers
- removed unnecessary aliasing of urls
- deleted unused (dif_files_path and pb_cal_path) runfolder accessors
- method to find file with globin genes inside transcriptome repo added
- changes to support gbs_plex repository
release 86.9
- to account for ephemeral staging areas on OpenStack, extend staging
directory path match expression in Apache httpd config file
- use bash shell when executing system calls from Perl
- copied srpipe::runfolder module (from SVN data_handling package)
as npg_tracking::illumina::runfolder to this package, removed tests
that depended on test folder hierarchy in the data_handling package
release 86.8
- remove redundant script for reporting sequencing failures to RT
- remove uptime info from instrument page - function no longer available
- correct handling of transition from 'down for service' to 'down for repair'
release 86.7
- stop building on Travis under Perl 5.16,
see https://rt.cpan.org/Public/Bug/Display.html?id=123310
- reference find role: when parsing a genome reference string
return the analysis, along with the organism, strain
and transcriptome version, if present
- refactor transcriptome::find to improve code reuse and add
new find file/path/index attributes and functionalities
- when inflating rpt key, cast id_run, position and tag index
(if any) values to integers and raise an error if conversion
gives a warning
- unused tracking server features removed:
simultaneous status update for multiple runs;
simultaneous status update for multiple instruments;
simultaneous mod update for multiple instruments;
XML feeds for individual run statuses with listing of
all runs at this status;
flowcell verification;
complex computation of width of padding id_run in run name
for display purposes
release 86.6
- unused db tables, which were renamed in the previous release,
are dropped; exclusions in teh DBIx class loader script removed
- Jenkins daemon - a bug in locating prod ssl server certificates fixed
- added a convenience factory class to transform rpt list strings
into a composition
release 86.5
- small fix in locate_runfolder for rna seqc links
- remove unused classes for image generation
- be compatible with ClearPress version 475.3.3 and above
- set explicit dist to precise in travis yml
- add dual configuration for apache mpm worker/event
- Jenkins' daemon with HTTPS as default configuration
- drop or rename orphaned tables from db
- drop
cost_code
cost_group
sensor_data_instrument
sensor_data
sensor_instrument
sensor
mail_run_project_followers
- rename
event_type_service to todrop_event_type_service
ext_service to todrop_ext_service
event_type_subscriber to todrop_event_type_subscriber
release 86.4
- reference find role: remove unused 'subset' attribute and
previously deprecated 'single_ref_found' method
- increate http tmeout for event notification
- run tracking server on two ports - one accepting http traffic, another
one https only; the same set of URLs on both ports
- tracking web GUI link to request LIMs page now changed to point to
a more useful lane page; request id will not be available if XML
LIMs feeds are discontinued
- link to auth script residing in the same virtual host
- sensor-related code removed
- mkfastq option added to make a samplesheet suitable for 10x mkfastq
release 86.3
- event notification - do LIMs notification first; if it fails and is run
multiple times as a consequence, users are not spammed
release 86.2
- event notification:
use faster ml warehouse st::api::lims driver (ml_warehouse_fc_cache)
find templates directory relative to the running script bin
fix npg tracking server url in reports' text
filter out undefined email addresses
bug fixes
- remove models for db tables that are due to be dropped
- update apache conf so server run with 1 thread/child w/mpm_worker_module
Config::IniFiles used by ClearPress::util does not seem to be thread safe
- replace image generation/manipulation routines by a stub
returning a simple static image (these routines are used
in npg_qc in web pages that are no longer used).
- st::api::lims helper over boolean values for a pool fixed to work
correctly for compositions
release 86.1
- update models, droping cost code and cost group
- updating templates to remove broken R&D watermark based on cost_code
- deleting fixtures and tests for cost code and cost group
- under modperl information about request url is not available in
env. variables; now getting it directy from the CGI module
- when possible, any subsequent requests should go to the server
from which the original page was served (makes sense in general
and helps to set up proper dev environment)
- web pages - drop buttons linking to non-existing pages
release 86.0
- httpd config files for httpd v.2.4.23 and mod-perl v.2
- simple uptime report for Illumina sequencing instruments
- reinmplementation of events reporting
- timing of emails to followers is changed from statuses
'run complete' and 'run archived' to 'qc review pending' and 'qc complete'.
- ClearPress library is pinned to v. 473.0.5
release 85.6
- code and tests changes to reduce number of warnings under Perl 5.22.2
- use mysql 5.7.13 in travis (installing from deb)
- Run web page
- removed image browser link
- Add EBI submissions link
- Build.PL fix to ensure that images under htdocs dir are copied to blib
when building.
- updated links to NPG QC since the pages moved to SeqQC
release 85.5
- removed all code that relates to instrument utilisation
- removed an option to create database entries via XML POST.
- removed custom error view
- deleted special privisions for the 'pipeline' user
- removed unused xml views for run lists
- Install all files, including cgi-related, within the install action.
- Remove instrument utilisation views for web pages.
- Deprecate 'request approval' instrument status, enable status change
from either of down statuses directly to 'wash required'.
- Remove code and tests for deprecated instrument statuses.
- Create flags to mark users and usergroups as deprecated,
deprecate 'approvers' group.
- User and usergroup lists are alphabetically sorted and include only
current entries.
- Fixed instrument status menu for batch updates to display only
current statuses.
- Composition-enabled LIMs object and reference finder.
- Subset support added to the rpt key based composition factory.
- Objects's attributes based composition factory sets only those components
that are either defined or required. Previously an undef value might
have been used for a component attribute even if the value did not have
to be set, i.e. the code was not future compatible with planned removal
of 'Maybe' qualifies from the tag_index and subset definition.
release 85.4
- Run/instrument tracking; added full support for HiSeq 4000 and limited support
for NextSeq and MiniSeq
- Add HiSeq 4000 to graphical instrument list
- Limit menu for changing run status, when not analyst, to those on instrument
- Composition framework - the composition object becomes immutable as long
as its components are immutable.
- Functionality for an incremental assembly of a composition moved to
a generic composition factory.
- Empty compositions are not allowed.
- Composition constructor returns a sorted composition.
- Public sort function is removed.
- Composition - find is implemented as a binary search.
release 85.3
- Add deployment to GitHub Releases
- In order to have correctly versioned files in a distribution,
changed the builder base class.
- Travis conf update to pre-load perl-dnap-utilities from github.
- Samplesheet daemon:
fixed logger;
changed reference repository root so that it can be run on a staging host.
- Changes to handle a different number of tiles in each lane
- Increased TAG_INDEX_MAX to 9999 to handle pools with >999 samples
release 85.2
- Added 'sample_accession_numbers' to lims shim
- Deprecated old accessors for qc outcomes.
release 85.1
- Added 'purpose' to lims shim
- utf8 option for test sqlite db
release 85.0
- lims shim
- endeavours to ensure consistency of primary info (e.g. id_run, position)
with drivers
- will pass through arguments to driver constructor if it takes them
release 84.12
- composition and component serialization to rpt
release 84.11
- Operations (inflation and deflation) on rpt strings and their hash
representation and lists of rpt values (based on npg_qc::autoqc::role::rpt_key)
- Factory to create composition from a string representation of an rpt list
- New filename method for the illumina component
- Alter "run complete" email alert text to make clear analysis and qc time
release 84.10
- create multi-attribute methods automatically
- added method st::api::lims::any_sample_consent_withdrawn
- removed potential warnings in event notification RT#495946
release 84.9
- transcriptome find updated for RNA-SeQC directory and its gtf file
- generic serializable sequence composition framework and illumina
sequencing specific component
- lims shim now works out driver type from a given driver
- sequencing failure RT emails cope with GCLP (no batch_id)
- add predicates for glossary roles
release 84.8
- jenkins daemon definition: remove hardcoded path to prod user home directory
- refactor snv finder to stop warnings when ref genome is undefined RT#481506
- make function for parsing reference gemone public since it is used outside
of the module where it is defined
- use RTAComplete file in fallback local (instead of ftp) determination of run
progress (which is consistent with ftp process and required for HiSeqX)
release 84.7
- accommodate LIMs-defined dual indexes
- Illumina samplesheet generation: do not truncate single index when shorter first
part of dual index is present; this was written with spiked Phix in mind,
however, spikes are not reflected in LIMs for MiSeq runs
release 84.6
- when changing ownership of run folder, set gid bit on every level
- reporting croaks without full stacktraces for folder and folder/location
- daemon control:
do not use deprecated load_class method
read a list of production users from a configuration file
release 84.5
- staging monitor extension:
move qc complete runfolders from analysis to outgoing
propagate group ownership inside the runfolder
- staging/instrument monitor:
remove superfluous run lanes
assume all runs are RTA
- perform runfolder name validation against the tracking db run record
rather than against xml feeds; compare to expected runfolder name
if this name is not in the database yet
- add sample_supplier_name, sample_cohort and sample_donor_id to
st::api::lims interface
release 84.4
- to avoid the need to create deep directory hierarchy containing www-live
for a live server, set dev env variable value for the Apache server
explicitly
release 81.3
- accessor methods for frequently used configuration file entries
- status monitor script and staging area daemon definition - allow for
farm-specific variations of path prefixes for staging areas
- staging area and status daemons use common log directory /nfs/{gs|sf}XX/log
release 81.2
- web server configuration generalisation to allow for running on
different clusters
- cross-cluster staging links for both tracking and seqqc links
- make sample name compalsory in samplesheet generation
- provide path substitution mechanism to cope with running code on a file
server where the abs_path may differ from the cannonical network path.
release 81.1
- extended samplesheet contains only the Data section, Illumina
on-instrument reference paths removed
- factor out parsing database configuration files into a separate
module
- when using MySQL database in testing, do not use database configuration
files implicitly from the user's home directory; force the caller to
supply the location of the file
- remove default repository root from npg::samplesheet
- no longer stop runs without batch id or with 13-digit entries in the
batch id field from moving from run mirrored to analysis pending (
allows automatic analysis of GCLP and QC runs reespectively).
- optionally change group ownership of certain analysis directories on
moving runfolder form incoming to analysis
release 81.0
- remove default config file
- allow environment variable NPG_REPOSITORY_ROOT to set repository root
- move config of staging areas and repository into a config file
- remove vestigial code for dealing with manual qc state
- remove good_bad column in run_lane table
- remove manual_qc_status table
- npg daemon 'local' host option avoids ssh
release 80.11
- removed staging area sf48
release 80.10
- default to in-memory option for SQLite in tests
- added a method to transparently return live or dev LIMs base URL
- reference genome values - trim white space
release 80.9
- do not try to infer phix reference from library/sample name
- correct regular expression for modern MiSeq reagent kits - previously
part of the id starting from teh dash was not returned by the flowcel_id accessor
in the short_info role
- samplesheet generation: library id shoudl be defined for all samples
release 80.8
- fix memory leak in the samplesheet driver for st::api::lims -
pass data hash by reference
release 80.7
- change reference repository host
release 80.6
- reenable and clean up cBot monitoring
- try to avoid recursion in st::api::lims
- cope with no reference genome info in XML
- provisions for ml_warehouse driver
- change host for development tracking server
- when building the package, change version in existing directories only
release 80.5
- remove redundant status update method from DBIx run_lane table binding
- remove unused methods from npg::api::xx objects
- remove auto-change of run statuses RT#420151
- status monitor enhancements RT#420193:
cope with runfolder move from outgoing to analysis
on detecting LatestSummary creation, check for existing status files
in the status folder
release 80.4
- list of staging areas updated
- reset monitor to use local disk if FTP to instrument fails
release 80.3
- report sample_name, library_id etc at pool level if non-spike children
give same value (to better cope with single plex pools)
release 80.2
- cope with old path being reported for a directory when it is moved
release 80.1
- glob expression fix for running on Ubuntu lucid with perl 5.14.2
release 80.0
- DBIx run and run-lane status update: ensure duplicate records are not created
allow to create a record for a status that is already current if the date
of the new record is earlier that the current status date
- status monitor bug fixes
- daemon for status monitor watcher
- propagate child/plex library level study data "alignment_in_bam" and
"separate_y_chromosome" to parent lane level as is done with
"contains_nonconsented_human" data
release 77.9
- DBIx run and run-lane status update refactoring to accommodate passing
status date and make setting these two status types symmetrical
- initial code for monitoring status files
- bug fix in inferring recalibrated path from the runfolder path - previously
the code returned basecalls directory path before getting to the recalibrated path
- following psd dev url change, update regular expression in the reflector url filter
release 77.8
- npg dev url is set to the url of the dev server that is actually used for API calls
- psd next release dev url replaced with training url, RT#412417
- wrapper object for the status and a script to write status to a file
- dual index refinements: ensure max length of "Index" used by MiSeq is the split
- dual indexes' size, and use custom default_expected_tag in preference to Index{,2}
when using samplesheet as source of LIMS info
- ensure samplesheet testing works without WTSI filesystem
- instrument location tests skipped if host name is too long
(TODO - increase db field size)
- travis ci configuration file added
- mysql db test configuration - password removed so that this configuration
can be used on travis ci
- implicit dependency on DBD::mysql driver made explicit to ensure all
dependencies are built on travis ci server
release 77.7
- run in progress tracking:
look for lane sub-directories in the BaseCalls directory if none found in the Intensities directory;
check for existence of a RTAComplete.txt file rather that a ImageAnalysis_Netcopy_complete_Read#.txt file
release 77.6
- added code to generate HiSeqX runfolder names
- updated list of directories to search for cycle sub-directories
release 77.5
- added new version value for HiSeqX RunInfo.xml
release 77.4
- changed list of top level directories for HiSeqX
release 77.3
- samplesheet creation now supports dual indexes
- sensor.pm now uses npg::api::request to allow for caching xml feeds when
retrieving sensor data
- web app and database adjustments for HiSeqX
- callbacks for uptime graphs removed
- removed the uptimes table from the textual listing of intruments page -
made page loading very slow
- removed upgrade schema scripts and redesign proposal file from MANIFEST
- merged instrument checker scripts for all existing sequencing instrument types
- removed instrument checker scripts for invividual seq instrument types
- all instrument checkers to use local libraries if available
release 77.2
- ensure module and package version starts with a digit
- npg_tracking::illumina::run::folder and its tests updated; the following accessors removed:
data_path (unused from outside),
tag_qseqs_path (redundant),
score_path (redundant),
qseq_location_path (redundant);
tests specific for old illumina mashines (IL) removed;
unused analysis_path accessor reimplemented to point to the top-level custom analysis folder
paths from analysis_path removed;
definitions of most paths placed together
public write accessors for bam_basecall and dif_files paths since they are used from the pipeline,
old private accessors retained for backwards compatibility
release 77.1
- transcriptome changed so that a default is not used if not supplied
- transcriptome version directory changed so tests changed accordingly
- reference find changed to handle transcriptome version parsing
- unused templates removed
- samplesheet regenerated when the existing file does not come from the currently
pending run
- samplesheet daemon moved to a staging server where samplesheets are stored
- copyright for all modules and script belongs to GRL - the copyright
notice edited where it was incorrect
- pass Illumina samplesheet study name field components through a filter to replace
commas
- use git describe directly to avoid need for gitver script
release 77.0
- the build file and module - code improvements, POD and PBP for the module
- scripts and modules:
version variable added where needed,
RSC keywords removed,
superfluous empty lines, 'Readonly' module import and other features removed,
where needed, 'use Readonly' added
- tests - remains of RSC keywords removed
- the distribution test does not perform pod tests (separate tests available),
checks that module version matches distribution version
- allow staging daemon to check progress by looking for .bcl.gz files (as
well as .bcl files)
release 76.5
- code action on Build only if install target specified
- global removal of [LastChanged]Version from POD; change VERSION svn declaration to ='0';
- added modules to handle locating of transcriptome files required by tophat2
- updated reference module with additions needed by transcriptome
- added new test for transcriptome
- updated reference test to check that transcriptome directory structure exists
- added files from a transcriptomics run for use in test
release 76.4
- replace uri_escape() by uri_escape_utf8()
- some unused code removed
- some small test scripts merged into related larger test scripts
- VERSION variable and headers with SVN properties removed from test files
- all test scripts are given .t extension
- unnecessary 'use' statements removed from tests
- server-wide definition for modperl custom perl libraries location in order
to provide consistent perl environment for all scripts at server start-up
time
release 76.3
- use staging area for staging daemon logs
release 76.2
- lims samplesheet driver bug fix - spiked phix tag index should be defined for plexes
- method to return the name of the env. var for the cached samplesheet file
- misleading xml lims driver POD entry fixed
release 76.1
- use scratch110 for references as scratch109 is hanging
- extend timeout for Jenkins auto-logout
release 76.0
- added defaults for Perl indentation in Emacs for this repository
- definition of the repository root for single nucleotide variation files
- a new module defining rules for finding single nucleotide variation files
- changes to the database schema and code to ensure compliance with sql
strict mode, relevant changes to tests and test data
- if batch_id is not supplied at run creation, enforce zero in the model,
wider tests to be performed if NULL needs to be used
- id_run_pair defaults to null; examining and setting this field removed
from the code, the concept of run-pair has not been used for a long time
- use ${JENKINS_HOME}/tmp instead of /tmp for jenkins tmpdir
- add perlcritic policy to prohibit tabs, convert tab to two white space, correct
indentation issues
- old and unused apache configuration files removed
- IL instruments removed from instrument utilisation menu (we do not have them any more)
- instrument uptime menu link removed - target graph is broken
- consolidated all instrument menu 70* tests in one 20 test
- removed code repetition from bin/ga_II_checker
- removed deprecated last_30_days method from npg::model::instrument_utilisation
- fixed the code where in test conditions operations on undefined variables
were performed
- removed excessive verbosity in most of the tests
- documentation in README, INSTALL and REDESIGN_PROPOSAL updated
release 75.6
- extended inline index to read 2
- hardcoded link for enigmatic authentication (not mod perl url) page so that the page
is accessible from the server on sfweb, where enigmatic is not deployed;
uncomment old code to restore the relative to the server root link for testing
- apache2.2 compatible configuration files to run the tracking server under
default apache version for Ubuntu precise; virtual host definitions in
stand-alone files allow for dynamic inclusion of virtual hosts
- fixed occasional authentication problem (inability of a particular thread to load
additional modules due to the fact that on a second invocation of npg script in the
same server thread @INC array is not extended since BEGIN block is not run) when
running under apache2.2 by extending the server @INC array at server start up time
- bam, cram, etc files access is prohibited from the runfolder directories listings,
however, these files are visible in listings
- more convenient sorting order of directories and files in the runfolder directories
listings
- directory index CSS changed to get a margin on the left; brighter color scheme
introduced
- Build.PL file extended to ensure that the shebang line of CGI scripts is
set to the Perl interpreter that is used for deployment
- explicit setting of PATH is removed from CGI scripts that do not use shell commands
- SVN-dependant VERSION removed from CGI scripts
- where necessary, PERL5LIB of CGI scripts is extended
- links to staging area server are picked up from config "staging_url" parameter,
and so are relative (to current host and port) if this is empty or non-existent
- switch to samplesheet driver if NPG_CACHED_SAMPLESHEET_FILE is set
- spiked_phix_tag_index accessor implementation for the samplesheet
driver to ensure that spiked_phix_tag_index is defined for a pool
release 75.5
- on Ubuntu precise daemon .pid files in /tmp directory get deleted after a week,
making daemons non-responsive; change modification time of the .pid
file when pinging the daemon, which is done at least once a day
release 75.4
- change of default reference repository location
release 75.3
- default UTF8 output for samplesheets
- add number of lanes to run list views
- new farm host for a samplesheet daemon
release 75.2
- host-agnostic jenkins daemon
- version of npg::api::request module updated to help tracking requests
release 75.1
- patch for samplesheet parser
- switch live npg api base URL to a different host in order to separate
human- and api-driven requests
- removed deprecated methods from npg::api::run_lane
- removed unused save2cache method from npg::api::util
- simplified logic for saving to cache in npg::api::request;
removed unused save2cache method
- staging area indexes defined in single place
release 75.0
- authentication through npg own enigmatic.cgi residing relative to
where the request comes from
- updated expected result for a live reference retrieval test following
a change in the reference genome study field
- add new staging areas 52 to 55
- don't bail out on finding no runfolders when FTP'ing to instrument for update info
- sample sheet generator extended to include npg specific fields and deal with multiple lanes
- use URI escaping for samplesheet contents instead of squashing whitespace and flipping , for |
release 74.14
- minor bug fix to lims sample_description
release 74.13
- data_access_group method to lims (study) to determine data access control requirements
- add new staging area (50)
release 74.12
- to ensure host-specific path is used when calling a script, use bare script name in staging
daemon definition
release 74.11
- samplesheet lims driver moved to st::api::lims namespace, contains parser only
- daemon utility: execute daemons in a bash login shell in order to pick up
target host environment; remove setting PERL5LIB and perl version
from definitions of daemons for samplesheet and staging
- removed Makefile.PL for web deployment - this functionality is now in Build.PL
release 74.10
- update live URL for getting/posting lims data as requested in RT#336085