forked from saltudelft/software-analytics-book
-
Notifications
You must be signed in to change notification settings - Fork 0
/
06-release-engineering-analytics.Rmd
1999 lines (1580 loc) · 84.7 KB
/
06-release-engineering-analytics.Rmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# Release Engineering Analytics
## Motivation
Release engineering is a software engineering discipline
concerned with the development, implementation, and improvement of
processes to deploy high-quality software reliably and predictably [@dyck2015a].
The changes made by developers of a software system
should eventually be integrated and deployed such that end users may benefit
from them.
In recent years, release engineers have developed and
adopted techniques to build infrastructures and pipelines which automate the
process of releasing software to an increasingly large degree.
These modern approaches have resulted
in various practices such as releasing new versions of a software system in
significantly shorter cycles.
Due to these developments being industry-driven, release engineering forms a
largely uncharted territory for software engineering research.
Efforts to close this gap would be relevant both for practitioners as well as
researchers [@adams2016a].
On the one hand, claims and rationales are presented by the industry to justify
practices in release engineering, but these are often not empirically validated.
For this reason, research should aim to build an understanding of the actual
effects of release engineering practices on the software development process.
On the other hand, software engineering researchers need to be aware of modern
release engineering practices in order to account for them in their analyses.
Otherwise, their lack of familiarity with these practices will likely result
in biases in their study results.
This systematic literature review aims to provide an overview of
the software analytics research that has been conducted so far on modern
release engineering.
Its main purpose is to identify the apparent gap
between research and practice, in order to guide further research efforts.
### Research Questions
Contrary to what is regularly the case, advances in release engineering
practices are driven by industry, instead of scientific research.
Building on this idea, our questions are constructed to identify in which ways
existing modern release engineering practices should still be studied in
software analytics research.
Our review thus aims to answer the following questions.
- **RQ1:** _How is modern release engineering done in practice?_
This question aims to identify the so-called "state of the practice" in
release engineering.
We will summarize practices that have been adopted to drive release
engineering forward. In addition, we will identify the tools utilized to bring
this about.
- **RQ2:** _What aspects of modern release engineering have been studied
in software analytics research so far?_
In order to answer this question, we investigate the practices that previous
empirical case studies have focused on.
In doing so, we identify the associated costs and benefits
that have been found, and the analysis methods used.
- **RQ3:** _What aspects of modern release engineering make for relevant
study objects in future software analytics research?_
In answering this question, we aim to identify the gap between practice and
research in release engineering.
This way, our intent is not only to guide but also to motivate future research.
## Research Protocol
Our research has been performed following the procedures for performing
systematic reviews by Kitchenham @kitchenham2004procedures.
In this process, we have set up strategies for searching, selecting and
quality-assessing studies.
Subsequently, we have extracted data from the selected studies and
synthesized the answers to our research questions.
All the papers that were found, were stored in a custom-built web-based tool for
conducting literature reviews.
The source code of this tool is published in a GitHub repository.
^[See https://github.com/jessetilro/research]
The tool was hosted on a virtual private server, such that all retrieved
publications were stored centrally, accessible to all reviewers.
In order to save space in this chapter of the book,
we have omitted the full research protocol from this chapter.
The interested can find our research protocol in detail in
[Section 7.5](#appendix).
## Answers
In this section, we will give an answer to each of the research questions
that we have presented in [Section 7.1.1](#research-questions-1).
### RQ1: Modern Release Engineering Practices
This section aims to answer to the question:
_How is modern release engineering done in practice?_
Adams et al. @adams2016a and Karvonen et al. @karvonen2017a have described release engineering
practices that are currently in use in the industry.
Adams et al. @adams2016a focused on modern release engineering,
while Karvonen et al. @karvonen2017a investigated agile release engineering,
which is a subset of all modern release engineering.
Both studies agreed that modern release engineering consists
of the following components:
- **Rapid Releases (RR).**
In contrast with traditional release cycles,
RRs regularly push new releases to users in a regular schedule.
As an example, FireFox releases a new version every six weeks.
- **DevOps.**
According to Dyck et al. @dyck2015a: "DevOps is an organizational approach
that stresses empathy and cross-functional collaboration
within and between teams (especially development and IT operations)
in software development organizations, in order to operate
resilient systems and accelerate delivery of changes."
- **Continuous Integration (CI).**
To quote Adams et al. @adams2016a: "[CI] refers to the activity of continuously polling the
[version control system] for new commits or merges,
checking these revisions out on dedicated build machines,
compiling them and running an initial set of tests to check for regressions."
- **Continuous Deployment or Continuous Delivery.**
When the CI tests pass, the code can be automatically deployed
to the production environment.
The difference between these terms is that continuous delivery
does not require that changes are automatically deployed,
but continuous deployment always automatically deploys changes.
However, the change might not yet be released,
because it can be hidden using feature toggles [@laukkanen2018a].
Besides these four components, Adams et al. @adams2016a also identified three other concepts
that are used in modern release engineering,
specifically to release software as often as possible
and thus enable continuous deployment or delivery:
- **Branching and Merging.**
There are several possible strategies of branching and merging.
Typically, merging must be done as often as possible
in order to release as often as possible.
- **Build System.**
Building the software must be done in a consistent way,
such that each build produces the same result.
With the build configuration stored inside the project,
every developer (or automated tool) only needs to issue a single command
in order to build the project,
instead of manually having to configure the build process every time.
- **Infrastructure-as-Code.**
In the same alley of "storing configuration",
infrastructure-as-code means that the server (or virtual machine)
on which the software product is running
can also be automatically configured with code,
instead of having to configure each server manually.
Besides these seven components that are more technical, Poo-Caamaño @poo-caamano2016a has
identified that there are also social aspects to modern release engineering. Specifically, most
large software projects have a dedicated Release Team that will decide on the release strategies
and communicate them to others.
### RQ2: Studied Parts of Release Engineering
This section aims to answer to the question:
_What aspects of modern release engineering
have been studied in software analytics research so far?_
Over the years, the software industry has come up with inventive approaches to deliver new
features and fixes in a more efficient and faster manner. This has resulted in case studies
being done to assess the associated risk and cost factors, and what benefits certain
strategies can give.
Khomh et al. @khomh2015a have looked into the effects of switching from traditional to rapid
release cycles
in the case of Mozilla Firefox. The paper has concluded that users do not experience
significantly more post-release bugs, bugs are fixed faster, but that users experience bugs
earlier in the software execution. Mantyla et al. @mantyla2015a have also considered data from
Mozilla
Firefox and has examined the impact of release engineering on testing efforts. Observations of
the paper conclude that the rapid release cycle performs more test executions per day, but
these tests focus on a smaller subset of the test case corpus and that testing happens closer
to release and is more continuous. A limitation of this study is that it measures correlation,
rather than causation. Da Costa et al. @da2014a has further zoomed into the integration of
addressed issues
and has considered data from Mozilla Firefox, as well as data from ArgoUML and Eclipse. The
paper found that addressed issues are usually delayed in a rapid release cycle and are often
excluded from releases.
Similar conclusions based on Mozilla Firefox were made by Da Costa et al. @da2016a,
who found that minor-traditional releases tend to have
less integration delay than major/minor-rapid releases.
Castelluccio et al. @castelluccio2017a has examined the practice of *patch uplifting* in the
release management at
Mozilla Firefox where patches that fix critical issues, or implement high-value features are
often promoted directly from the development channel to a stabilization channel. The paper
evaluated the characteristics of patch uplift decisions and interviewed three Mozilla release
managers. The paper concluded that the majority of patch uplift decisions are made due to a
wrong functionality or crash. The specificity and code author of patches that are requested to
be uplifted are also a major factor for release managers.
In response to case studies being done on many prominent open source software projects,
Teixeira @teixeira2017a has described OpenStack's shift to a liberal six-month release cycle. As
this
is an ongoing study, the results given by the paper are preliminary and only observe the
process. OpenStack's release process can be considered as a hybrid of feature-based and
time-based releases. OpenStack encourages regular releases but also attempts to include new
features at each regular release.
Rather than focussing on topics such as issue and delays, Poo-Caamaño @poo-caamano2016a focusses
on the
communication in release engineering in the cases of GNOME and OpenStack. Through analyzing
over 2.5 years of communication, the paper has made a number of observations.
The paper found that developers tend to communicate through
blogs, bug trackers, conferences, and hackfests.
Another finding is that a release team is set to define
requirements, quality standards, and coordination through (direct) communication.
Although only the mailing lists of the projects were studied,
defined challenges include keeping everyone informed and engaged,
monitoring changes and setting priorities in cross-project coordination.
Laukkanen et al. @laukkanen2018a have described what effects modern release engineering
have on software with different organizational contexts.
This study specifically focusses on continuous deployment practices.
The paper has found that high internal quality standards combined with the
large distributed organizational context of large corporations slowed the verification process
down and therefore had a negative impact on release capability. However, in small
corporations, the lack of internal verification measures due to a lack of resources was
mitigated by code review, disciplined CI and external verification by customers in customer
environments. More about the factors that can play a role is addressed by Rodríguez et al.
@rodriguez2017a,
where an overview of contributing factors in continuous deployment are defined and categorized
based on literature between 2001 and 2014.
As rapid release cycles and continuous deployment are topics that are new and emerging, not
enough research has been done to generalize any conclusions that are made in the case studies
discussed in this section. This is why all the empirical studies in this survey have one major
sidenote in common: more case studies are needed. Open challenges such as these will be
discussed in the next section.
### RQ3: Future Research
This section aims to answer the question:
_What aspects of modern release engineering make for
relevant study objects in future software analytics research?_
#### General Suggestions
The body of literature that we analyzed for this survey mostly comprised case
studies that employed quantitative analysis methods.
From these studies, interesting conclusions have been drawn about the effects of
release engineering practices on software development processes in specific
contexts.
However, the generalizability of the findings in these case studies is very
limited.
Therefore, in general many studies suggest that future research efforts focus on
performing additional case studies, both to verify existing findings and to
study new relationships and new contexts [@karvonen2017a; @teixeira2017a;
@khomh2015a; @claes2017a; @laukkanen2018a; @adams2016a; @castelluccio2017a].
It also seems worthwhile to triangulate findings by complementing data analyses
with other quantitative (e.g. a survey) or qualitative (e.g. an interview)
methods [@karvonen2017a].
Finally, additional literature reviews will allow researchers to keep an
overview of the most recent developments and findings in the area of release
engineering [@rodriguez2017a; @laukkanen2018a].
Apart from verifying results, it might be worthwhile to leverage them by
constructing analysis tools for practitioners.
For example, Castelluccio et al. @castelluccio2017a suggest exploring possibilities to leverage
their research by building classifiers capable of automatically assessing the
risk associated with patch uplift candidates and recommend patches that can be
uplifted safely.
Also, companies seem to be struggling with the adoption of continuous delivery
and deployment, so a checklist for analyzing readiness for these practices
might be developed [@karvonen2017a].
The review by Karvonen et al. @karvonen2017a makes a number of general suggestions for future
research.
In particular, there should be more attention to comprehensively reporting how
practices are implemented and in which context they are embedded, instead of
just stating that they are used.
Also, the viewpoints of different stakeholders other than developers can be
taken into account.
For example, the customer perceptions regarding the adoption of a certain
practice can be investigated.
#### Directions for Specific Practices
**Rapid Releases**
As established, rapid releases are a prevalent topic in current research on
modern release engineering.
However, it will be useful to verify these results given the fact that this
research mainly involves case studies (most of which are only concerned with
Mozilla Firefox due to the availability of data).
To this end, there are opportunities to further investigate the effects of
switching to rapid releases on:
- code integration [@da2014a; @da2016a; @souza2015a; @castelluccio2017a],
- testing efforts [@mantyla2015a],
- software quality [@khomh2015a; @khomh2012a],
- (library) adoption [@fujibayashi2017a]
- time pressure and work patterns [@claes2017a].
**DevOps**
When it comes to DevOps, future research is needed to refine its definition such
that it is uniform and valid for many situations [@dyck2015a]. According to
Karvonen et al. @karvonen2017a, it seems that the goals in DevOps are congruent with those
in release engineering, and future research on this topic is therefore highly
relevant in order to study modern release engineering.
**Continuous Delivery / Deployment**
Research on continuous deployment seems to be still in its infancy, therefore Rodríguez et al.
@rodriguez2017a have suggested a significant number of different concrete
opportunities for future research.
In general, they conclude that the topic needs an increase in the number and
rigor of empirical studies, and thus it presents opportunities for software
analytics research.
In a systematic literature review, Laukkanen et al. @laukkanen2017a identified 40 problems,
28 causal relationships and 29 solutions related to the adoption of continuous
delivery.
These problems and solutions can be studied further to deepen the
understanding of the nature of these problems and how to apply their solutions.
Some of the problems that are more of a human or organizational nature might
be involved with a broader spectrum of changes, so it should also be
investigated to what extent these problems are specific to continuous delivery.
**Continuous Integration / Build system**
One of the main issues that seems to be obstructing organizations in adopting
modern release engineering practices is build design [@laukkanen2017a]. In a
case study by Laukkanen et al. @laukkanen2018a, it was found that a complex automated build and
integration system led to a more undisciplined one, which in turn slowed down
the verification and release processes. Therefore, future research might
investigate how developers can make their builds more maintainable and of
higher quality, how anti-patterns in the design of the build can be refactored,
and how continuous integration can be made faster and more energy efficient
[@adams2016a].
## Conclusion
In this literature survey,
we have provided an answer to the following three research questions:
- **RQ1:** _How is modern release engineering done in practice?_
We found that there are six important technical aspects to
modern release engineering: Rapid Releases, DevOps, Continuous Integration,
Continuous Deployment, Branching and Merging, Build Configuration and
Infrastructure-as-code.
The most important social aspect of modern release engineering is communication.
- **RQ2:** _What aspects of modern release engineering have been studied
in software analytics research so far?_
At this point in time, case studies have mainly focussed on
the resulting factors of switching from a traditional release cycle
to a rapid release cycle, and what effects this has in various
organizational contexts.
As all included studies suggest, more empirical studies are needed to be
able to make general conclusions in the novel field of release engineering.
- **RQ3:** _What aspects of modern release engineering make for relevant
study objects in future software analytics research?_
In general, more empirical research is required to validate and generalize
the results of many previous case studies within the field of release
engineering.
In addition to performing case studies involving quantitative analyses,
it may be beneficial to triangulate results using various research methods.
Also, future research should more comprehensively describe how practices
are implemented, and consider different stakeholders.
For each practice, future research is suggested on the one hand to further
investigate their effects on the development process, and on the other hand
to investigate problems involved with their adoption.
## Appendix
This appendix contains sections that were part of our process during
this literature survey.
They are not directly needed to answer the research questions,
but are still relevant in order to validate our survey.
This Appendix contains our project timetable, the research protocol in full detail,
and the raw extracted data from the selected studies.
### Project Timetable
The literature review was conducted over the course of four weeks. We worked iteratively and
planned for four weekly milestones.
+-----------+----------+-------------------------------------------------------+
| Milestone | Deadline | Goals |
+===========+==========+=======================================================+
| 1 | 16/9/18 | - Develop the search strategy |
| | | - Collect initial publications |
+-----------+----------+---+
| 2 | 23/9/18 | - Write full research protocol |
+-----------+----------+---+
| 3 | 30/9/18 | - Collect additional literature according to the protocol |
| | | - Perform data extraction |
+-----------+----------+---+
| 4 | 7/10/18 | - Perform data synthesis |
| | | - Write final version of the chapter |
+-----------+----------+---+
### Research Protocol
In this appendix, we will describe in detail how we applied the protocol for
performing systematic literature reviews by Kitchenham @kitchenham2004procedures.
In order, we will go over the search strategy, study selection,
study quality assessment, and data extraction.
The last subsection will list which studies we included in this review
and which we have found, but excluded from the review for a specific reason.
#### Search Strategy
Since release engineering is a relatively new research topic,
we took an exploratory approach in collecting any literature revolving around
the topic of release engineering from the perspective of software analytics.
This aided us to determine a more narrow scope for our survey,
subsequently to allow us to find additional literature to fit this scope.
At the start of this project, we were provided with an initial seed of five
papers as a starting point for our literature survey
[@adams2016a; @da2014a; @da2016a; @khomh2012a; @khomh2015a].
We collected other publications using two search engines:
Scopus and Google Scholar.
Each of the two search engines comprises several databases such as
ACM Digital Library, Springer, IEEE Xplore and ScienceDirect.
The main query that we constructed is displayed in Figure 1.
The publications found using this query were:
- @kaur2019a
- @kerzazi2013a
- @castelluccio2017a
- @karvonen2017a
- @claes2017a
- @fujibayashi2017a
- @souza2015a
- @laukkanen2018a
- @dyck2015a
```
TITLE-ABS-KEY(
(
"continuous release" OR "rapid release" OR "frequent release"
OR "quick release" OR "speedy release" OR "accelerated release"
OR "agile release" OR "short release" OR "shorter release"
OR "lightning release" OR "brisk release" OR "hasty release"
OR "compressed release" OR "release length" OR "release size"
OR "release cadence" OR "release frequency"
OR "continuous delivery" OR "rapid delivery" OR "frequent delivery"
OR "fast delivery" OR "quick delivery" OR "speedy delivery"
OR "accelerated delivery" OR "agile delivery" OR "short delivery"
OR "lightning delivery" OR "brisk delivery" OR "hasty delivery"
OR "compressed delivery" OR "delivery length" OR "delivery size"
OR "delivery cadence" OR "continuous deployment" OR "rapid deployment"
OR "frequent deployment" OR "fast deployment" OR "quick deployment"
OR "speedy deployment" OR "accelerated deployment" OR "agile deployment"
OR "short deployment" OR "lightning deployment" OR "brisk deployment"
OR "hasty deployment" OR "compressed deployment" OR "deployment length"
OR "deployment size" OR "deployment cadence"
) AND (
"release schedule" OR "release management" OR "release engineering"
OR "release cycle" OR "release pipeline" OR "release process"
OR "release model" OR "release strategy" OR "release strategies"
OR "release infrastructure"
)
AND software
) AND (
LIMIT-TO(SUBJAREA, "COMP") OR LIMIT-TO(SUBJAREA, "ENGI")
)
AND PUBYEAR AFT 2014
```
_Figure 1. Query used for retrieving release engineering publications via Scopus._
In addition to querying search engines as described above,
references related to retrieved papers were analyzed.
These reference lists were obtained from Google Scholar and from the
_References_ section in the papers themselves.
We selected all papers on release engineering that are citing or being cited
by the initial set of papers.
Using this approach, we have found six additional papers.
The results of the reference analysis are listed in Table 1.
_Table 1. Papers found indirectly by investigating citations of/by other papers._
+-----------------+-------------+-------------------+
| Starting point | Type | Result |
+=================+=============+===================+
| @souza2015a | has cited | @plewnia2014a |
| | | @mantyla2015a |
+-----------------+-------------+-------------------+
| @khomh2015a | is cited by | @poo-caamano2016a |
| | | @teixeira2017a |
+-----------------+-------------+-------------------+
| @mantyla2015a | is cited by | @rodriguez2017a |
| | | @cesar2017a |
+-----------------+-------------+-------------------+
| @laukkanen2018a | has cited | @laukkanen2017a |
+-----------------+-------------+-------------------+
All the papers that were found, were stored in a custom-built web-based tool for
conducting literature reviews.
The source code of this tool is published in a GitHub repository.
^[See https://github.com/jessetilro/research]
The tool was hosted on a virtual private server, such that all retrieved
publications were stored centrally, accessible to all reviewers.
#### Study Selection
We selected the studies that we wanted to include in the survey
with aid of the aforementioned tool for storing the papers.
In this tool,
it is possible to label papers with tags and leave comments and ratings.
Every paper is reviewed based on the inclusion and exclusion criteria.
Based on this, the tool allowed to filter out all papers
that appeared not to be relevant for this literature survey.
We only used one exclusion criteria: studies that are published before 2014,
will not be included in our survey (this is enforced by our search query).
The inclusion criteria are as follows:
- The study must show (at least) one release engineering technique.
- The study must not just show a release engineering technique,
but analyze its performance compared to other techniques.
The last subsection of this appendix lists which studies
were selected and which were discareded.
#### Study Quality Assessment
Based on Kitchenham @kitchenham2004procedures, the quality of a paper will be assessed
by the evidence it provides, based on the following scale.
All levels of quality in this scale will be accepted,
except for level 5 (evidence obtained from expert opinion).
1. Evidence obtained from at least one properly-designed
randomised controlled trial.
2. Evidence obtained from well-designed pseudo-randomised controlled trials
(i.e. non-random allocation to treatment).
3. Comparative studies in a real-world setting:
1. Evidence obtained from comparative studies with concurrent controls and
allocation not randomised, cohort studies, case-control studies or
interrupted time series with a control group.
2. Evidence obtained from comparative studies with historical control,
two or more single arm studies,
or interrupted time series without a parallel control group.
4. Experiments in artificial settings:
1. Evidence obtained from a randomised experiment performed in an
artificial setting.
2. Evidence obtained from case series,
either post-test or pre-test/post-test.
3. Evidence obtained from a quasi-random experiment performed in an
artificial setting.
5. Evidence obtained from expert opinion based on theory or consensus.
Also, the studies will be examined to see if they contain any type of bias.
For this, the same types of biases will be used as described by
Kitchenham@kitchenham2004procedures:
- Selection/Allocation bias: Systematic difference between comparison groups
with respect to treatment.
- Performance bias: Systematic difference is the conduct of comparison groups
apart from the treatment being evaluated.
- Measurement/Detection bias: Systematic difference between the groups in how
outcomes are ascertained.
- Attrition/Exclusion bias: Systematic differences between comparison groups in
terms of withdrawals or exclusions of participants from the study sample.
The studies will be labeled by their quality level and possible biases.
This information can be used during the Data Synthesis phase
to weigh the importance of individual studies [@kitchenham2004procedures].
#### Data Extraction
To accurately capture the information contributed by each publication in our
survey, we will use a systematic approach to extracting data.
To guide this process, we will be using a data extraction form which describes
what aspects of a publication are crucial to record.
Besides general publication information (title, author etc.), the form contains
questions that are based on our defined research questions.
Furthermore, the form contains a section for quantitative research, where
aspects such as population and evaluation will be documented.
The form that is used for this is shown below:
```
General information:
- Name of person extracting data:
- Date form completed (dd/mm/yyyy):
- Publication title:
- Author information:
- Publication type:
- Conference/Journal:
- Type of study:
What practices in release engineering does this publication mention?
Are these practices to be classified under dated, state of the art or state of
the practice? Why?
What open challenges in release engineering does this publication mention?
What research gaps does this publication contain?
Are these research gaps filled by any other publications in this survey?
Quantitative research publications:
- Study start date:
- Study end date or duration:
- Population description:
- Method(s) of recruitment of participants:
- Sample size:
- Evaluation/measurement description:
- Outcomes:
- Limitations:
- Future research:
Notes:
```
#### Data Synthesis
To summarize the contributions and limitations of each of the included
publications, we will apply a descriptive synthesis approach.
In this part of our survey, we will compare the data that was extracted of the
included publications.
Publications with similar findings will be grouped and evaluated, and
differences between groups of publications will be structured and elaborated on.
In this we will compare them using specifics such as their study types, time of
publication and study quality.
If the extracted data allows for a structured tabular visualization of
similarities and differences between publications this we serve as an additional
form of synthesis. However, this depends on the final included publications of
this survey.
#### Included and Excluded Studies
**Included:**
- @adams2016a
- @castelluccio2017a
- @cesar2017a
- @claes2017a
- @da2014a
- @da2016a
- @dyck2015a
- @fujibayashi2017a
- @karvonen2017a
- @kerzazi2013a
- @khomh2015a
- @laukkanen2017a
- @laukkanen2018a
- @mantyla2015a
- @plewnia2014a
- @poo-caamano2016a
- @rodriguez2017a
- @souza2015a
- @teixeira2017a
**Excluded:**
- @khomh2012a has been excluded, because it presents the same results as
@khomh2015a, while the latter is more extensive because it is a journal article
instead of a conference article.
- @kaur2019a has been excluded, because we could not obtain the actual paper
since it has not yet been officially released.
### Raw Extracted Data
#### Understanding the impact of rapid releases on software quality -- The Case of Firefox
Reference: @khomh2015a
General information:
- Name of person extracting data: Maarten Sijm
- Date form completed: 27-09-2018
- Author information: Foutse Khomh, Bram Adams, Tejinder Dhaliwal, Ying Zou
- Publication type: Paper in Conference Proceedings
- Conference: Mining Software Repositories (MSR)
- Type of study: Quantitative, empirical case study
What practices in release engineering does this publication mention?
- Changing from traditional to rapid release cycles in Mozilla Firefox
Are these practices to be classified under dated,
state of the art or state of the practice? Why?
- State of the practice, because they study Firefox
and Firefox is still using rapid release cycles.
However, it is dated because the data is six years old.
What open challenges in release engineering does this publication mention?
- More case studies are needed
What research gaps does this publication contain?
- More case studies are needed
Are these research gaps filled by any other publications in this survey?
-
Quantitative research publications:
- Study start date: 01-01-2010 (Firefox 3.6)
- Study end date or duration: 20-12-2011 (Firefox 9.0)
- Population description: Mozilla Wiki, VCS, Crash Repository, Bug Repository
- Method(s) of recruitment of participants: N/A (case study)
- Sample size: 25 alpha versions, 25 beta versions,
29 minor versions and 7 major versions.
Amount of bugs/commits/etc. is not specified.
- Evaluation/measurement description: Wilcoxon rank sum test
- Outcomes:
- With shorter release cycles,
users do not experience significantly more post-release bugs
- Bugs are fixed faster
- Users experience these bugs earlier during software execution
(the program crashes earlier)
- Limitations: Results are specific to Firefox
- Future research: More case studies are needed
#### On the influence of release engineering on software reputation
Reference: @plewnia2014a
General information:
- Name of person extracting data: Maarten Sijm
- Date form completed: 27-09-2018
- Author information: Christian Plewnia, Andrej Dyck, Horst Lichter
- Publication type: Paper in Conference Proceedings
- Conference: 2nd International Workshop on Release Engineering
- Type of study: Quantitative, empirical case study on multiple software
What practices in release engineering does this publication mention?
- Rapid releases
Are these practices to be classified under dated,
state of the art or state of the practice? Why?
- Dated practice, data is from before 2014
What open challenges in release engineering does this publication mention?
- Identifying software reputation can better be done using a qualitative study.
What research gaps does this publication contain?
- Identifying software reputation can better be done using a qualitative study.
Are these research gaps filled by any other publications in this survey?
-
Quantitative research publications:
- Study start date: Q3 2008
- Study end date or duration: Q4 2013
- Population description: Chrome, Firefox, Internet Explorer
- Method(s) of recruitment of participants: N/A (case study)
- Sample size: 3 browsers
- Evaluation/measurement description: No statistical analysis,
just presenting market share results
- Outcomes:
- Chrome's market share increased after adopting rapid releases
- Firefox's market share decreased after adopting rapid releases
- IE's market share decreased
- Limitations:
- Identifying software reputation can better be done
using a qualitative study.
- Future research:
- Identifying software reputation can better be done
using a qualitative study.
#### On rapid releases and software testing: a case study and a semi-systematic literature review
Reference: @mantyla2015a
General information:
- Name of person extracting data: Maarten Sijm
- Date form completed: 28-09-2018
- Author information: Mäntylä, Mika V. and Adams, Bram and Khomh, Foutse and Engström, Emelie
and Petersen, Kai
- Publication type: Journal/Magazine Article
- Journal: Empirical Software Engineering
- Type of study: Empirical case study and semi-systematic literature review
What practices in release engineering does this publication mention?
- Impact of rapid releases on testing effort
Are these practices to be classified under dated,
state of the art or state of the practice? Why?
- State of the practice for the case study
- State of the art for the literature review
What open challenges in release engineering does this publication mention?
- Future work should focus on empirical studies of these factors
that complement the existing qualitative observations
and perceptions of rapid releases.
What research gaps does this publication contain?
- See open challenges
Are these research gaps filled by any other publications in this survey?
-
Quantitative research publications:
- Study start date: June 2006 (Firefox 2.0)
- Study end date or duration: June 2012 (Firefox 13.0)
- Population description: System-level test execution data
- Method(s) of recruitment of participants: N/A (case study)
- Sample size: 1,547 unique test cases, 312,502 executions,
performed by 6,058 individuals on 2,009 software builds,
22 OS versions and 78 locales.
- Evaluation/measurement description: Wilcoxon rank-sum test, Cliff's delta,
Cohen's Kappa for Firefox Research Question (FF-RQ) 5.
- Outcomes (FF-RQs; RR = rapid release; TR = traditional release):
1. RRs perform more test executions per day, but these tests focus on a
smaller subset of the test case corpus.
2. RRs have less testers, but they have a higher workload.
3. RRs test fewer, but larger builds.
4. RRs test fewer platforms in total,
but test each supported platform more thoroughly.
5. RRs have higher similarity of test suites and testers
within a release series than TRs had.
6. RR testing happens closer to the release date and is more continuous,
yet these findings were not confirmed by the QA engineer.
- Limitations:
- Study measures correlation, not causation
- Not generalizable, as it is a case study on FF
- Future research: More empirical studies
Semi-systematic literature survey:
- Study date: Unknown (before 2015)
- Population description: Papers with main focus on:
- Rapid Releases (RRs)
- Aspect of software engineering largely impacted by RRs
- An agile, lean or open source process having results of RRs
- Excluding: opinion papers without empirical data on RRs
- Method(s) of recruitment of participants: Scopus queries
- Sample size: 24 papers
- Outcomes:
- Evidence is scarce.
Often RRs are implemented as part of agile adoption.
This makes it difficult to separate the impact of RRs
from other process changes.
- Originates from several software development paradigms: Agile, FOSS, Lean,
internet-speed software development
- Prevalence
- Practiced in many software engineering domains, not just web applications
- Between 23% and 83% of practitioners do RRs
- (Perceived) Problems:
- Increased technical debt
- RRs are in conflict with high reliability and high test coverage
- Customers might be dipleased with RRs (many updates)
- Time-pressure / Deadline oriented work
- (Perceived) Benefits:
- Rapid feedback leading to increased quality focus of the devs and testers
- Easier monitoring of progress and quality
- Customer satisfaction
- Shorter time-to-market
- Continuous work / testing
- Enablers:
- Sequential development where multiple releases are under work simultaneously
- Tools for automated testing and efficient deployment
- Involvement of product management and productive customers
- Limitations:
- Not all papers that present results about RRs,
have "rapid release" mentioned in the abstract.
- Future research:
- Systematically search for agile and lean adoption papers
#### Release management in free and open source software ecosystems
Reference: @poo-caamano2016a
General information:
- Name of person extracting data: Maarten Sijm
- Date form completed: 28-09-2018
- Author information: Germán Poo-Caamaño
- Publication type: PhD Thesis
- Type of study: Empirical case study on two large-scale FOSSs: GNOME and OpenStack
What practices in release engineering does this publication mention?
- Communication in release engineering
Are these practices to be classified under dated,
state of the art or state of the practice? Why?
- State of the practice, because case study
What open challenges in release engineering does this publication mention?
- Is the ecosystem [around the studied software] shrinking or expanding?
- How have communications in the ecosystem changed over time?
What research gaps does this publication contain?
- More case studies are needed
Are these research gaps filled by any other publications in this survey?
-
Quantitative research publications (GNOME):
- Study start date: January 2009 (GNOME 2.x)
- Study end date or duration: August 2011 (GNOME 3.x)
- Population description: Mailing lists
- Method(s) of recruitment of participants:
GNOME's website recommends this channel of communication.
IRC is also recommended, but its history is not stored.
- Sample size: 285 mailing lists, 6947 messages, grouped into 945 discussions.
- Evaluation/measurement description: Counting
- Outcomes:
- Developers also communicate via blogs, bug trackers, conferences,
and hackfests.
- The Release Team has direct contact with almost all participants
in the mailing list
- The tasks of the Release Team:
- defining requirements of GNOME releases
- coordinating and communicating with projects and teams
- shipping a release within defined quality and time specifications
- Major challenges of the Release Team:
- coordinate projects and teams of volunteers without direct power over them
- keep the build process manageable
- monitor for unplanned changes
- monitor for changes during the stabilization phase
- test the GNOME release
- Limitations:
- Only mailing list was investigated, other channels were not
- Possible subjective bias in manually categorizing email subjects
- Not very generalizable, as it's just one case study
- Future research:
- Fix the limitations
Quantitative research publications (OpenStack):
- Study start date: May 2012
- Study end date or duration: July 2014
- Population description: Mailing lists
- Method(s) of recruitment of participants: Found on OpenStack's website
- Sample size: 47 mailing lists, 24,643 messages, grouped into 7,650 discussions.
Filtered data: 14,486 messages grouped into 2,682 discussions.
- Evaluation/measurement description: Counting
- Outcomes:
- Developers communicate via email, blogs, launchpad, wiki, gerrit,
face-to-face, IRC, video-conferences, and etherpad.
- Project Team Leaders and the Release Team members are the key players
in the communication and coordination across projects in the context of
release management
- The tasks for the Release Team and Project Team Leaders:
- defining the requirements of an OpenStack release
- coordinating and communicating with projects and teams to reach
the objectives of each milestone
- coordinating feature freeze exceptions at the end of a release
- shipping a release within defined quality and time specifications
- Major challenges of these teams:
- coordinate projects and teams without direct power over them
- keep everyone informed and engaged
- decide what becomes part of of the integrated release
- monitor changes
- set priorities in cross-project coordination
- overcome limitations of the communication infrastructure
- Limitations:
- Only studies mailing list, to compare with GNOME case study
- Possible subjective bias in manually categorizing email subjects
- Not very generalizable, as it's just one case study