-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
2748 lines (2746 loc) · 128 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<!-- Last Published: Thu Sep 08 2022 21:27:35 GMT+0000 (Coordinated Universal Time) -->
<html
data-wf-page="630554c6691ab923537ddaa4"
data-wf-site="62f68e72843db6420ca4bfdf"
>
<head>
<meta charset="utf-8" />
<title>Tailslide</title>
<meta content="width=device-width, initial-scale=1" name="viewport" />
<link href="css/normalize.css" rel="stylesheet" type="text/css" />
<link href="css/webflow.css" rel="stylesheet" type="text/css" />
<link href="css/tailslide.css" rel="stylesheet" type="text/css" />
<link href="css/syntax.css" rel="stylesheet" type="text/css" />
<link
rel="stylesheet"
href="https://cdn.jsdelivr.net/npm/prismjs@1.28.0/plugins/line-numbers/prism-line-numbers.min.css"
/>
<script
src="https://ajax.googleapis.com/ajax/libs/webfont/1.6.26/webfont.js"
type="text/javascript"
></script>
<script type="text/javascript">
WebFont.load({
google: {
families: [
"Poppins:regular,500,600,700,800",
"Source Code Pro:regular",
],
},
});
</script>
<script type="text/javascript">
!(function (o, c) {
var n = c.documentElement,
t = " w-mod-";
(n.className += t + "js"),
("ontouchstart" in o ||
(o.DocumentTouch && c instanceof DocumentTouch)) &&
(n.className += t + "touch");
})(window, document);
</script>
<link href="images/favicon.png" rel="shortcut icon" type="image/x-icon" />
<link href="images/256_logo.jpg" rel="apple-touch-icon" />
</head>
<body data-scroll-time="0">
<div
data-collapse="medium"
data-animation="default"
data-duration="400"
data-easing="ease-in"
data-easing2="ease-out"
data-doc-height="1"
role="banner"
class="nav w-nav"
>
<div class="nav-container w-container">
<a href="#landing" class="logo w-nav-brand"
><img
alt="Nav Logo"
width="180"
data-w-id="d33e92b0-a0cc-9a61-8693-e9d36fee16d5"
src="images/logo_white_left.svg"
/></a>
<nav role="navigation" class="nav-menu w-nav-menu">
<a href="#about" class="nav-link w-nav-link">About</a>
<a href="#case-study" class="nav-link w-nav-link">Case Study</a>
<a href="#presentation" class="nav-link w-nav-link">Presentation</a>
<a href="#team" class="nav-link w-nav-link">Team</a>
<a href="/documentation" class="nav-link w-nav-link">Docs</a>
<a
href="https://github.com/tailslide-io#"
target="_blank"
class="w-inline-block"
><img
alt="Github Nav Icon"
width="35"
loading="lazy"
src="images/github_white.svg"
class="github-logo"
/></a>
</nav>
<div class="w-nav-button">
<div class="w-icon-nav-menu"></div>
</div>
</div>
</div>
<div id="landing" class="landing-card">
<img
src="images/arrow_down.svg"
loading="lazy"
style="
-webkit-transform: translate3d(0, 0px, 0) scale3d(1, 1, 1) rotateX(0)
rotateY(0) rotateZ(0) skew(0, 0);
-moz-transform: translate3d(0, 0px, 0) scale3d(1, 1, 1) rotateX(0)
rotateY(0) rotateZ(0) skew(0, 0);
-ms-transform: translate3d(0, 0px, 0) scale3d(1, 1, 1) rotateX(0)
rotateY(0) rotateZ(0) skew(0, 0);
transform: translate3d(0, 0px, 0) scale3d(1, 1, 1) rotateX(0)
rotateY(0) rotateZ(0) skew(0, 0);
opacity: 0;
"
data-w-id="a124c267-8c76-5e56-0069-e18fe6f053e7"
alt="Arrow Down"
class="down-arrow"
/>
<div class="landing-wrap">
<div class="landing-left">
<img
src="images/graphic_white.svg"
width="800"
alt="Tailslide Logo"
loading="eager"
class="landing-logo"
/>
</div>
<div class="landing-right">
<img
src="images/name_white.svg"
loading="eager"
width="372"
alt="Landing Page Logo Text"
class="landing-logo-title"
/>
<h2 class="heading-6">
<strong class="bold-text-light">An open-source </strong
><span class="text-span"
><strong class="bold-text-dark"
>feature flag framework</strong
></span
><strong class="bold-text-light">
for easier code deployment with automated </strong
><span class="text-span"
><strong class="bold-text-dark">failure protection</strong></span
><strong class="bold-text-light">.</strong>
</h2>
<a href="#case-study" class="button w-button">Read the Case Study</a>
</div>
</div>
</div>
<div
id="about"
data-w-id="52df9d9f-fef8-7b24-3a9c-31facb64d911"
class="flag-card"
>
<div class="card-wrapper">
<div class="card-text-div">
<h2 class="card-header-light">Feature Flag Management</h2>
<p class="card-p-light">
A feature flag system purpose-built for new feature deployment with
granular control over rollout and developer whitelists.
</p>
</div>
<div class="card-image-div">
<img
src="images/flag_dashboard2.webp"
sizes="(max-width: 767px) 90vw, (max-width: 1439px) 48vw, 550px"
srcset="
images/flag_dashboard2-p-500.png 500w,
images/flag_dashboard2-p-800.png 800w,
images/flag_dashboard2.webp 1039w
"
loading="lazy"
width="724"
alt="Flag Dashboard View"
class="screenshot"
/>
</div>
</div>
</div>
<div data-w-id="c64219d1-5c9c-bbde-2f99-11ff9b45fafd" class="circuit-card">
<div class="card-wrapper">
<div class="card-image-div">
<img
src="images/graph_dashboard2.webp"
loading="lazy"
srcset="
images/graph_dashboard2-p-500.png 500w,
images/graph_dashboard2-p-800.png 800w,
images/graph_dashboard2.webp 1039w
"
alt="Graph Dashboard View"
sizes="(max-width: 767px) 90vw, (max-width: 1439px) 48vw, 550px"
class="screenshot"
/>
</div>
<div class="card-text-div">
<h2 class="card-header-dark">Automated Failure Protection</h2>
<p class="card-p-dark">
Give developers peace of mind and reduce MTTR with a built-in
Circuit Breaker and fully automated recovery.
</p>
</div>
</div>
</div>
<div data-w-id="42dbcbdb-a4e8-853f-b58e-3918f86cdebd" class="sdk-card">
<div class="card-wrapper">
<div class="card-text-div">
<h2 class="card-header-light">Easy Integration</h2>
<p class="card-p-light">
SDK available in popular back-end languages for simple configuration
within your app. Flag Evaluation and Circuit Observation are a
method call away.
</p>
</div>
<div class="card-image-div">
<div class="code-block">
<div class="code-block-header">
<div id="sdk-header" class="text-block">tailslideSDK.js</div>
</div>
<div class="sdk-icon-div w-richtext">
<div class="w-embed">
<pre
class="line-numbers"
><code class="language-js" id="sdk-code">const FlagManager = require('tailslide');
const config = {
natsServer: 'nats://localhost:4222',
natsStream: 'flags_ruleset',
appId: 1,
userContext: '375d39e6-9c3f-4f58-80bd-e5960b710295',
sdkKey: 'myToken',
redisHost: 'http://localhost',
redisPort: 6379,
};
const manager = new FlagManager(config);
await manager.initialize();
</code></pre>
</div>
</div>
</div>
<div class="div-block-3">
<img
src="images/js-icon.webp"
loading="lazy"
alt="Javascript Icon"
id="sdk-js"
class="sdk-icon"
/><img
src="images/ruby-icon.webp"
loading="lazy"
alt="Ruby Icon"
id="sdk-ruby"
class="sdk-icon"
/><img
src="images/python-icon.webp"
loading="lazy"
alt="Python Icon"
id="sdk-python"
class="sdk-icon"
/><img
src="images/go-icon.webp"
loading="lazy"
alt="Go Icon"
id="sdk-go"
class="sdk-icon"
/>
</div>
</div>
</div>
</div>
<div
data-w-id="e6b297b3-47a4-eaa0-a422-626df36637fa"
class="deployment-card"
>
<div class="card-wrapper">
<div class="card-image-div">
<video
autoplay
loop
muted
playsinline
width="550px"
class="deploy-img"
>
<source src="images/tailslide_terminal_loop.mp4" type="video/mp4" />
Your browser does not support the HTML5 Video element.
</video>
<!-- <img
src="images/tailslide_terminal-copy-3.svg"
loading="lazy"
alt="Dock Deploy Example"
width="550"
class="deploy-img"
/> -->
</div>
<div class="card-text-div">
<h2 class="card-header-dark">Simple Deployment</h2>
<p class="card-p-dark">
Deploy Tailslide on any cloud provider or your own infrastructure
with a simple docker-compose command.
</p>
</div>
</div>
</div>
<section id="case-study" class="case-study wf-section">
<div class="case-study-container">
<div class="case-study-wrapper">
<div
id="w-node-_932a788b-82c3-a588-cd76-f478efbb41e2-537ddaa4"
class="case-study-toc"
>
<div class="toc-header">Table of Contents</div>
<div id="toc" class="toc"></div>
</div>
<div
id="w-node-_932a788b-82c3-a588-cd76-f478efbb41e6-537ddaa4"
class="case-study-text"
>
<div
target="_blank"
id="content"
class="casestudy w-node-_932a788b-82c3-a588-cd76-f478efbb41e7-537ddaa4 w-richtext"
>
<h1>Case Study</h1>
<h2>1. Introduction</h2>
<h3>1.1 Tailslide</h3>
<figure
style="max-width: 200px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-logo.webp"
loading="lazy"
alt="Tailslide Logo"
/>
</div>
<figcaption>Fig 1.1 Tailslide logo</figcaption>
</figure>
<p>
Tailslide is an open-source, lightweight, self-hosted feature
flag management solution designed for organizations with a
microservices architecture to easily and safely deploy new
backend features with built-in circuit-breaking. Tailslide
monitors the performance of feature flags and leverages a
customizable circuit-breaking pattern to toggle off problematic
features and direct users to a pre-defined fallback service or
feature. If circuits trip, indicating that features are not
working as intended - developers are immediately notified via a
Slack notification, so they can respond appropriately. Tailslide
speeds up the deployment cycle without sacrificing reliability.
</p>
<p>
Using Tailslide, developers can perform "dark
launches" - allowing for experimentation of new features in
a live production environment, where only internal test accounts
or beta users are served the new feature. Tailslide also allows
for the safe and gradual rollout of individual features to a
subset of the user base, with the ability to easily ramp up,
ramp down or shut off all traffic to a specified feature.
</p>
<figure
style="max-width: 450px"
class="w-richtext-align-center w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-components.webp"
loading="lazy"
alt="Tailslide components"
/>
</div>
<figcaption>
Fig 1.2 Tailslide architecture, user interface dashboard, and
supported server SDKs
</figcaption>
</figure>
<h3>1.2 Deployment and Feature Release</h3>
<figure
style="max-width: 450px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-release-cycle.webp"
loading="lazy"
alt="Software release cycle"
/>
</div>
<figcaption>Fig 1.3 Software release cycle</figcaption>
</figure>
<p>
To first understand the “why” behind feature flags, it’s
important to first understand the context surrounding feature
flags. Let’s first describe deployment and feature release,
before discussing the traditional software release cycle and why
software teams should strive toward more frequent deployments.
</p>
<p>
<strong>Deployment</strong> is the process of distributing code
from a development or staging environment into a production
environment [1].
</p>
<p>
<strong>Feature Release</strong> refers to making features in
deployed code available to users [1].
</p>
<p>
As Tailslide’s focus is on server-side feature flags, in this
write up we will use <strong>feature</strong> to refer to any
new functionality or service being developed on the backend of a
web application.
</p>
<figure
class="w-richtext-align-center w-richtext-figure-type-image"
>
<div>
<video
autoplay
loop
muted
playsinline
preload="metadata"
style="max-width: 350px"
>
<source
src="images/cs-deployment-loop.mp4"
type="video/mp4"
/>
Your browser does not support the HTML5 Video element.
</video>
<video
autoplay
loop
muted
playsinline
preload="metadata"
style="display: block; margin: auto; max-width: 200px"
>
<source src="images/cs-release-loop.mp4" type="video/mp4" />
Your browser does not support the HTML5 Video element.
</video>
</div>
<figcaption>Fig 1.4 Deployment and feature release</figcaption>
</figure>
<h3>1.3 Traditional Software Release Cycle</h3>
<figure
style="max-width: 300px"
class="w-richtext-align-floatleft w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-deployment.webp"
loading="lazy"
alt="Code Deployment"
/>
</div>
<figcaption>
Fig 1.5 Once code is deployed, all users receive the new
features.
</figcaption>
</figure>
<p>
In a traditional software release cycle, feature release occurs
simultaneously with deployment. Once code is deployed, all users
will receive the features within that feature release. Although
new features may appear to work in a staging environment,
unexpected issues often arise in production - such as when a
server fails to handle higher-than-expected user traffic.
</p>
<p>
The downside of releasing features alongside deployment is that
when features break, developers must scramble to deploy hotfixes
on top of the problematic deployment or roll back the deployment
altogether. Even if there is only one offending feature, the
entire deployment could have to be rolled back.
</p>
<p>
This leads to unpredictable repair wait times or extended
service downtime, negatively impacting both the development team
and the end user. The tight coupling between deployment and
feature release means that developers have little control over
the active state of their new features after deployment. As a
result, traditional deployment and feature release processes
contribute to long software release cycles with negative
consequences [2].
</p>
<h3>1.4 Benefits of Frequent Deployment</h3>
<p>
Delivering new features in a timely and reliable manner is
critical for the success of a software company. This often
requires efficient software development teams capable of
frequently deploying new features while guaranteeing service
uptime for end users [3]. One way to expedite the deployment
process is by allowing teams to safely deploy into production on
demand, allowing them to experiment and see how their code
behaves in a production environment.
</p>
<p>
In order to perform this safely, two criteria must be satisfied:
</p>
<ul role="list">
<li>
First, the impact of deployments on end users should be
controllable, so that only target users receive the new
features without affecting the experience of regular users.
</li>
<li>
Second, the deployment process should have a low mean time to
recovery (MTTR) [4] so that service outages from failed
deployment do not impact business operations.
</li>
</ul>
<h4>1.4.1 Deployment Flexibility</h4>
<p>
To meet these criteria, it is important to separate deployment
from feature release. Feature flags are one way to separate
deployment from feature release. Feature flags work by
abstracting feature releases as conditionals in the application
code. This means developers introduce feature release logic to
target which users receive the new features and control the
impact radius of their new features. This abstraction allows
developers to instantly shut off all traffic to problematic new
features without needing additional hotfixes or deployment
rollbacks.
</p>
<h4>1.4.2 Deployment Reliability</h4>
<p>
Deployment reliability is just as important as frequency. A tool
that implements the well-established circuit breaking pattern
can automatically turn new features off when problems arise,
thus lowering MTTR. This automatic fail-safe helps to reduce the
burden of monitoring the performance of new features on
development teams, allowing them to safely release code into
production without worrying about service interruptions. They
can rest assured that fallbacks are in place when new features
fail.
</p>
<h4>1.4.3 Deployment Feedback</h4>
<p>
Real-world feedback is especially informative in a microservices
architecture, where testing the interactions among microservices
in staging environments can be challenging and time-consuming.
Allowing new features to fail early but safely in the production
environment enables developers to observe how their systems
behave in unexpected production scenarios that simulated tests
cannot anticipate, such as unexpectedly high traffic,
infrastructure problems, or unpredictable user behavior. Testing
in production is an invaluable resource for developers to learn
from, in order to iterate and improve on new features quickly.
</p>
<h2>2. Hypothetical</h2>
<p>
To illustrate how feature flags can improve deployment, let’s
describe the challenges that a hypothetical company, Healthbar,
faced before discovering feature flags as a deployment strategy.
</p>
<figure
style="max-width: 400px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-healthbar.webp"
loading="lazy"
alt="Healthbar Teams"
/>
</div>
<figcaption>Fig 2.1 Healthbar Teams</figcaption>
</figure>
<p>
Healthbar is a startup that provides platform-as-a-service
(PaaS) related to information technology for hospitals, medical
and healthcare companies. Healthbar uses a microservices
architecture. Due to the severe consequences of service downtime
for its customers, Healthbar takes precautions to avoid
widespread service blackouts. Healthbar wants to ensure that its
customers are being serviced at all times. Healthbar’s processes
include deploying code in automated test environments and a
robust code review approval process, involving multiple rounds
of approval amongst developers, testers, and managers. Although
these safeguards have been effective in minimizing service
blackouts, they have slowed deployment, and teams often spend
over a month between deployments.
</p>
<p>
Healthbar’s management wants to better serve customer needs as
more companies enter the space. They encourage engineering teams
to release new features faster, to retain and grow existing
market share. Currently, using the traditional deployment
strategy greatly limits the teams’ progress.
</p>
<h3>2.1 Canary Deployment</h3>
<p>
Recently, Healthbar discovered Canary Deployment, a deployment
strategy that releases an application or service incrementally
to a subset of users, before rolling out the change to all users
if no issues arise [5]. With canary deployments, two versions of
an application are deployed onto two separate production
environments: the latest production release, and the canary
version (version with new features). A load balancer is used to
direct some traffic to the canary, with the remaining traffic
going to the latest release. If no issues occur, the traffic
directed to the canary increases, until all users are routed to
the canary. If problems arise with the canary - the rollback
strategy is simply to reroute all users back to the current
version, until problems with the canary have been fixed. With
Canary Deployments, the determination of which version a user is
served occurs at the infrastructure layer, via the load
balancer.
</p>
<figure
style="max-width: 800px"
class="w-richtext-align-center w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-canary.webp"
loading="lazy"
alt="Canary Deployment Diagram"
/>
</div>
<figcaption>
Fig 2.2 Initially, only a few users receive and test the new
features. Gradually, more users are directed toward new
features. If errors occur, users are redirected back to the
old features.
</figcaption>
</figure>
<h4>2.1.1 Benefits of Canary Deployments</h4>
<p>
Canary Deployments help Healthbar to increase its deployment
frequency and allow them to safely test its new services in
production. By controlling the traffic to a canary to only a
small subset of users with a gradual rollout, Healthbar reduces
the risk of a problematic deployment leading to a widespread
service blackout that affects all users. Healthbar loosens some
of its stringent approval processes, resulting in more frequent
deployments. Frequent deployment means smaller changes in each
deployment, fewer things that may break, and fewer checklists
for teams to cross off or monitor. Management is happy to see an
increase in productivity. With the new features becoming more
mature, it wants the teams to integrate those new features into
core business offerings.
</p>
<h4>2.1.2 Shortcomings of Canary Deployment</h4>
<p>
Canary Deployments still suffer from a tight coupling of
deployment and feature release, a similar limitation faced in a
traditional software release. Developers lack the ability to
control the state of individual new features after deployment.
If the canary consists of multiple new features, but only one
non-essential feature is problematic, there is no way to switch
off traffic routed towards only that problematic feature.
Instead, they must reroute all users from the canary back to the
existing version.
</p>
<figure
style="max-width: 450px"
class="w-richtext-align-floatleft w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-deploy-combo.webp"
loading="lazy"
alt="Deployment Combinations"
/>
</div>
<figcaption>
Fig 2.3 Each combination of features for feature integration
requires additional deployments and infrastructures.
</figcaption>
</figure>
<p>
Another shortfall of canary deployments arises in feature
integration. As the number of features involved in integration
tasks increases, the number of canary deployments also grows.
This is because experimenting with different combinations of
features, requires multiple deployments and additional
infrastructures for hosting those deployments, increasing both
complexity and costs of deployment.
</p>
<p>
For instance, a potential solution may involve any combination
of Features A, B, and C, all of which would require their own
separate canary deployment for testing. Ultimately, due to the
nature of microservices architectures, testing the integration
of new features in production is the only way to gain assurance
that features are functioning correctly under real-world
conditions, such as high traffic load, infrastructure problems,
and unpredictable user behavior.
</p>
<p>
With Healthbar’s limited resources, the developers need another
deployment strategy that offers ease of feature integration in a
single deployment, the granular control to selectively toggle
individual features on or off without entire version rollback,
and the ability to safely and reliably test in production with
automatic fail-safes.
</p>
<h3>2.2 Feature Flags</h3>
<h4>2.2.1 Benefits of Feature Flags</h4>
<figure
style="max-width: 275px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-ifelse.webp"
loading="lazy"
alt="If Else Code Example"
/>
</div>
<figcaption>Fig 2.4 Conditional logic</figcaption>
</figure>
<p>
With further research and discussion within Healthbar teams,
they came up with Feature Flags. Feature flags decouple feature
release from infrastructure [6]. At its simplest, feature flags
embed conditional logic into applications which allow developers
to dynamically control the on or off state of individual
features at runtime. Fundamentally, feature flags allow
developers to decouple deployment from feature release.
Deployment is now only about moving code into production, and
feature release solely depends on runtime logic. Since feature
flags evaluate feature release at the application layer,
developers are free to test different combinations of their
features in their feature integration tasks with just a single
deployment.
</p>
<p>
With feature flags, the developers only need to make a single
deployment that covers both the new and current features on the
same infrastructure. The determination of which version a user
is served occurs at the application layer, based on the runtime
logic. In this way, deployment is tied to the underlying
infrastructure, and feature release is manipulated by server
application logic.
</p>
<h4>2.2.2 Additional Feature Flags Capabilities</h4>
<p>
Developers at Healthbar identify additional elements they are
seeking in a feature flag solution in order to safely and
reliably test in production, as their features move through
varying levels of maturity.
</p>
<ul role="list">
<li>
<strong>Targeted Testing</strong> - In the early stages, the
developers want the ability to expose specific internal test
users with the new features, without affecting regular users.
In addition, developers want granular control over which
specific features to toggle on or off in order to test
different combinations of features. While a canary deployment
can target internal testers, the full feature set must either
be on or off, which limits developers’ ability to test new
features.
</li>
<li>
<strong>Gradual Rollout</strong> - From their experience with
canary deployments, the developers now also want the power to
safely and gradually roll out an individual feature to a
subset of users, and slowly expose it to the entire user base
if no issues arise. This is similar to the rollout strategy
for a canary deployment, except the rollout control is
finer-grained, and occurs at the individual feature level.
</li>
<li>
<strong>Automated Failsafe</strong> - Once ready for release
to the entire user-base, Healthbar’s priority remains focussed
on the reduction of widespread service blackout. If calls to a
new feature or service fail unexpectedly - perhaps slower
response times due to higher-than-usual traffic - Healthbar
would like an automated fault-tolerant solution, that will
direct users to a pre-defined fallback service or feature. The
automated nature of this failsafe process reduces the burden
of monitoring the performance of new features on development
teams, allowing them to safely release code into production
without worrying about service interruptions. One stability
pattern that provides this automated failover is the Circuit
Breaker pattern.
</li>
</ul>
<h3>2.3 Circuit Breaker</h3>
<p>
Healthbar is concerned with resiliency when deploying new
features and quick recovery when services do fail, as calls to a
deployed feature over a network introduce an additional point of
failure. Whether the feature itself fails or the connection
between services degrades, this can lead to a poor end-user
experience or further outages within a distributed application.
</p>
<figure
style="max-width: 350px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-circuit-pattern.webp"
loading="lazy"
alt="Circuit Breaker Pattern"
/>
</div>
<figcaption>
Fig 2.5 The Circuit Breaker pattern defines 3 Circuit States:
Closed, Open and Half-Open, corresponding to full traffic, no
traffic, and reduced traffic being directed to a service. The
Half-Open state is an automated recovery process.
</figcaption>
</figure>
<p>
Implementation of a stability pattern can help Healthbar protect
against these failures and safely recover. Stability patterns
are used to promote resiliency in distributed systems [7] - the
Circuit Breaker is a stability pattern popularized by Michael
Nygard in his book ‘Release it,’ [8] and it is built to protect
against failures as well as recover from them.
</p>
<p>
A Circuit Breaker in software is modeled after an electrical
circuit breaker and is a component responsible for monitoring
the flow of information from one service to another - in this
case, keeping track of the failure rate of requests between two
services or within a service. A Circuit Breaker controls the
flow of information by changing the state of the circuit,
cutting off all traffic to a service if failure rate reaches a
specified threshold, and then gradually self-testing the
connection to a failed service in order to re-close the circuit,
allowing traffic to resume to the now-functioning service if a
suitable rate of requests is successful.
</p>
<h4>2.4 Feature Flag Landscape</h4>
<p>
Healthbar developers are excited to find that there are existing
feature flag solutions that meet their deployment needs. Some
even come with a full set of observability, monitoring, and
safety features. Because Healthbar is a small company with
limited resources, the teams have to justify to management which
feature flag solution to pursue.
</p>
<figure
style="max-width: 400px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-competitors.webp"
loading="lazy"
alt="Competitor Comparison"
/>
</div>
<figcaption>
Fig 2.6 Feature flag competitive landscape, from enterprise
solutions such as LaunchDarkly to feature-rich open source
frameworks such as Unleash, and automated failure protection
oriented Tailslide.
</figcaption>
</figure>
<p>
LaunchDarkly is a prominent enterprise solution. It is easy to
set up and has a feature-rich platform, including integration
with observability tools and circuit-breaking capabilities.
However, it costs significantly more than other competing open
source frameworks. Since it is hosted on LaunchDarkly platform,
sensitive patient information may be stored on its cloud.
Furthermore, the solution is not open source, meaning developers
may not be able to configure the software to their exact needs.
</p>
<p>
Open-source frameworks, such as Unleash, typically offer either
a self-hosted option or the use of their cloud platform.
Although easy to set up, they do not offer the circuit-breaking
functionalities Healthbar is seeking.
</p>
<p>
Finally, they found Tailslide, an open-source feature flag
solution that is fully self-hosted and easy to set up. It is
simple to use, and only comes with the essential feature flag
elements they are looking for, such as dark launches and gradual
rollouts. Most importantly, Tailslide includes an automated
fault tolerant solution with built-in circuit-breaking.
Healthbar decides to move forward with Tailslide as their
feature flag management solution.
</p>
<h2>3. Tailslide Architecture</h2>
<h3>3.1 Tailslide Architecture Overview</h3>
<figure
style="max-width: 450px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-ff-arch.webp"
loading="lazy"
alt="Feature Flag Architecture"
/>
</div>
<figcaption>
Fig 3.1 Services involved in Feature Flag solution.
</figcaption>
</figure>
<p>
Tailslide’s architecture was built to facilitate two major uses:
the feature flag ruleset data transmission and automated circuit
breaking.
</p>
<p>
<em
>Note that services in orange are user-provided, and services
in blue are part of Tailslide’s architecture.</em
>
</p>
<p>
The feature flag component is focused on managing CRUD
operations on a per-flag basis, and propagating the feature flag
ruleset data to all user microservices with software development
kits (SDKs) embedded within them. By using conditional branching
logic within the user microservice, the real-time transmission
of feature flag ruleset data allows the developer to dynamically
control the flow of application logic.
</p>
<p>
The circuit breaking component can be broken down into three
major steps, which include: the emission and storage of
success/failure data, the evaluation of error rate against
user-configured thresholds, and the circuit state change
propagation.
</p>
<figure
style="max-width: 450px"
class="w-richtext-align-center w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-circuit-arch.webp"
loading="lazy"
alt="Circuit Breaker Architecture"
/>
</div>
<figcaption>
Fig 3.2 Services involves in Circuit Breaking.
</figcaption>
</figure>
<p>
Broadly, there are four key pieces of Tailslide’s architecture.
These include:
</p>
<ol role="list">
<li>
<strong>Tower</strong> - A full-stack application that handles
the CRUD functionality related to feature flag management.
</li>
<li>
<strong>NATS JetStream</strong> - Connective technology,
enabling asynchronous communication.
</li>
<li>
<strong>Tailslide SDKs</strong> - Provides user microservices
with flag state to ensure appropriate logic evaluation at
runtime and emission of circuit-breaking data.
</li>
<li>
<strong>Aerobat</strong> - A lightweight Node application
responsible for all circuit-breaking logic.
</li>
</ol>
<p>
We will now provide an overview of the role of each component.
</p>
<h4>3.1.1 Tower</h4>
<figure
style="max-width: 125px"
class="w-richtext-align-floatright w-richtext-figure-type-image"
>
<div>
<img
src="images/cs-tower.webp"
loading="lazy"
alt="Tower Architecture"
/>
</div>
<figcaption>Fig 3.3 Tower</figcaption>
</figure>
<p>
Tower is a full-stack application that handles the functionality
related to feature flag management.
</p>
<p>
It consists of a React user interface that allows a user to
perform basic CRUD functionality, such as creating an app and
flag, making edits to flags, toggling a flag on and off, and
viewing logs related to those flags. Each flag has a title, an
optional description, an assigned rollout percentage, a field to
list white-listed users, a toggle to turn the flag on/off with a
single click and a variety of circuit-breaking configuration
settings.
</p>
<p>
Circuit breaking settings include setting the Error Threshold,
the Initial Recovery Percentage, the Recovery Increment
Percentage, the Recovery Delay, the Recovery Rate, and setting
the Recovery Profile as either Linear or Exponential. By
default, circuit breaking is disabled, but can be enabled with a
toggle switch.
</p>
<p>
Within the frontend, a user can also generate SDK Keys for use
in Tailslide, view event logs of a flag’s history, and view