-
Notifications
You must be signed in to change notification settings - Fork 0
/
better-local-development-for-serverless-functions-b96b5a4cfa8f.html
250 lines (196 loc) · 11.9 KB
/
better-local-development-for-serverless-functions-b96b5a4cfa8f.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge" />
<meta name="HandheldFriendly" content="True" />
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<meta name="robots" content="" />
<link href="https://fonts.googleapis.com/css2?family=Source+Code+Pro:ital,wght@0,400;0,700;1,400&family=Source+Sans+Pro:ital,wght@0,300;0,400;0,700;1,400&display=swap" rel="stylesheet">
<link rel="stylesheet" type="text/css" href="./theme/stylesheet/style.min.css">
<link id="pygments-light-theme" rel="stylesheet" type="text/css"
href="./theme/pygments/github.min.css">
<link rel="stylesheet" type="text/css" href="./theme/font-awesome/css/fontawesome.css">
<link rel="stylesheet" type="text/css" href="./theme/font-awesome/css/brands.css">
<link rel="stylesheet" type="text/css" href="./theme/font-awesome/css/solid.css">
<link href="https://shanedowling.com/feeds/all.atom.xml" type="application/atom+xml" rel="alternate" title="Shane Dowling Atom">
<link href="https://shanedowling.com/feeds/all.rss.xml" type="application/rss+xml" rel="alternate" title="Shane Dowling RSS">
<script type="text/javascript">
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'G-Z9P3V6D0SV', 'auto');
ga('send', 'pageview');
</script>
<meta name="author" content="Shane Dowling" />
<meta name="description" content="Lambda is a terrific piece of kit for all the benefits listed on the AWS product page and Serverless is a very useful framework for developing Lambda functions. However, developing serverless applications locally is a total pain if what you're solving isn't totally trivial. When things get complicated and your …" />
<meta name="keywords" content="">
<meta property="og:site_name" content="Shane Dowling"/>
<meta property="og:title" content="Better local development for Serverless Functions"/>
<meta property="og:description" content="Lambda is a terrific piece of kit for all the benefits listed on the AWS product page and Serverless is a very useful framework for developing Lambda functions. However, developing serverless applications locally is a total pain if what you're solving isn't totally trivial. When things get complicated and your …"/>
<meta property="og:locale" content="en_US"/>
<meta property="og:url" content="./better-local-development-for-serverless-functions-b96b5a4cfa8f.html"/>
<meta property="og:type" content="article"/>
<meta property="article:published_time" content="2019-02-11 20:15:38.791000+00:00"/>
<meta property="article:modified_time" content=""/>
<meta property="article:author" content="./author/shane-dowling.html">
<meta property="article:section" content="Tech"/>
<meta property="og:image" content="/images/logo.png">
<title>Shane Dowling – Better local development for Serverless Functions</title>
</head>
<body class="light-theme">
<aside>
<div>
<a href="./">
<img src="/images/logo.png" alt="Shane Dowling" title="Shane Dowling">
</a>
<h1>
<a href="./">Shane Dowling</a>
</h1>
<p>Platform Engineering Manager</p>
<nav>
<ul class="list">
<li>
<a target="_self"
href="./pages/contact.html#contact">
Contact
</a>
</li>
<li>
<a target="_self"
href="./pages/now-page.html#now-page">
Now Page
</a>
</li>
<li>
<a target="_self"
href="./pages/statuslog.html#statuslog">
Statuslog
</a>
</li>
<li>
<a target="_self" href="https://hybridcloudshow.com/" >podcast</a>
</li>
<li>
<a target="_self" href="https://pemonthly.com/" >newsletter</a>
</li>
</ul>
</nav>
<ul class="social">
<li>
<a class="sc-github"
href="http://github.com/shano"
target="_blank">
<i class="fa-brands fa-github"></i>
</a>
</li>
<li>
<a class="sc-mastodon"
rel="me" href="http://social.lol/@sed"
target="_blank">
<i class="fa-brands fa-mastodon"></i>
</a>
</li>
<li>
<a class="sc-linkedin"
href="https://www.linkedin.com/in/shanedow/"
target="_blank">
<i class="fa-brands fa-linkedin"></i>
</a>
</li>
</ul>
</div>
</aside>
<main>
<article class="single">
<header>
<h1 id="better-local-development-for-serverless-functions-b96b5a4cfa8f">Better local development for Serverless Functions</h1>
<p>
Posted on Mon 11 February 2019 in <a href="./category/tech.html">Tech</a>
</p>
</header>
<div>
<p><img alt="" src="images/1__roedigbmFjRYkZobdZWuKg.jpeg"></p>
<p>Lambda is a terrific piece of kit for all the benefits listed on the <a href="https://aws.amazon.com/lambda/">AWS product page</a> and <a href="https://serverless.com/">Serverless</a> is a very useful framework for developing Lambda functions. However, developing serverless applications locally is a total pain if what you're solving isn't totally trivial.</p>
<p>When things get complicated and your Lambda functions start to integrate with other AWS services, things really begin to break down. There are a few things that look like silver-bullets, I'll share them here and explain why they didn't work for me, then give you a working example that I myself struggled to find(hence me writing this).</p>
<h3>Localstack</h3>
<p><a href="https://github.com/localstack/localstack">Localstack</a> is definitely the biggest attempt at a silver bullet here. Basically it emulates a huge chunk of AWS locally and you connect your serverless functions into that. Why I didn't use it:</p>
<ol>
<li>Bits worked, but if you have any sort of half complicated setup(like a region outside us-east-1 or eu-west-1), it will break down. Uploading a whole cloudformation template was a disaster of broken links between services. I got to the point of editing localstack code to try fix issue after issue I was coming across and gave up. The product would be cool if it wasn't trying to do something so super complicated.</li>
<li>It's also a huge resource drain on a developer's machine.</li>
<li>Fundamentally, you shouldn't have to emulate half of AWS to test your code.</li>
</ol>
<h3>Serverless Offline</h3>
<p>Serverless offline is fairly straightforward. It should allow you to spin up your functions locally and call them as needed. Where this fell down for me was that it required you to have HTTP endpoints for your lambda functions, all of our Lambda functions were driven either by SQS events or a Cron. We also instantiate our SNS topics in our code, so it would constantly attempt to create SNS topics and AWS would throw errors.</p>
<p>I also did try using <a href="https://www.npmjs.com/package/serverless-offline-sqs">serverless-offline-sqs</a> but could not get it to drive events into our Lamdba functions.</p>
<p>I was burning a lot of time getting a dev environment setup and I realised that this was fundamentally the wrong approach. I needed to start writing proper unit tests and use mocking to emulate the infrastructure(I mean ideally they'd be isolated properly and infrastructure/functions would be totally ignorant of each other, but that's for another day).</p>
<h3>The Example</h3>
<p>Say we're developing an application that makes use of:</p>
<ul>
<li>Lambda</li>
<li>MongoDB</li>
<li>SQS</li>
<li>SNS</li>
</ul>
<p>Your lambda function connects to your MongoDB, does a thing and fires off occasional SNS notifications/SQS messages to other services.</p>
<p><strong>Lambda</strong></p>
<p>For the actual code being run, I was going to use <a href="https://www.npmjs.com/package/jest">jest</a> to start writing tests. Jest works well and integrates nicely into serverless.</p>
<p><strong>MongoDB</strong></p>
<p>To emulate MongoDB, I'm going to use an <a href="https://www.npmjs.com/package/mongodb-memory-server">in-memory version of MongoDB</a> that ties nicely into our Mongoose models.</p>
<p><strong>SQS/SNS</strong></p>
<p>For any Amazon infrastructure, we're going to use <a href="https://www.npmjs.com/package/aws-sdk-mock">aws-sdk-mock</a>. This is an excellent wrapper around <a href="https://www.npmjs.com/package/sinon">sinon</a> for mocking the AWS infrastructure, that's super useful for unit tests as you'll see later.</p>
<h3>Tieing it all together</h3>
<p>So firstly lets install our dependencies.</p>
<p><code>npm install aws-sdk-mock mongodb-memory-server sinon jest --save-dev</code></p>
<p>Let's also create a tests folder to keep all this test code in.</p>
<p>mkdir tests</p>
<p><strong>Setting up MongoDB</strong></p>
<p>Now, for our code to integrate with our in-memory Mongo-DB server, we're going to need to add a few setup/teardown functions and a Mongo Environment. Clone this <a href="https://github.com/vladgolubev/jest-mongodb">repo</a> and put:</p>
<ul>
<li><code>setup.js</code></li>
<li><code>teardown.js</code></li>
<li><code>mongo-environment.js</code></li>
</ul>
<p>into your new test folder.</p>
<p>In your root directory add a <code>jest.config.js</code> with this content.</p>
<div class="highlight"><pre><span></span><code><span class="nx">module</span><span class="p">.</span><span class="nx">exports</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span>
<span class="w"> </span><span class="nx">globalSetup</span><span class="o">:</span><span class="w"> </span><span class="s1">'./tests/setup.js'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">globalTeardown</span><span class="o">:</span><span class="w"> </span><span class="s1">'./tests/teardown.js'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">testEnvironment</span><span class="o">:</span><span class="w"> </span><span class="s1">'./tests/mongo-environment.js'</span><span class="p">,</span>
<span class="w"> </span><span class="nx">roots</span><span class="o">:</span><span class="w"> </span><span class="p">[</span><span class="s1">'./tests/'</span><span class="p">],</span>
<span class="p">};</span>
</code></pre></div>
<p>This should be enough to get MongoDB up and running. Now onto our actual tests scripts.</p>
<p><strong>Setting up our tests</strong></p>
<p>Now, finally, you can tie it all together with this snippet:</p>
<p>Add jest to your package.json</p>
<div class="highlight"><pre><span></span><code><span class="nt">"scripts"</span><span class="p">:{</span>
<span class="w"> </span><span class="nt">"test"</span><span class="p">:</span><span class="w"> </span><span class="s2">"jest"</span>
<span class="p">},</span>
</code></pre></div>
<p>And run with <code>npm test</code>.</p>
<p>One major point to note is that aws-mock-sdk requires a very specific way of instantiating AWS resources(within the scope of the function tests), so if you see any errors around aws regions, double check <a href="https://github.com/dwyl/aws-sdk-mock#use-in-your-tests">you're following the rules</a>.</p>
</div>
<div class="tag-cloud">
<p>
</p>
</div>
</article>
<footer>
<p>© </p>
<p>
Built with <a href="http://getpelican.com" target="_blank">Pelican</a> using <a href="http://bit.ly/flex-pelican" target="_blank">Flex</a> theme
</p></footer> </main>
<script type="application/ld+json">
{
"@context" : "http://schema.org",
"@type" : "Blog",
"name": " Shane Dowling ",
"url" : ".",
"image": "/images/logo.png",
"description": ""
}
</script>
</body>
</html>