How to test async flow

[ekeren]

Naively, I wanted to write a test that verifies that on_message works

bring cloud;

let q = new cloud.Queue();
let c = new cloud.Counter();

q.on_message(inflight (m: str): str => {
  c.inc();
  print("on_message got triggered with ${m}");
});

new cloud.Function(inflight () => {
  print("before push counter is ${c.peek()}");
  q.push("a messgae");
  print("after push counter is ${c.peek()}");
  assert(1 == c.peek());
}) as "test:something";

The result of wing test is the following:

fail ┌ a.w » root/test:something
     │ before push counter is 0
     │ after push counter is 0
     │ Error: assertion failed: '(1 === (await c.peek()))'
     │     at evalmachine.<anonymous>:34:23
     │     at Handler.handle (evalmachine.<anonymous>:35:15)
     │     at async exports.handler (evalmachine.<anonymous>:12:10)
     │     at async Object.withTrace (/Users/eyalkeren/.nvm/versions/node/v18.12.1/lib/node_modules/winglang/node_modules/@winglang/sdk/lib/testing/simulator.js:114:38)
     │     at async Simulator.runTest (/Users/eyalkeren/.nvm/versions/node/v18.12.1/lib/node_modules/winglang/node_modules/@winglang/sdk/lib/testing/simulator.js:303:13)
     │     at async Simulator.runAllTests (/Users/eyalkeren/.nvm/versions/node/v18.12.1/lib/node_modules/winglang/node_modules/@winglang/sdk/lib/testing/simulator.js:278:26)
     │     at async testOne (/Users/eyalkeren/.nvm/versions/node/v18.12.1/lib/node_modules/winglang/dist/commands/test.js:26:21)
     └     at async test (/Users/eyalkeren/.nvm/versions/node/v18.12.1/lib/node_modules/winglang/dist/commands/test.js:12:9)

The reason for this test to fail is because the on_message handler will be called async.

How should I test the above scenario?

[Chriscbr]

I think to test this scenario you would need to add some kind of sleep statement after you push to the queue. This reflects the same reality as if you were to run the test on the cloud. (Pushing to the queue would not instantaneously update the counter)

[yoav-steinberg]

Welcome to distributed computing :)
Behaves perfectly as far as I can tell.

[skinny85]

This is a common problem with async operations. There are many libraries that allow you to express waiting on a condition easily - for an example, see Awaitility in Java.

You can add some built-in helpers to Wing (either to the standard library of the language, or the SDK), so your users can do something like:

new cloud.Function(inflight () => {
  print("before push counter is ${c.peek()}");
  q.push("a message");
  print("after push counter is ${c.peek()}");
  assert.waitingUntil(() => c.peek() == 1, {
    atMost: 2.seconds,
  });
}) as "test:something";

Or something similar 🙂.

[skinny85]

An alternative would be to add some concurrency primitives to Wing, in this case, a semaphore:

bring cloud;

let q = new cloud.Queue();
let c = new cloud.Counter();
let s = cloud.Semaphore.closed(1); // starts closed, and needs 1 raise to open

q.on_message(inflight (m: str): str => {
  c.inc();
  print("on_message got triggered with ${m}");
  s.raise();
});

new cloud.Function(inflight () => {
  print("before push counter is ${c.peek()}");
  q.push("a messgae");
  print("after push counter is ${c.peek()}");
  s.pass();
  assert(1 == c.peek());
}) as "test:something";

But the implementation of this might be pretty involved.

[yoav-steinberg]

+1 for cloud sync primitives.

[ekeren]

With extern and some workarounds I was able to write a test that:

• ✅ works on sim
• ✅ works on tf-aws

queue/on_message.w

bring cloud;

let q = new cloud.Queue();
let c = new cloud.Counter();

// workaround for $stdlib is not defined compiler error
// had to wrap the inflight ():bool => { c.peek == 2 } inflight method with a the Predicate resource
resource Predicate {
  c: cloud.Counter;
  init(c: cloud.Counter){
    this.c = c;
  }

  inflight test(): bool{
    return this.c.peek() == 2;
  }
}

resource TestHelper {
  init(){}
  extern "helper.js" static inflight sleep(milli: num);
  
  /** 
   * test predicate every 100 miliseconds until a minute passes
  **/
  inflight assert(predicate: Predicate) { 
    let var i = 0;
    while i<600 {
      i = i + 1;
      if predicate.test() {
        assert(predicate.test());
        return;
      } 
      TestHelper.sleep(100);
    }
    assert(predicate.test());
  }
}

q.on_message(inflight (msg: str): str => {
  c.inc();
});

let js = new TestHelper();

let predicate = new Predicate(c);
new cloud.Function(inflight () => {
  q.push("hello");
  q.push("world");
  js.assert(predicate);
}) as "test:on_message";

helper.js

exports.sleep = function(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Running the tests

tf-aws

Running directly from rybickic/test-engine branch, because this functionality was not merged yet

% npx nx wing -- test -t tf-aws  queue/on_message.w


   ✔    6/6 dependent project tasks succeeded [6 read from cache]

   Hint: you can run the command with --verbose to see the full dependent project outputs

 —————————————————————————————————————————————————————————————————————————————


> nx run winglang:copy  [existing outputs match the cache, left as is]

../../target/wasm32-wasi/debug/wingc.wasm -> ./wingc.wasm

> nx run winglang:build  [existing outputs match the cache, left as is]


> winglang@0.0.0 build
> tsc


> nx run winglang:wing test -t tf-aws queue/on_message.w

- Compiling to tf-aws...
✔ Compiling to tf-aws...
- terraform init
✔ terraform init
- terraform apply
✔ terraform apply
- Setting up test runner...
✔ Setting up test runner...
- Running tests...
✔ Running tests...
pass ─ on_message.tfaws » root/Default/env0/test:on_message
- terraform destroy
✔ terraform destroy

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Successfully ran target wing for project winglang and 8 tasks it depends on (2m)

         With additional flags:
           test queue/on_message.w
           --t=tf-aws

   Nx read the output from the cache instead of running the command for 8 out of 9 tasks.

sim

% wing test on_message.w
pass ─ a.w » root/test:on_message