Use With AWS Lambda

[nburdett19]

Hello,

Is it possible to use this within a Lambda function?

I inserted my nodejs code that was working on my local machine into an AWS Lambda function and received the error "Error: Cannot find module 'node-redshift'.

I then uploaded the zipped module after downloading from git as it's own Lambda function and received the error "errorMessage": "Cannot find module '/var/task/index'". Is there a way around both these issues, to enable the use of node-redshift within an AWS Lambda function?

Thanks,

Nick

[dmanjunath]

Hey Nick, I haven't tried using the package with lambda but I don't see why it wouldn't work. There's no step to enable it to work for lambda per se, it's all pretty standard. All the code in the package is self contained so it should have everything you need to just run it on lambda. I'll try to reproduce this to test it out because this is a good question, but first two questions. 1. The zip that you uploaded to lambda, in the node_modules folder can you verify that node-redshift is installed? 2. How are you calling the package? Do you have a function I could run to test it out?

…

On Aug 4, 2017, at 1:06 PM, nburdett19 ***@***.***> wrote: Hello, Is it possible to use this within a Lambda function? I inserted my nodejs code that was working on my local machine into an AWS Lambda function and received the error "Error: Cannot find module 'node-redshift'. I then uploaded the zipped module after downloading from git as it's own Lambda function and received the error "errorMessage": "Cannot find module '/var/task/index'". Is there a way around both these issues, to enable the use of node-redshift within an AWS Lambda function? Thanks, Nick — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[nburdett19]

Hi Dheeraj,

I do not see "node-redshift" in the folder node_modules folder. Do you know which sub folder of node_modules it would be in? I downloaded the .zip from this link on git: https://github.com/dmanjunath/node-redshift/archive/master.zip

Here is a script that pulls data from a table called tickets in the tic schema that I use, and works on my windows (win 10) laptop and returns data in JSON format:

//var aws = require('aws-sdk');
var Redshift = require('node-redshift');
//AWS.config.update ({region: "us-east-1"});

var client = {
	user: 'username',
	database: 'database',
	password: 'password',
	port: 'portnumber',
	host: 'server connection string',
	ssl: 'true',
	keepAlive: 'true'
};

var redshift = new Redshift(client, {rawConnection: true});

// using callbacks
 redshift.connect(function(err){ //create connection manually
   if(err) throw err;
   else{
     redshift.query('select * from tic."tickets"', {raw: true}, function(err, data){ //query redshift
       if(err) throw err;
       else{
         console.log(JSON.stringify(data, null, 4));

         redshift.close();
       }
     });
   }
 });

[nburdett19]

Hello,

I have an update. I restructured the Zip folder to remove the node-redshift-master folder that contained the index.js file. The zip folder now looks like:

node-redshift-master.zip > bin,examples,lib,node_modules,index.js etc...

Where it used to be structured as:

node-redshift-masater.zip > node-redshift-master > bin,examples,lib,node_modules,index.js etc...

I now am getting an error of "errorMessage": "Handler 'handler' missing on module 'index'".

Would you know what would cause this error in the index.js file?

[dmanjunath]

Sorry I should have been more specific. I meant to ask is there a node-redshift folder in the zip you uploaded to lambda? I didnt mean to say did the node-redshift master from github have a node-redshift folder in node_modules. That would be recursive and very confusing.

So in essence, I guess my question is, in the zip you uploaded to lambda, what does the structure look like?

[nburdett19]

I restructured the Zip folder to remove the node-redshift-master folder that contained the index.js file. The zip folder now looks like:

node-redshift-master.zip > bin,examples,lib,node_modules,index.js etc...

Where it used to be structured as:

node-redshift-masater.zip > node-redshift-master > bin,examples,lib,node_modules,index.js etc...

I now am getting an error of "errorMessage": "Handler 'handler' missing on module 'index'".

Would you know what would cause this error in the index.js file?

[dmanjunath]

So I just uploaded my test repo to lambda to test and it works. But I did run into two problems, I think you may be running into the same problems.

The first is that if you right click the folder and say Compress on Mac, it apparently doesn't work https://stackoverflow.com/questions/41750026/aws-lambda-error-cannot-find-module-var-task-index. That's what caused the "Cannot find module '/var/task/index'".

The second is you need an exported handler function in your index.js file at the root of your directory. So here's how my root level index.js(file that contains the lambda handler) looks like

var Redshift = require('node-redshift');

var client = {
  user:user,
  database:database,
  password:password,
  port:port,
  host:host
};

var redshift = new Redshift(client);

exports.handler = function index(event, context, callback) {
  redshift.query('SELECT * FROM "Table" LIMIT 5;', {raw: true}, function(err, data){
    if(err) console.error(err);
    else{
      console.log(data);
    }
  });
};

[nburdett19]

Ok, so I added the zip file to a Lambda Function and it stopped giving me the "Cannot find module '/var/task/index'" error. GOOD NEWS!!

But, I'm running into an error when running my other lambda function containing my node-redshift query. I am getting the "Cannot find module 'node-redshift'", after adding the exports.handler to my code.

var Redshift = require('node-redshift');
var client = {
  user:user,
  database:database,
  password:password,
  port:port,
  host:host
};
var redshift = new Redshift(client, {rawConnection: true});

exports.handler = function index(event, context, callback) {
  redshift.query('select * from tic."tickets"', {raw: true}, function(err, data){
    if(err) console.error(err);
    else{
      console.log(data);
    }
  });
};

So my understanding is that you're putting the code above in another lambda function, and it is returning data from your select statement inside the redshift.query() code? Are you calling the lambda.invoke() function to run your node-redshift module on your code provided in your most recent comment?

[dmanjunath]

The code I attached is my lambda function in its entirety. All I'm doing is calling that query in the handler when invoked. It's a very simple lambda function, tried to keep it as easy to test as possible 🙂 If it would help I can try to find a way to upload my zip file so you can take a look.

…

On Aug 9, 2017, at 5:57 PM, nburdett19 ***@***.***> wrote: Ok, so I added the zip file to a Lambda Function and it stopped giving me the "Cannot find module '/var/task/index'" error. GOOD NEWS!! But, I'm running into an error when running my other lambda function containing my node-redshift query. I am getting the "Cannot find module 'node-redshift'", after adding the exports.handler to my code. var Redshift = require('node-redshift'); var client = { user:user, database:database, password:password, port:port, host:host }; var redshift = new Redshift(client, {rawConnection: true}); exports.handler = function index(event, context, callback) { redshift.query('select * from tic."tickets"', {raw: true}, function(err, data){ if(err) console.error(err); else{ console.log(data); } }); }; So my understanding is that you're putting the code above in another lambda function, and it is returning data from your select statement inside the redshift.query() code? Are you calling the lambda.invoke() function to run your node-redshift module on your code provided in your most recent comment? — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[nburdett19]

If you wouldn't mind it would help me with understanding. From what I've researched, I would need to add the node-redshift zip as its own lambda function (which i have), and then put the following code in another lambda function and invoke the node-redshift function. When I run your example, i get the error of: "errorMessage": "Cannot find module 'node-redshift'". Are you adding your code similar to mine into the zip folder before uploading it to Lambda?

//var aws = require('aws-sdk');
var Redshift = require('node-redshift');
//AWS.config.update ({region: "us-east-1"});

var client = {
	user: 'username',
	database: 'db',
	password: 'pw',
	port: 'port',
	host: 'host',
	ssl: 'true',
	keepAlive: 'true'
};

var redshift = new Redshift(client, {rawConnection: true});

exports.handler = function index(event, context, callback) {
  redshift.query('select * from tix."open_tickets_by_member"', {raw: true}, function(err, data){
    if(err) console.error(err);
    else{
      console.log(data);
    }
  });
};

[dmanjunath]

Oh I see the problem. I didn't add node-redshift as it's own lambda function. It's all in a single zip that's uploaded to lambda. Here's what I had:

node_redshift_lambda.zip

The way you're trying to do it currently, with one redshift lambda and another lambdas calling it, is a microservice like architecture. So you would only initialize redshift and call redshift.query() from one lambda. For all the other lambdas, you'd never use the node-redshift module. Instead, you'd use the aws api to call lambda.invoke from the caller lambda to the redshift lambda. You can pass along the string of the query to the redshift lambda and it will pass back the payload when it's done

[nburdett19]

Great, thank you! just to compare, what were your results when running the function? I recieved the value of "null". But returning a JSON string of data from the table seems like just a tweak to the console.log in the else function.

[dmanjunath]

I actually got data from the table as expected. The lambda console showed a formatted JSON object.

…

On Aug 10, 2017, at 4:00 PM, nburdett19 ***@***.***> wrote: Great, thank you! just to compare, what were your results when running the function? I recieved the value of "null". But returning a JSON string of data from the table seems like just a tweak to the console.log in the else function. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[Tomarr]

Hi
I am trying to use thie same module to access redshift from Node.js programmatically from a aws lambda console but I am getting getaddrinfo ENOTFOUND. Can you tell me why? I was not able to find any significant info regarding this.

Thanks.

[DanielSamsonraj]

Hi,
I am facing an issue while working it with AWS lambda, i just copied the code from above comment and tested it but it doesn't seem to be working, all the lines after redshift.query() doesn't execute. There's no issue with the VPC configs & because the same worked in python, so I am pretty sure there's an issue in JS.