Upload file to Google Storage

[felpsio]

Hi,

I'm having lots of problems to be able to upload a file using Google Storage inside Hydrogen since its workers doesn't support the official google library.

My store has a few features powered by Google AI. So I do need to store a few things there to be able to use their AI APIs properly.

I'm wondering if anyone has some working code for Hydrogen to be able to upload it to Google Storage. It can be inside the backend or in the frontend using signed Urls.

[blittle]

Is it a package from Google that you are unable to use inside workers? If so, which package? Does google offer a web API of some sort that you could just post data to?

[felpsio]

Yep, here the code that I would use in a normal NodeJs environment:

import { NextApiRequest, NextApiResponse } from "next";
import { Storage } from "@google-cloud/storage";
import { v4 as uuidv4 } from "uuid";

const PROJECT_ID = process.env.PROJECT_ID;
const PRIVATE_KEY = process.env.PRIVATE_KEY;
const CLIENT_EMAIL = process.env.CLIENT_EMAIL;
const BUCKET_NAME = process.env.BUCKET_NAME;

const storage = new Storage({
  projectId: PROJECT_ID,
  credentials: {
    private_key: PRIVATE_KEY,
    client_email: CLIENT_EMAIL
    }
  });
   

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  const { fileType } = req.body;

  const objectName = uuidv4();
  const bucketName = BUCKET_NAME;
  const contentType = fileType;

  if (!objectName || !contentType) {
    return res.status(400).json({ error: "Filename and content type are required." });
  }

  try {
    const [url] = await storage
      .bucket(bucketName)
      .file(objectName)
      .getSignedUrl({
        action: "write",
        expires: Date.now() + 15 * 60 * 1000, // 15 minutes from now
        contentType: contentType,
      });

    res.status(200).json({ signedUrl: url });
  } catch (error) {
    res.status(500).json({ error: error.message });
  }
}

There is a way to do it without the library, but it's not trivial and I wasn't able to make it work so far :(.

[blittle]

What's the specific error you are seeing? My guess is that Storage isn't supported in a worker environment. It looks like you are using Vercel for hosting too. This is more an issue of what your hosting provider than it is of Hydrogen. You can deploy Hydrogen to a NodeJS and use NodeJS libraries directly.

[felpsio]

I'm using Shopify Hydrogen for hosting. The snippet above is what I developed in Nextjs to get the same result. But whenever I try to import some similar library, I get an error.

The specific error:

[mf:err] TypeError: globalThis.XMLHttpRequest is not a constructor
    at checkTypeSupport (/Users/felipe/PROJECT/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js:20:11)
    at node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js (/Users/felipe/PROJECT/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/capability.js:39:45)
    at /Users/felipe/PROJECT/dist/worker/index.js:14:56
    at node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/request.js (/Users/felipe/PROJECT/node_modules/rollup-plugin-node-polyfills/polyfills/http-lib/request.js:1:1)
    at /Users/felipe/PROJECT/dist/worker/index.js:14:56
    at node-modules-polyfills:https (/Users/felipe/PROJECT/dist/worker/node-modules-polyfills:https:30:1)
    at /Users/felipe/PROJECT/dist/worker/index.js:14:56
    at node-modules-polyfills-commonjs:https (/Users/felipe/PROJECT/dist/worker/node-modules-polyfills-commonjs:https:2:18)
    at /Users/felipe/PROJECT/dist/worker/index.js:17:50
    at node_modules/gaxios/build/src/gaxios.js (/Users/felipe/PROJECT/node_modules/gaxios/src/gaxios.ts:16:1)

[blittle]

Do you mean Oxygen for hosting? Oxygen (and most workers, for example Cloudflare) does not support XMLHttpRequest, and instead uses fetch to make subrequests.

[blittle]

I think you have 2 options if you'd like to keep using Oxygen as a deployment environment:

1. Directly make requests to Google's API bypassing this library
2. You could attempt to polyfill XMLHTTPRequest with fetch (which is a bit backwards, because for years we have been polyfilling fetch with XMLHTTPRequest

[felpsio]

@blittle, you're right. It's a limitatiom from Oxygen, not Hydrogen. PS: Not fly.io or Vercel seems to be supporting the hello word template from Hydrogen out of the box. I tested both and also tried to follow the recommendations from the docs (remove oxygen lib and replace the adapter with the related provider (like @verce/remix). To be able to use another hosting provider it's necessary a deeper change in the server setup.

For now I'm just using firebase-functions to help me when I need to use a specific library. It's kind of annoying (another tool to manage and slower responses), but it's easier for now.

I tried 1. but it's much more complicated and not well documented as well and more complicated to maintain. 2 I didn't find a good polyfill in a quick search. I'm open to suggestions.

If anyone else is facing a similar problem I recommended using firebase functions or aws lambda.

Would be nice if the official template generated by hydrogen works out of the box on vercel/fly.io or if there was a better guide on how to adapt to one of these platforms. I wasn't able to quickly adapt them o vercel or fly.io

[blittle]

I'd love to have an official template for Vercel or Fly. We are a little stretched at the moment though, so I don't know if we'll be able to get to it, or long term maintain it, anytime soon :(

[jamalsoueidan]

I haven't read all the posts, but Oxygen should NOT be responsible for uploading your files.

Why

You should use "fetch" in your components, not in your loader or action. You must upload the file directly from the user browser to Google Cloud, or whatever service you're using.

I'm using Shopify's FileCreate to upload my files to Shopify. People often forget that Shopify won't allow wastage of resources. There are millions of people using Shopify, and they aim to minimize a lot of server handling tasks.

Steps

1. request the "upload URL" from Shopify.
2. allow the user to pick a file.
3. use the user's browser to send the file to the "upload URL" that Shopify provided.
4. retrieve the file URL from the response of the "upload URL" after the file has been uploaded.

I hope it make sense.

How

I'm pretty sure whatever service you are using, can provide you with the url to DIRECTLY upload the file to them in the browser without interaction with your servers.

[YES] USER <> GOOGLE
[NO] User < Oxygen > Google

[felpsio]

@jamalsoueidan, I'm not uploading the file on Oxygen.

I generate an url to upload to Google Storage and the user uploads to the storage directly. I just need oxygen backend to be able to generate the signed Url for this :).

[felpsio]

I’m using AI with a custom model in Google. So I need some files to be stored in Google to be able to interoperate with Googles APIs easier. For the files I don’t need for AI I can store in Shopify without problems :)

[jamalsoueidan]

Shopify store the files at google when you upload them, you actually get google url :)

[felpsio]

interesting. @jamalsoueidan do you know if the final file is accessible directly with the google URL? I need to be able to access it from my backend later. If Shopify protects the direct access or make it more complicated it may not be very useful for the integration with Google AI APIs

[jamalsoueidan]

You can access it, but it will expire after couple of days just to let you know.

https://storage.googleapis.com/shopify-staged-uploads/tmp/68240605458/products/cea3fcb0-ddcb-43c1-b9a8-c9e57ba23752/7106990342471_customer_profile_1686032438869.jpg

I just uploaded my own pic :)

[felpsio]

😍. Thanks for such friendly reply :D. Unfortunately, I'll need to store the file in google storage anyway later so I can use the image to keep improving the ML model. So the temporary Shopify link doesn't solve my problem. But hopefully I'll have other contexts where this may be useful. Thanks again :D