Cannot bundle .node files #14289
Labels
Comments
|
|
|
This issue is preventing me from using many modules. No workaround or package has worked. |
|
I managed to bundle nodejs-polars using this config but it's incredibly hacky. |
|
Well, that's one hell of a hack. But I'll have to study and use it I guess. Thank you so much. Resolving .node files should be a feature in Vite. |
|
The same problem occurs with I have confirmed that it works by adding the following settings to vite: {
optimizeDeps: {
exclude: ["fsevents"],
},
},Error log |
|
|
|
|
We've managed to get |
|
Are there any plans on this one? Even though @goastler shared a nice blog, it doesn't solve anything for the general public. |
|
The handling of Further, there's more complications with files other than tldr; probably better for someone who knows more about these files to implement a plugin / build support in vite |
|
related #5688 |
|
For those searching for a solution, I (chatgpt to be honest) came up with a quick plugin: function nativeFilesPlugin(): PluginOption {
const files = new Map<string, { readonly fileName: string; readonly fileContent: Buffer }>();
return {
name: 'node-binaries-plugin',
async load(id) {
if (!id.endsWith('.node')) {
return null;
}
const fileContent = await fs.readFile(id);
const hash = createHash('sha256').update(fileContent).digest('hex').slice(0, 8);
const fileName = `${path.basename(id, '.node')}.${hash}.node`;
files.set(id, { fileName, fileContent });
return `export default require('./${fileName}');`;
},
generateBundle(_, bundle) {
for (const [id, { fileName, fileContent }] of files.entries()) {
this.emitFile({ type: 'asset', fileName, source: fileContent });
delete bundle[id];
}
},
};
}I'm building into build: {
rollupOptions: {
output: {
format: 'cjs',
},
},
commonjsOptions: {
requireReturnsDefault: 'auto',
}
} |
|
It looks like an even simpler version also works: {
name: 'require-node-binaries-plugin',
transform: (code, id) => {
if (id.endsWith('.node')) {
return code.replace(/export default "(.+)"/, 'export default require(".$1")');
}
return;
},
}paired with |
|
Small update here. Native assets didn't work in my project because I'm using So I had to revert to the original naive plugin, and, unfortunately, add a special hard-coded case for function nativeFilesPlugin(): PluginOption {
interface NativeFile {
readonly fileName: string;
readonly fileContent: Buffer;
}
const nativeFiles = new Map<string, NativeFile>();
const uws = require.resolve('uWebSockets.js');
const readNativeFile = async (filePath: string): Promise<NativeFile> => {
const fileContent = await readFile(filePath);
const hash = createHash('sha256').update(fileContent).digest('hex').slice(0, 8);
const fileName = `${path.basename(filePath, '.node')}.${hash}.node`;
return { fileName, fileContent };
};
return {
name: 'native-files-plugin',
async load(id) {
if (id === uws) {
// Special handling for the ESM-wrapper around CJS-module with dynamic requires (uWebSockets.js).
const nativeFile = await readNativeFile(
// Yes, build the file name at build time, not runtime 🤷♂️(we can't do dynamic `import {} from ''`)
path.resolve(path.dirname(uws), './uws_' + process.platform + '_' + process.arch + '_' + process.versions.modules + '.node'),
);
nativeFiles.set(id, nativeFile);
return `export default require('./${nativeFile.fileName}')`;
} else if (id.endsWith('.node')) {
const nativeFile = await readNativeFile(id);
nativeFiles.set(id, nativeFile);
return `export default require('./${nativeFile.fileName}');`;
}
return;
},
generateBundle(_, bundle) {
for (const [id, { fileName, fileContent }] of Array.from(nativeFiles.entries())) {
this.emitFile({ type: 'asset', fileName, source: fileContent });
delete bundle[id];
}
},
};
}I believe the issue should be handled on rollup's side, not on vite's, as I believe it's purely rollup's job to bundle everything into a module graph that vite would understand. There's also an on-going work on Rolldown (a Rust replacement for rollup), so maybe it should even go there, or maybe it's already supported there, I can't test it right now. |
|
I installed import { defineConfig } from 'vite';
import electron from 'vite-plugin-electron/simple';
import vue from '@vitejs/plugin-vue';
import { nodePolyfills } from 'vite-plugin-node-polyfills';
export default defineConfig({
plugins: [
vue(),
electron({
main: {
entry: 'electron/main.ts',
vite: {
build: {
rollupOptions: {
external: ['@ultimateshadsform/universal-media'],
},
},
},
},
preload: {
input: 'electron/preload.ts',
},
renderer: {},
}),
nodePolyfills(),
],
resolve: {
alias: {
path: 'path-browserify',
},
},
optimizeDeps: {
exclude: ['@ultimateshadsform/universal-media'],
},
ssr: {
noExternal: ['@ultimateshadsform/universal-media'],
},
build: {
rollupOptions: {
external: ['@ultimateshadsform/universal-media', 'electron'],
},
},
});Note
Putting With the external option it shuts up about that. My electron main: import { app, BrowserWindow, ipcMain } from 'electron';
import { fileURLToPath } from 'node:url';
import path from 'node:path';
import { MediaController } from '@ultimateshadsform/universal-media';
// Create singleton instance
const mediaController = new MediaController();
// Store subscription
let currentSubscription: { stop: () => void } | null = null;
const __dirname = path.dirname(fileURLToPath(import.meta.url));
// The built directory structure
//
// ├─┬─┬ dist
// │ │ └── index.html
// │ │
// │ ├─┬ dist-electron
// │ │ ├── main.js
// │ │ └── preload.mjs
// │
process.env.APP_ROOT = path.join(__dirname, '..');
// 🚧 Use ['ENV_NAME'] avoid vite:define plugin - Vite@2.x
export const VITE_DEV_SERVER_URL = process.env['VITE_DEV_SERVER_URL'];
export const MAIN_DIST = path.join(process.env.APP_ROOT, 'dist-electron');
export const RENDERER_DIST = path.join(process.env.APP_ROOT, 'dist');
process.env.VITE_PUBLIC = VITE_DEV_SERVER_URL
? path.join(process.env.APP_ROOT, 'public')
: RENDERER_DIST;
let win: BrowserWindow | null;
function createWindow() {
win = new BrowserWindow({
icon: path.join(process.env.VITE_PUBLIC, 'electron-vite.svg'),
webPreferences: {
preload: path.join(__dirname, 'preload.mjs'),
},
});
// Test active push message to Renderer-process.
win.webContents.on('did-finish-load', () => {
win?.webContents.send('main-process-message', new Date().toLocaleString());
});
if (VITE_DEV_SERVER_URL) {
win.loadURL(VITE_DEV_SERVER_URL);
} else {
// win.loadFile('dist/index.html')
win.loadFile(path.join(RENDERER_DIST, 'index.html'));
}
}
// Add IPC handlers
function setupIPC() {
ipcMain.handle('media:play', () => mediaController.play());
ipcMain.handle('media:pause', () => mediaController.pause());
ipcMain.handle('media:stop', () => mediaController.stop());
ipcMain.handle('media:next', () => mediaController.next());
ipcMain.handle('media:previous', () => mediaController.previous());
ipcMain.handle('media:getMediaInfo', () => mediaController.getMediaInfo());
ipcMain.handle('media:getSystemVolume', () =>
mediaController.getSystemVolume()
);
ipcMain.handle('media:getSystemMute', () => mediaController.getSystemMute());
ipcMain.handle('media:setSystemVolume', (_event, volume: number) =>
mediaController.setSystemVolume(volume)
);
ipcMain.handle('media:setSystemMute', (_event, mute: boolean) =>
mediaController.setSystemMute(mute)
);
// Handle subscriptions
ipcMain.handle('media:subscribe', (event) => {
try {
// Clean up previous subscription
if (currentSubscription) {
currentSubscription.stop();
}
// Create new subscription and forward events to renderer
currentSubscription = mediaController.subscribeToEvents(
(eventData: any) => {
event.sender.send('media:event', eventData);
}
);
} catch (error) {
console.error('Error in subscribe:', error);
}
});
ipcMain.handle('media:unsubscribe', () => {
if (currentSubscription) {
currentSubscription.stop();
currentSubscription = null;
}
});
}
// Quit when all windows are closed, except on macOS. There, it's common
// for applications and their menu bar to stay active until the user quits
// explicitly with Cmd + Q.
app.on('window-all-closed', () => {
if (process.platform !== 'darwin') {
app.quit();
win = null;
}
});
app.on('activate', () => {
// On OS X it's common to re-create a window in the app when the
// dock icon is clicked and there are no other windows open.
if (BrowserWindow.getAllWindows().length === 0) {
createWindow();
}
});
app.whenReady().then(() => {
createWindow();
setupIPC();
});That's about it. My project is build using napi-rs so this should work. The trick here is using Here is the auto generated index.js file napi-rs generates: /* tslint:disable */
/* eslint-disable */
/* prettier-ignore */
/* auto-generated by NAPI-RS */
const { existsSync, readFileSync } = require('fs')
const { join } = require('path')
const { platform, arch } = process
let nativeBinding = null
let localFileExisted = false
let loadError = null
function isMusl() {
// For Node 10
if (!process.report || typeof process.report.getReport !== 'function') {
try {
const lddPath = require('child_process').execSync('which ldd').toString().trim()
return readFileSync(lddPath, 'utf8').includes('musl')
} catch (e) {
return true
}
} else {
const { glibcVersionRuntime } = process.report.getReport().header
return !glibcVersionRuntime
}
}
switch (platform) {
case 'android':
switch (arch) {
case 'arm64':
localFileExisted = existsSync(join(__dirname, 'universal-media.android-arm64.node'))
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.android-arm64.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-android-arm64')
}
} catch (e) {
loadError = e
}
break
case 'arm':
localFileExisted = existsSync(join(__dirname, 'universal-media.android-arm-eabi.node'))
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.android-arm-eabi.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-android-arm-eabi')
}
} catch (e) {
loadError = e
}
break
default:
throw new Error(`Unsupported architecture on Android ${arch}`)
}
break
case 'win32':
switch (arch) {
case 'x64':
localFileExisted = existsSync(
join(__dirname, 'universal-media.win32-x64-msvc.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.win32-x64-msvc.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-win32-x64-msvc')
}
} catch (e) {
loadError = e
}
break
case 'ia32':
localFileExisted = existsSync(
join(__dirname, 'universal-media.win32-ia32-msvc.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.win32-ia32-msvc.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-win32-ia32-msvc')
}
} catch (e) {
loadError = e
}
break
case 'arm64':
localFileExisted = existsSync(
join(__dirname, 'universal-media.win32-arm64-msvc.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.win32-arm64-msvc.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-win32-arm64-msvc')
}
} catch (e) {
loadError = e
}
break
default:
throw new Error(`Unsupported architecture on Windows: ${arch}`)
}
break
case 'darwin':
localFileExisted = existsSync(join(__dirname, 'universal-media.darwin-universal.node'))
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.darwin-universal.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-darwin-universal')
}
break
} catch {}
switch (arch) {
case 'x64':
localFileExisted = existsSync(join(__dirname, 'universal-media.darwin-x64.node'))
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.darwin-x64.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-darwin-x64')
}
} catch (e) {
loadError = e
}
break
case 'arm64':
localFileExisted = existsSync(
join(__dirname, 'universal-media.darwin-arm64.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.darwin-arm64.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-darwin-arm64')
}
} catch (e) {
loadError = e
}
break
default:
throw new Error(`Unsupported architecture on macOS: ${arch}`)
}
break
case 'freebsd':
if (arch !== 'x64') {
throw new Error(`Unsupported architecture on FreeBSD: ${arch}`)
}
localFileExisted = existsSync(join(__dirname, 'universal-media.freebsd-x64.node'))
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.freebsd-x64.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-freebsd-x64')
}
} catch (e) {
loadError = e
}
break
case 'linux':
switch (arch) {
case 'x64':
if (isMusl()) {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-x64-musl.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-x64-musl.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-x64-musl')
}
} catch (e) {
loadError = e
}
} else {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-x64-gnu.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-x64-gnu.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-x64-gnu')
}
} catch (e) {
loadError = e
}
}
break
case 'arm64':
if (isMusl()) {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-arm64-musl.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-arm64-musl.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-arm64-musl')
}
} catch (e) {
loadError = e
}
} else {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-arm64-gnu.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-arm64-gnu.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-arm64-gnu')
}
} catch (e) {
loadError = e
}
}
break
case 'arm':
if (isMusl()) {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-arm-musleabihf.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-arm-musleabihf.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-arm-musleabihf')
}
} catch (e) {
loadError = e
}
} else {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-arm-gnueabihf.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-arm-gnueabihf.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-arm-gnueabihf')
}
} catch (e) {
loadError = e
}
}
break
case 'riscv64':
if (isMusl()) {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-riscv64-musl.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-riscv64-musl.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-riscv64-musl')
}
} catch (e) {
loadError = e
}
} else {
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-riscv64-gnu.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-riscv64-gnu.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-riscv64-gnu')
}
} catch (e) {
loadError = e
}
}
break
case 's390x':
localFileExisted = existsSync(
join(__dirname, 'universal-media.linux-s390x-gnu.node')
)
try {
if (localFileExisted) {
nativeBinding = require('./universal-media.linux-s390x-gnu.node')
} else {
nativeBinding = require('@ultimateshadsform/universal-media-linux-s390x-gnu')
}
} catch (e) {
loadError = e
}
break
default:
throw new Error(`Unsupported architecture on Linux: ${arch}`)
}
break
default:
throw new Error(`Unsupported OS: ${platform}, architecture: ${arch}`)
}
if (!nativeBinding) {
if (loadError) {
throw loadError
}
throw new Error(`Failed to load native binding`)
}
const { EventType, MediaController } = nativeBinding
module.exports.EventType = EventType
module.exports.MediaController = MediaController |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
I am trying to create a server side bundle that includes packages containing .node files. In this specific instance, the problematic package is nodejs-polars.
I have tried various plugins to resolve this but none of them seem to work.
My config is as follows:
Reproduction
https://stackblitz.com/edit/vitejs-vite-xp1ggj?file=vite.config.ts,package.json&terminal=dev
Steps to reproduce
Run
vite build --config vite.config.tsSystem Info
System: OS: Linux 5.4 Ubuntu 20.04.6 LTS (Focal Fossa) CPU: (8) x64 AMD FX(tm)-8350 Eight-Core Processor Memory: 9.66 GB / 31.31 GB Container: Yes Shell: 5.8 - /usr/bin/zsh Binaries: Node: 18.14.2 - ~/.nvm/versions/node/v18.14.2/bin/node Yarn: 1.22.19 - ~/.nvm/versions/node/v18.14.2/bin/yarn npm: 8.9.0 - ~/.nvm/versions/node/v18.14.2/bin/npm Browsers: Brave Browser: 115.1.56.14 Chrome: 116.0.5845.96The text was updated successfully, but these errors were encountered: