Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bitsandbytes be decoupled from huggingface #549

Closed
RanchiZhao opened this issue Jun 28, 2023 · 3 comments
Closed

bitsandbytes be decoupled from huggingface #549

RanchiZhao opened this issue Jun 28, 2023 · 3 comments

Comments

@RanchiZhao
Copy link

I'd like to know if bitsandbytes can be decoupled from huggingface, or if they have to be used together. In addition, is the int4 quantization process completed during the get_accelerate_model phase and unrelated to the subsequent training with Trainer? Or, at what point does the dequantization process occur? This is because I've noticed that int4 quantization alters the shape of the linear layer.

@RanchiZhao
Copy link
Author

Recently, I have finally adapted QLoRa for frameworks other than HuggingFace. The focus of the adaptation is primarily on the operators in Linear4bit and a series of quantized optimizers. Of course, during this process there are quite al lot dtype errors.

@rayrayraykk
Copy link

could you share your ideas or repo? Many thanks!

@RanchiZhao
Copy link
Author

could you share your ideas or repo? Many thanks!

you could see this, i hope it will be helpful: RanchiZhao/bmtrain_qlora#1 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants