Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BLAKE2b gadget #1767

Merged
merged 56 commits into from
Oct 15, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
d357853
add blake2b
boray Jul 18, 2024
cb93722
add or gadget and improvements
boray Jul 22, 2024
31abb6e
Merge branch 'feature/blake2b' of github.com:o1-labs/o1js into featur…
boray Jul 23, 2024
2846396
add comments and tests
boray Jul 23, 2024
9dd79aa
fix bitwise test
boray Jul 24, 2024
24c743a
add comments to gadgets
boray Jul 24, 2024
044cbe1
dump vks
boray Jul 24, 2024
81bc5a9
add doccoments and test
boray Jul 26, 2024
58a07a0
add yoni's fix #1763
boray Jul 26, 2024
72877d7
Merge remote-tracking branch 'origin/main' into feature/blake2b
boray Jul 26, 2024
10836c9
add UInt32.or()
boray Jul 26, 2024
00cdb75
Update src/lib/provable/gadgets/bitwise.ts
boray Jul 28, 2024
5db6974
update vk-regression.json
boray Jul 28, 2024
63b939c
fixes and styling
boray Jul 29, 2024
e6317f1
remove question marks
boray Jul 30, 2024
95a71d3
Merge remote-tracking branch 'origin/feature/divmod32-quotientbits-bo…
boray Jul 30, 2024
90daf62
optimize g function additions
boray Aug 21, 2024
1938405
hardcode IV as UInt64
boray Aug 21, 2024
5e2b28a
dump vks
boray Aug 21, 2024
a86a106
simplify or
boray Aug 22, 2024
c38118e
fix formatting
boray Aug 22, 2024
5bba739
fix divMod64 range check
boray Aug 22, 2024
e9c5a29
Merge remote-tracking branch 'origin/v2' into feature/blake2b
boray Aug 22, 2024
555070b
fix elliptic curve to match v2
boray Aug 23, 2024
d071e7b
fix formatting
boray Aug 23, 2024
9b32c4d
style(gadgets.ts): wrap comments for readability
boray Aug 29, 2024
70169e3
fix(bitwise.unit-test.ts): correct typo
boray Sep 5, 2024
acd32bb
perf: optimize divmod rangechecks
boray Sep 5, 2024
ec5b6a7
fix: remove incorrect optimization
boray Sep 5, 2024
b741027
fix(arithmetic.ts): fix range assertion
boray Sep 6, 2024
a34d8c2
docs: clarify output description
boray Sep 6, 2024
6ba2968
fix(blake2b.ts): add range check for digestLength
boray Sep 7, 2024
d841a51
fix(blake2b.ts): range check input byte length
boray Sep 7, 2024
d999c09
test: add test vector
boray Sep 9, 2024
e5bd574
feat: add last block flag and state type
boray Sep 9, 2024
1c5f3bd
fix: counter logic
boray Sep 9, 2024
7f650b1
chore: update reference
boray Sep 9, 2024
e545799
docs: add comments
boray Sep 9, 2024
368ae88
docs: add more comments
boray Sep 10, 2024
cb80ccf
perf: reduce row number
boray Sep 20, 2024
20a0eac
fix: uint64 construction
boray Sep 20, 2024
cceeef5
test: remove in-circuit equivalence tests
boray Sep 20, 2024
4304c0f
test: add in-circuit tests for blake2b
boray Sep 29, 2024
567e2ac
test: keep first two equivalentAsync
boray Oct 6, 2024
7125718
test: convert field to uint64
boray Oct 6, 2024
4d897aa
docs(gadgets.ts): fix ranges
boray Oct 6, 2024
5e67f23
docs(gadgets.ts): fix input ranges
boray Oct 7, 2024
6e469c1
refactor: change type of t and refactor little endian convert
boray Oct 7, 2024
3fe8f42
test: add provable test
boray Oct 7, 2024
2dbf186
chore: dump-vks
boray Oct 7, 2024
b972e7a
test: fix incorrect hash in test vectors
boray Oct 8, 2024
cd102fc
Merge branch 'v2' into feature/blake2b
boray Oct 14, 2024
ab34d29
test: fix unit tests
boray Oct 14, 2024
4e83d46
test: fix blake2b example
boray Oct 14, 2024
fc3aa3e
docs: update changelog-v2
boray Oct 15, 2024
1da0ed0
fix: styling and typing
boray Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/bindings
18 changes: 18 additions & 0 deletions src/examples/crypto/blake2b/blake2b.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { Bytes, Gadgets, ZkProgram } from 'o1js';

export { BLAKE2BProgram, Bytes12 };

class Bytes12 extends Bytes(12) {}

let BLAKE2BProgram = ZkProgram({
name: 'blake2b',
publicOutput: Bytes(32),
methods: {
blake2b: {
privateInputs: [Bytes12],
async method(xs: Bytes12) {
return Gadgets.BLAKE2B.hash(xs, 32);
},
},
},
});
23 changes: 23 additions & 0 deletions src/examples/crypto/blake2b/run.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import { Bytes12, BLAKE2BProgram } from './blake2b.js';

console.time('compile');
await BLAKE2BProgram.compile();
console.timeEnd('compile');

let preimage = Bytes12.fromString('hello world!');

console.log('blake2b rows:', (await BLAKE2BProgram.analyzeMethods()).blake2b.rows);

console.time('prove');
let proof = await BLAKE2BProgram.blake2b(preimage);
console.timeEnd('prove');
let isValid = await BLAKE2BProgram.verify(proof);

console.log('digest:', proof.publicOutput.toHex());

if (
proof.publicOutput.toHex() !==
'4fccfb4d98d069558aa93e9565f997d81c33b080364efd586e77a433ddffc5e2'
)
throw new Error('Invalid blake2b digest!');
if (!isValid) throw new Error('Invalid proof');
6 changes: 6 additions & 0 deletions src/lib/provable/crypto/hash.ts
Original file line number Diff line number Diff line change
Expand Up @@ -138,4 +138,10 @@ const Hash = {
return Keccak.preNist(512, bytes);
},
},

BLAKE2B: {
hash(bytes: Bytes) {
return Gadgets.BLAKE2B.hash(bytes);
},
},
};
54 changes: 52 additions & 2 deletions src/lib/provable/gadgets/arithmetic.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ import { provableTuple } from '../types/struct.js';
import { Field } from '../wrapped.js';
import { assert } from '../../util/errors.js';
import { Provable } from '../provable.js';
import { rangeCheck32, rangeCheckN } from './range-check.js';
import { rangeCheck32, rangeCheck64, rangeCheckN } from './range-check.js';

export { divMod32, addMod32 };
export { divMod32, addMod32, divMod64, addMod64 };

function divMod32(n: Field, nBits = 64) {
assert(
Expand Down Expand Up @@ -56,3 +56,53 @@ function divMod32(n: Field, nBits = 64) {
function addMod32(x: Field, y: Field) {
return divMod32(x.add(y), 33).remainder;
}


function divMod64(n: Field, nBits = 128) {
assert(nBits >= 0 && nBits <= 255, `nBits must be in the range [0, 255], got ${nBits}`)
boray marked this conversation as resolved.
Show resolved Hide resolved

// calculate the number of bits allowed for the quotient to avoid overflow
const quotientBits = Math.max(0, nBits - 64);

if (n.isConstant()) {
assert(
n.toBigInt() < 1n << 128n,
`n needs to fit into 128 bit, but got ${n.toBigInt()}`
);
let nBigInt = n.toBigInt();
let q = nBigInt >> 64n;
let r = nBigInt - (q << 64n);
return {
querolita marked this conversation as resolved.
Show resolved Hide resolved
remainder: new Field(r),
quotient: new Field(q),
};
}

let [quotient, remainder] = Provable.witness(
provableTuple([Field, Field]),
() => {
let nBigInt = n.toBigInt();
let q = nBigInt >> 64n;
let r = nBigInt - (q << 64n);
return [q, r] satisfies [bigint, bigint];
}
);

if (quotientBits === 1) {
quotient.assertBool();
querolita marked this conversation as resolved.
Show resolved Hide resolved
} else {
rangeCheckN(quotientBits, quotient);
}
boray marked this conversation as resolved.
Show resolved Hide resolved
rangeCheck64(remainder);

n.assertEquals(quotient.mul(1n << 64n).add(remainder));

return {
remainder,
quotient,
};
}

function addMod64(x: Field, y: Field) {
return divMod64(x.add(y), 128).remainder;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reasoning behind nBits being 128 in this case? (Instead of a larger or smaller value perhaps). Is it to prevent the quotient from being larger than 64 bits upon reduction?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah if both inputs are 64 bits then the result is 65 bits, that would cause an efficient (boolean) check for the quotient

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the question: can't we assume that the inputs are 64 bits and therefore use nBits=65?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it should be 65. It used to be 128 at some point. I adapted the changes made in #1763 and forgot to update nBits.

}
boray marked this conversation as resolved.
Show resolved Hide resolved
30 changes: 30 additions & 0 deletions src/lib/provable/gadgets/bitwise.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ export {
rotate64,
rotate32,
and,
or,
rightShift64,
leftShift64,
leftShift32,
Expand Down Expand Up @@ -173,6 +174,35 @@ function and(a: Field, b: Field, length: number) {
return outputAnd;
}

function or(a: Field, b: Field, length: number) {
boray marked this conversation as resolved.
Show resolved Hide resolved
// Validate at 240 bits to ensure padLength (next multiple of 16) doesn't exceed 254 bits,
// preventing potential underconstraint issues in the circuit
validateBitLength(length, 240, 'or');

// obtain pad length until the length is a multiple of 16 for n-bit length lookup table
let padLength = Math.ceil(length / 16) * 16;

// handle constant case
if (a.isConstant() && b.isConstant()) {
let max = 1n << BigInt(padLength);

assert(a.toBigInt() < max, `${a} does not fit into ${padLength} bits`);
assert(b.toBigInt() < max, `${b} does not fit into ${padLength} bits`);

return new Field(a.toBigInt() | b.toBigInt());
}

// calculate expect and output
let outputOr = Provable.witness(Field, () => a.toBigInt() | b.toBigInt());

outputOr.assertEquals(
not(and(not(a, length), not(b, length), length), length)
);

// return the result of the or operation
return outputOr;
}

function rotate64(
field: Field,
bits: number,
Expand Down
190 changes: 190 additions & 0 deletions src/lib/provable/gadgets/blake2b.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
// https://www.blake2.net/blake2.pdf
import { UInt64, UInt8 } from '../int.js';
import { FlexibleBytes } from '../bytes.js';
import { Bytes } from '../wrapped-classes.js';
import { Gadgets } from './gadgets.js';

export { BLAKE2B };

const BLAKE2BConstants = {
IV: [
UInt64.from(0x6a09e667f3bcc908n),
UInt64.from(0xbb67ae8584caa73bn),
UInt64.from(0x3c6ef372fe94f82bn),
UInt64.from(0xa54ff53a5f1d36f1n),
UInt64.from(0x510e527fade682d1n),
UInt64.from(0x9b05688c2b3e6c1fn),
UInt64.from(0x1f83d9abfb41bd6bn),
UInt64.from(0x5be0cd19137e2179n),
],

SIGMA: [
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
[14, 10, 4, 8, 9, 15, 13, 6, 1, 12, 0, 2, 11, 7, 5, 3],
[11, 8, 12, 0, 5, 2, 15, 13, 10, 14, 3, 6, 7, 1, 9, 4],
[7, 9, 3, 1, 13, 12, 11, 14, 2, 6, 5, 10, 4, 0, 15, 8],
[9, 0, 5, 7, 2, 4, 10, 15, 14, 1, 11, 12, 6, 8, 3, 13],
[2, 12, 6, 10, 0, 11, 8, 3, 4, 13, 7, 5, 15, 14, 1, 9],
[12, 5, 1, 15, 14, 13, 4, 10, 0, 7, 6, 3, 9, 2, 8, 11],
[13, 11, 7, 14, 12, 1, 3, 9, 5, 0, 15, 4, 8, 6, 2, 10],
[6, 15, 14, 9, 11, 3, 0, 8, 12, 2, 13, 7, 1, 4, 10, 5],
[10, 2, 8, 4, 7, 6, 1, 5, 15, 11, 9, 14, 3, 12, 13, 0],
],
};

const BLAKE2B = {
hash(data: FlexibleBytes, digestLength = 64) {
querolita marked this conversation as resolved.
Show resolved Hide resolved
const state = initialize(digestLength);
update(state, Bytes.from(data).bytes);
const out = final(state);
return Bytes.from(out);
},
get IV() {
return BLAKE2BConstants.IV;
},
};

function initialize(outlen: number): {
h: UInt64[];
t: UInt64[];
f: UInt64[];
buf: UInt8[];
buflen: number;
outlen: number;
} {
const h = BLAKE2B.IV.slice(); // shallow copy IV to h
h[0] = UInt64.from(0x01010000).xor(UInt64.from(outlen)).xor(h[0]);
querolita marked this conversation as resolved.
Show resolved Hide resolved
return {
querolita marked this conversation as resolved.
Show resolved Hide resolved
h,
t: [UInt64.zero, UInt64.zero],
querolita marked this conversation as resolved.
Show resolved Hide resolved
f: [UInt64.zero, UInt64.zero],
buf: [],
buflen: 0,
outlen,
};
}

function G(
v: UInt64[],
a: number,
b: number,
c: number,
d: number,
x: UInt64,
y: UInt64
) {
v[a] = UInt64.Unsafe.fromField(Gadgets.divMod64(v[a].value.add(v[b].value.add(x.value)), 128).remainder);
v[d] = v[d].xor(v[a]).rotate(32, 'right');

v[c] = UInt64.Unsafe.fromField(Gadgets.divMod64(v[c].value.add(v[d].value), 128).remainder);
v[b] = v[b].xor(v[c]).rotate(24, 'right');

v[a] = UInt64.Unsafe.fromField(Gadgets.divMod64(v[a].value.add(v[b].value.add(y.value)), 128).remainder);
v[d] = v[d].xor(v[a]).rotate(16, 'right');

v[c] = UInt64.Unsafe.fromField(Gadgets.divMod64(v[c].value.add(v[d].value), 128).remainder);
v[b] = v[b].xor(v[c]).rotate(63, 'right');
}

function compress(state: {
h: UInt64[];
t: UInt64[];
f: UInt64[];
buf: UInt8[];
buflen: number;
outlen: number;
}): void {
const { h, t, f, buf } = state;
const v = h.concat(BLAKE2B.IV); // initalize local work vector. First half from state and second half from IV.

v[12] = v[12].xor(t[0]); // low word of the offset
v[13] = v[13].xor(t[1]); // high word of the offset
v[14] = v[14].xor(f[0]);
v[15] = v[15].xor(f[1]);

const m: UInt64[] = [];
for (let i = 0; i < 16; i++) {
m.push(
querolita marked this conversation as resolved.
Show resolved Hide resolved
UInt64.from(
buf[i * 8]
.toUInt64()
.or(buf[i * 8 + 1].toUInt64().leftShift(8))
.or(buf[i * 8 + 2].toUInt64().leftShift(16))
.or(buf[i * 8 + 3].toUInt64().leftShift(24))
.or(buf[i * 8 + 4].toUInt64().leftShift(32))
.or(buf[i * 8 + 5].toUInt64().leftShift(40))
.or(buf[i * 8 + 6].toUInt64().leftShift(48))
.or(buf[i * 8 + 7].toUInt64().leftShift(56))
mitschabaude marked this conversation as resolved.
Show resolved Hide resolved
)
);
}

for (let i = 0; i < 12; i++) {
const s = BLAKE2BConstants.SIGMA[i % 10];
G(v, 0, 4, 8, 12, m[s[0]], m[s[1]]);
G(v, 1, 5, 9, 13, m[s[2]], m[s[3]]);
G(v, 2, 6, 10, 14, m[s[4]], m[s[5]]);
G(v, 3, 7, 11, 15, m[s[6]], m[s[7]]);

G(v, 0, 5, 10, 15, m[s[8]], m[s[9]]);
G(v, 1, 6, 11, 12, m[s[10]], m[s[11]]);
G(v, 2, 7, 8, 13, m[s[12]], m[s[13]]);
G(v, 3, 4, 9, 14, m[s[14]], m[s[15]]);
}

for (let i = 0; i < 8; i++) {
h[i] = v[i].xor(v[i + 8]).xor(h[i]);
}
}

function update(
state: {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

h: UInt64[];
t: UInt64[];
f: UInt64[];
buf: UInt8[];
buflen: number;
outlen: number;
},
input: UInt8[]
): void {
for (let i = 0; i < input.length; i++) {
if (state.buflen === 128) {
state.t[0] = state.t[0].add(128);
if (state.t[0].equals(UInt64.zero)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you explain in what situations this conditional would succeed? I am asking because it seems like the counter t[0] gets updated using a normal add, whereas the t[1] uses a modular addition instead. Are you thinking of a UInt64 overflow? If so, maybe it's better to use addMod64 here as well.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw @boray this is a bug. you can't use a Bool as condition, it's an object, always truthy!

I highly recommend to do out-of-circuit logic on plain JS values, like bigint to avoid mistakes like this, i.e. here I would use

if (state.t[0].toBigint() === 0n) {

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah but wait, is this supposed to be circuit code and state.t[0] is a variable? then you fundamentally can't use a JS if condition anyway

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

state.t[0] is a variable but I am not sure if that logic should be in-circuit or not. I added the in-circuit version as a comment.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I implemented this part with reference to the snippet below from the BLAKE2B RFC.

      ctx->t[0] += ctx->c;                // mark last block offset
       if (ctx->t[0] < ctx->c)             // carry overflow
           ctx->t[1]++;                    // high word

state.t[1] = state.t[1].addMod64(UInt64.one);
}
compress(state);
state.buflen = 0;
}
state.buf[state.buflen++] = input[i];
}
}

function final(state: {
h: UInt64[];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: state type here too

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

t: UInt64[];
f: UInt64[];
buf: UInt8[];
buflen: number;
outlen: number;
}): UInt8[] {
state.t[0] = state.t[0].add(state.buflen);
if (state.t[0].equals(UInt64.zero)) {
state.t[1] = state.t[1].add(UInt64.zero);
querolita marked this conversation as resolved.
Show resolved Hide resolved
}
state.f[0] = UInt64.from('0xFFFFFFFFFFFFFFFF');

while (state.buflen < 128) {
querolita marked this conversation as resolved.
Show resolved Hide resolved
state.buf[state.buflen++] = UInt8.from(0);
}
compress(state);

const out: UInt8[] = [];
for (let i = 0; i < state.outlen; i++) {
out[i] = UInt8.from(
state.h[i >> 3].rightShift(8 * (i & 7)).and(UInt64.from(0xff))
querolita marked this conversation as resolved.
Show resolved Hide resolved
boray marked this conversation as resolved.
Show resolved Hide resolved
);
}
return out;
}
3 changes: 1 addition & 2 deletions src/lib/provable/gadgets/elliptic-curve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ import {
} from '../../../bindings/crypto/bigint-helpers.js';
import {
CurveAffine,
GroupAffine,
affineAdd,
affineDouble,
} from '../../../bindings/crypto/elliptic-curve.js';
Expand Down Expand Up @@ -394,7 +393,7 @@ function multiScalarMulConstant(
// TODO dedicated MSM
let s = scalars.map(Field3.toBigint);
let P = points.map(Point.toBigint);
let sum: GroupAffine = Curve.zero;
let sum = Curve.zero;
for (let i = 0; i < n; i++) {
if (useGlv) {
sum = Curve.add(sum, Curve.Endo.scale(P[i], s[i]));
Expand Down
Loading