docs: Create pull_request_template.md

[jlkravitz]

This is the template for PRs, for us and for the outside world (to, e.g., add models and tasks). MTEB has a dataset and task checklist. For reference, below is the template they use.

---

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

Adding datasets checklist

Reason for dataset addition: ...

• I have run the following models on the task (adding the results to the pr). These can be run using the mteb -m {model_name} -t {task_name} command.
  • sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2
  • intfloat/multilingual-e5-small
• I have checked that the performance is neither trivial (both models gain close to perfect scores) nor random (both models gain close to random scores).
• If the dataset is too big (e.g. >2048 examples), considering using self.stratified_subsampling() under dataset_transform()
• I have filled out the metadata object in the dataset file (find documentation on it here).
• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

Adding a model checklist

• I have filled out the ModelMeta object to the extent possible
• I have ensured that my model can be loaded using
  • mteb.get_model(model_name, revision_id) and
  • mteb.get_model_meta(model_name, revision_id)
• I have tested the implementation works on a representative set of tasks.

[scopello]

Looks good, thanks!

[jlkravitz]

anything from the DGEB checklist you wanna add?

…

On Mon, Jul 15, 2024 at 9:03 AM scopello ***@***.***> wrote: ***@***.**** approved this pull request. Looks good, thanks! — Reply to this email directly, view it on GitHub <#13 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAWWZK2CLMPW4JBBSTWFLZTZMPCA7AVCNFSM6AAAAABKXZ3TYWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCNZXGY2TQNJRGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ninjha01]

Two thoughts:

1. This is totally a nit, but do we care about all of these distinctions? On my end, all I care about is breaking change/non breaking change + motivation for the PR.

- [ ] Bugfix
- [ ] Feature (e.g., including new dataset or model)
- [ ] Code style update (formatting, local variables)
- [ ] Refactoring (no functional changes, no api changes)
- [ ] Build related changes
- [ ] CI related changes
- [ ] Documentation content changes
- [ ] Other... Please describe:

Perhaps contradictorily, I'd like to have a section for the reviewer to go through:

[ ] CI passes
[ ] docs updated
[ ] manual spotcheck

2. Perhaps off topic, but should we have a separate template for the actual results submission? Or could we fold that into this? I can create an issue and take this on if we think it's a good idea.

[jlkravitz]

Two thoughts:

1. This is totally a nit, but do we care about all of these distinctions? On my end, all I care about is breaking change/non breaking change + motivation for the PR.

- [ ] Bugfix
- [ ] Feature (e.g., including new dataset or model)
- [ ] Code style update (formatting, local variables)
- [ ] Refactoring (no functional changes, no api changes)
- [ ] Build related changes
- [ ] CI related changes
- [ ] Documentation content changes
- [ ] Other... Please describe:

Perhaps contradictorily, I'd like to have a section for the reviewer to go through:

[ ] CI passes
[ ] docs updated
[ ] manual spotcheck

2. Perhaps off topic, but should we have a separate template for the actual results submission? Or could we fold that into this? I can create an issue and take this on if we think it's a good idea.

On 1, I think my thought is that I would like to appropriately tag the commit, and these correspond 1:1 with the conventional commit standard. I'm open to changing them or making them more aligned with this project, though. Thoughts?

Imo, CI passes: that's automated and required to merge.
docs updated: added
manual spotcheck: not sure if this adds anything?

On 2, I think we could fold into this. It seems like it can get annoying to have multiple PR templates. I'll merge this, and then we can iterate on it if that works!

[ninjha01]

works for me!

On Mon, Jul 15, 2024, at 12:58 PM, Joshua Kravitz wrote: > Two thoughts: > > 1. This is totally a nit, but do we care about all of these distinctions? On my end, all I care about is breaking change/non breaking change + motivation for the PR. > `- [ ] Bugfix > - [ ] Feature (e.g., including new dataset or model) > - [ ] Code style update (formatting, local variables) > - [ ] Refactoring (no functional changes, no api changes) > - [ ] Build related changes > - [ ] CI related changes > - [ ] Documentation content changes > - [ ] Other... Please describe:

`

> Perhaps contradictorily, I'd like to have a section for the reviewer to go through: > > `[ ] CI passes > [ ] docs updated > [ ] manual spotcheck

`

…

> 1. Perhaps off topic, but should we have a separate template for the actual results submission? Or could we fold that into this? I can create an issue and take this on if we think it's a good idea. On 1, I think my thought is that I would like to appropriately tag the commit, and these correspond 1:1 with the conventional commit standard <https://www.conventionalcommits.org/en/v1.0.0/>. I'm open to changing them or making them more aligned with this project, though. Thoughts? Imo, CI passes: that's automated and required to merge. docs updated: added manual spotcheck: not sure if this adds anything? On 2, I think we could fold into this. It seems like it can get annoying to have multiple PR templates. I'll merge this, and then we can iterate on it if that works! — Reply to this email directly, view it on GitHub <#13 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGA4JHE4NBZ7GJHUUWMMLJ3ZMQSXJAVCNFSM6AAAAABKXZ3TYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRZGI3TMOJUGQ>. You are receiving this because your review was requested.Message ID: ***@***.***>