Events for exams IDA #268

ilee2u · 2023-09-19T21:03:46Z

Description: We are developing a new backend for exams called edx-exams. This backend will be utilized by instructor dashboards to interact with student exam attempts, and will store data regarding those attempts.

We are currently working to implement the downstream effects triggered whenever an exam attempt is submitted or reviewed. For example, when an exam attempt is submitted, we will want to make sure edx-platform knows to mark the exam subsection.

We have added several event definitions based on different actions that can be performed on exam attempts:

EXAM_ATTEMPT_SUBMITTED
EXAM_ATTEMPT_REJECTED
EXAM_ATTEMPT_VERIFIED
EXAM_ATTEMPT_ERRORED
EXAM_ATTEMPT_RESET

These events will be consumed by various services in edx-platform in order to perform tasks such as creating grade overrides, invalidating certificates, changing credit requirement statuses, and more.

ISSUE: MST-2083 (Internal)

Dependencies: None.

Merge deadline: List merge deadline (if any)

Installation instructions: List any non-trivial installation
instructions.

Testing instructions:

Open page A
Do thing B
Expect C to happen
If D happened instead - check failed.

Reviewers:

tag reviewer
tag reviewer

Merge checklist:

Post merge:

Create a tag
Check new version is pushed to PyPI after tag-triggered build is
finished.
Delete working branch (if not needed anymore)

Author concerns: List any concerns about this PR - inelegant
solutions, hacks, quick-and-dirty implementations, concerns about
migrations, etc.

Zacharis278 · 2023-09-21T18:05:05Z

openedx_events/learning/data.py

+        credit_requirement_status (str): status to change the student's CreditRequirement to
+    """
+
+    user_id = attr.ib(type=int)


I noticed @MichaelRoytman's PR uses the UserData object here. Do you think we should use that? The LMS can look it up but it might be nice to match how other events are sending user id / email / etc

Looking at the parameters of the downstream functions, it looks like most of them take user_id as a parameter anyways, so I think user_id is the way to go here.

Also I believe Michael's PR just has placeholder data for now.

I think it would make sense to use the UserData. It seems to be more common to use that than a user_id. But I do see uses of both, so there's precedence to use both.

I don't think we have to model our events off the way they're being consumed now. Ideally, they shouldn't be so context aware. I think, for me, it's less, "the Instructor Service currently only expects a user_id so that's what we should put in the event" and more "how can I best and most generically represent the concept of a user in my event"?

Is sending PII when we don't need to a consideration?

I think there's a good chance future downstream logic would use the pii. This probably gets hooked up to name affirmation at some point.

openedx_events/learning/signals.py

Zacharis278

One comment on how we send user info. Otherwise this looks good to me

openedx_events/learning/signals.py

openedx_events/learning/data.py

MichaelRoytman

Nice job!

openedx_events/learning/data.py

robrap · 2023-09-21T19:53:48Z

openedx_events/learning/signals.py

+
+# .. event_type: org.openedx.learning.exam.attempt.submitted.v1
+# .. event_name: EXAM_ATTEMPT_SUBMITTED
+# .. event_description: Emitted when an exam attempt is submitted by a learner.


Some high-level questions:

Does the platform have other types of "exams" that would not trigger these events? If so, how do we differentiate? Note that an event is defined by its event trigger + event data.

For the exams covered by these events, are there multiple exam types? Would it be useful to know that as part of the event data?

If there are different exam types, do some of these events only apply to certain types of exam types? If so, would that be useful to note?

kind of. The existing exam system (edx-proctoring) does not trigger these events, the service that emits these is intended to replacement. There are no other types of exams outside that.

Good point. There are multiple exam types and edx-platform is mostly unaware of this since all the state/ui flows are manage by the exams/proctoring service. That said, I think there is value including this. Consumers may likely care if an exam is proctored or not. We might have actually missed a need for this. I think Credits depend on if an exam is proctored.

Yup some of these events are impossible depending on the exam type. A timed exam cannot be rejected since there is no video review to reject for example. We should add those details to the event descriptions

Are there non-timed exams, and are those included in this as well? Or are these just timed-exams, proctored and not-proctored? Just curious.

🤔 I guess this doesn't include 'all' exams just 'special exams' which have a different UX and workflow. Right now, all special exam types are a subset of timed. This wouldn't include normal sections instructors are using as or are graded as exams however. We could possibly reintroduce the term 'special exam' for these events which could clear this up.

If you think of this more as an API discussion around exams and special-exams, who do you think should be a part of this discussion? Maybe https://openedx.atlassian.net/wiki/spaces/COMM/pages/3554214289/Arch+Coordination+-+How+to+announce+and+tune-in will help you get the right discussion going?

^ I think this is a good idea to have before we merge. I could see people getting confused on two points

these events only pertain to 'special' exams

these events are not published by the existing exams system

I've made the following changes to address this:

Changed the data name to "SpecialExamAttemptData"

Added a comment about how events using this data are only emitted for special exams from the newer exams backend, edx-exams

That's better! Thank you folks.

I don't want to push hard on this, because it's not my domain of expertise, but the naming difference between EXAM_ATTEMPT_SUBMITTED and SpecialExamAttemptData just catches my attention. The obvious other option is to rename to SPECIAL_EXAM_ATTEMPT_SUBMITTED , and I guess the question I have is around the pros/cons of going with the more specific name.

🤔 🤔 It's a good question, I'd almost lean towards not using 'special' in the name at all and just commenting to the meaning? Otherwise, we need to consistently use the term across Open edX. I don't think 'special exam' has any additional meaning to anyone working outside this domain unless it's described. There was a actually discussion a while back that we stop using this term because it's just confusing, trying to dig that up.

I'm not sure how to best describe the fact that this intends to include all content that is treated with an exam user experience, but not content that might be titled (on the section or grading policy) by course teams as 'exam'. The latter is ultimately no different from other content, the platform doesn't do anything functionally different. (hence not special)

openedx_events/learning/data.py

mariajgrimaldi

Hi there folks. I left a few minor comments. Thanks!

CHANGELOG.rst

mariajgrimaldi

Just a minor doc fix before we merge.

ilee2u · 2023-09-29T13:53:51Z

Hi, we've been informed that v8.6.0, which this PR is based off, is having issues, so please do not merge this until we find a solution/workaround for those issues.

CHANGELOG.rst

openedx_events/learning/data.py

Zacharis278 · 2023-09-29T18:36:24Z

docs/decisions/0013-special-exam-submission-and-review-events.rst

+Decision
+********
+
+* Since `edx-exams` is independent, we will want to prevent circular dependencies between `edx-platform` and `edx-exams`.


I don't think this is a decision that impacts this events library and is better explained in detail by the ADR you already linked above.

Zacharis278 · 2023-09-29T18:51:40Z

docs/decisions/0013-special-exam-submission-and-review-events.rst

+
+* Since `edx-exams` is independent, we will want to prevent circular dependencies between `edx-platform` and `edx-exams`.
+* We have defined the events that we plan to define in `this ADR <https://github.com/edx/edx-exams/blob/main/docs/decisions/0004-downstream-effect-events.rst>`_ in `edx-exams` to use the event bus, as the new exams backend is independent in this ADR
+* We plan for these events to be produced in `edx-exams`, and consumed in various `edx-platform` services (e.g. certificates, credit, instructor, grades).


I think we can flip our focus here to be less about edx-exams and more how these events impact edx-platform. Maybe

exam events will be consumed by edx-platform to handle all behavior as it pertains to the state of an exam subsection.

the edx-exams service will produce these events

there is no plan to have edx-proctoring produce these events

Zacharis278 · 2023-09-29T19:02:39Z

docs/decisions/0013-special-exam-submission-and-review-events.rst

+
+**Draft** 2023-09-29
+
+Context


I think the context section might be a good place to explain the meaning of exam / special exam.

Rough points to hit

course subsections with an exam_type have additional logic that governs completion, grading, credit requirements, and more based on the type of exam.

these sections are sometimes referred to as special exams

course subsections without an exam type configured may still have a grading policy named 'Exam'. This type of content does not have the exam user experience and is not governed by any exam specific logic.

- Also changed user_id to student_user

- Specified that the events are only emitted when special exams go through edx-exams. - Re-ordered params in data docstring

- I literally forgot a period oi

docs/decisions/0013-special-exam-submission-and-review-events.rst

MichaelRoytman mentioned this pull request Sep 20, 2023

Emit the ExamAttemptSubmitted Open edX event when an exam is submitted. edx/edx-exams#178

Merged

6 tasks

ilee2u force-pushed the exams-ida-hackathon branch from 423b066 to 6424a40 Compare September 21, 2023 15:40

ilee2u changed the title ~~Events for exams IDA (draft/experimental), might not be PR'd this way.~~ Events for exams IDA Sep 21, 2023

Zacharis278 reviewed Sep 21, 2023

View reviewed changes

openedx_events/learning/signals.py Outdated Show resolved Hide resolved

Zacharis278 reviewed Sep 21, 2023

View reviewed changes

openedx_events/learning/signals.py Outdated Show resolved Hide resolved

Zacharis278 reviewed Sep 21, 2023

View reviewed changes

MichaelRoytman reviewed Sep 21, 2023

View reviewed changes

openedx_events/learning/signals.py Outdated Show resolved Hide resolved

MichaelRoytman reviewed Sep 21, 2023

View reviewed changes

openedx_events/learning/data.py Outdated Show resolved Hide resolved

Zacharis278 approved these changes Sep 21, 2023

View reviewed changes

MichaelRoytman approved these changes Sep 21, 2023

View reviewed changes

ilee2u marked this pull request as ready for review September 21, 2023 18:44

ilee2u requested a review from a team as a code owner September 21, 2023 18:44

robrap reviewed Sep 21, 2023

View reviewed changes

openedx_events/learning/data.py Outdated Show resolved Hide resolved

robrap reviewed Sep 21, 2023

View reviewed changes

mariajgrimaldi reviewed Sep 28, 2023

View reviewed changes

openedx_events/learning/data.py Outdated Show resolved Hide resolved

mariajgrimaldi reviewed Sep 28, 2023

View reviewed changes

ilee2u force-pushed the exams-ida-hackathon branch from 37dd373 to 7e5cabd Compare September 28, 2023 19:28

mariajgrimaldi reviewed Sep 29, 2023

View reviewed changes

CHANGELOG.rst Outdated Show resolved Hide resolved

mariajgrimaldi approved these changes Sep 29, 2023

View reviewed changes

ilee2u mentioned this pull request Sep 29, 2023

feat: add/remove course staff role events #267

Merged

10 tasks

mariajgrimaldi reviewed Sep 29, 2023

View reviewed changes

CHANGELOG.rst Outdated Show resolved Hide resolved

mariajgrimaldi added the blocked by other work PR cannot be finished until other work is complete label Sep 29, 2023

robrap reviewed Sep 29, 2023

View reviewed changes

openedx_events/learning/data.py Show resolved Hide resolved

ilee2u force-pushed the exams-ida-hackathon branch 2 times, most recently from a360649 to ca3ff5e Compare September 29, 2023 18:23

Zacharis278 reviewed Sep 29, 2023

View reviewed changes

feat: add/remove course staff role events

a6bbecd

ilee2u added 22 commits October 2, 2023 10:27

feat: signals complete

f99c3b3

feat: improved docstrings pt 1

8e858fa

feat: improved event defs + avro

e39f330

fix: rebasing off main only

3ed95a5

fix: made signal purpose broader

e148d0f

fix: removed unecessary event param

e61f729

- Also changed user_id to student_user

fix: avro

0615aaf

fix: removed unecessary docstring

0f31223

chore: version bump

2e3de7a

feat: added exam_type to ExamAttemptData

6a2647f

fix: corrected exam_type attr.ib

121f6ba

feat: Added "Special" to data name

c3074b8

- Specified that the events are only emitted when special exams go through edx-exams. - Re-ordered params in data docstring

chore: avro

8917946

chore: move data import

d3dbf5a

chore: changelog correction

2de9b4a

chore: 8.70 -> 8.7.0

d9f9e1b

- I literally forgot a period oi

docs: clarified signal name prefix EXAM_

2a2299a

feat: removed "Special" from data name

918a641

docs: Added ADR Draft

d3c6581

chore: quality

c1a196d

docs: removed non-event related decision info

d7ce2e5

fix: improved ADR

9e4744c

ilee2u force-pushed the exams-ida-hackathon branch from 273f27a to 9e4744c Compare October 2, 2023 14:28

Zacharis278 reviewed Oct 2, 2023

View reviewed changes

docs/decisions/0013-special-exam-submission-and-review-events.rst Outdated Show resolved Hide resolved

ilee2u added 2 commits October 2, 2023 10:31

chore: nit

6304946

chore: Add ADR link to exam event data

80629ce

mariajgrimaldi merged commit b36ea94 into openedx:main Oct 2, 2023
8 checks passed

mariajgrimaldi removed the blocked by other work PR cannot be finished until other work is complete label Oct 2, 2023

MichaelRoytman mentioned this pull request Oct 4, 2023

Add grades event bus event handler for rejected special exam. openedx/edx-platform#33412

Merged

timmc-edx mentioned this pull request Oct 18, 2023

Add event bus consumers for edx-platform edx/edx-arch-experiments#474

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Events for exams IDA #268

Events for exams IDA #268

ilee2u commented Sep 19, 2023 •

edited

Loading

Zacharis278 Sep 21, 2023 •

edited

Loading

ilee2u Sep 21, 2023

MichaelRoytman Sep 21, 2023 •

edited

Loading

Zacharis278 Sep 21, 2023

Zacharis278 left a comment

MichaelRoytman left a comment

robrap Sep 21, 2023

Zacharis278 Sep 21, 2023

robrap Sep 21, 2023

Zacharis278 Sep 22, 2023

robrap Sep 22, 2023

Zacharis278 Sep 28, 2023

ilee2u Sep 28, 2023

mariajgrimaldi Sep 29, 2023

robrap Sep 29, 2023

Zacharis278 Sep 29, 2023

mariajgrimaldi left a comment

mariajgrimaldi left a comment

ilee2u commented Sep 29, 2023

Zacharis278 Sep 29, 2023 •

edited

Loading

Zacharis278 Sep 29, 2023

Zacharis278 Sep 29, 2023 •

edited

Loading


		Draft 2023-09-29

		Context

Events for exams IDA #268

Events for exams IDA #268

Conversation

ilee2u commented Sep 19, 2023 • edited Loading

Zacharis278 Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelRoytman Sep 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zacharis278 left a comment

Choose a reason for hiding this comment

MichaelRoytman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mariajgrimaldi left a comment

Choose a reason for hiding this comment

mariajgrimaldi left a comment

Choose a reason for hiding this comment

ilee2u commented Sep 29, 2023

Zacharis278 Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zacharis278 Sep 29, 2023 • edited Loading

Choose a reason for hiding this comment

ilee2u commented Sep 19, 2023 •

edited

Loading

Zacharis278 Sep 21, 2023 •

edited

Loading

MichaelRoytman Sep 21, 2023 •

edited

Loading

Zacharis278 Sep 29, 2023 •

edited

Loading

Zacharis278 Sep 29, 2023 •

edited

Loading