qna2021

Questions from the STAK Conference 2021

Are the ratings for the questions fixed by the teacher, or can they be dynamically adjusted based on student performance?

The ratings change depending on how often they are answer correctly compared to other questions. However, it is possible to fix the ratings, by setting the K-factor to zero. Each time a rating changes, the change is proportional to K.

And if they’re fixed, how are you deciding on the ratings?

Whether the ratings are fixed or not, the teacher needs to estimate the difficulty and set a starting rating, so that the first questions the student sees are at least reasonably easy.

Rating estimation is a difficult problem which has been studied in the literature on Computer Adaptive Testing (CAT). CAP is less sensitive to these estimates, because we do not use them for quantitative assessment, and we allow the estimates to improve over time.

Have you used this at a large scale?

In my first semester mathematics module for business students, I have had 150-180 students and and respectively 42 and 59 questions per activity. The first activity used mainly Calculated questions, while the second had about 40 STACK questions and the rest multiple choice.

Can the thresholds for stars be changed once the quiz is set up? Say one realises that it might be too hard to obtain a particular rating with the current questions.

Yes. Everything can be reconfigured after the quis has started. However, the students may have to answer some more questions after the change to benefit. It may have to be tested.

I think you said this already works with STACK questions - how does it deal with random variants within STACK questions?

CAPQuiz is oblivious to the randomisation. If the question knows that it should be randomised, it is also randomised when used within CAPQuiz.

However, the typical STACK use cases as discussed in the community only need 5-10 pre-instantiated question instances. In CAPQuiz, because the students need to do the same problems over and over again until they get it right, we should have a lot more different variants than this. Our experience with CAPQuiz is that STACK scales poorly in the number of instances. After having run quizzes with 100-120 variants per question, I have concluded that the questions should not be pre-instantiated at all when used in CAPQuiz.