Add support for significant figures (sig_figs) parameter - #273
Conversation
…ation and documentation
peterbjohnson
left a comment
There was a problem hiding this comment.
Questions:
-
units: are there tests for physical quantities with different units, e.g. 1 mile vs 1609 m, combined with sig_figs? How does it work?
-
symbols: can symbols be used with sig_figs? E.g. x**2?
-
relaxed: we previously discussed two versions of sig_figs. One is the scientific version, implemented here, the other is the more relaxed sense of 'correct within x sig_fig'. Are we offering both, or only the first? What's our reasoning?
-
clashes: is this the first time we've had incompatible parameters? Any thoughts on user experience, given that we provide atol/rtol by default?
I've also made a minor inline comment
| # must still fail, so this can't be gated behind "ordinary equality already returned False". | ||
| lhs_string = criterion.children[0].content_string().strip() | ||
| rhs_string = criterion.children[1].content_string().strip() | ||
| if {lhs_string, rhs_string} == {"response", "answer"}: |
There was a problem hiding this comment.
Is this different to the test in the physical_quantity context line 277, which permitted the opposite variant too (both of response==answer and answer==response)?
Problem
This was a requested feature (#270). At the moment, the evaluation function can only check a numeric answer using absolute or relative tolerance. Some questions need something more specific: they need to check that a student's answer is written to a required number of significant figures, not just that the value is close enough.
For example, if the correct answer is
92and the question asks for 4 significant figures,92.00should be accepted but92should not, even though they represent the same number. Tolerance alone cannot tell these apart, since it only checks how close the value is, not how precisely it was written.Changes
app/utility/expression_utilities.py: added four new functions.round_to_sig_figs: rounds a number to a given number of significant figures.split_numeric_string: checks that a response is written as a plain number (e.g."92.00"), and splits it into its whole-number part, decimal part, and whether a decimal point was written.count_sig_figs: counts the significant figures in those parts, following the standard rules (leading zeros do not count, trailing zeros after a decimal point do).sig_figs_match: combines the three functions above to decide whether a response is both numerically correct and written to the right number of significant figures. To compare the rounded response and rounded answer, it uses Python'smath.ulpfunction as the allowed error, instead of comparing with==. Rounding a number to a fixed number of significant figures does not always produce an exact result in floating point arithmetic, so two values that should be treated as equal can end up differing by a tiny amount.math.ulp(x)gives the smallest possible gap betweenxand the next representable floating point number, so using it as the allowed error accepts only that unavoidable floating point rounding error, without being loose enough to accept a genuinely different value. This is the same idea asnumpy.spacing, which is what the equivalent Is-Similar feature uses, butmath.ulpis part of the standard library, so no new dependency (numpy) needed to be added to this project.app/evaluation.py: added thesig_figsparameter, withsignificant_figuresaccepted as an alternative name. Added checks that raise an error ifsig_figsis used together withatol/rtol, or ifsig_figsis not a positive whole number.app/context/symbolic.py: incheck_equality, added a check at the very start of the function. Ifsig_figsis set and the comparison is a directresponse = answer(not some other custom comparison), the function now usessig_figs_matchinstead of its normal equality check. This has to happen before the normal check, not after, because92.00and92would otherwise already be treated as equal (they are the same number) before the significant figures were ever counted.app/context/physical_quantity.py: made the equivalent change insidequantity_match, which is the function that decides whether a response matches the answer for physical quantities. Whensig_figsis set, it replaces the normal value check withsig_figs_match, using the response's value as originally written (before unit conversion) so the significant figures can be counted correctly. The unit comparison itself is unchanged. Also updated the existing logic that estimates a tolerance from the number of significant figures in the answer, so it is skipped wheneversig_figsis set, keeping the two features independent.app/docs/user.md: added a section explaining the newsig_figs/significant_figuresparameter, with an example and the significant figure counting rules.app/docs/dev.md: added a short technical note for developers on how the feature fits into the existing code.app/tests/expression_utilities_test.py: added tests for the four new functions.app/tests/symbolic_evaluation_test.pyandapp/tests/physical_quantity_evaluation_test.py: added tests that call the full evaluation function withsig_figsset, covering correct answers, wrong values, wrong precision, non-numeric answers, and the new error cases.Checklist
app/docs.md) if user-facing behaviour changedCloses #270