Skip to content

fix: give the LCB sandbox's fake stdout a binary buffer - #1465

Open
abhishekcpp007 wants to merge 1 commit into
huggingface:mainfrom
abhishekcpp007:fix/lcb-capturing-stdout-buffer
Open

abhishekcpp007 wants to merge 1 commit into
huggingface:mainfrom
abhishekcpp007:fix/lcb-capturing-stdout-buffer

Conversation

@abhishekcpp007

Copy link
Copy Markdown

What does this PR do?

Capturing swaps sys.stdout for a plain StringIO while a candidate solution runs, and the LCB grader reads the solution's answer back out of it. The real sys.stdout also exposes a binary .buffer, and a StringIO does not, so a correct solution that writes with sys.stdout.buffer.write(...) raises AttributeError inside the sandbox and is graded as wrong.

Writing bytes straight to the buffer is an ordinary fast-output idiom in competitive programming, so the grader was scoring solutions on how they emit their answer rather than on the answer.

Before:

print                      -> '6'
sys.stdout.write           -> '6'
sys.stdout.buffer.write    -> AttributeError: '_io.StringIO' object has no attribute 'buffer'

Fix

The fake stdout now carries a .buffer that decodes writes into the text stream Capturing reads from, so buffered output lands in the captured result and keeps its order relative to print(). Undecodable bytes are replaced rather than raising, since a grader should not crash on a solution's output.

RawIOBase rather than BytesIO is deliberate, and it is the part worth reviewing. BytesIO implements writelines in C against its own internal buffer, so it bypasses an overridden write and the output is dropped with no error:

class WriteThrough(BytesIO):
    def __init__(self, text):
        super().__init__()
        self._text = text

    def write(self, data):
        self._text.write(bytes(data).decode())
        return len(data)

t = StringIO()
b = WriteThrough(t)
b.write(b"one\n")
b.writelines([b"two\n", b"three\n"])
print(repr(t.getvalue()))   # 'one\n'  <- two and three are gone

That would trade a loud AttributeError for a quiet wrong answer, which is the worse outcome for a grader. RawIOBase routes writelines through the overridden write.

Tests

tests/unit/tasks/test_lcb_capturing.py covers print(), sys.stdout.write, sys.stdout.writelines, and both buffer write paths. All of them print the same answer and all are required to capture identically. It also covers interleaving of text and binary writes, the byte count write returns, restoration of the real sys.stdout, and undecodable bytes.

The buffer cases fail on main with AttributeError and pass here.

Relationship to #1255 and #1257

This is the output-side counterpart of #1255, which reports the same defect on sys.stdin. That one is already being fixed in #1257, so I have deliberately left call_method untouched and the two do not overlap. #1257 changes call_method, this changes Capturing, and they add different test files.

Closes #1464

`Capturing` swaps sys.stdout for a StringIO while a solution runs, and the
grader reads the solution's answer back out of it. A real sys.stdout also
exposes a binary `.buffer`, and a StringIO does not, so a correct solution
that writes with sys.stdout.buffer.write() raises AttributeError inside the
sandbox and is graded as wrong. Writing bytes straight to the buffer is an
ordinary fast-output idiom in competitive programming, so the grader was
scoring solutions on how they emit their answer rather than on the answer.

The fake stdout now carries a `.buffer` that decodes writes into the text
stream Capturing reads from, so buffered output lands in the captured result
and keeps its order relative to print(). Undecodable bytes are replaced
rather than raising, since a grader should not crash on a solution's output.

RawIOBase rather than BytesIO is deliberate. BytesIO implements writelines
in C against its own buffer, so it bypasses an overridden write and drops
that output silently, which would turn a visible AttributeError into a
wrong answer that is much harder to notice.

Tests cover print(), sys.stdout.write, sys.stdout.writelines and both buffer
write paths, plus interleaving, the returned byte count, restoration of the
real stdout, and undecodable bytes.

This is the output-side counterpart of huggingface#1255, which is about sys.stdin and
is already being fixed in huggingface#1257. The two do not overlap.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] LCB grader fails correct solutions that write via sys.stdout.buffer

1 participant