Monkeypatching pandas series to_csv with pytest

[Tim-Kirkwood]

Hello,

I am testing a function, write_query(), in module.py. My test is in test_module.py, which is carried out using pytest. I am using the pytest monkeypatch fixture to monkeypatch Series.to_csv(), in order to prevent file writing. It works in part - my monkeypatched function performs checks that seem to complete, and raises an error that is expected and indicates a passed test (I did this so I could explicitly include a test in the test function, rather than calling the function in a test and have the test performed in the function alone). However, it is still writing a file, doesnt_matter, and I'm not sure why. Can anyone explain? Please note, I have tried raising this on stack overflow but haven't had much traction. Please let me know if I can provide any more information.

Thanks!

Tim

folder/module.py

import pandas as pd

def write_query(df, outpath, entry_name_col):
    
    #Make a list of queries.  NB - no pandas objects/methods involved in 
    #build_query()
    entry_names = build_query(df, entry_name_col) 
    
    #Write the queries to a csv
    with open(outpath, 'w') as csv_file: 
        pd.Series(entry_names).to_csv(csv_file,
                                      index=False, 
                                      lineterminator='\n', 
                                      header = False)

folder/test_module.py

import os
import pytest
from folder.module import build_query, write_query

def temp_to_csv(self, csv_file, index, lineterminator, header):

    #check csv_file is correct
    assert csv_file.name == 'doesnt_matter'

    #check values to be written are correct
    if list(self.values) == ['(id:"entry_name_1")', '(id:"entry_name_2")']:

        #if they are, raise an error with a specific message that can be checked 
        #in the main test.  
        raise ValueError('Correct values passed to to_csv') 

    else: 
        raise ValueError
        
class TestWriteQuery:
    def test_normal(self, 
                    good_input_queries,   #dataframe fixture in conftest.py 
                    monkeypatch):

        #check file doesnt exist before function is called
        assert 'doesnt_matter' not in os.listdir(os.getcwd())  #This passes
        
        #monkeypatch Series.to_csv() - have also tried module.pd.Series.to_csv, 
        #pandas.Series.to_csv, pd.Series.to_csv, Series.to_csv
        monkeypatch.setattr('folder.module.pd.Series.to_csv', 
                            temp_to_csv)

        #call function and test raised error
        with pytest.raises(ValueError) as e:                   #This passes
            write_query(good_input_queries, 
                        outpath = 'doesnt_matter', 
                        entry_name_col = 'Entry name')
        #e is <ExceptionInfo ValueError('Correct values passed to to_csv') tblen=3>
        assert 'Correct values passed to to_csv' in str(e)     #This passes
        
        #check file has not been written
        assert 'doesnt_matter' not in os.listdir(os.getcwd())  #This fails

[portyu9]

Hi @Tim-Kirkwood

The file is being created by this line:

with open(outpath, "w") as csv_file:

not by:

pd.Series(entry_names).to_csv(...)

Opening a path with mode "w" creates the file if it doesn't exist (and truncates it if it does).

So even if you successfully patch:

pd.Series.to_csv

this still happens:

with open(outpath, "w") as csv_file:
    # file already exists at this point

If your goal is to test that to_csv() is called correctly, then the existence of the empty file is expected.

For example:

mock_to_csv.assert_called_once()

can verify the pandas interaction independently.

If you want a test with no filesystem interaction at all, mock open() as well:

from unittest.mock import mock_open, patch


def test_write_query(monkeypatch):
    mock_to_csv = ...

    monkeypatch.setattr(
        "your_module.pd.Series.to_csv",
        mock_to_csv,
    )

    with patch("builtins.open", mock_open()):
        write_query(df, "output.csv", "name")

    mock_to_csv.assert_called_once()

If the module imported open under another name, patch the name where your code actually looks it up.

For this function, though, I would probably use pytest's tmp_path and test the observable result instead:

def test_write_query(tmp_path):
    output = tmp_path / "query.csv"

    write_query(df, output, "name")

    assert output.read_text() == "expected\ncontent\n"

That tests the real behavior of write_query() without touching your actual filesystem and avoids mocking pandas implementation details.

So the monkeypatch is not creating the file — open(outpath, "w") is.

Please mark this answer as accepted if it helped, thank you.

[Tim-Kirkwood]

Thanks very much for your reply and suggested fixes!