03-methods.tex

\section{Methods}

\subsection{Reflexive Interviews with Employers}

For our conversations with employers, we employed reflexive interviewing, a collaborative technique that extends beyond traditional interview methods by engaging both interviewer and interviewee in the co-construction of meaning and understanding. Unlike conventional interviews in which researchers primarily extract information, reflexive interviewing creates a dialogical space where participants collaborate in exploring, elaborating, and interpreting their own experiences and perspectives. The interview process is iterative, allowing for deeper investigation of themes that surface during the conversation \citep{pessoaUsingReflexiveInterviewing2019}.

Technology professionals involved in hiring entry-level software engineers were recruited from program partner companies, meetup attendees, and our personal networks to participate in individual online meetings using a purposive sampling approach. During these one-on-one sessions, participants engaged in interviews exploring the specific traits and competencies they prioritized when evaluating junior developer candidates. Interviews were framed around ``ideal'' candidates. Specifically, we asked the participants to answer the question, ``What would you need to see on a resume so that you would feel like you made a mistake if you didn't interview this person?'' Participants were then asked to provide concrete examples of how these desired traits could be demonstrated through open source contributions made by college students.

During each interview, the insights were synthesized into individualized ``hiring manager agreements'' that captured each participant's desired qualities and evidence criteria. These documents underwent iterative editing as participants reflected on and refined their stated requirements in partnership with the researcher. The iterative nature of this approach allowed for deeper exploration of initially ambiguous or contradictory requirements, with participants able to clarify and elaborate on their expectations through continued dialogue.

Surveys of employers are frequently subject to response bias, especially politeness bias \citep{leeBestAnswerReally2019}, social desirability bias \citep{furnhamResponseBiasSocial1986}, and acquiescence bias \citep{knowlesAcquiescentRespondingSelfReports1997}. In an effort to reduce these biases, we asked each industry professional to e-sign an agreement committing to interviewing an agreed number of students who met their documented requirements. We also expected that this would provide additional motivation for students.

\subsection{Student Motivation Survey}

Student motivation was evaluated through a survey using EVT instruments from two sources---\citet{ecclesMindActorStructure1995} and \citet{flakeMeasuringCostForgotten2015}---but adapted to refer to an open source contribution rather than mathematical skills (as in \citet{ecclesMindActorStructure1995}) or a specific class (\citet{flakeMeasuringCostForgotten2015}). These instruments were selected and modified in prior work by the authors and based on the work of others \citep{olewnikCocurricularEngagementEngineering2023}. The final instrument is presented in Appendix~\ref{tab:motivationSurvey}.

As part of our program, students were already assigned to sections on the basis of time availability; one of these sections was randomly selected as the experimental group. The control group comprised the students in all other sections in the same program in the same year. All students were enrolled in an extracurricular program that included interview preparation, resume advice, and the opportunity to contribute to open source projects \citep{narayananScalableApproachSupport2023}.

Students in the experimental group were shown the hiring manager agreements as part of a regularly scheduled workshop and provided with a link allowing them to read the agreements in detail, before being asked to complete the motivation survey. Responses to the student motivation survey were compared between students in the experiment and control groups. After the survey was completed, all students were provided with equal access to the hiring manager agreements.