99-panel-feedback.tex

\section{Panel Summary}

\textbf{PANEL RECOMMENDATION:} Highly Competitive

Through experiential, mentored learning in Artificial Intelligence and Machine Learning, and through microinternships, the investigators plan to increase the partnership among higher education, workforce preparation credentialling, and AI/tech industry in Washington State through an annual ten-month program, of which eight months will be mentored. In so doing they hope to create a talent pipeline and launch the careers of 150 students from community and technical colleges (CTCs) in three annual cohorts.

\subsection{Intellectual Merit}

\subsubsection{Strengths}

The team of investigators has tried out the microinternship format successfully through PI Menezes’ CodeDay effort, and draws on their businesses’ experience in building the proposed project. Particular strengths:

\begin{itemize}
\item The feedback loop created by the mentoring sessions, and the flexibility of the formats for these, promised a well-supported experience for all.
\item Methods of education and training are pedagogically supported.
\item Appropriate remuneration for the time commitment required of the students is budgeted. The fact that the majority of the budget is designated to support students resonated with the RFP.
\item Not only the students, but the career mentors (based on protocols developed by P.I. Wang’s Mentors in Tech (MinT) program), are trained and know what they are doing. Moreover, the students are accorded time to reflect and change before reaching the end of the program. One reviewer noted that this proposal does well at fulfilling the eight principles put forth by the Society of Experiential Education. https://www.nsee.org/standards-and-practice
\end{itemize}

\subsubsection{Concerns}

The microinternship model proposed here relies on open source projects, which the investigators say promotes diversity. Concern was expressed that open source might not be as welcoming as anticipated, and that this would need to be monitored more closely than is indicated. P.I. Menezes’ company, CodeDay, has built partnerships with AI Open Source projects for these internships, but which open source projects would be included in this project? The level of tasks that students would contribute to open source was also questioned. The investigators describe these as simple, but the panel was concerned that student work might not be important and complex enough to prove their competency to future employers. More details about the open source tasks or an example of one or two would have strengthened the proposal.

The panel also expressed concern about the difficulty of locating authentic open source projects that students could both learn from and contribute to, suggesting that the issue represents a disconnect between the project’s goals and the way industry works. More details about the tasks or an example of tasks would have strengt disconnect between them.

While the investigators state they have other programs that will help them recruit enough students, they don’t say how they will deal with getting too many. What will their criteria be for choosing among them? If, for instance, they began to look at GPAs, that could represent a barrier to the students they wish to attract.

Another concern: the investigators say they expect 90\% participation from their selected cohort. We wondered whether there was a contract for participation – and why a more ambitious 100\% wasn’t required. Do the investigators feel they are taking a risk? Or is this expectation based on their prior experience with this format?

In terms of the required number of mentors, the statement is made that students will be grouped in twos and threes with their mentor based on mentor availability. We weren’t sure how this would work out in practice, and were concerned about the investigators’ ability to ensure this intended ratio of mentor to students.

\subsection{Broader Impacts}

\subsubsection{Strengths}

The intention is to train 150 students representing diverse backgrounds from community and technical colleges. The investigators intend to expand learning opportunities for their students while enlarging their current curriculum programs. They are already working with many state and regional programs and will use and expand on these as they develop their proposed project. Particularly compelling was the projection that the planned experiential learning might sustain engagement that might lead to further education and credentialing for these students. The ExLENT section is proposed as an enlargement and iteration of what has been achieved on a smaller scale, but the new proposal is specific to AI/ML, and the previous project was not.

\subsubsection{Concerns}

The stipend allotted for students did not take into account the potential need for personal computers, a potential barrier to participation in this program. The panel was concerned that, without addressing this issue, the project might not be able to assure that students would attain the necessary practical training.

\subsection{Solicitation Specific Review Criteria}

\subsubsection{To what extent does the project create on-ramps for diverse individuals into careers in emerging technology fields and to what extent does the project provide participants with a path beyond the ExLENT program?}

Through structured mentoring and microinternships, this program will support and train 150 underrepresented students from community and technical colleges with the goal of launching their careers.

\subsubsection{To what extent does the project reduce barriers so that members of groups historically underrepresented or underserved in STEM can acquire the training and learning needed for careers in emerging technology?}

The project will match students with two independent mentors, with whom they will make monthly structured reflections on the scheduled workshops, trainings, job applications, and other aspects of the program.

\subsubsection{To what extent does the project develop the interests, motivations, skills, knowledge, and/or proficiencies of workers in emerging technology?}

The proposal incorporates coursework, personal mentoring, and best practices concerning job training and engagement with industrial partners.

\subsection{Additional Areas of Evaluation} 

\subsubsection{Data Management Plan}

The panel reviewed the proposal’s data management plan and found it to be adequate, although one panel member wondered whether the open source licenses involved would be permissive enough to be useful to students.

\subsubsection{Postdoc Mentoring Plan}

N/A

\subsubsection{Results From Prior NSF Support}

N/A

\subsection{Summary Statement}

The panel ranked this proposal highly competitive. The panel believed that the issues raised above could be properly addressed. The primary concern to be addressed is the quality and content of the microinternships and how employers will view the relevance to future careers. Such a program, which takes into account the diversity of background, experience, education, and skill levels of community college and technical students, without imposing barriers to participation, has the potential to transform the working lives of those involved, as well as to fill growing needs for those with AI/ML abilities as those fields expand. The funds requested for mentoring through the Mentor in Tech organization is appropriate; the plan is well-detailed, with a strategy for orientation, networking, individual plans, and one-on-one mentoring. Additionally, the program meets the RFP goal of bringing non-higher education organizations to the table. The summary was read by the panel, and the panel members concurred that the summary accurately reflects the panel discussion.

\newpage
\section{Review 1}
\textbf{Rating:} Excellent

\subsection{Intellectual Merit}

The project aims to build a diverse AI/ML workforce by training students from community and technical colleges. The training enables students to become enablers and developers of AI/ML and provides practical skills in application of AI/ML tools for real-world applications in academia and industry. Success of the training is measured through experiential learning in terms of micro-internship, industry mentoring, and diverse career opportunities in both academia and industry. The evaluation plan has necessary elements to be successful and based on the PI team expertise, the project has high likelihood of being successful. 

The budget is about evenly distributed across PI and clerical support and the participant support cost. The students – especially minority, underserved, or underrepresented – may not have access to personal resources to acquire personal computers (or laptops) for training and internship activities or knowhow for such activities. Lack of access to suitable computing will limit their success irrespective of the intellectual merits of the proposed activities. 

\subsection{Broader Impacts}

Up to 150 students will be trained across three cohorts during the project period. The program aligns with various state and regional workforce development arms for broadening participation and impact. Moreover, the program leverages products from recently completed grants (eg. AppConnect NW: DUE#1700629). In summary, the project has high likelihood of success. The recipe is scalable and can help developing globally competitive workforce. For example, the key ingredients currently lacking in any training are convergence thinking, adapting AI/ML tools across applications, and coding experience.

The students may not have access to personal computers or have financial resources to secure them for the training. The stipend (\$25/hr) may motivate them to buy their own device or may become undue burden on them as a part of the program requirement. 


\subsection{Solicitation-Specific Review Criteria}

The project evaluation and mentoring plans are adequate. The training in AI/ML, open-source software use and development, and coding experience in direct context with real-world applications provide students on-ramps for diverse careers in AI/ML tool development and applications and path beyond ExLENT program. Experiential learning via micro-internships provide access to widespread opportunities while reducing the learning and engagement barriers for disadvantaged groups including minorities and underrepresented or underserved communities. The connection between AI/ML tools and their real-world applications established through industry mentoring assists in motivating and skill building opportunities among the trainees, provides new networking opportunities for academic and professional success pathways. 

\subsection{Summary Statement}

The project aims to enable, build, and develop future domestic workforce in AI/ML technologies for industry and academia. The trainees will be drawn from community and technical colleges and represent minority, underserved, or underrepresented communities. They will engage in experiential learning through micro-internships and mentorship with industry. A total of 150 trainees will be prepared during the project. The model leverages results from federal grants, state and regional programs for broadening participation and impact. The project is unique in terms of addressing skills shortage in translating application needs into AI/ML technologies through open-source software development. The project has potential for enabling globally competitive workforce. However, the students engaged through this program are assumed to have access to computers for programming and engagement in the project goals. In reality, the underserved groups may not have resources to maintain personal computers or have understanding of the scope of these capabilities for their job readiness.

\newpage
\section{Review 2}
\textbf{Rating:} Excellent

\subsection{Intellectual Merit}

For this proposal, the primary intellectual merit applies to the beneficiaries of the project, the 150 Community/Technical College students who will be recruited to take part. While they will be brought up to speed in terms of skills and their understanding of Artificial Intelligence/Machine Learning, emphasis will be placed upon experiences that will build their sense of the role of related industries, and, just as vital, of their own potential to contribute to them. Their identities as current and future contributors will be developed in concrete and emotional terms, as they learn not only how their attributes can benefit these fields, but also how to go about building relationships and confidence, allowing them to move forward independently. 

\subsection{Broader Impacts}

The broader impacts benefits of this project apply to the industries that report a struggle to find candidates from underrepresented minorities to hire as they attempt to increase the new perspectives needed, as well as to the CTC students who heretofore have had difficulty finding the pathways into these industries or fields. The scale at which the project is projected suggests its potential to change the impact as well as the image of CTCs by more firmly establishing them as resources to the AI/ML industries. 

\subsection{Summary Statement} 

I find no weaknesses. 

\subsubsection{Strengths}

\begin{itemize}
\item The development of a broader and more impactful talent pipeline based on existing processes. The principal investigators and their collaborator at the CTC level know what they are doing, and support it with their own research and understanding of how to proceed, as well as the bigger picture of other's research into career education and pathways. Thus their project will involve implementing a "fully-fledged" program informed by best practices that have already been proved effective. 

\item Their plan is concise, specific, and sustainable. The proposal conveys a vivid understanding of the barriers faced by the students it addresses, lays out a plan involving precise time demand and compensation, and sites the work in an environment formed by an appropriate number of industrial partners prepared to provide the needed mentorship -- and includes a plan B. The investigators demonstrate that they know how to scale and sustain this project in order to impact thousands of students, well beyond the initial cohorts. 

\item The proposal is specific about the objectives micro internships can foster or achieve, and will be intentional about reaching their goals. 

\item The plan for evaluation is direct, demonstrating not only criteria but how to measure it and at what phase in the project. Their independent evaluator is competent.

\item The PIs are established in this work and well experienced to create what they propose. Each has already guided many students through this kind of program. For example, CodeDay has already built partnerships with AI open source projects for internships. This project will further the current work of both PIs.
\end{itemize}

\newpage
\section{Review 3}
\textbf{Rating:} Excellent

\subsection{Intellectual Merit} 

\subsubsection{Strengths}

The proposal for this program is very well written and thought out with the required details being provided along with the support of the partnerships that are mentioned. The Experiential Learning activities that are mentioned are participation in micro-internships and career mentoring. Through the internship, there is a low level of time commitment and the stipends provided to those involved which helps with some of the barriers that are identified in the community college and technical college student body. Not only does the micro-internship provide these students with a hands-on real-world experience as they will be working on an open-source AI project, but it allows them to interact with an industry mentor during the project. It is mentioned that the industry mentor is not made aware of the project ahead of time so that the students in the micro-internship and the mentors are able to consider each other colleagues versus having a mentor that would be “teaching” the students in the micro-internship. This is highly important because research shows that a successful experiential learning internship is one where the person who provides the experience, in this case, Career mentoring, becomes a co-learner with those who are participating in the experiential learning activity. With the aspect of not making the mentors aware of the projects ahead of time, this allows for them to be co-learners with the students which provides the students the experience of working with someone. 

The proposal also focuses on the training that takes place as an onboarding process for both the students going through the program and those who are mentors in the program. This allows everyone to have the appropriate training to fully understand their expectations and goals of the project and be able to work through the micro-internship. Second, there is a period of reflection for the students that go through the program and the focus of the reflection is discussed based on how students should complete the reflection, blog, or video, and those reflections will be shared with industry partners. 

The evaluations and assessment of the program have key questions being asked that are not only focused on the learning of AI/ML concepts from the students but also in association with their skills and views of the workforce, which is important as there is a career-mentoring piece associated with this project. 

The PI and Co-PI for this project are with organizations that have had successes with other endeavors and they also have the background and experience to lead this project into success. 

\subsubsection{Concerns}

Within the proposal, it is mentioned and a timeline is provided that the program is an 8-month program. The proposal documents what will occur during the first 4 weeks, which is when the students will have completed their first micro-internship. The remaining time is shown on their timeline as the place where students complete their second micro-internship and continue their Career mentoring, but outside of the image there are not any details provided so one is left to assume that the program continues similar to what was set up during the first 2 months.

The PIs also mention that due to other programs that they have had they are certain that they will not have any issues in being able to recruit the 150 students for the program, however, the selection process if more than 150 students apply is not discussed. 

It is not mentioned if the program will have student earn any type of course credit toward their community college or technical college curriculum. It also does not state if there is a certificate or participation or some other way that the participation in the micro-internship will be recognized. 

\subsection{Broader Impacts}

\subsubsection{Strengths}

The proposal highlights the barriers that students in community colleges and technical colleges face. Keeping the time commitment to a minimal requirement and providing a stipend addresses the challenges faced by these students. The proposed structure for their program also seems to be scalable and adaptable. This makes it so that it may be replicated by other organizations/institutions with the appropriate partnerships. 
The PIs have also partnered with significant organizations within Washington state that can assist in expanding the program throughout the state. 

\subsubsection{Concerns}

The only concern with the broader impact is the fact that the proposal mentions that there are over 150 students who are predicted to be interested in the program. There are no details associated with how the selection of students will be made for the program if there are over 50 students interested each year. 


\subsection{Solicitation-Specific Review Criteria}

\subsubsection{To what extent does the project create on-ramps for diverse individuals into careers in emerging technology fields and to what extent does the project provide participants with a path beyond the ExLENT program?}

The program will begin with community college and technical college students. The project can extend beyond the program as students can then go on to four-year degrees. While there is no connection to the 4 year schools, through the project students will meet industry partners through the career mentoring program and their reflections. These opportunities may allow them to make connections that can be valuable once they are ready to enter the workforce. 

\subsubsection{To what extent does the project reduce barriers so that members of groups historically underrepresented or underserved in STEM can acquire the training and learning needed for careers in emerging technology?}

The PIs have taken into consideration the demographic and the barriers faced by community college and technical college students. Through the minimal time requirement and the stipend provided by the project, they are assisting to reduce the barriers for the students who will participate. 

\subsubsection{To what extent does the project develop the interests, motivations, skills, knowledge, and/or proficiencies of workers in emerging technology?}

The project develops interests, motivations, skills, knowledge, and proficiencies throughout. From the beginning training that participants receive, being able to work on an open source project that contributes to the AI project, and then having the mentoring from a career professional and ability to work with them on the given projects. The participants are able to learn while doing, and at the same time learn about the workforce from their mentor. At the same time, the mentor has the opportunity to learn from the students, not just because they are working on the projects together but because the students are essentially having the opportunity to showcase themselves to the career mentors. 

\subsection{Summary Statement} 

I am very much supportive of this proposal. The PIs have taken into consideration all aspects of the items being requested and have considered the barriers for the students that they are attempting to recruit to the program. 

The only parts of concern that I have with this proposal, budget, etc. is that it is not mentioned if this program is essentially an “extra-curricular” opportunity for students if it will count towards their degree somehow, or how the selection will be made if more than 50 students apply for the project per year. 

Otherwise, I am very confident in the proposal and that the PIs will have success with the proposal. They have “key” organizations that are listed as partners and have provided letters of support for the proposal.

\newpage
\section{Review 4}
\textbf{Rating:} Very Good

\subsection{Intellectual Merit}

\subsubsection{Strengths}

\begin{itemize}
\item The team and the partner institutions (CodeDay, MinT, CTC associations, Washington State industry associations etc.) have the background/expertise to carry out the work essential to the project/proposal and the plan seems to have a likelihood of being successful. The team has a background that highlighted the efficacy of micro-internships as an instrument for general computer science students from CTCs and state colleges. It is likely that with suitable modifications for emerging AI technologies, a similar approach (as proposed) might successfully guide CTC students towards careers in emerging technology fields.

\item The project identifies and tackles a critical issue with CTC students getting a technology job post-graduation — internships or lack thereof. CTC students find it challenging to overcome issues related to time commitment required for an internship. The micro-internship format is very promising: with the flexibility of asynchronous feedback / mentoring built into the project plan and a decent wage for the time-commitment, it has the potential to elicit student interest and make student recruitment quite easy.
\end{itemize}

\subsubsection{Concerns}
\begin{itemize}
\item The authors claim that open-source work increases diversity and cite references that contributions to open-source projects can impart the students with a sense of contributing to society. While the latter may indeed be true, open-source projects have a long history of being unwelcoming (even spectacularly successful projects like Linux), and despite the presence of “code of conduct” for many open-source projects, online feedback from community members can often be off-putting to put it mildly. Thus, the choice of open-source projects that the student cohorts will contribute to, as well as the selection of mentors is crucial (and challenging). There is also a chance that the experience of contributing to the open-source project can vary based on the student’s background. 

\item The proposal mentions that the tasks selected for the micro-internship (i.e. open-source AI/ML projects) are expected to be very simple. It might be challenging to present the work completed in the micro-internship to potential employers and claim its significance. Related to the point above, the choice of the open-source project affects this as well. The authors point us towards reference \#20 for open source AI-projects/partnerships that are in place but that reference points to an ITiCSE paper that gives Crates.io as an exemplar open-source project and that does not really work as an AI/ML project.
\end{itemize}

\subsection{Broader Impacts}

\subsubsection{Strengths}

The project focuses on an underrepresented group in emerging technology careers, namely CTC students. Additionally, the substantial number of direct learners (150) enhances the project's impact. The knowledge produced, including curricular and instructional materials, has the potential to extend beyond the project's intended geographic scope in Washington State. The micro-internships can facilitate connections between academia and industry. The project addresses concerns about time commitments and renumeration for CTC students that can potentially be adopted/adapted for CTC students across the country. 

\subsubsection{Concerns}

N/A 

\subsection{Solicitation-Specific Review Criteria}

\subsubsection{1. To what extent does the project create on-ramps for diverse individuals into careers in emerging technology fields and to what extent does the project provide participants with a path beyond the ExLENT program?}

The project creates experiential learning opportunities for at least 150 CTC students from diverse backgrounds and provides them an on-ramp to emerging AI and AI-assisted fields. Components like community building, mentoring and interview preparation for CTC students can extend well beyond the ExLENT program. 

\subsubsection{To what extent does the project reduce barriers so that members of groups historically underrepresented or underserved in STEM can acquire the training and learning needed for careers in emerging technology?}

By alleviating concerns regarding time commitment for CTC students wrt internships, the project reduces significant barriers to learner adoption. The work on open-source AI/ML projects should also help the students build self-sufficiency, and reduce the barrier to finding industry/career opportunities. 

\subsubsection{To what extent does the project develop the interests, motivations, skills, knowledge, and/or proficiencies of workers in emerging technology?}

This proposal aligns well with this particular aspect of the solicitation. The micro-internships involve mentoring sessions, practice interviews etc. that are very pertinent to developing learner interest, motivations and skills with respect to AI and AI-assisted technologies. 

\subsection{Summary Statement} 

Strengths of the proposal include the team's expertise, background, and the potential success of micro-internships as catalysts for CTC students being successful in emergent technology careers. This proposal seems to align itself well with the goals of the ExLENT program with few caveats (e.g. choice of open source projects).