Repository navigation
|
TLDR: Could you please consider reducing the amount of github hosted runner time this project is consuming? 🙏 Hi, I am a member of the Apache DataFusion and Arrow PMCs. Our CI jobs that run on github hosted runners are taking hours to start due to other jobs running in the (shared) apache organization which is severely impacting our project's progress. While investigating what our jobs were waiting for, I was continually pointed at this repository. 117 Runner-Days of time in the last 24 hours, 20x more than DataFusion (updated)I have a script that computes the total amount of github hosted runner time in hours spend for each project (attached below) The airflow project is currently using almost 20x more than the next highest project I can find, to the tune of almost 120 days of runner time in each 24 hour window. On average, at any time, Apache Airflow has 128 jobs running on the shared runners andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 24h apache/airflow
1535 runs of apache/airflow created since 2026-10-21T09:36:30Z (24.0 h window)
jobs loaded: 1469 runs from cache, 66 from the API, 0 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
success 19020 58.7% 169845 117.9
skipped 10557 32.6% 0 0.0
cancelled 2417 7.5% 12016 8.3
failure 256 0.8% 2941 2.0
queued 70 0.2% 0 0.0
in_progress 69 0.2% 0 0.0
total 32389 184802 128.3
average concurrent jobs over the window: 128Here is the script (and the data to save you some GH API credits): usage-airflow.zip You can see more analysis / discussion on these issues
I looked at this repo briefly to see if there was something obvious that was consuming all the time that maybe could reduce the time spent. I think it may just be that each commit runs significant amount of work on CI. I noticed that each commit in stacked PRs kicks off the entire CI suite, for example that I discussed with @ephraimbuddy in this PR) but I don't think this explains the entire usage cc @kaxil and @potiuk as I see your names on many approvals / PRs. Thank you for your consideration |
Replies: 11 comments 14 replies
|
+1 from apache/iggy PMC. it would be great if you would consider to tone down the usage, due to recent long queues i created issue to infra. |
I'm going to check this but I don't think it's each commit triggering every other PRs CI in the stack. I think when the PRs are restacked is when all their respective CIs kicks off. |
|
Thanks for this issue - it is absolutely what we are doing now. In case you have not noticed we are well in the process of doing just that (and already implemented some of earlier - easier optimisations). I thin everyone (including @hubcio and @alamb in your projects) - you should have similar approach we do - where we, proactively - before issues like that - plan and relentlessly execute optimizations efforts. I would really love if all the projects do that - including those projects which experienced the queuing time now. Everyone's small optimisation count, and I hope Airflow blueprinit on how to do that might be followed. All of the optimizations we tried and plan are also described in this PR to the infra website - apache/infrastructure-website#320 where we share and ask others to try similar things.
My estimate is that it will reduce Airflow usage by about 80% overall. Let me close the issue - because it's been outdated before it's been opened. |
|
Thank you -- it is great you are already working on it. I agree that it will be helpful for all projects to reduce the runner usage We are madly trying to do the same thing in our projects. For example, we recently switch DataFusion so CI mostly runs in contributor forksL |
|
And to be clear DataFusion has used lots of runner time in the past as well, though mostly of our heavy jobs are now on Hosted RunsOn runners |
|
And yes - we usually have 10x more traffic / PRs than the next biggest project - but last week is an outlier as we prepare for 3.4.0 release and there is a lot of cherry-picking involved. Usually - and i keep on monitoring it at https://infra-reports.apache.org/#ghactions - we hover around 6 FT runners, but this last 7 days we have 17 - that is indeed "a lot". If you look at last 30 days average is 10 FT. I hope to reduce it to ~ 5 at peak. Most of the "Release preparation" jobs will be anyhow taken over by our self-hosted runners when we enable it - I hope this weekend. |
|
cc: @shahar1 -> let's focus on switching to the self-hosted over the weekend. Ping me on slack any time and we will complete it faster - this will defintely help @alamb and @hubcio and others - and possibly we can share our "blueprint" - step-by-step how to do the RunsOn enablement when you have AWS credits. This might be super helpful for infra and other ASF teams to follow it. |
|
BTW my post above mis stated -- this repo uses 20x more runner time than Datafusion, not the next highest repo The next highest repo I found was apache/beam actually which uses like 1/2 of this runner time |
|
As of this morning, airflow has still used 106 days of runner time in the last 24 hours 😞 |
|
Of course - we have a lot of PRs |
|
And really - there is no point in posting it over and over - we are working on making things better for weeks - and we are close to conclusion. And the best way - rather than annoying us with posting obvious things is to be patient @alamb. Our improvements - which BTW we spent a lot of volunteer hours on - will come when they are ready - and not because there are frequent nudges. |
|
BTW. I also posted improvement proposals for Arrow @alamb - > are you working on them ? Shall I open an issue for that? |
|
I think running PR's CI on the fork owner's runners should be very helpful. I want to file similar PRs for other arrow repos (CPP and go) |
|
My plan is actually to create shared actions when we get it rolled out to airflow. But @alamb - since you've already done it for Arrow it seems - maybe you can generalise it and contribute to https://github.com/apache/infrastructure-actions ? That would help people like @sdf-jkl ? Airflow design docs about it and AIP https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451974711/AIP-120+Run+CI+for+external+contributions+in+contributor+forks - bear a lot of complexity as Airflow's CI was already super complex before, but maybe you can extract it easier from Arrow? WDYT @alamb ? |
|
A shared action would be awesome! Would make it way easier for everyone to make the switch |
|
I wasn't involved in the initial Arrow CI setup -- it uses external runners due to the fact there used to be a company (ursa labs) that sponsored running a bunch of infrastrcture pre-github runners (or before the runners got good) |
|
@alamb - you can always nudge the right people. Hopefully they won't be annoyed. |
Thanks for this issue - it is absolutely what we are doing now. In case you have not noticed we are well in the process of doing just that (and already implemented some of earlier - easier optimisations). I thin everyone (including @hubcio and @alamb in your projects) - you should have similar approach we do - where we, proactively - before issues like that - plan and relentlessly execute optimizations efforts.
I would really love if all the projects do that - including those projects which experienced the queuing time now. Everyone's small optimisation count, and I hope Airflow blueprinit on how to do that might be followed.
All of the optimizations we tried and plan are also described i…