The primary spam problem isn't that a single account opens many pull requests on a single repo, but that spammer accounts open many pull requests spread across many repositories. So limiting accounts to a couple of open PRs on my repository won't help much.
I'd rather enforce a limit based on the number of PRs that account opened across all public repositories it doesn't have write access to within the last week. And PRs that were closed without getting merged should be held against the account somehow (perhaps via a "close as unwelcome" option for the maintainer).
> And PRs that were closed without getting merged should be held against the account somehow
That strikes me as a bad solution. I've sent plenty of PRs over the last two decades that were things I wasn't sure if upstream wanted or not, but I did the work and wanted to offer it to them. If you get penalized for not having a PR merged, it's going to incentivize selfishness
Hence the cooldown period? I think the mechanism proposed here should be perfectly fine for targeted PRs, while mitigating those that sit above baseline.
That's why I suggested an explicit "close as unwelcome" option (label to be bikeshed). And the impact of the rejection should decay over time.
In any case, my proposal is a rough sketch of how I'd approach the problem, not a production ready algorithm. But I'd expect even that basic approach to work a lot better than github's approach.
Ultimately what kills any effort to curb this behaviour is the fact that the perpetrator can always open another account.
If I was a maintainer of an open-source project, I would have a two-tier system:
-PRs from previous contributors.
-All others, sorted by lines of code, ascending.
Reasoning:
-Large PRs from someone without a track record are rare.
-It's not a huge ask to have people first solve a smaller problem.
-Small PRs are easy to verify - it's especially easy to tell if a given one-liner is impactful or just spam. Should also be easier to summarise it in the title.
-Don't quote me on that but I think LLMs are still bad at clear, concise, meaningful changes.
I assume the idea is that you probably weren't doing that 20+ times a day. For me, I was raising at most one or two open source PRs per week at most, and I only had time to focus on one or two repos during that time. I think thats a good baseline, with a big overhead for exceptional circumstance, but I don't see a world where someone should be able to make 10 PRs a day, every day, for weeks
Good point. I've (even with agents) never made more than like 5 PRs in one day internal to a company and if I have they typically included accompanying proto or submodule changes. Heck give a factor of safety of 2x and cap at 10 daily PRs per account for repos that youre "untrusted"
Your proposal wouldn't help me at all. I wouldn't say that the problem I'm having is even "spam" per se. (For context I receive hundreds of PRs each week across my OSS projects like mise)
In my case I sometimes get a flurry of PRs from over-exuberant contributors, not necessarily low quality even! Using this I can at least put some back-pressure on that and help keep things more fair across my contributors.
Github needs a "Human Trust" score where "AI Slop" label applied to a PR reduces the score and denies all PR's for the next X hours, delay increasing exponentially for every PR labelled with "AI Slop". Repeated "AI Slop" labels given by a maintainer reduces "Human Trust" Score like dropping off a cliff.
Every successful merge for a PR spread across N days slowly increases "Human Trust" score. (So a slow of fake merged PR's cannot fake increase Human Trust score). Just like the real world, Human Trust should be hard to gain and easy to lose.
If your Human Trust Score becomes negative due to too much "AI Slop", then you are banned from all PR submissions for a quarter. Your profile picture is also replaced with the Robot Identicon to indicate to the world that your human brain has been replaced by AI and urgent health-check is needed.
That could be abused the other way against well-intentioned humans. Simply let someone accumulate PRs at a not-abusive rate, and then once enough exist falsely report them all in a day.
> banned from all PR submissions
So then the person can't even make a PR against their own repository? Or when we're there a maintainer, known contributor, or a member of an organization that might be their workplace?
If you didn't take the time to write it, why should I take the time to read it?
This is a band-aid. Maybe even a good band-aid, because it'll keep individual contributors from flooring the zone. But the core problem is Github's model that assumes code is worth reading.
I'm much rather see the agent logs stapled to PRs. Make it easy to understand if there's a brain behind the suggested changes before engaging.
> If you didn't take the time to write it, why should I take the time to read it?
This is the fundamental problem. You have to look at the equilibrium. When you submit a PR, you're asking for some of my time. I have to figure out if it's likely to be worth it for me. If you have a track record of producing useful software that I have merged before, you're putting your reputation at risk when you submit a new PR, so it's probably good. If you start sending AI slop, I'm going to downgrade your reputation.
If you have no track record though, I'll probably at least take a glance since even if I'm not sure, at least you had to spend some time to write the code and put together the PR. Now that's not true.
My guess is we're going to have to create some new systems for reputation, maybe bond posting, maybe "sponsored" PRs, where someone trusted vouches for it, etc.
Incidentally, this doesn't just apply to PRs. It's emails, all kinds of other messages, reports, etc.
I think this is a really solid move. This gives OSS contributors a lot of flexibility. You could set the limit to 0, and manually add contributors. You could set it to 1-3 to allow people to get their foot in the door. But the de facto limit today is infinite, which is spammed. Imagine if GMail did this! If I don't whitelist or reply within `n` emails, youre done. I would KILL for that.
I don't often give GitHub credit, because I work with it every day and I encounter something frustrating or broken nearly every day ending in "day", but kudos to them for working on addressing the some of the big problems.
I also like the other features mentioned in the blog post. It won't make a difference to me and my daily work, but I'm glad that they are taking the criticisms seriously.
Though I have to admit that I'm a bit conflicted about this. Part of me also wants more people to move off of GitHub to help break their monopoly on code on the web, but I also don't want the people making and maintaining open source to give up their projects due to burnout and slop spam.
Somebodies are already making “loops” <facepalm> that will add the noise back. If PR isn’t merged in <time> close and open a new one, either the same one or a new one of higher priority.
If <time> is set low enough, the noise still exists
We should have agents to triage PRs. Their "smarter bypass signals" is already implemented by Mitchell Hashimoto's Vouch system: https://github.com/mitchellh/vouch
The primary spam problem isn't that a single account opens many pull requests on a single repo, but that spammer accounts open many pull requests spread across many repositories. So limiting accounts to a couple of open PRs on my repository won't help much.
I'd rather enforce a limit based on the number of PRs that account opened across all public repositories it doesn't have write access to within the last week. And PRs that were closed without getting merged should be held against the account somehow (perhaps via a "close as unwelcome" option for the maintainer).
> And PRs that were closed without getting merged should be held against the account somehow
That strikes me as a bad solution. I've sent plenty of PRs over the last two decades that were things I wasn't sure if upstream wanted or not, but I did the work and wanted to offer it to them. If you get penalized for not having a PR merged, it's going to incentivize selfishness
Hence the cooldown period? I think the mechanism proposed here should be perfectly fine for targeted PRs, while mitigating those that sit above baseline.
That's why I suggested an explicit "close as unwelcome" option (label to be bikeshed). And the impact of the rejection should decay over time.
In any case, my proposal is a rough sketch of how I'd approach the problem, not a production ready algorithm. But I'd expect even that basic approach to work a lot better than github's approach.
Really it should be a "report as spam" option
Ultimately what kills any effort to curb this behaviour is the fact that the perpetrator can always open another account.
If I was a maintainer of an open-source project, I would have a two-tier system:
-PRs from previous contributors.
-All others, sorted by lines of code, ascending.
Reasoning:
-Large PRs from someone without a track record are rare.
-It's not a huge ask to have people first solve a smaller problem.
-Small PRs are easy to verify - it's especially easy to tell if a given one-liner is impactful or just spam. Should also be easier to summarise it in the title.
-Don't quote me on that but I think LLMs are still bad at clear, concise, meaningful changes.
I assume the idea is that you probably weren't doing that 20+ times a day. For me, I was raising at most one or two open source PRs per week at most, and I only had time to focus on one or two repos during that time. I think thats a good baseline, with a big overhead for exceptional circumstance, but I don't see a world where someone should be able to make 10 PRs a day, every day, for weeks
Good point. I've (even with agents) never made more than like 5 PRs in one day internal to a company and if I have they typically included accompanying proto or submodule changes. Heck give a factor of safety of 2x and cap at 10 daily PRs per account for repos that youre "untrusted"
Your proposal wouldn't help me at all. I wouldn't say that the problem I'm having is even "spam" per se. (For context I receive hundreds of PRs each week across my OSS projects like mise)
In my case I sometimes get a flurry of PRs from over-exuberant contributors, not necessarily low quality even! Using this I can at least put some back-pressure on that and help keep things more fair across my contributors.
why not both?
Github needs a "Human Trust" score where "AI Slop" label applied to a PR reduces the score and denies all PR's for the next X hours, delay increasing exponentially for every PR labelled with "AI Slop". Repeated "AI Slop" labels given by a maintainer reduces "Human Trust" Score like dropping off a cliff.
Every successful merge for a PR spread across N days slowly increases "Human Trust" score. (So a slow of fake merged PR's cannot fake increase Human Trust score). Just like the real world, Human Trust should be hard to gain and easy to lose.
If your Human Trust Score becomes negative due to too much "AI Slop", then you are banned from all PR submissions for a quarter. Your profile picture is also replaced with the Robot Identicon to indicate to the world that your human brain has been replaced by AI and urgent health-check is needed.
That could be abused the other way against well-intentioned humans. Simply let someone accumulate PRs at a not-abusive rate, and then once enough exist falsely report them all in a day.
> banned from all PR submissions
So then the person can't even make a PR against their own repository? Or when we're there a maintainer, known contributor, or a member of an organization that might be their workplace?
"AI slop" reporting also comes with a budget. You can only apply N "AI slop" labels to a user per week.
Obviously ban is only for PRs against other repositories belonging to different users/organization. Not PRs towards same user/org.
At the end they mentioned they're exploring global trust signals
If you didn't take the time to write it, why should I take the time to read it?
This is a band-aid. Maybe even a good band-aid, because it'll keep individual contributors from flooring the zone. But the core problem is Github's model that assumes code is worth reading.
I'm much rather see the agent logs stapled to PRs. Make it easy to understand if there's a brain behind the suggested changes before engaging.
> If you didn't take the time to write it, why should I take the time to read it?
This is the fundamental problem. You have to look at the equilibrium. When you submit a PR, you're asking for some of my time. I have to figure out if it's likely to be worth it for me. If you have a track record of producing useful software that I have merged before, you're putting your reputation at risk when you submit a new PR, so it's probably good. If you start sending AI slop, I'm going to downgrade your reputation.
If you have no track record though, I'll probably at least take a glance since even if I'm not sure, at least you had to spend some time to write the code and put together the PR. Now that's not true.
My guess is we're going to have to create some new systems for reputation, maybe bond posting, maybe "sponsored" PRs, where someone trusted vouches for it, etc.
Incidentally, this doesn't just apply to PRs. It's emails, all kinds of other messages, reports, etc.
I think this is a really solid move. This gives OSS contributors a lot of flexibility. You could set the limit to 0, and manually add contributors. You could set it to 1-3 to allow people to get their foot in the door. But the de facto limit today is infinite, which is spammed. Imagine if GMail did this! If I don't whitelist or reply within `n` emails, youre done. I would KILL for that.
I get a lot of emails that I want to get but never reply to. I would not want to have to remember to whitelist all of those.
Yeah fair, then you could set it higher, even 100. Or default it off.
HEY email service does something like that:
https://www.hey.com/features/the-screener/
> Imagine if GMail did this!
It would be very annoying? Spammers can still spam one message, but now your friends can't email you twice. Awesome.
This is a barely-better-than-nothing blunt tool.
I don't often give GitHub credit, because I work with it every day and I encounter something frustrating or broken nearly every day ending in "day", but kudos to them for working on addressing the some of the big problems.
I also like the other features mentioned in the blog post. It won't make a difference to me and my daily work, but I'm glad that they are taking the criticisms seriously.
Though I have to admit that I'm a bit conflicted about this. Part of me also wants more people to move off of GitHub to help break their monopoly on code on the web, but I also don't want the people making and maintaining open source to give up their projects due to burnout and slop spam.
Somebodies are already making “loops” <facepalm> that will add the noise back. If PR isn’t merged in <time> close and open a new one, either the same one or a new one of higher priority.
If <time> is set low enough, the noise still exists
also, close all issues and open them as you plan to work on them.
We should have agents to triage PRs. Their "smarter bypass signals" is already implemented by Mitchell Hashimoto's Vouch system: https://github.com/mitchellh/vouch