Incident Report: Organizations Team privileges

On April 14, 2025 security@pypi.org was notified of a potential security concernrelating to privileges granted to a PyPI User via Organization Teams membershippersisting after the User was removed from the PyPI Organization the Team belongs to.

We validated the report as a true finding, identified all cases where this scenariohad occurred, notified impacted parties, and released a fix.A full audit determined that all instances were accounted for,with no unauthorized actions taken as a result of the issue.

Timeline of events

• 2025-04-14 16:37 UTC A PyPI User who has been testing out our Organizations features noticed the issue and reported it according to our Security Policy to security@pypi.org.
• 2025-04-14 17:02 UTC PyPI Security acknowledges receipt.
• 2025-04-14 17:22 UTC PyPI Security validates the report as a true finding.
• 2025-04-14 17:58 UTC PyPI Security validating test and hot fix prepared for internal review.
• 2025-04-14 18:30 UTC PyPI Security removes invalid Team Membership and notifies the owners of the only other actively impacted Organization. public PR opened with fix.
• 2025-04-14 18:33 UTC Hot fix is merged.
• 2025-04-14 18:39 UTC Hot fix deployed and live on PyPI.
• 2025-04-14 19:06 UTC Security audit complete, validating that only two instances of this had occurred, with no unauthorized actions taken as a result of the persisted privileges.

Details

PyPI Organizations have been a feature on PyPI since they were first enabledon April 20, 2023.This issue was introduced in the initial development of Organizations features,and was mitigated April 14, 2025.

PyPI Organizations are quickly seeing more use as we (finally) exit our public betaperiod. In the last month we have gone from 70 Community Organization beta testersto 1,935 active Organizations¹, so it is of little surprise that we are surfacing a fewmore issues as a result.

Thanks to PyPI’s strong test coverage identifying and validating the issue was rathertrivial, and getting a fix prepared and out the door was straight forward.

In total, this incident was resolved in 2 hours and 2 minutes from the time of report.

Response

Given that this is an otherwise straightforward bug, I thought I would take a momentto share how the issue was validated as well as how we audited.I’ve replaced the specific organization, team, and user strings below,but otherwise all of this is copied and pasted from the terminal session usedas I worked this report.

I spun up a local development environment ofpypi/warehousefrom the current main branch locally and followed the reporter’s steps to reproduce:

The basic reproduce steps were:

1. Add a user to an organization as a member
2. Add that member to a organization team
3. Remove the member from the organization

Noting that indeed, the User’s team role persisted, and they could continue to actwith those privileges on PyPI.

At that point the reporter and PyPI Administrators team were notified that we had afinding, and that review would be needed shortly to get a fix merged and deployed.

From there, I added afailing testwhich further validated the issue, and got to work creating apatchwhich turned the test green.

Now, with time to wait while a volunteer PyPI Admin returned I focused on assessingif this was actively impacting any other organizations:

warehouse=> select o.name as organization, t.name as team_name, u.username as user, tr. role_name as team_role, ors. role_name as organization rolefrom team_roles tr join teams t on t.id=tr.team_id join organizations o on t.organization_id=o.id join users u on u.id=tr.user_id left outer join organization roles ors on ors.organization_id=t.organization_id and ors.user_id=tr.user_idwhere ors. role_name is null; organization | team_name | user | team_role | organization_role--------------+-------------+-----------+-----------+------------------- spam | Spam-owners | spamlover | Member | (1 row)

This query showed me that one instance of a User having an Organization Team Rolewithout being a Member of that Organization still existed on PyPI².The reporter made clear that they had already resolved the instance from their testing.

I drafted a notice to the five users with role Owner on the impacted Organization, and took a moment to realize that this was our first time emailing OrganizationOwners as a group, and that we needed to account for the fact that Users on PyPIdo not necessarily already know one-another’s email addresses, as it is not requiredto invite them to a Project or Organization. A quick gut-check in the PyPI Moderatorschannel validated my plan to Bcc: all the Owners rather than To: them as agroup.³

By that point, the volunteer PyPI Administrator was available to review the PR anddrafted e-mail. We notified the impacted Organization, and then coordinated toopen the PR publicly and approve/merge it hastily before completing a more in-depthaudit.

Luckily this audit was straightforward using our internal security recordscombined with the fact that there has been minimal churn in the Organization membershipin the short time that Organizations has been in broader use.

warehouse=> select o.name, time, tag, u.usernamefrom organization_events oe join users u on (additional->>'target_user_id')::uuid=u.id join organizations o on oe.source_id=o.idwhere tag in ('organization:team_role:remove', 'organization:organization_role:remove')order by time; name | time | tag | username ------------+----------------------------+---------------------------------------+------------------- lumberjack | 2023-05-02 03:01:18.935901 | organization:organization_role:remove | sirrobin holygrail | 2023-07-06 12:55:43.261593 | organization:organization_role:remove | blackknight ni | 2023-09-18 12:07:17.389244 | organization:organization_role:remove | shrubbery parrot | 2024-02-04 19:23:25.354344 | organization:organization_role:remove | exparrot spam | 2024-08-24 01:40:22.405746 | organization:organization_role:remove | spamlover spam | 2025-02-09 18:14:13.891224 | organization:team_role:remove | eggandspam albatross | 2025-03-07 06:55:29.446617 | organization:organization_role:remove | nudge albatross | 2025-03-07 06:55:37.271176 | organization:organization_role:remove | wink cheese | 2025-03-13 18:25:54.650905 | organization:team_role:remove | gorgonzola cheese | 2025-03-13 18:26:02.525162 | organization:team_role:remove | camembert ministry | 2025-03-20 07:53:45.616404 | organization:organization_role:remove | sillywalks argument | 2025-03-31 15:52:18.186223 | organization:organization_role:remove | contradiction fishslap | 2025-04-14 15:12:14.023183 | organization:organization_role:remove | danceking fishslap | 2025-04-14 15:24:54.208641 | organization:organization_role:remove | danceking fishslap | 2025-04-14 15:27:22.954624 | organization:team_role:remove | danceking

Here, we see the spamlover user being removed from the spam Organizationon 2024-08-24, without being removed from the team, confirming our finding from theearlier query.

We also see the User danceking from the fishslap Organization being removed fromthe Organization multiple times, before the reporter removed them from their assignedTeam.

This allowed us to confirm that beyond the already identified incidents,no other Organizations had found this problem before without letting us know.

Thanks

First and foremost, thanks to our reporters, Matthew Treinish and Jake Lishmanof IBM Quantum for finding and reporting this issue.

We are grateful for the entire community of security researchers and users whofind and report security issues to PyPI in accordance with ourSecurity Policy.PyPI relies on the efforts of our community to help us find and resolve issues likethese before they become critical issues.Cooperation between all parties helps to improve the security of open source,and none of us could do it alone.

The tools and capabilities we’ve evolved in PyPI over the past six years have reallycome to be an asset in situations like these. I’m grateful to all the contributorsand admins who have helped us to build them 💜.

Ee Durbin is the Director of Infrastructure atthe Python Software Foundation.They have been contributing to keeping PyPI online, available, andsecure since 2013.