feat: follow the organizations users follow on GitHub, now or once they show up

KNOWLEDGE_BASE.md

@@ -94,7 +94,7 @@ Livestream history: the team agreed on GitHub-only sign-in and importing profile
 
 ### First-time setup
 
-The first GitHub sign-in opens a short onboarding flow: review your mirrored profile (after editing it on GitHub, **Sync from GitHub** pulls the changes in without leaving setup), choose a color, see who came over from your GitHub network and follow suggested developers, and learn about the composer, keyboard shortcuts, and terminal commits. Skip setup at any step or finish it to continue to the app. The follow step shows what the one-time GitHub import brought over: the people you follow on GitHub who are already on /dev/null, whom you now follow, and your GitHub followers who are here, who now follow you. Completed or skipped setup does not appear again. New users also start with Tom as a mutual follow; Tom is the platform's fictional welcome bot. Tag @tom in a public commit to ask about the platform or find posts on a topic.
+The first GitHub sign-in opens a short onboarding flow: review your mirrored profile (after editing it on GitHub, **Sync from GitHub** pulls the changes in without leaving setup), choose a color, see who came over from your GitHub network and follow suggested developers, and learn about the composer, keyboard shortcuts, and terminal commits. Skip setup at any step or finish it to continue to the app. The follow step shows what the one-time GitHub import brought over: the people and organizations you follow on GitHub that are already on /dev/null, which you now follow, and your GitHub followers who are here, who now follow you. Completed or skipped setup does not appear again. New users also start with Tom as a mutual follow; Tom is the platform's fictional welcome bot. Tag @tom in a public commit to ask about the platform or find posts on a topic.
 
 Sources: [onboarding page](resources/js/pages/onboarding/show.tsx), [sign-in tests](tests/Feature/Controllers/SessionControllerTest.php).
 
@@ -166,7 +166,7 @@ Sources: [badge definitions](app/Enums/Achievement.php), [unlock conditions and
 
 ### Following developers and suggestions
 
-Follow or unfollow from a developer's profile or a follow control elsewhere in the app. If someone already follows you and you do not follow them yet, their profile, hover card, and connections lists offer **Follow back**. Following puts their commits in your Following log. Following or unfollowing from a log updates the controls, counts, and live-following sidebar without navigating away: Latest and Trending keep their loaded commits and ranking, while Following updates its commits and empty-state suggestions. You cannot follow yourself or follow across a block. GitHub follows are imported once: connections to developers already on /dev/null are added in both directions, with up to 1,000 GitHub connections checked each way. This is not continuous synchronization, so unfollowing someone on /dev/null sticks.
+Follow or unfollow from a developer's profile or a follow control elsewhere in the app. If someone already follows you and you do not follow them yet, their profile, hover card, and connections lists offer **Follow back**. Following puts their commits in your Following log. Following or unfollowing from a log updates the controls, counts, and live-following sidebar without navigating away: Latest and Trending keep their loaded commits and ranking, while Following updates its commits and empty-state suggestions. You cannot follow yourself or follow across a block. GitHub follows are imported once: connections to developers already on /dev/null are added in both directions, with up to 1,000 GitHub connections checked each way. The organizations you follow on GitHub come along too: you follow the ones already on /dev/null, and each of the others as soon as it shows up here. This is not continuous synchronization, so unfollowing someone on /dev/null sticks, and following someone on GitHub later does not carry over.
 
 **Who to follow** suggests developers in the side column or within the log on smaller screens, each with the first line of their bio and why they are suggested (how many people you follow follow them, or their follower count). Follow from the icon next to each suggestion; for someone who already follows you, it follows back. The X icon marks a suggestion **Not interested**: that developer is never suggested to you again, without muting them, and the next suggestion takes their place. Refresh for different suggestions, or use **… → Don't show again** to hide the block, with an immediate **Undo** option. Show or hide these suggestions in Settings → Appearance. Suggested developers must have a visible top-level commit. On profiles, suggestions prioritize the people that developer follows; on commit pages, they prioritize visible thread participants. Regular suggestions fill any remaining places. You can also follow back from the notifications list.
 

app/Actions/CreateExport.php

@@ -49,6 +49,7 @@
  *     dismissed_suggestions: Generator<int, ExportUser>,
  *     organizations: array<int, array{handle: string, role: string, joined_at: string}>,
  *     followed_organizations: array<int, array{handle: string, followed_at: string}>,
+ *     pending_organization_follows: list<string>,
  *     messages: Generator<int, ExportMessage>,
  *     reports: Traversable<int, array<string, mixed>>,
  *     remote_pins: array<int, array{to: string|null, pinned_at: string|null}>,
@@ -119,6 +120,7 @@ public function handle(User $user): array
             'dismissed_suggestions' => $this->handles($user->dismissedSuggestions()),
             'organizations' => $this->organizations($user),
             'followed_organizations' => $this->followedOrganizations($user),
+            'pending_organization_follows' => $this->pendingOrganizationFollows($user),
             'messages' => $this->messages($user),
             'reports' => CommitReport::query()->where('reporter_id', $user->id)->oldest('id')->cursor()->map(fn (CommitReport $report): array => [
                 'commit_id' => $report->commit_id,
@@ -359,6 +361,17 @@ private function followedOrganizations(User $user): array
         ])->all();
     }
 
+    /**
+     * The GitHub IDs of the organizations the user follows on GitHub, to follow here once they are on /dev/null.
+     *
+     * @return list<string>
+     */
+    private function pendingOrganizationFollows(User $user): array
+    {
+        /** @var list<string> */
+        return DB::table('pending_organization_follows')->where('user_id', $user->id)->orderBy('github_id')->pluck('github_id')->all();
+    }
+
     /**
      * Moderators stay anonymous, as they are on the notifications page, and level-ups are the user's own doing.
      *

app/Actions/CreateUserGitHubFollows.php

@@ -5,20 +5,24 @@
 namespace App\Actions;
 
 use App\Contracts\Services\GitHubProvider;
+use App\Enums\ProfileType;
+use App\Models\Profile;
 use App\Models\User;
 use Illuminate\Database\Eloquent\Collection;
 use Illuminate\Support\Facades\DB;
 
 /**
- * Brings a user's GitHub network over, once: they follow the people they follow on GitHub, and the people
- * following them on GitHub follow them here. It only ever adds follows, and never runs twice, so an unfollow sticks.
+ * Brings a user's GitHub network over, once: they follow the people and the organizations they follow on GitHub, and
+ * the people following them on GitHub follow them here. The organizations not on /dev/null yet are kept, so the user
+ * follows them once they are. It only ever adds follows, and never runs twice, so an unfollow sticks. Organizations are
+ * marked as imported apart, so the users imported before organizations could be followed get them on another run.
  */
 final readonly class CreateUserGitHubFollows
 {
     /**
      * Up to 1,000 accounts each way, so a popular account can't use up the rate limit on its own.
      */
-    private const int MaxPages = 10;
+    public const int MaxPages = 10;
 
     public function __construct(
         private GitHubProvider $github,
@@ -29,22 +33,20 @@ public function __construct(
 
     public function handle(User $user, bool $notify = true): void
     {
-        if ($user->github_follows_imported_at !== null) {
-            return;
-        }
-
-        $login = $this->github->login($user->github_id);
-
-        if ($login === null) {
-            $user->update(['github_follows_imported_at' => now()]);
+        $importsPeople = $user->github_follows_imported_at === null;
+        $importsOrganizations = $user->github_organization_follows_imported_at === null;
 
+        if (! $importsPeople && ! $importsOrganizations) {
             return;
         }
 
-        $following = $this->users($user, $this->github->followingIds($login, self::MaxPages));
-        $followers = $this->users($user, $this->github->followerIds($login, self::MaxPages));
+        $login = $this->github->login($user->github_id);
+        $followingIds = $login === null ? ['users' => [], 'organizations' => []] : $this->github->followingIds($login, self::MaxPages);
+        $following = $importsPeople ? $this->users($user, $followingIds['users']) : new Collection();
+        $followers = $importsPeople && $login !== null ? $this->users($user, $this->github->followerIds($login, self::MaxPages)) : new Collection();
+        $organizationIds = collect($followingIds['organizations'])->unique()->values();
 
-        DB::transaction(function () use ($user, $following, $followers, $notify): void {
+        DB::transaction(function () use ($user, $notify, $following, $followers, $organizationIds, $importsPeople, $importsOrganizations): void {
             $followedIds = $user->following()->syncWithoutDetaching($following->modelKeys())['attached'];
             $followerIds = $user->followers()->syncWithoutDetaching($followers->modelKeys())['attached'];
 
@@ -58,7 +60,25 @@ public function handle(User $user, bool $notify = true): void
                 }
             }
 
-            $user->update(['github_follows_imported_at' => now()]);
+            if ($importsOrganizations) {
+                // Waits for an organization being created from GitHub right now, which follows the pending rows it finds
+                // (UpdateOrCreateOrganization), so the follow is made on one side or the other.
+                DB::select('select pg_advisory_xact_lock(hashtextextended(?, 0))', [UpdateOrCreateOrganization::FollowsLock]);
+
+                $organizations = Profile::query()->where('type', ProfileType::Organization)->whereIn('github_id', $organizationIds)->get();
+
+                $user->followedOrganizations()->syncWithoutDetaching($organizations->modelKeys());
+
+                DB::table('pending_organization_follows')->insertOrIgnore($organizationIds
+                    ->reject(fn (string $id): bool => $organizations->contains('github_id', $id))
+                    ->map(fn (string $id): array => ['user_id' => $user->id, 'github_id' => $id, 'created_at' => now()])
+                    ->all());
+            }
+
+            $user->update(array_filter([
+                'github_follows_imported_at' => $importsPeople ? now() : null,
+                'github_organization_follows_imported_at' => $importsOrganizations ? now() : null,
+            ]));
         });
     }
 

app/Actions/UpdateOrCreateOrganization.php

@@ -13,14 +13,21 @@
 
 final readonly class UpdateOrCreateOrganization
 {
+    /**
+     * Held while an organization is created and while a GitHub import keeps the organizations it doesn't find pending,
+     * so neither misses the other's rows. Both are short and rare, so one lock for every organization is enough.
+     */
+    public const string FollowsLock = 'github-organization-follows';
+
     public function __construct(private UpdateStaleHandles $updateStaleHandles)
     {
         //
     }
 
     /**
      * Mirrors a GitHub organization on its profile, from an account of the GitHub REST API, such as
-     * `GET /users/{username}`. Another profile holding its handle renamed away on GitHub.
+     * `GET /users/{username}`. Another profile holding its handle renamed away on GitHub. A new organization is
+     * followed by the users following it on GitHub whose follows were imported before it showed up.
      *
      * @param  array<array-key, mixed>  $account
      */
@@ -31,16 +38,30 @@ public function handle(array $account): Organization
         assert(is_int($githubId));
 
         return DB::transaction(function () use ($profile, $githubId): Organization {
+            DB::select('select pg_advisory_xact_lock(hashtextextended(?, 0))', [self::FollowsLock]);
+
             $this->updateStaleHandles->handle($profile->handle, (string) $githubId);
 
             // A profile never changes type: a user's account GitHub has since turned into an organization fails on its
             // GitHub ID, until that case is handled...
-            $organization = Profile::query()->updateOrCreate(['github_id' => (string) $githubId, 'type' => ProfileType::Organization], [
+            $organizationProfile = Profile::query()->updateOrCreate(['github_id' => (string) $githubId, 'type' => ProfileType::Organization], [
                 ...$profile->attributes(),
                 'github_synced_at' => now(),
             ]);
 
-            return Organization::query()->firstOrCreate(['id' => $organization->id]);
+            $organization = Organization::query()->firstOrCreate(['id' => $organizationProfile->id]);
+
+            if ($organization->wasRecentlyCreated) {
+                $pending = DB::table('pending_organization_follows')->where('github_id', (string) $githubId);
+
+                DB::table('organization_follows')->insertUsing(
+                    ['user_id', 'organization_id', 'created_at'],
+                    $pending->clone()->select('user_id')->selectRaw('?, ?', [$organization->id, now()]),
+                );
+                $pending->delete();
+            }
+
+            return $organization;
         });
     }
 }

app/Console/Commands/ImportGitHubFollowsCommand.php

@@ -5,23 +5,41 @@
 namespace App\Console\Commands;
 
 use App\Actions\CreateUserGitHubFollows;
+use App\Contracts\Services\GitHubProvider;
 use App\Models\User;
 use Illuminate\Console\Attributes\Description;
 use Illuminate\Console\Attributes\Signature;
 use Illuminate\Console\Command;
+use Illuminate\Database\Eloquent\Builder;
 use Illuminate\Http\Client\ConnectionException;
 use Illuminate\Http\Client\RequestException;
 
 #[Signature('follows:import')]
-#[Description('Follow, for every user not imported yet, who they follow and who follows them on GitHub, without notifying anyone')]
+#[Description('Follow, for every user not imported yet, who and which organizations they follow and who follows them on GitHub, without notifying anyone')]
 final class ImportGitHubFollowsCommand extends Command
 {
-    public function handle(CreateUserGitHubFollows $createUserGitHubFollows): int
+    /**
+     * Below this many requests left, the command stops, leaving the rest to the app until the rate limit resets: above
+     * the floor where looking up mentioned organizations pauses, so they carry on meanwhile.
+     */
+    private const int RateLimitFloor = 1500;
+
+    public function handle(CreateUserGitHubFollows $createUserGitHubFollows, GitHubProvider $github): int
     {
         $imported = 0;
         $failed = 0;
 
-        foreach (User::query()->whereNull('github_follows_imported_at')->lazyById() as $user) {
+        foreach ($this->users()->lazyById() as $user) {
+            $rateLimit = $github->rateLimit();
+
+            // A user takes one request for their login, and up to the maximum of pages each way.
+            if ($rateLimit !== null && $rateLimit['remaining'] < self::RateLimitFloor + 1 + 2 * CreateUserGitHubFollows::MaxPages) {
+                $this->components->warn("Stopped after {$imported} users, to keep ".self::RateLimitFloor." GitHub requests for the rest of the app: {$rateLimit['remaining']} left.");
+                $this->components->info("{$this->users()->count()} users left. Run the command again after {$rateLimit['resets_at']->utc()->toDateTimeString()} UTC ({$rateLimit['resets_at']->diffForHumans()}), when the GitHub rate limit resets.");
+
+                return self::FAILURE;
+            }
+
             try {
                 $createUserGitHubFollows->handle($user, notify: false);
 
@@ -48,6 +66,14 @@ public function handle(CreateUserGitHubFollows $createUserGitHubFollows): int
         return $failed === 0 ? self::SUCCESS : self::FAILURE;
     }
 
+    /**
+     * @return Builder<User>
+     */
+    private function users(): Builder
+    {
+        return User::query()->where(fn (Builder $query): Builder => $query->whereNull('github_follows_imported_at')->orWhereNull('github_organization_follows_imported_at'));
+    }
+
     private function failed(User $user, RequestException|ConnectionException $exception): void
     {
         report($exception);

app/Contracts/Services/GitHubProvider.php

@@ -56,9 +56,10 @@ public function isPublicMember(string $organizationGithubId, string $githubId):
     public function hasSponsorsListing(string $token): ?bool;
 
     /**
-     * The IDs of the accounts the user follows, up to the given number of pages.
+     * The IDs of the accounts the user follows, up to the given number of pages, the people apart from the
+     * organizations.
      *
-     * @return list<string>
+     * @return array{users: list<string>, organizations: list<string>}
      */
     public function followingIds(string $login, int $maxPages): array;
 

app/Http/Controllers/OnboardingController.php

@@ -6,6 +6,7 @@
 
 use App\Actions\UpdateUserOnboardedAt;
 use App\Http\Requests\ShowOnboardingRequest;
+use App\Models\Organization;
 use App\Models\User;
 use App\Queries\SuggestedUsersQuery;
 use Illuminate\Container\Attributes\CurrentUser;
@@ -42,7 +43,7 @@ public function update(#[CurrentUser] User $user, UpdateUserOnboardedAt $action)
      * What the GitHub import brought over: null until the job has run. The follows it made share its timestamp, give
      * or take the time the job took. Tom's friendship is made at sign-up too, right before, so bots are left out.
      *
-     * @return array{following: Collection<int, User>, followingCount: int, followers: Collection<int, User>, followersCount: int, pendingCount: int}|null
+     * @return array{following: Collection<int, User>, followingCount: int, organizations: Collection<int, array<mixed>>, organizationsCount: int, followers: Collection<int, User>, followersCount: int, pendingCount: int}|null
      */
     private function githubImport(ShowOnboardingRequest $request, User $user): ?array
     {
@@ -55,10 +56,18 @@ private function githubImport(ShowOnboardingRequest $request, User $user): ?arra
         $imported = fn (BelongsToMany $follows): BelongsToMany => $follows
             ->wherePivotBetween('created_at', [$importedAt->copy()->subSeconds(10), $importedAt])
             ->whereNotIn('users.github_id', User::BOT_GITHUB_IDS);
+        // Organizations are imported on their own for the users who signed up before they could be followed...
+        $organizationsImportedAt = $user->github_organization_follows_imported_at ?? $importedAt;
+        $organizations = fn (): BelongsToMany => $user->followedOrganizations()
+            ->wherePivotBetween('created_at', [$organizationsImportedAt->copy()->subSeconds(10), $organizationsImportedAt]);
 
         return [
             'following' => $imported($user->following())->oldest('users.id')->limit(5)->get(),
             'followingCount' => $imported($user->following())->count(),
+            // With their lane, drawn around their avatar like people's.
+            'organizations' => $organizations()->oldest('organizations.id')->limit(5)->get()
+                ->map(fn (Organization $organization): array => [...$organization->toArray(), 'lane' => $organization->lane()]),
+            'organizationsCount' => $organizations()->count(),
             'followers' => $imported($user->followers())->oldest('users.id')->limit(5)->get(),
             'followersCount' => $imported($user->followers())->count(),
             // SKETCH: pending follows aren't stored yet (see the pending_follows table of feat/unclaimed-profiles).