fix(scraper): stop setting bogus openclaw/skills githubUrl#40
Open
oratis wants to merge 1 commit into
Open
Conversation
scrape-skill-details.ts extracted githubUrl from github.com/openclaw/skills/tree/main/skills/<author>/<name> links on clawskills.sh pages — a repo that does not exist (HTTP 404). This had polluted 5028 of 5385 Skill.githubUrl rows in prod (now NULLed out in a separate one-time DB remediation). clawskills.sh exposes no per-skill source repo: the canonical per-skill link is the ClawHub page (already stored as clawHubUrl/clawSkillsUrl), and the only GitHub link on a page is the site-wide "Star" button to the aggregate list VoltAgent/awesome-openclaw-skills. So the scraper now leaves githubUrl untouched (NULL) for clawskills-sourced skills; genuine githubUrl values come only from source=GITHUB_SCRAPE skills. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Owner
Author
Review — LGTM ✅改动小而准,root cause 分析扎实:
一个小提醒(不阻塞):prod 数据已在带外 NULL 掉,但如果哪天用旧备份恢复数据库,这批脏数据会回来——备份恢复 runbook 里值得记一笔。 可以合并。 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
scripts/scrape-skill-details.tsextractedgithubUrlfromgithub.com/openclaw/skills/tree/main/skills/<author>/<name>links on clawskills.sh pages — a repo that does not exist (HTTP 404). This had polluted 5028 of 5385Skill.githubUrlrows in prod. The/api/cron/check-githubroute was accurately reporting them as dead links.Root cause
clawskills.sh exposes no per-skill source repo. Verified against the live site and both data-origin scrapers (
parse-skills.ts,scrape-clawskills.ts):clawHubUrl/clawSkillsUrl).github.com/VoltAgent/awesome-openclaw-skills— not a per-skill source.So bogus values could not be corrected to real repos (none exist per-skill); they could only be nulled.
Change
The scraper no longer populates
githubUrlfor clawskills-sourced skills — it's leftNULL. GenuinegithubUrlvalues continue to come only fromsource=GITHUB_SCRAPEskills (scrape-github-skills.ts, realrepo.htmlUrl). Removed the interface field, the DOM query, the return field, and the update assignment, with an explanatory comment.Prod data remediation (already applied, out of band — not in this PR)
A one-time DB update NULLed
githubUrl+ cleared staleghStatus/ghCheckedAton the 5028 affected rows (backup taken; exactly 5028 rows asserted before mutating). Post-state: 0 bogus, 222 real URLs preserved. All affected skills retain theirclawSkillsUrlfallback, which the UI (SkillCard) already prefers whengithubUrlis null — so no user-visible link was lost.🤖 Generated with Claude Code