Skip to content

fix(scraper): stop setting bogus openclaw/skills githubUrl#40

Open
oratis wants to merge 1 commit into
mainfrom
claude/competent-gagarin-2f988a
Open

fix(scraper): stop setting bogus openclaw/skills githubUrl#40
oratis wants to merge 1 commit into
mainfrom
claude/competent-gagarin-2f988a

Conversation

@oratis

@oratis oratis commented Jul 4, 2026

Copy link
Copy Markdown
Owner

Problem

scripts/scrape-skill-details.ts extracted githubUrl from github.com/openclaw/skills/tree/main/skills/<author>/<name> links on clawskills.sh pages — a repo that does not exist (HTTP 404). This had polluted 5028 of 5385 Skill.githubUrl rows in prod. The /api/cron/check-github route was accurately reporting them as dead links.

Root cause

clawskills.sh exposes no per-skill source repo. Verified against the live site and both data-origin scrapers (parse-skills.ts, scrape-clawskills.ts):

  • The canonical per-skill link is the ClawHub page (already stored as clawHubUrl / clawSkillsUrl).
  • The only GitHub link on a skill page is the site-wide "Star" button to the aggregate list github.com/VoltAgent/awesome-openclaw-skills — not a per-skill source.

So bogus values could not be corrected to real repos (none exist per-skill); they could only be nulled.

Change

The scraper no longer populates githubUrl for clawskills-sourced skills — it's left NULL. Genuine githubUrl values continue to come only from source=GITHUB_SCRAPE skills (scrape-github-skills.ts, real repo.htmlUrl). Removed the interface field, the DOM query, the return field, and the update assignment, with an explanatory comment.

Prod data remediation (already applied, out of band — not in this PR)

A one-time DB update NULLed githubUrl + cleared stale ghStatus/ghCheckedAt on the 5028 affected rows (backup taken; exactly 5028 rows asserted before mutating). Post-state: 0 bogus, 222 real URLs preserved. All affected skills retain their clawSkillsUrl fallback, which the UI (SkillCard) already prefers when githubUrl is null — so no user-visible link was lost.

🤖 Generated with Claude Code

scrape-skill-details.ts extracted githubUrl from
github.com/openclaw/skills/tree/main/skills/<author>/<name> links on
clawskills.sh pages — a repo that does not exist (HTTP 404). This had
polluted 5028 of 5385 Skill.githubUrl rows in prod (now NULLed out in a
separate one-time DB remediation).

clawskills.sh exposes no per-skill source repo: the canonical per-skill
link is the ClawHub page (already stored as clawHubUrl/clawSkillsUrl),
and the only GitHub link on a page is the site-wide "Star" button to the
aggregate list VoltAgent/awesome-openclaw-skills. So the scraper now
leaves githubUrl untouched (NULL) for clawskills-sourced skills; genuine
githubUrl values come only from source=GITHUB_SCRAPE skills.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@oratis

oratis commented Jul 11, 2026

Copy link
Copy Markdown
Owner Author

Review — LGTM ✅

改动小而准,root cause 分析扎实:

  • 确认了 clawskills.sh 页面上唯一的 GitHub 链接是站级 Star 按钮(聚合列表),github.com/openclaw/skills/... 这个 per-skill 路径根本不存在(404),所以"修正为真实 repo"不可行、只能不写 —— 结论成立。
  • 删除路径完整:interface 字段、DOM 提取、返回值、updateData.githubUrl 赋值四处全清,没有残留。留下的注释把"为什么这里不抓 githubUrl"和 5028 行污染的历史讲清楚了,能防止未来有人"好心"把提取逻辑加回来。
  • 真实 githubUrl 的唯一来源收敛到 scrape-github-skills.ts(source=GITHUB_SCRAPE),与 prod 修复后的状态(222 条真实 URL 保留)一致。
  • UI 侧 SkillCardgithubUrl 为 null 时回退 clawSkillsUrl,用户无感。

一个小提醒(不阻塞):prod 数据已在带外 NULL 掉,但如果哪天用旧备份恢复数据库,这批脏数据会回来——备份恢复 runbook 里值得记一笔。

可以合并。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant