Suggest article selectors with OpenAI on request

CLAUDE.md

@@ -39,9 +39,10 @@ web/           → Go HTML templates (HTMX v2), static assets
 **Key interfaces:**
 - `extractor.Rules` (defined consumer-side in `extractor/readability.go`), implemented by `datastore.RulesDAO`. Mock generated with `//go:generate moq` in extractor package.
 - `extractor.Retriever` (defined in `extractor/retriever.go`) — abstracts URL content fetching. Two implementations: `HTTPRetriever` (default, standard HTTP GET with Safari user-agent) and `CloudflareRetriever` (Cloudflare Browser Rendering API for JS-rendered pages). When `UReadability.Retriever` is nil, defaults to `HTTPRetriever`.
-- `extractor.OpenAIClient` and `extractor.Summaries` (defined in `extractor/summary.go`): the go-openai client and the summary cache (`datastore.SummariesDAO`, TTL index on `expires_at`). `UReadability.OpenAI` is set only when `--openai.api-key` is given; nil means summaries are off. Mocks in `extractor/mocks/`; moq cannot read Go 1.27 export data, so regenerate them with a newer moq or edit by hand.
+- `extractor.SuggestRule` (`extractor/improve.go`) backs `POST /api/content-parsed-wrong`: the model suggests a selector from a page outline (`extractor/outline.go`), the selector is checked against the page, and it is stored as a disabled rule when the domain has none. It never runs during normal extraction.
+- `extractor.OpenAIClient` and `extractor.Summaries` (defined in `extractor/summary.go`): the go-openai client and the summary cache (`datastore.SummariesDAO`, TTL index on `expires_at`). `UReadability.OpenAI` is set only when `--openai.api-key` is given; nil means OpenAI features (summaries, `SuggestRule`) are off. Mocks in `extractor/mocks/`; moq cannot read Go 1.27 export data, so regenerate them with a newer moq or edit by hand.
 
-Optional OpenAI flags, all in the `--openai.*` group: `api-key` / `OPENAI_API_KEY` (enables summaries), `model` / `OPENAI_MODEL` (default `gpt-6-luna`), `summary-prompt`, `max-content-length` (characters, default 10000).
+Optional OpenAI flags, all in the `--openai.*` group: `api-key` / `OPENAI_API_KEY` (enables summaries and selector suggestions), `model` / `OPENAI_MODEL` (default `gpt-6-luna`), `summary-prompt`, `max-content-length` (characters, default 10000).
 
 ## Content Extraction Flow
 

README.md

@@ -21,13 +21,13 @@
 | cf-route-all | CF_ROUTE_ALL    | `false`        | route every request through Cloudflare Browser Rendering |
 | dbg          | DEBUG           | `false`        | debug mode                                            |
 
-#### OpenAI summaries (optional)
+#### OpenAI (optional)
 
-Setting `--openai.api-key` enables article summaries; without it no OpenAI calls are made.
+Setting `--openai.api-key` enables article summaries and selector suggestions for pages that are parsed wrong; without it no OpenAI calls are made.
 
 | Command line              | Environment               | Default      | Description                                                        |
 |---------------------------|---------------------------|--------------|--------------------------------------------------------------------|
-| openai.api-key            | OPENAI_API_KEY            | none         | OpenAI API key, enables summaries                                  |
+| openai.api-key            | OPENAI_API_KEY            | none         | OpenAI API key, enables summaries and selector suggestions         |
 | openai.model              | OPENAI_MODEL              | `gpt-6-luna` | OpenAI model                                                       |
 | openai.summary-prompt     | OPENAI_SUMMARY_PROMPT     | built-in     | custom system prompt for summaries                                 |
 | openai.max-content-length | OPENAI_MAX_CONTENT_LENGTH | `10000`      | max characters of article text sent for summarizing (0 for no limit) |
@@ -46,9 +46,12 @@ When Cloudflare credentials are not set, the service uses a standard HTTP client
     GET /api/content/v1/parser?token=secret&url=http://aa.com/blah - extract content (emulate Readability API parse call)
     GET /api/content/v1/parser?token=secret&url=http://aa.com/blah&summary=true - extract content with AI-generated summary
     POST /api/extract {url: http://aa.com/blah}  - extract content
+    POST /api/content-parsed-wrong?url=http://aa.com/blah - ask OpenAI for a rule for a page that is parsed wrong (credentials required)
 
 Summary generation requires the server token to be set and an OpenAI API key. Summaries are cached in MongoDB for 30 days, keyed by model, prompt and article text; a TTL index removes expired ones. When the key is set, rule previews in the web UI also show a summary.
 
+`POST /api/content-parsed-wrong` uses the same credentials as the rule management routes and calls OpenAI only when it is requested, never during normal extraction. The model gets an outline of the page body and suggests a CSS selector for the article. The selector is applied to the page and accepted only when it matches a plausible amount of text, not nearly the whole page and not what is extracted now; up to three suggestions are tried. The response holds the selector, the page extracted with it and `saved`. When the domain has no rule yet, the selector is stored as a **disabled** rule, so nothing changes for readers until it is reviewed and enabled in the web UI. When the domain already has a rule, enabled or not, that rule is left as it is and only the suggestion is returned.
+
 ## Development
 
 ### Running tests

datastore/rules.go

@@ -18,6 +18,7 @@ const (
 	fieldID        = "_id"
 	fieldExpiresAt = "expires_at"
 	opSet          = "$set"
+	opSetOnInsert  = "$setOnInsert"
 )
 
 // RulesDAO data-access obj for custom parsing rules, implements Rules
@@ -77,17 +78,7 @@ func (r RulesDAO) GetByID(ctx context.Context, id bson.ObjectID) (Rule, bool) {
 
 // Save upsert rule
 func (r RulesDAO) Save(ctx context.Context, rule Rule) (Rule, error) {
-	// normalise nil slices to empty ones so cleared multi-value fields are stored as BSON arrays
-	// ([]) rather than null (the driver encodes nil slices as null), keeping the stored type stable.
-	if rule.MatchURLs == nil {
-		rule.MatchURLs = []string{}
-	}
-	if rule.Excludes == nil {
-		rule.Excludes = []string{}
-	}
-	if rule.TestURLs == nil {
-		rule.TestURLs = []string{}
-	}
+	rule = withEmptySlices(rule)
 	ch, err := r.UpdateOne(ctx, bson.M{fieldDomain: rule.Domain}, bson.M{opSet: rule}, options.UpdateOne().SetUpsert(true))
 	if err != nil {
 		log.Printf("[WARN] failed to save, error=%v, article=%v", err, rule)
@@ -109,6 +100,33 @@ func (r RulesDAO) Save(ctx context.Context, rule Rule) (Rule, error) {
 	return rule, err
 }
 
+// SaveIfAbsent stores the rule only when the domain has no rule yet, enabled or not, and reports
+// whether it did. An existing rule is never touched, unlike Save which replaces it.
+func (r RulesDAO) SaveIfAbsent(ctx context.Context, rule Rule) (bool, error) {
+	rule = withEmptySlices(rule)
+	ch, err := r.UpdateOne(ctx, bson.M{fieldDomain: rule.Domain}, bson.M{opSetOnInsert: rule}, options.UpdateOne().SetUpsert(true))
+	if err != nil {
+		log.Printf("[WARN] failed to save, error=%v, article=%v", err, rule)
+		return false, err
+	}
+	return ch.UpsertedID != nil, nil
+}
+
+// withEmptySlices normalises nil slices to empty ones so cleared multi-value fields are stored as BSON
+// arrays ([]) rather than null (the driver encodes nil slices as null), keeping the stored type stable.
+func withEmptySlices(rule Rule) Rule {
+	if rule.MatchURLs == nil {
+		rule.MatchURLs = []string{}
+	}
+	if rule.Excludes == nil {
+		rule.Excludes = []string{}
+	}
+	if rule.TestURLs == nil {
+		rule.TestURLs = []string{}
+	}
+	return rule
+}
+
 // Disable marks enabled=false, by id
 func (r RulesDAO) Disable(ctx context.Context, id bson.ObjectID) error {
 	_, err := r.UpdateOne(ctx, bson.M{fieldID: id}, bson.M{opSet: bson.M{fieldEnabled: false}})

datastore/rules_test.go

@@ -2,6 +2,7 @@ package datastore
 
 import (
 	"context"
+	"fmt"
 	"math/rand/v2"
 	"testing"
 
@@ -92,6 +93,48 @@ func TestRulesSave(t *testing.T) {
 	})
 }
 
+func TestRulesSaveIfAbsent(t *testing.T) {
+	rules := setupRules(t)
+	ctx := context.Background()
+
+	t.Run("new domain is stored", func(t *testing.T) {
+		rule := Rule{Domain: randDomain(), Content: "div.copy", User: "openai", TestURLs: []string{"https://example.com/a"}}
+		saved, err := rules.SaveIfAbsent(ctx, rule)
+		require.NoError(t, err)
+		assert.True(t, saved)
+
+		var stored Rule
+		require.NoError(t, rules.FindOne(ctx, bson.M{fieldDomain: rule.Domain}).Decode(&stored))
+		assert.Equal(t, "div.copy", stored.Content)
+		assert.Equal(t, "openai", stored.User)
+		assert.False(t, stored.Enabled)
+		assert.Equal(t, []string{"https://example.com/a"}, stored.TestURLs)
+		assert.Equal(t, []string{}, stored.MatchURLs, "nil slices are stored as arrays")
+	})
+
+	for _, enabled := range []bool{true, false} {
+		t.Run(fmt.Sprintf("existing rule is kept, enabled=%v", enabled), func(t *testing.T) {
+			domain := randDomain()
+			orig, err := rules.Save(ctx, Rule{Domain: domain, Content: "original", Author: "me", Enabled: enabled})
+			require.NoError(t, err)
+
+			saved, err := rules.SaveIfAbsent(ctx, Rule{Domain: domain, Content: "suggested", User: "openai"})
+			require.NoError(t, err)
+			assert.False(t, saved)
+
+			var stored []Rule
+			cursor, err := rules.Find(ctx, bson.M{fieldDomain: domain})
+			require.NoError(t, err)
+			require.NoError(t, cursor.All(ctx, &stored))
+			require.Len(t, stored, 1)
+			assert.Equal(t, orig.ID, stored[0].ID)
+			assert.Equal(t, "original", stored[0].Content)
+			assert.Equal(t, "me", stored[0].Author)
+			assert.Equal(t, enabled, stored[0].Enabled)
+		})
+	}
+}
+
 func TestRulesGet(t *testing.T) {
 	rules := setupRules(t)
 

extractor/improve.go

@@ -0,0 +1,206 @@
+package extractor
+
+import (
+	"cmp"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/PuerkitoBio/goquery"
+	log "github.com/go-pkgz/lgr"
+	"github.com/sashabaranov/go-openai"
+
+	"github.com/ukeeper/ukeeper-readability/datastore"
+)
+
+// ErrOpenAIDisabled is returned by SuggestRule when no OpenAI client is configured
+var ErrOpenAIDisabled = errors.New("OpenAI is not configured")
+
+// ErrNoSelector is returned by SuggestRule when no attempt produced a selector that passed the checks
+var ErrNoSelector = errors.New("no usable selector was suggested")
+
+// errInvalidAnswer marks a model reply that is not the requested JSON
+var errInvalidAnswer = errors.New("reply is not the requested JSON")
+
+const (
+	suggestAttempts     = 3
+	suggestCallTimeout  = 60 * time.Second // one model call
+	suggestedRuleUser   = "openai"
+	maxSelectorLen      = 200
+	minSuggestedLen     = 200 // characters of text a selector has to match
+	maxWholePagePercent = 90  // a selector matching more of the page text than this is the page, not the article
+	currentTextLen      = 500 // characters of the current extraction shown to the model
+)
+
+const suggestPrompt = `You find the main article on a web page. You get an outline of the page body: one line per element ` +
+	`with its tag, id, classes, the number of characters of visible text and the number of paragraphs. ` +
+	`Reply with JSON only: {"selector": "<CSS selector>"}. ` +
+	`The selector must match the element that holds the whole article text and nothing else: ` +
+	`no navigation, comments, related links, share buttons or footer. ` +
+	`Prefer an id or a stable class over positional selectors.`
+
+// Suggestion is what SuggestRule found for a URL
+type Suggestion struct {
+	Selector string    `json:"selector"`
+	Saved    bool      `json:"saved"`    // true when the selector was stored as a disabled rule for review
+	Response *Response `json:"response"` // the page extracted with the selector
+}
+
+// attempt is a rejected selector together with the reason, passed back to the model
+type attempt struct {
+	selector string
+	reason   string
+}
+
+// SuggestRule asks the model for a CSS selector of the article on the page, for a page the general
+// parser got wrong. The selector is applied to the page and has to match a plausible amount of text
+// before it is accepted. When the domain has no rule yet, it is stored as a disabled rule, so
+// nothing changes for readers until someone reviews and enables it. When the domain already has a
+// rule, the existing one is left alone and the suggestion is only returned.
+func (f *UReadability) SuggestRule(ctx context.Context, reqURL string) (*Suggestion, error) {
+	if f.OpenAI == nil {
+		return nil, ErrOpenAIDisabled
+	}
+	u, err := url.Parse(reqURL)
+	if err != nil {
+		return nil, fmt.Errorf("parse url %q: %w", reqURL, err)
+	}
+
+	var rule *datastore.Rule // the enabled rule, as extraction would use it
+	if f.Rules != nil {
+		if r, found := f.Rules.Get(ctx, reqURL); found {
+			rule = &r
+		}
+	}
+
+	rb, body, err := f.fetch(ctx, reqURL, rule)
+	if err != nil {
+		return nil, err
+	}
+	doc, err := goquery.NewDocumentFromReader(strings.NewReader(body))
+	if err != nil {
+		return nil, err
+	}
+
+	current, _, _ := f.getContent(ctx, body, reqURL, rule)
+	current = f.getText(current, "")
+	outline := pageOutline(doc.Nodes[0])
+	pageLen := visibleTextLen(findBody(doc.Nodes[0]))
+
+	var rejected []attempt
+	for range suggestAttempts {
+		selector, err := f.askSelector(ctx, reqURL, current, outline, rejected)
+		if errors.Is(err, errInvalidAnswer) {
+			rejected = append(rejected, attempt{reason: "the reply was not the requested JSON"})
+			continue
+		}
+		if err != nil {
+			return nil, err
+		}
+		if reason := f.checkSelector(body, selector, current, pageLen); reason != "" {
+			log.Printf("[INFO] selector %q for %s rejected: %s", selector, reqURL, reason)
+			rejected = append(rejected, attempt{selector: selector, reason: reason})
+			continue
+		}
+
+		resp, err := f.parse(ctx, reqURL, body, rb, &datastore.Rule{Content: selector, Enabled: true})
+		if err != nil {
+			return nil, err
+		}
+		return &Suggestion{Selector: selector, Saved: f.saveSuggested(ctx, u.Host, reqURL, selector), Response: resp}, nil
+	}
+	return nil, ErrNoSelector
+}
+
+// askSelector makes one model call and returns the selector from the reply
+func (f *UReadability) askSelector(ctx context.Context, reqURL, current, outline string, rejected []attempt) (string, error) {
+	var sb strings.Builder
+	fmt.Fprintf(&sb, "URL: %s\n\n", reqURL)
+	if current != "" {
+		if r := []rune(current); len(r) > currentTextLen {
+			current = string(r[:currentTextLen])
+		}
+		fmt.Fprintf(&sb, "The text extracted now is reported as wrong. It starts with:\n---\n%s\n---\n\n", current)
+	}
+	fmt.Fprintf(&sb, "Page body outline:\n---\n%s\n---\n", outline)
+	for _, a := range rejected {
+		if a.selector == "" {
+			fmt.Fprintf(&sb, "\nAn earlier attempt failed: %s. Try again.", a.reason)
+			continue
+		}
+		fmt.Fprintf(&sb, "\nThe selector %q was rejected: %s. Suggest a different one.", a.selector, a.reason)
+	}
+
+	callCtx, cancel := context.WithTimeout(ctx, suggestCallTimeout)
+	defer cancel()
+	resp, err := f.OpenAI.CreateChatCompletion(callCtx, openai.ChatCompletionRequest{
+		Model: cmp.Or(f.OpenAIModel, DefaultOpenAIModel),
+		Messages: []openai.ChatCompletionMessage{
+			{Role: openai.ChatMessageRoleSystem, Content: suggestPrompt},
+			{Role: openai.ChatMessageRoleUser, Content: sb.String()},
+		},
+		ResponseFormat: &openai.ChatCompletionResponseFormat{Type: openai.ChatCompletionResponseFormatTypeJSONObject},
+	})
+	if err != nil {
+		return "", fmt.Errorf("openai selector request: %w", err)
+	}
+	if len(resp.Choices) == 0 {
+		return "", fmt.Errorf("openai returned no choices: %w", errInvalidAnswer)
+	}
+
+	var answer struct {
+		Selector string `json:"selector"`
+	}
+	if err := json.Unmarshal([]byte(resp.Choices[0].Message.Content), &answer); err != nil || strings.TrimSpace(answer.Selector) == "" {
+		return "", errInvalidAnswer
+	}
+	return strings.TrimSpace(answer.Selector), nil
+}
+
+// checkSelector applies the selector to the page and returns why it can't be the article, or "" when it can
+func (f *UReadability) checkSelector(body, selector, current string, pageLen int) string {
+	if len(selector) > maxSelectorLen {
+		return "it is too long"
+	}
+	res, err := extractWithSelector(body, selector)
+	if err != nil || res == "" {
+		return "it matches nothing on the page"
+	}
+	doc, err := goquery.NewDocumentFromReader(strings.NewReader(res))
+	if err != nil {
+		return "what it matches can't be parsed"
+	}
+	n := visibleTextLen(findBody(doc.Nodes[0]))
+	switch {
+	case n < minSuggestedLen:
+		return fmt.Sprintf("it matches only %d characters of text", n)
+	case n*100 > pageLen*maxWholePagePercent:
+		return "it matches almost all the text of the page"
+	case f.getText(res, "") == current:
+		return "it extracts the same text as the current result"
+	}
+	return ""
+}
+
+// saveSuggested stores the selector as a disabled rule when the domain has none. Both enabled and
+// disabled rules count, and an existing rule is never replaced.
+func (f *UReadability) saveSuggested(ctx context.Context, domain, reqURL, selector string) bool {
+	if f.Rules == nil {
+		return false
+	}
+	rule := datastore.Rule{Domain: domain, Content: selector, TestURLs: []string{reqURL}, User: suggestedRuleUser}
+	saved, err := f.Rules.SaveIfAbsent(ctx, rule)
+	switch {
+	case err != nil:
+		log.Printf("[WARN] failed to save suggested rule for %s: %v", domain, err)
+	case !saved:
+		log.Printf("[INFO] %s already has a rule, suggested selector %q is not saved", domain, selector)
+	default:
+		log.Printf("[INFO] saved disabled rule for %s with selector %q", domain, selector)
+	}
+	return err == nil && saved
+}