[simonchrz]

Moves the change feed readers from git-pkgs/proxy#625 here, as asked in the review there. A new changes/ package reads the feeds in which registries list the packages that changed since a given point, starting with npm (replicate.npmjs.com/registry/_changes) and Packagist (metadata/changes.json).

type Feed interface {
	Ecosystem() string
	Start(ctx context.Context) (Cursor, error)
	Changes(ctx context.Context, cursor Cursor) (Changes, Cursor, error)
}

• Feeds are stateless. The caller keeps the Cursor (a string, so it can be stored) and moves on to the returned one after acting on the changes. On error Changes returns the cursor it was given.
• NewNPM and NewPackagist take WithClient (a *client.Client, so requests get its retries and limits) and WithURL. PackagistFeedURL builds the URL for a Packagist instance.
• Packagist: Start reads the timestamp from the 400 a request without since gets. A delete is reported as vendor/pkg and vendor/pkg~dev, and a resync as Changes.All.
• npm: Start reads update_seq from the database document, and sequences are passed back as given. Changes reads up to ten pages of 10000 and skips _design/ ids.
• Fixed while moving: when a page after the first failed or came without last_seq, the names of the earlier pages were lost. Changes now returns them with the cursor after the last good page and no error.

Tests run against fake Packagist and replicate servers: start position, update/delete/resync, errors returning the cursor, paging and the page cap, a later page failing either way, and string sequences. The proxy side is ready against this branch and will bump registries once there's a tag.

[andrew]

Please fix npm startup for feed URLs containing query parameters and add a regression test through NewNPM(WithURL(...)).Start.

[simonchrz]

Fixed in 88ec8f0. Start now parses the feed URL and trims _changes from its path only, keeping the query. TestNPMQueryInURL runs Start and Changes against a feed that requires ?token=example; it fails without the fix.