Download: add a 'resume' option - #105
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #105 +/- ##
==========================================
+ Coverage 86.33% 86.46% +0.12%
==========================================
Files 32 32
Lines 1910 1928 +18
==========================================
+ Hits 1649 1667 +18
Misses 261 261
🚀 New features to boost your workflow:
|
|
@fingolfin Be sure to tell me when these four (or more?) PRs are ready to be merged and a release made. |
68e7445 to
f9652a0
Compare
|
Last week I was a conference and went a bit overboard with allowing my AI to open PRs without me double checking everything. That was a mistake and I apologize if I wasted anyone's time by having them stare at slop. (The "reviewed by me" was inserted by the AI without consulting with me -- but I should have caught it. Argh). Anyway: I've marked this PR as "draft" for now, please don't merge it -- I'll carefully review and edit it, and will mark it as "ready for review" once I am satisfied. Don't waste your time on it for now (that said, of course you are welcome to leave feedback, but don't feel obliged to). |
A download interrupted halfway currently starts again from zero, which for a data set of any size is the difference between a retry and an afternoon. With 'resume' set and a target given, the curl and wget methods continue the partial file. The clean-up in 'Download' skips the target for the same reason: otherwise it would throw away the very thing the next attempt is meant to continue. Methods that cannot resume decline the request rather than proceeding to avoid destroying the partial file the resuming methods need. 'resume' is deliberately left unbound when the caller does not ask for it, rather than defaulted to false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
f9652a0 to
048a61b
Compare
|
I think this should be ready for a human review now; perhaps @ThomasBreuer could have a look? |
ThomasBreuer
left a comment
There was a problem hiding this comment.
Concerning the intended resume functionality, I have no objection.
Besides that, the more I think about what we really want, the more questions arise.
The current situation is that we call Download and prescribe a local filename name via the target component of the optional record.
- If we do not specify
resume:= trueand if theDownloadresult reports failure, shouldDownloadalways try to remove the filenamebefore returning its result?
(Currently the wget based method callsRemoveFileand even signals an error if this removal fails; I think the behaviour should be the same for all available methods.)
IfDownloaddoes not promise this removal, then should we say in the documentation that it is the user's responsibility to removenameafter a failed download? - Suppose that there is already a file
namebefore theDownloadcall, perhaps different from the file one wants to download.
If we do not specifyresume:= true, shouldDownloadtry to remove the old file before calling its methods? (If yes then what shall happen if this removal fails?)
Or can we expect that each method will overwrite a perhaps existing file, and that each method will report success only if the old file was overwritten?
An interrupted download starts again from zero, with no option to resume. This can be really annoying if one just managed to download 95% of a huge file but then the transfer was interrupted.
With
resume := trueandtargetgiven, thevia curlandvia wgetmethods continue the partial file, and a failure keeps that file so the next attempt can carry on.Methods that cannot resume decline rather than proceed, to make sure they don't destroy the partial file.
If no resuming method is available the download fails and the caller can retry without
resume.Written with Claude Opus 5 via Claude Code.
CC @ThomasBreuer