fstree: add compressed object decompression tool#4079
Conversation
End-rey
requested review from
carpawell,
cthulhu-rider and
roman-khimov
as code owners
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4079 +/- ##
==========================================
+ Coverage 26.28% 26.66% +0.38%
==========================================
Files 671 674 +3
Lines 45862 45860 -2
==========================================
+ Hits 12056 12230 +174
+ Misses 32674 32448 -226
- Partials 1132 1182 +50 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| // | ||
| // The method preserves combined physical files by rewriting all hard links to | ||
| // the same new combined file. The storage node must not write to this FSTree | ||
| // while the migration is running. |
There was a problem hiding this comment.
Wrong assumption, it will. And the idea is to let it do so.
There was a problem hiding this comment.
The migration can now run while the storage node is running on Linux. On other platforms, the node must be stopped because safe concurrent deletion handling is not available.
| return RewriteCompressedStats{}, fmt.Errorf("flush pending writes: %w", err) | ||
| } | ||
|
|
||
| files, err := t.compressedRewriteFiles() |
There was a problem hiding this comment.
Please don't create these registries. Our mainnet shards have ~16M objects routinely, just listing them is already a problem and it takes a lot of time. You need to do the job while iterating over the tree.
| } | ||
|
|
||
| func rewriteHardlinkedObjectFile(group []rewriteCompressedFile, data []byte) error { | ||
| tmp, err := os.CreateTemp(filepath.Dir(group[0].path), ".rewrite-compressed-*") |
There was a problem hiding this comment.
Use O_TMPFILE, just like regular FSTree does.
| if err = os.Link(tmpPath, linkPath); err != nil { | ||
| return fmt.Errorf("link temporary object file to %q: %w", linkPath, err) | ||
| } | ||
| if err = os.Rename(linkPath, group[i].path); err != nil { |
There was a problem hiding this comment.
Be prepared for the node deleting the same file as you walk over the FSTree.
| // The method preserves combined physical files by rewriting all hard links to | ||
| // the same new combined file. The storage node must not write to this FSTree | ||
| // while the migration is running. | ||
| func (t *FSTree) RewriteCompressed() (RewriteCompressedStats, error) { |
There was a problem hiding this comment.
if it is an FSTree method, why do we need this utility then? it can be a background routing for the next release
There was a problem hiding this comment.
Theoretically, yes, I also had this thought, but:
- either it needs to be synchronized with other activities (deletion first of all to avoid object resurrection) which requires some effort
- or it keeps the same logic as an outsider, but then having a separate process is easier, can be controlled by shard and can even be performed before the release
Add fstree-decompress for rewriting old zstd-compressed FSTree objects as plain canonical object bytes. Support single FSTree roots via `--path` and node configs via `--config`. Signed-off-by: Andrey Butusov <andrey@nspcc.io>
3 participants
No description provided.