Skip to content

tee: own page caches using memfd for 2+ outputs - #15854

Open
oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:tee-memfd
Open

oech3 wants to merge 1 commit into
uutils:mainfrom
oech3:tee-memfd

Conversation

@oech3

@oech3 oech3 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Closes #15803 .
Note that this causes large performance drop:

> yes|tee /dev/null|pv>/dev/null
 133GiB 0:00:02 [66.8GiB/s]
> yes|tee-memfd /dev/null|pv>/dev/null
17.7GiB 0:00:02 [8.86GiB/s]
> yes|tee-rw /dev/null|pv>/dev/null
10.1GiB 0:00:02 [5.09GiB/s]

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

GNU testsuite comparison:

GNU test failed: tests/tee/tee. tests/tee/tee is passing on 'main'. Maybe you have to rebase?

@oech3
oech3 force-pushed the tee-memfd branch 4 times, most recently from 5681e0d to e0a00ac Compare October 26, 2026 04:24
@oech3

This comment was marked as resolved.

@oech3
oech3 marked this pull request as draft October 26, 2026 04:36
@oech3 oech3 changed the title tee: restrict fast-path to 1 output tee: own page caches using memfd for 2+ outputs Oct 26, 2026
@oech3
oech3 force-pushed the tee-memfd branch 11 times, most recently from 77d41f4 to b860187 Compare October 26, 2026 09:46
@oech3
oech3 marked this pull request as ready for review October 26, 2026 09:46
Comment thread tests/by-util/test_tee.rs Outdated
@oech3
oech3 force-pushed the tee-memfd branch 8 times, most recently from 15cd37b to 7027117 Compare October 26, 2026 14:43
@oech3
oech3 force-pushed the tee-memfd branch 3 times, most recently from 4ad7452 to 389987f Compare October 26, 2026 16:36
@oech3

oech3 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

Passing tee.sh now.

@oech3
oech3 marked this pull request as draft October 26, 2026 23:54
@oech3

oech3 commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

I think we should keep using tee(2) for stream input as it is always reliable (and consider that it is provider's bug if input can be changed diring copy): #15894 .

@oech3
oech3 marked this pull request as ready for review October 27, 2026 03:52
@xtqqczze

xtqqczze commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

yes|tee /dev/null|pv>/dev/null

what performance result does gnu-tee give?

@oech3

oech3 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

Why? but

$ yes | pv>/dev/null
 105GiB 0:00:01 [ 105GiB/s]
$ yes | tee-memfd | pv>/dev/null
78.6GiB 0:00:01 [78.6GiB/s]
$ yes | '/home/a/bin/gnu-git/tee' /dev/null | pv>/dev/null
2.94GiB 0:00:01 [2.94GiB/s] 
$ yes | tee-rw /dev/null | pv>/dev/null
4.95GiB 0:00:01 [4.95GiB/s]
$ yes | tee-memfd /dev/null | pv>/dev/null
8.80GiB 0:00:01 [8.80GiB/s]

Note that pv must be latest version to let pv use 1 MiB pipe, and yes should be uutils version since it is faster tham GNU.

I'm sad to lost because of we need to doubt source, but it would be difficult to make tee-memfd an overhead of pipeline on the real world until people started using memfd + mmap at many utils.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

tee: zero-copy lets stdout and FILE differ if input changes after tee exits

2 participants