Skip to content

Add response compression to replicator - #6106

Open
lacklacklack wants to merge 1 commit into
apache:mainfrom
lacklacklack:replicator-response-compression
Open

Add response compression to replicator#6106
lacklacklack wants to merge 1 commit into
apache:mainfrom
lacklacklack:replicator-response-compression

Conversation

@lacklacklack

Copy link
Copy Markdown
Contributor

Overview

Adds gzip compression for outbound HTTP responses when the client sends
Accept-Encoding: gzip.

The replicator now sends Accept-Encoding: gzip on outbound requests
(_bulk_docs, _revs_diff, _bulk_get) and automatically decompresses
gzip responses received from the remote server.

Testing recommendations

Run a replication and check the stats:

# Start CouchDB
./dev/run --admin=admin:admin

# Create DBs, add docs, replicate
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db
curl -X PUT http://admin:admin@127.0.0.1:25984/target_db
curl -X PUT http://admin:admin@127.0.0.1:15984/source_db/doc1 \
  -H "Content-Type: application/json" \
  -d '{"data":"'$(python3 -c "print('x'*2000)")'"}'
curl -X POST http://admin:admin@127.0.0.1:15984/_replicate \
  -H "Content-Type: application/json" \
  -d '{"source":"http://admin:admin@127.0.0.1:15984/source_db","target":"http://admin:admin@127.0.0.1:25984/target_db"}'

# Check stats — value should be > 0
curl -s http://admin:admin@127.0.0.1:15984/_node/_local/_stats/couch_replicator/responses_decompressed | python3 -m json.tool

Automated tests: couch_replicator_compression_tests in src/couch_replicator/test/eunit/.

Related Issues or Pull Requests

Add request compression to replicator - #6013

Checklist

  • This is my own work, I did not use AI, LLM's or similar technology
  • Code is written and works correctly
  • Changes are covered by tests
  • Any new configurable parameters are documented in rel/overlay/etc/default.ini
  • Documentation changes were made in the src/docs folder
  • Documentation changes were backported (separated PR) to affected branches

@nickva nickva left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good start! But I think we'll still need the main bits on the server that do the compression. To make things much easier for us mochiweb already includes options to compress responses both for chunks and one-shots.

Chunks may be neat to try but a bit tricky at first. Let's see if we can do one-shot ones at least.

maybe_decompress_response(_Headers, <<>>) ->
<<>>;
maybe_decompress_response(Headers, Body) ->
case lists:keyfind("content-encoding", 1, [{string:to_lower(K), V} || {K, V} <- Headers]) of

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See how we use mochiweb_header make/get_value that has a facility to fetch headers in a case insensitive way

.. versionadded:: 3.6

Decompress gzip-encoded inbound response bodies (``_bulk_docs``,
``_revs_diff``, ``_bulk_get``) received from the replication source.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor nit: _revs_diff and _bulk_docs calls are for the target

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants