diff --git a/Makefile b/Makefile index 46ae0191..29be0869 100644 --- a/Makefile +++ b/Makefile @@ -50,14 +50,16 @@ else -curl -sSL https://install.python-poetry.org | python3 - endif +# grpcio-tools pinned: generated stubs refuse grpcio older than the tools version, +# keep in sync with the grpcio floor declared in pyproject.toml .PHONY: gen gen: - poetry run pip install 'grpcio-tools>=1.68.0' packaging + poetry run pip install 'grpcio-tools~=1.83.0' packaging poetry run python3 tools/grpc_code_gen.py .PHONY: gen-basic gen-basic: - python3 -m pip install 'grpcio-tools>=1.68.0' packaging + python3 -m pip install 'grpcio-tools~=1.83.0' packaging python3 tools/grpc_code_gen.py .PHONY: install diff --git a/docs/en/setup/CLI.md b/docs/en/setup/CLI.md index adfeb870..0eb3e391 100644 --- a/docs/en/setup/CLI.md +++ b/docs/en/setup/CLI.md @@ -53,7 +53,7 @@ or `uwsgi --die-on-term --http 0.0.0.0:5000 --http-manage-expect --master --workers 3 --enable-threads --threads 3 --manage-script-name --mount /=main:app` -Please change it to (**the `-p` option starts one agent in each process, which is the correct behavior**): +Please change it to (**the `-p` option starts one agent in each worker process and none in the master, which is the correct behavior**): **Important:** if the call to uwsgi/gunicorn is prefixed with other commands, this approach will fail since agent currently looks for the command line input at index 0 for safety as an experimental feature. @@ -72,6 +72,13 @@ Note that `sw-python` also work with spawned subprocess (os.exec*/subprocess) as Additionally, `sw-python` started agent works well with `os.fork` when your application forks workers, as long as the `SW_AGENT_EXPERIMENTAL_FORK_SUPPORT` is turned on. (It will be automatically turned on when gunicorn is detected) +**Important**: with the default gRPC reporter, explicit `os.fork()` is NOT reliable on grpcio >= 1.80: the agent's +background reporters enter gRPC at any time (heartbeat, segment flush) independent of application requests, and a +fork while a channel is live is subject to open upstream races (grpc/grpc#43055, grpc/grpc#43062) that can silently +break reporting in either process. Applications that fork should use `SW_AGENT_PROTOCOL=http` (or `kafka`). +Gunicorn via `sw-python run -p` is NOT affected: there the agent creates its channels only after the fork, +which is gRPC's supported model. + ## Configuring the agent You would normally want to provide additional configurations other than the default ones. diff --git a/docs/en/setup/Configuration.md b/docs/en/setup/Configuration.md index 42362d91..bfefb890 100644 --- a/docs/en/setup/Configuration.md +++ b/docs/en/setup/Configuration.md @@ -38,9 +38,9 @@ export SW_AGENT_YourConfiguration=YourValue | agent_collector_heartbeat_period | SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD | | 30 | The agent will exchange heartbeat message with SkyWalking OAP backend every `period` seconds | | agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds | | agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | | | A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}` | -| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. This feature may not work when a precise combination of gRPC + Python 3.7 + subprocess (not fork) is used together. The agent will output a warning log when using on Python 3.7 for such a reason. | +| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. | | agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. | -| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. | +| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. Not compatible with pre-forking servers (`sw-python run -p`): the agent refuses to start under a Gunicorn master. | ### SW_PYTHON Auto Instrumentation CLI | Configuration | Environment Variable | Type | Default Value | Description | | :------------ | :------------ | :------------ | :------------ | :------------ | diff --git a/docs/en/setup/Plugins.md b/docs/en/setup/Plugins.md index c430fb85..0321bd60 100644 --- a/docs/en/setup/Plugins.md +++ b/docs/en/setup/Plugins.md @@ -51,7 +51,7 @@ or a limitation of SkyWalking auto-instrumentation (welcome to contribute!) | [urllib3](https://urllib3.readthedocs.io/en/latest/) | Python >=3.12 - NOT SUPPORTED YET; Python >=3.10 - ['1.26', '1.25']; | `sw_urllib3` | | [urllib3](https://urllib3.readthedocs.io/en/latest/) | Python >=3.12 - ['2.3', '2.0']; | `sw_urllib3_v2` | | [urllib_request](https://docs.python.org/3/library/urllib.request.html) | Python >=3.7 - ['*']; | `sw_urllib_request` | -| [websockets](https://websockets.readthedocs.io) | Python >=3.7 - ['10.3', '10.4']; | `sw_websockets` | +| [websockets](https://websockets.readthedocs.io) | Python >=3.11 - ['10.3', '10.4', '13.1', '17.0.1']; Python >=3.7 - ['10.3', '10.4', '13.1']; | `sw_websockets` | ### Notes - The celery server running with "celery -A ..." should be run with the HTTP protocol as it uses multiprocessing by default which is not compatible with the gRPC protocol implementation @@ -60,6 +60,8 @@ in SkyWalking currently. Celery clients can use whatever protocol they want. Hug is believed to be abandoned project, use this plugin with a bit more caution. Instead of Hug, plugin test should move to test actual Falcon. - Falcon 3.x/4.x plugin. For legacy hug-based instrumentation, see sw_falcon. +- The agent package itself depends on grpcio >= 1.83, which is therefore the +effective minimum version of the instrumented library as well. - The Neo4j plugin integrates neo4j python driver 5.x.x versions which support both Neo4j 5 and 4.4 DBMS. - Sanic 21.9+ plugin using signal listeners. @@ -68,7 +70,9 @@ Note: Sanic's touchup system recompiles handle_request at startup, so we use signal listeners instead of monkey-patching handle_request. - urllib3 1.x plugin. For urllib3 2.x, see sw_urllib3_v2. - urllib3 2.x plugin. For urllib3 1.x, see sw_urllib3. -- The websocket instrumentation only traces client side connection handshake, +- Both the legacy (websockets.legacy, websockets <= 13) and the new asyncio +(websockets.asyncio, websockets >= 13) client implementations are instrumented. +The websocket instrumentation only traces client side connection handshake, the actual message exchange (send/recv) is not traced since injecting headers to socket message body is the only way to propagate the trace context, which requires customization of message structure and extreme care. (Feel free to add this feature by instrumenting the send/recv methods commented out in the code diff --git a/docs/en/setup/advanced/AsyncEnhancement.md b/docs/en/setup/advanced/AsyncEnhancement.md index 6f220784..09ba04c2 100644 --- a/docs/en/setup/advanced/AsyncEnhancement.md +++ b/docs/en/setup/advanced/AsyncEnhancement.md @@ -6,6 +6,10 @@ Since `1.1.0`, the Python agent supports asynchronous reporting of ALL telemetry export SW_AGENT_ASYNCIO_ENHANCEMENT=true ``` +> Limitation: this option is incompatible with pre-forking servers via `sw-python run -p` (e.g. Gunicorn) — +> the asyncio agent has no fork support, so the agent refuses to start and the application runs without +> observability. See the [Gunicorn FAQ](../faq/How-to-use-with-gunicorn.md). + ## Why we need this feature Before version `1.1.0`, SkyWalking Python agent had only an implementation with the Threading module to provide data reporters. Yet with the growth of the Python agent, it is now fully capable and requires more resources than when only tracing was supported (we start many threads and gRPC itself creates even more threads when streaming). diff --git a/docs/en/setup/faq/How-to-use-with-gunicorn.md b/docs/en/setup/faq/How-to-use-with-gunicorn.md index e65a5b7d..3c1a9fc9 100644 --- a/docs/en/setup/faq/How-to-use-with-gunicorn.md +++ b/docs/en/setup/faq/How-to-use-with-gunicorn.md @@ -14,7 +14,7 @@ serves requests. > Note: Python 3.10+ is required. Earlier versions (3.7-3.9) are no longer supported. -**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and all Gunicorn workers and master will get their own working agent. +**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and every Gunicorn worker will get its own working agent (the master is instrumented only and runs no agent). **Important:** if the call to gunicorn is prefixed with other commands, this approach will fail since agent currently looks for the command line input at index 0 for safety as an experimental feature. @@ -29,17 +29,34 @@ sw-python run -p gunicorn gunicorn_consumer_prefork:app --workers 2 --worker-cla By specifying the -p or --prefork option in sw-python CLI, the `agent_experimental_fork_support` agent option will be turned on automatically. Startup flow: -sw-python -> gunicorn -> master process (agent starts) -> fork -> worker process (agent restarts due to os.register_at_fork) +sw-python -> gunicorn -> master process (instrumentation only) -> fork -> worker process (full agent starts due to os.register_at_fork) -The master process will get its own agent, although it won't report any trace, since obviously it doesn't take requests, -it still reports metrics that is useful for debugging +The master process does not run a full agent: it only installs instrumentation, and the reporters plus the +gRPC channel are created in each forked worker. Therefore the master does not appear as a service instance +(it takes no requests anyway). A gRPC channel created before fork() is unsafe with grpcio >= 1.80, see +[apache/skywalking#13958](https://github.com/apache/skywalking/issues/13958). > A runnable example can be found in the demo folder of skywalking-python GitHub repository +### Known issue with agent <= 1.2.0 and grpcio >= 1.80 + +Agent versions up to 1.2.0 started a full agent (including a gRPC channel) in the Gunicorn master before forking. +With grpcio >= 1.80 this produces continuous `Kick Failure (eventfd_write: Bad file descriptor)` errors +and can silently hang workers. Tracing generally keeps working — the errors come from the gRPC client polling +engine and are unrelated to the OAP version. Workarounds on old agents: pin `grpcio<1.80` or use `SW_AGENT_PROTOCOL=http`. +Fixed agent versions require `grpcio >= 1.83` and never create a gRPC channel in the master. + +### Incompatible with the asyncio enhancement + +`SW_AGENT_ASYNCIO_ENHANCEMENT=true` is incompatible with `sw-python run -p gunicorn`: the asyncio agent has no +fork support, so the agent refuses to start (an error is logged) and the application serves WITHOUT observability. +Remove the asyncio enhancement option, or run Gunicorn without `-p`. + ## Manual Approach (only use when sw-python doesn't work) -**Limitation**: Using normal postfork hook will not add observability to the master process, you could also define a prefork hook to -start an agent in the master process, with a instance name like `instance-name-master()` +**Limitation**: Using normal postfork hook will not add observability to the master process. +Do NOT start an agent in the master process (e.g. from a prefork hook): a gRPC channel created before +fork() is unsafe with grpcio >= 1.80 and can deadlock workers. The following is just an example, since Gunicorn's automatic injection approach is likely to work in many situations. diff --git a/docs/en/setup/faq/How-to-use-with-uwsgi.md b/docs/en/setup/faq/How-to-use-with-uwsgi.md index 46878655..69c83d7a 100644 --- a/docs/en/setup/faq/How-to-use-with-uwsgi.md +++ b/docs/en/setup/faq/How-to-use-with-uwsgi.md @@ -13,8 +13,8 @@ Some of the original discussion can be found here: > You can always fall back to the manual approach. > (although it's also possible to pass postfork hook without changing code, which is essentially how sw-python is implemented) -> Limitation: regardless of the approach used, uWSGI master process cannot be safely monitored. Since it doesn't take any requests, it is generally acceptable. -> Alternatively, you could switch to Gunicorn, where its master process can be monitored properly along with all child workers. +> Limitation: regardless of the approach used, the uWSGI master process is not monitored. Since it doesn't take any requests, it is generally acceptable. +> Gunicorn behaves the same way — only worker processes are monitored, which is the fork-safe design. **Important**: The `--enable-threads` and `--master` option must be given to allow the usage of post_fork hooks and threading in workers. In the `sw-python` CLI, these two options will be automatically injected for you in addition to the post_fork hook. diff --git a/poetry.lock b/poetry.lock index 88a59c2c..9876a839 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1628,155 +1628,135 @@ test = ["objgraph", "psutil", "setuptools"] [[package]] name = "grpcio" -version = "1.80.0" +version = "1.83.0" description = "HTTP/2-based RPC framework" optional = false -python-versions = ">=3.9" +python-versions = ">=3.10" groups = ["main", "plugins"] files = [ - {file = "grpcio-1.80.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:886457a7768e408cdce226ad1ca67d2958917d306523a0e21e1a2fdaa75c9c9c"}, - {file = "grpcio-1.80.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:7b641fc3f1dc647bfd80bd713addc68f6d145956f64677e56d9ebafc0bd72388"}, - {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:33eb763f18f006dc7fee1e69831d38d23f5eccd15b2e0f92a13ee1d9242e5e02"}, - {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:52d143637e3872633fc7dd7c3c6a1c84e396b359f3a72e215f8bf69fd82084fc"}, - {file = "grpcio-1.80.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c51bf8ac4575af2e0678bccfb07e47321fc7acb5049b4482832c5c195e04e13a"}, - {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:50a9871536d71c4fba24ee856abc03a87764570f0c457dd8db0b4018f379fed9"}, - {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:a72d84ad0514db063e21887fbacd1fd7acb4d494a564cae22227cd45c7fbf199"}, - {file = "grpcio-1.80.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f7691a6788ad9196872f95716df5bc643ebba13c97140b7a5ee5c8e75d1dea81"}, - {file = "grpcio-1.80.0-cp310-cp310-win32.whl", hash = "sha256:46c2390b59d67f84e882694d489f5b45707c657832d7934859ceb8c33f467069"}, - {file = "grpcio-1.80.0-cp310-cp310-win_amd64.whl", hash = "sha256:dc053420fc75749c961e2a4c906398d7c15725d36ccc04ae6d16093167223b58"}, - {file = "grpcio-1.80.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:dfab85db094068ff42e2a3563f60ab3dddcc9d6488a35abf0132daec13209c8a"}, - {file = "grpcio-1.80.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:5c07e82e822e1161354e32da2662f741a4944ea955f9f580ec8fb409dd6f6060"}, - {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ba0915d51fd4ced2db5ff719f84e270afe0e2d4c45a7bdb1e8d036e4502928c2"}, - {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:3cb8130ba457d2aa09fa6b7c3ed6b6e4e6a2685fce63cb803d479576c4d80e21"}, - {file = "grpcio-1.80.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:09e5e478b3d14afd23f12e49e8b44c8684ac3c5f08561c43a5b9691c54d136ab"}, - {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:00168469238b022500e486c1c33916acf2f2a9b2c022202cf8a1885d2e3073c1"}, - {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:8502122a3cc1714038e39a0b071acb1207ca7844208d5ea0d091317555ee7106"}, - {file = "grpcio-1.80.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:ce1794f4ea6cc3ca29463f42d665c32ba1b964b48958a66497917fe9069f26e6"}, - {file = "grpcio-1.80.0-cp311-cp311-win32.whl", hash = "sha256:51b4a7189b0bef2aa30adce3c78f09c83526cf3dddb24c6a96555e3b97340440"}, - {file = "grpcio-1.80.0-cp311-cp311-win_amd64.whl", hash = "sha256:02e64bb0bb2da14d947a49e6f120a75e947250aebe65f9629b62bb1f5c14e6e9"}, - {file = "grpcio-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:c624cc9f1008361014378c9d776de7182b11fe8b2e5a81bc69f23a295f2a1ad0"}, - {file = "grpcio-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:f49eddcac43c3bf350c0385366a58f36bed8cc2c0ec35ef7b74b49e56552c0c2"}, - {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d334591df610ab94714048e0d5b4f3dd5ad1bee74dfec11eee344220077a79de"}, - {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0cb517eb1d0d0aaf1d87af7cc5b801d686557c1d88b2619f5e31fab3c2315921"}, - {file = "grpcio-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4e78c4ac0d97dc2e569b2f4bcbbb447491167cb358d1a389fc4af71ab6f70411"}, - {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:2ed770b4c06984f3b47eb0517b1c69ad0b84ef3f40128f51448433be904634cd"}, - {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:256507e2f524092f1473071a05e65a5b10d84b82e3ff24c5b571513cfaa61e2f"}, - {file = "grpcio-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9a6284a5d907c37db53350645567c522be314bac859a64a7a5ca63b77bb7958f"}, - {file = "grpcio-1.80.0-cp312-cp312-win32.whl", hash = "sha256:c71309cfce2f22be26aa4a847357c502db6c621f1a49825ae98aa0907595b193"}, - {file = "grpcio-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:9fe648599c0e37594c4809d81a9e77bd138cc82eb8baa71b6a86af65426723ff"}, - {file = "grpcio-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e9e408fc016dffd20661f0126c53d8a31c2821b5c13c5d67a0f5ed5de93319ad"}, - {file = "grpcio-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:92d787312e613754d4d8b9ca6d3297e69994a7912a32fa38c4c4e01c272974b0"}, - {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac393b58aa16991a2f1144ec578084d544038c12242da3a215966b512904d0f"}, - {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:68e5851ac4b9afe07e7f84483803ad167852570d65326b34d54ca560bfa53fb6"}, - {file = "grpcio-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:873ff5d17d68992ef6605330127425d2fc4e77e612fa3c3e0ed4e668685e3140"}, - {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:2bea16af2750fd0a899bf1abd9022244418b55d1f37da2202249ba4ba673838d"}, - {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:ba0db34f7e1d803a878284cd70e4c63cb6ae2510ba51937bf8f45ba997cefcf7"}, - {file = "grpcio-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:8eb613f02d34721f1acf3626dfdb3545bd3c8505b0e52bf8b5710a28d02e8aa7"}, - {file = "grpcio-1.80.0-cp313-cp313-win32.whl", hash = "sha256:93b6f823810720912fd131f561f91f5fed0fda372b6b7028a2681b8194d5d294"}, - {file = "grpcio-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:e172cf795a3ba5246d3529e4d34c53db70e888fa582a8ffebd2e6e48bc0cba50"}, - {file = "grpcio-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:3d4147a97c8344d065d01bbf8b6acec2cf86fb0400d40696c8bdad34a64ffc0e"}, - {file = "grpcio-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d8e11f167935b3eb089ac9038e1a063e6d7dbe995c0bb4a661e614583352e76f"}, - {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f14b618fc30de822681ee986cfdcc2d9327229dc4c98aed16896761cacd468b9"}, - {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4ed39fbdcf9b87370f6e8df4e39ca7b38b3e5e9d1b0013c7b6be9639d6578d14"}, - {file = "grpcio-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2dcc70e9f0ba987526e8e8603a610fb4f460e42899e74e7a518bf3c68fe1bf05"}, - {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:448c884b668b868562b1bda833c5fce6272d26e1926ec46747cda05741d302c1"}, - {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a1dc80fe55685b4a543555e6eef975303b36c8db1023b1599b094b92aa77965f"}, - {file = "grpcio-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:31b9ac4ad1aa28ffee5503821fafd09e4da0a261ce1c1281c6c8da0423c83b6e"}, - {file = "grpcio-1.80.0-cp314-cp314-win32.whl", hash = "sha256:367ce30ba67d05e0592470428f0ec1c31714cab9ef19b8f2e37be1f4c7d32fae"}, - {file = "grpcio-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:3b01e1f5464c583d2f567b2e46ff0d516ef979978f72091fd81f5ab7fa6e2e7f"}, - {file = "grpcio-1.80.0-cp39-cp39-linux_armv7l.whl", hash = "sha256:aacdfb4ed3eb919ca997504d27e03d5dba403c85130b8ed450308590a738f7a4"}, - {file = "grpcio-1.80.0-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:a361c20ec1ccd3c3953d20fb6d7b4125093bdd10dff44c5e2bbb39e58917cedc"}, - {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:43168871f170d1e4ed16ae03d10cd21efa29f190e710a624cee7e5ae07da6f4f"}, - {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:1b97cd29a8eda100b559b455331c487a80915b6ea6bd91cf3e89836c4ee8d957"}, - {file = "grpcio-1.80.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bac1d573dfa84ce59a5547073e28fa7326d53352adda6912e362da0b917fcef4"}, - {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:4560cf0e86514595dbbd330cd65b7afad4b5c4b8c4905c041cfffa138d45e6fd"}, - {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:ec0a592e926071b4abad50c1495cd0d0d513324b3ff5e7267067c33ba27506e4"}, - {file = "grpcio-1.80.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:deb10a1528473c11f72a0939eed36d83e847d7cbb63e8cc5611fb7a912d38614"}, - {file = "grpcio-1.80.0-cp39-cp39-win32.whl", hash = "sha256:627fb7312171cdc52828bd6fac8d7028ff2a64b89f1957b6f3416caa2218d141"}, - {file = "grpcio-1.80.0-cp39-cp39-win_amd64.whl", hash = "sha256:05d55e1798756282cddd52d56c896b3e7d673e3a8798c2f1cd05ba249a3bb4de"}, - {file = "grpcio-1.80.0.tar.gz", hash = "sha256:29aca15edd0688c22ba01d7cc01cb000d72b2033f4a3c72a81a19b56fd143257"}, + {file = "grpcio-1.83.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:fba099b716e73512d61b97f71ea3c31a72abb36904036e316bf4dd148ca8dcc8"}, + {file = "grpcio-1.83.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:6755ed67cc3e454d51ae9f6e1915b80d3942fa4de956ef48dacd45ab7f40b727"}, + {file = "grpcio-1.83.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5882c1a721b50ce0123ee5e839e1ab059ad72a7ade76cdf2d5bd833b56791acf"}, + {file = "grpcio-1.83.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:4e3eedfc92b6b9f2960115e7e620cf0cbf80bb7849a51ce3820dc54dfd88b6b9"}, + {file = "grpcio-1.83.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4fcaa7c45c45b4a89e2867d1f1785d9481a788399d915e341ed2eb49aeef9dd4"}, + {file = "grpcio-1.83.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:6b6c666a1d5613ff360c9e90f44665e3a88b25a815209ddbc0917eec281931cb"}, + {file = "grpcio-1.83.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:6be5c807b717be3dd649446f021301fd7907e376318675d2147823071034112a"}, + {file = "grpcio-1.83.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:c834e86d8fd2f03d7e4db49a027f7c5b89c5b88eed305543a5295bd6fee61e40"}, + {file = "grpcio-1.83.0-cp310-cp310-win32.whl", hash = "sha256:35a5b1c192496b6c25956eebfa963468935612206fd2543ac3ce981e6a5e0f03"}, + {file = "grpcio-1.83.0-cp310-cp310-win_amd64.whl", hash = "sha256:8f6c395e493d20c39b29392ca200e9aaeb78d0bc2f04db0c0a7da7ddc939aa57"}, + {file = "grpcio-1.83.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:8ff0b8767ddd62704e0d9571c1890af08d84a3a689ebba1807e62519d0b3277f"}, + {file = "grpcio-1.83.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:4772402f43517b4824980be4b3b2274a81eec0004a70009473c31b340d43e223"}, + {file = "grpcio-1.83.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f4cee5fc86e84a0cf7ad1574b454c3320e087c07f55b7df5dc0ac6a873fb90c0"}, + {file = "grpcio-1.83.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:f5e822a7e7d03282f6ad225e710493c48b9057a353358344a5f7c42b2b37618d"}, + {file = "grpcio-1.83.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f5f410d7c2903eabb34789dfd6342eef04af1ad459943936b7e09a9f5bd417b9"}, + {file = "grpcio-1.83.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ee94a4016fdf8699fb1fd8a38652475ff677f1c72074cee44deeeb9a7e95e745"}, + {file = "grpcio-1.83.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:c6444666317338e903093c7c756e6cc88eee59f798cb8dd41e87725bf54e1617"}, + {file = "grpcio-1.83.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:aa074041231f03959cb097dd5517b0677b8ea49215bae01d5710a7b69dd59969"}, + {file = "grpcio-1.83.0-cp311-cp311-win32.whl", hash = "sha256:cb056f6e171c42639a50460b2929c82241fda51f71cf3dcdd68090fe45095a45"}, + {file = "grpcio-1.83.0-cp311-cp311-win_amd64.whl", hash = "sha256:7416952ca770477990257206276999056f8316d79196f2f25942393e58a20b49"}, + {file = "grpcio-1.83.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:28f6c35ac8fcf10e4594f138e468f194360089dde40d126a7033e863fc479930"}, + {file = "grpcio-1.83.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:33898e6a28e4ae598f1577cb1c4fec2a15c033d0ec52b9b45a09610dd045b9da"}, + {file = "grpcio-1.83.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:6fb8a1dd0c6f0f931e69e9d0dc6d1c406ed2a44fa963414eafba07b7fb685d16"}, + {file = "grpcio-1.83.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:2b5e75c34842cd9c1b95285ca395c6a569664b81e3ffa6b714125922942abaaf"}, + {file = "grpcio-1.83.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:aeb339838db07600481ef869507279b75326c75eac6d10f7afa62a0da1d2bcdd"}, + {file = "grpcio-1.83.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:f47d62808b4c0a97b78bff88a6d4ca283a2a492b9a04a87d814af95ca3b9c19c"}, + {file = "grpcio-1.83.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:62003babc444a606dcd1f009cd16391ce23669ae4ad6ec267a873da7937a69f5"}, + {file = "grpcio-1.83.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:1aa567f8c3f19850ffd5d2858c9a8ea7c80f0db6c01186b71eb31e923ec984f5"}, + {file = "grpcio-1.83.0-cp312-cp312-win32.whl", hash = "sha256:cb2906c61db4f9c64cc360054b5df70eeb81846228e9e56a4944bd415a63dadc"}, + {file = "grpcio-1.83.0-cp312-cp312-win_amd64.whl", hash = "sha256:1c699bbb20f143c8f2bff219de578aa2dc1f919399d67dc702b038b986ee62df"}, + {file = "grpcio-1.83.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:6662f3b1e07cc7493d437351860dc867bddc6a93c83ecf33bbfdaf0c217ab2d0"}, + {file = "grpcio-1.83.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:74fe6f9e8a35c7dbf32255ee154d15e3e5338a81ed39173d079d594d2e544cd1"}, + {file = "grpcio-1.83.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:10b3fa0475eb572c9a81a6fe37fa16a9c500c0c91cfc148cac15692b7e3c2867"}, + {file = "grpcio-1.83.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:5f20a988480b0f28207f057f7f7ae1313393c3cef0adcfeae8248f9947eaf881"}, + {file = "grpcio-1.83.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7bd82671b39065ba18cd536e9cd45b27ff649053f81ddd2c6a966d595067080f"}, + {file = "grpcio-1.83.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:bc60215b5cb9fc8ca72942c498b551ac2305bd08f6ef8d4e3f0d21b64fbecd61"}, + {file = "grpcio-1.83.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:f1c3e5689d4b90987b1d72022bcfe866a9a3dc66197484cf856d96b6150e7f45"}, + {file = "grpcio-1.83.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a21cb4eeeba124443f399be2e8b624943cde864dcbe588cb42e5c483a52a906c"}, + {file = "grpcio-1.83.0-cp313-cp313-win32.whl", hash = "sha256:8fe04f1050a59f875601eb55d42b4f66946fe89817f967e34db1462ccd07dadf"}, + {file = "grpcio-1.83.0-cp313-cp313-win_amd64.whl", hash = "sha256:6e01ecd9d8ef280abe1365138a4dc318f9a5287f4cb1b41d07816f796653f735"}, + {file = "grpcio-1.83.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:3f351629f6ae16ecc0ec3553e586a6763ffd9f6114044286d0cbec3e09241bfa"}, + {file = "grpcio-1.83.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d05ff664100d429335b93c91b8b34ddf9e94a112205e7fa06dede309e44a4e4c"}, + {file = "grpcio-1.83.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7936f2a56cf04f6514705c0fedf400971de01b6aa1719327e4718f410a765e2b"}, + {file = "grpcio-1.83.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b0a0be840e51b6b7ee9df9269770faf77bdf4b771053c257c21d12bad607714c"}, + {file = "grpcio-1.83.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:009667eaf3dcd5224c713589cdc98e7ca4ed0ff0b61132c6b276e930eb83a2df"}, + {file = "grpcio-1.83.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:bb669918fd88936b15599caff4160a77ab74bdeb25f2231f6e45b61282d6107b"}, + {file = "grpcio-1.83.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:c19b454d3d3f28db81f2c7c4dbaee96e7f6fd149721733ffe79d6bc530f17404"}, + {file = "grpcio-1.83.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:61007cd08640abc5c54547ee32505474c482cd733a53cb87551ea81faa6350af"}, + {file = "grpcio-1.83.0-cp314-cp314-win32.whl", hash = "sha256:32e11c37f5285b0c6fa3042c05fe06903696689749833fc64e67dec71b9bbe33"}, + {file = "grpcio-1.83.0-cp314-cp314-win_amd64.whl", hash = "sha256:2bb48cb5e6dd005ca12b89ce4b6ac0b48ff3112c747542ee7986ef611a8ca6d9"}, + {file = "grpcio-1.83.0.tar.gz", hash = "sha256:7674587248fbbb2ac6e4eecf83a8a0f3d91a928f941de571acfd3a2f007fbc24"}, ] [package.dependencies] typing-extensions = ">=4.12,<5.0" [package.extras] -protobuf = ["grpcio-tools (>=1.80.0)"] +protobuf = ["grpcio-tools (>=1.83.0)"] [[package]] name = "grpcio-tools" -version = "1.80.0" +version = "1.83.0" description = "Protobuf code generator for gRPC" optional = false -python-versions = ">=3.9" +python-versions = ">=3.10" groups = ["main"] files = [ - {file = "grpcio_tools-1.80.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:727477b9afa4b53f5ec70cafb41c3965d893835e0d4ea9b542fe3d0d005602bf"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:85fe8d15f146c62cb76f38d963e256392d287442b9232717d30ae9e3bbda9bc3"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:95f0fffb5ca00519f3b602f938169b4dfa04b165e03258323965a9dfe8cc4d80"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:7a0106af212748823a6ebd8ffbd9043414216f47cae3835f3187de0a62c415d3"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:31fd01a4038b5dfc4ec79504a17061344f670f851833411717fef66920f13cd7"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:57da9e19607fac4a01c48ead333c0dd15d91ed38794dce1194eda308f73e2038"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:90968f751851abb8b145593609800fa70c837e1c93ba0792c480b1c8d8bc29ef"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:b69dc5d6376ab43406304d1e2fc61ccf960b287d4325d77c3d45448c37a9d2da"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-win32.whl", hash = "sha256:3e8dcfebe34cb54df095de3d5871a4562a85a29f26d0f8bb41ee2c3dcfb11c3c"}, - {file = "grpcio_tools-1.80.0-cp310-cp310-win_amd64.whl", hash = "sha256:fc622ed4ca400695f41c9eae3266276c6ba007e4c28164ce53b44e7ccc5e492b"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:1c43e5c768578fe0c6de3dbfaabe64af642951e1aa05c487cacedda63fa6c6c4"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:a225348456575f3ac7851d8e23163195e76d2a905ee340cf73f33da62fba08aa"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a9396f02820d3f51c368c2c9dee15c55c77636c91be48a4d5c702e98d6fe0fdc"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:797c08460cae16b402326eac329aec720dccf45c9f9279b95a352792eb53cf0f"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1872a867eb6217de19edb70a4ce4a374ced9d94293533dfd42fa649713f55bf4"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:db122ba5ee357e3bb14e8944d69bbebcbdae91d5eace29ed4df3edc53cbc6528"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:ddefd48c227e6f4d640fe576fac5fb2c4a8898196f513604c8ec7671b3b3d421"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:970ec058fa469dd6dae6ebc687501c5da670d95dead75f62f5b0933dce2c9794"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-win32.whl", hash = "sha256:526b4402d47a0e9b31cd6087e42b7674784617916cc73c764e0bc35ed41b4ee5"}, - {file = "grpcio_tools-1.80.0-cp311-cp311-win_amd64.whl", hash = "sha256:ee101ecda7231770f6a5da1024a9a6ed587a7785f8fe23ab8283f4a1acb3ffe6"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:d19d5a8244311947b96f749c417b32d144641c6953f1164824579e1f0a51d040"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:fb599a3dc89ed1bb24489a2724b2f6dd4cddbbf0f7bdd69c073477bab0dc7554"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:623ee31fc2ff7df9a987b4f3d139c30af17ce46a861ae0e25fb8c112daa32dd8"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b46570a68378539ee2b75a5a43202561f8d753c832798b1047099e3c551cf5d6"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:51caf99c28999e7e0f97e9cea190c1405b7681a57bb2e0631205accd92b43fa4"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:cdaa1c9aa8d3a87891a96700cadd29beec214711d6522818d207277f6452567c"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:3399b5fd7b59bcffd59c6b9975a969d9f37a3c87f3e3d63c3a09c147907acb0d"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:9c6abc08d3485b2aac99bb58afcd31dc6cd4316ce36cf263ff09cb6df15f287f"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-win32.whl", hash = "sha256:18c51e07652ac7386fcdbd11866f8d55a795de073337c12447b5805575339f74"}, - {file = "grpcio_tools-1.80.0-cp312-cp312-win_amd64.whl", hash = "sha256:ac6fdd42d5bb18f0d903a067e2825be172deff70cf197164b6f65676cb506c9b"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:e7046837859bbfd10b01786056145480155c16b222c9e209215b68d3be13060e"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:a447f28958a8fe84ff0d9d3d9473868feb27ee4a9c9c805e66f5b670121cec59"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:75f00450e08fe648ad8a1eeb25bc52219679d54cdd02f04dfdddc747309d83f6"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:3db830eaff1f2c2797328f2fa86c9dcdbd7d81af573a68db81e27afa2182a611"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7982b5fe42f012686b667dda12916884de95c4b1c65ff64371fb7232a1474b23"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6451b3f4eb52d12c7f32d04bf8e0185f80521f3f088ad04b8d222b3a4819c71e"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:258bc30654a9a2236be4ca8e2ad443e2ac6db7c8cc20454d34cce60265922726"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:865a2b8e6334c838976ab02a322cbd55c863d2eaf3c1e1a0255883c63996772a"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-win32.whl", hash = "sha256:f760ac1722f33e774814c37b6aa0444143f612e85088ead7447a0e9cd306a1f1"}, - {file = "grpcio_tools-1.80.0-cp313-cp313-win_amd64.whl", hash = "sha256:7843b9ac6ff8ca508424d0dd968bd9a1a4559967e4a290f26be5bd6f04af2234"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:12f950470449dbeec78317dbc090add7a00eb6ca812af7b0538ab7441e0a42c3"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:d3f9a376a29c9adf62bb56f7ff5bc81eb4abeaf53d1e7dde5015564832901a51"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1ba1ffbf2cff71533615e2c5a138ed5569611eec9ae7f9c67b8898e127b54ac0"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:13f60f8d9397c514c6745a967d22b5c8c698347e88deebca1ff2e1b94555e450"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:88d77bad5dd3cd5e6f952c4ecdd0ee33e0c02ecfc2e4b0cbee3391ac19e0a431"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:017945c3e98a4ed1c4e21399781b4137fc08dfc1f802c8ace2e64ef52d32b142"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a33e265d4db803495007a6c623eafb0f6b9bb123ff4a0af89e44567dad809b88"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6c129da370c5f85f569be2e545317dda786a60dd51d7deea29b03b0c05f6aac3"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-win32.whl", hash = "sha256:25742de5958ae4325249a37e724e7c0e5120f8e302a24a977ebd1737b48a5e97"}, - {file = "grpcio_tools-1.80.0-cp314-cp314-win_amd64.whl", hash = "sha256:bbf8eeef78fda1966f732f79c1c802fadd5cfd203d845d2af4d314d18569069c"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-linux_armv7l.whl", hash = "sha256:4c615f3b5c6f7e8e0b06f60e3fa9cebf88372296255268db9e9a23e72bb698bf"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:3954b5d07ac19d752ee70c7d63ee0ba0f9a840c33e042decf355f04b1ff41d93"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9a765334d3080d147ecf7b8ab04900e56108f6457dde0a3ba7f68c270f9d6efc"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:c18def9c38d36767946932d2cc7baf39dcae5fea5a02843ea34399871f981a09"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4534022e4d5dd3d7d2183ff5846bf950cbaf889af0ea5290f94212001f7cad84"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:1816e8e512402ed0b3fe4a336aaff14f9cb42455aa88fa86f754d53973668bd6"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-musllinux_1_2_i686.whl", hash = "sha256:e3b6d09f87eb87a8cab58f7e99cae3551467f51b2bcbab17a2fe931e94e7efef"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:6c6ce08167fd77fa057dc44fea8501c66d108eeef536073dba55c8fd3684c7a9"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-win32.whl", hash = "sha256:5de4eb2d08bddeee28265c10369934b2d23b8c4acc39d419ee6a58afe34d754f"}, - {file = "grpcio_tools-1.80.0-cp39-cp39-win_amd64.whl", hash = "sha256:6a35a73042dc4bbcdd7aafc141ee9966c8ae97bf4b9f0f49e10e3e1aa54139ac"}, - {file = "grpcio_tools-1.80.0.tar.gz", hash = "sha256:26052b19c6ce0dcf52d1024496aea3e2bdfa864159f06dc7b97b22d041a94b26"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:f281cb706999676bb841bcd57129a69091b9286236c89d6114c752ebf6cd5a1b"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:3647c6adae9528dd56183061371151e3a96f71c299dc69c540318c3af2233a88"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b6346c688d25bcf264e55f0c5f48ae825f7a2906ab969ed3b4a93df53e48bf07"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:281d5d056d7ba839f4fff9b63f9ad239fe3353fe10e28e782251bdae6ba68306"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:35af1be2fe409abf9817ec43c34aab8a99189b15530eb78f3a94a6b1266d8b12"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:7f5f5b6e1a91422069601fbf94f7fc970a7647fa69d2bc9f59e38913523117af"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:cad9333c0d5afcc2ffb26bebc5e8f097e3218b964758c3c65609dbcb77ec2aa7"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:2106b29b9dae5068acab7ee2f6b104d3054b4f31289f105b4afe1fcfbf1966c4"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-win32.whl", hash = "sha256:a47e674e6afac5d73ee3a87d57ed53f7b79cc01f44d602fe6ee90919aa171583"}, + {file = "grpcio_tools-1.83.0-cp310-cp310-win_amd64.whl", hash = "sha256:88fc53ee3ce28d3ea2fe8fe1d3ed57854d0d25d6dac18e74c1f24e0a377bb509"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:72471a4a46909f1d798836c0a0aa2f568e10f6404585d7b22ac7330dd6a7bc74"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:7da547dd1e0b1fe3d6d5677e9c1848969ecdbd51a92c342cf82d885c5935de7d"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:33350bd94c4913f8eaf6ca79ce69cc673bb46ca5f800e6474b1d6899ed321dad"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0f27da6dc58c910f31dcee4fcbfd05659c10c14b9e132f4f17da48d867e75eb4"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:b3a5000e8540efb80f74d9c760f36ed9408294d76edb0fa4b87fd287b0a8a258"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:f710032264ed114c9d3b52f4c5cf71d68ccf70f25c4cb5776fe60388374bc01b"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:ad00de87334154901e9af50a7f3f95261a0133502c6c0cea1f4e6107245154c3"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:de2b0c645363f7c4b005f145e06e790870dfb21bb7e162dec6fee2f054de9291"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-win32.whl", hash = "sha256:6d1a1c9e62689d04b63b227558b759a55dce8fb81a3934d3b5f95d1ee26a2b45"}, + {file = "grpcio_tools-1.83.0-cp311-cp311-win_amd64.whl", hash = "sha256:a6ac3cc2c2d77f869a96dfaf2b1315852878babddfe2dc49b9fd47afbf502865"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:bd93bbe18c4424805fd2e39854f75d76f80655175621254dc43cb45ec8e91e85"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:dc2d370563ee1ee6c1769e49df35ef3f6e75cea8f25acf4ac6e54b335e6f788f"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8350e236470700b02bc4ba7f27a8796559630170e6527dda41c0344fbe988e56"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:b1649f47c4675c1540ad2a77005f4d08392c06202686a0b1bb6b894f96cb75fe"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7d4edc6ba9fdca70bbf585ff6ab5971b8cd6140b4b316df66fd91d168bc1b617"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:762a9f8a4a4a39bda02feebed94efb8d778e0e5a82d0c8f786dce5ddcb950c7f"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:c6c469c928a183f1a99ab26e263fe307347ee7023fa623b55bc778846b2f51b9"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:7980b3ca9dd31c42468c5af8cec97037f83715ea6efbe1b936ecb9c6832ac0f5"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-win32.whl", hash = "sha256:fd2ff46917f566b3b63dae191d1b05ef2188fe51e756ef321cbdd707ab29dbfb"}, + {file = "grpcio_tools-1.83.0-cp312-cp312-win_amd64.whl", hash = "sha256:92d2343806b5c21162a57fbaad24fcd8d935ef530f95a0f706a4e2546fdc0662"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:3277cfbb7cbbd2d72921fbcd7aba6c8ab1c91a9ab27e8045ac0a0f2e0517cec9"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:1aa9617ce9c2bcbfb8f2fa08e6259e1b3cadaab0316e41f71496847af2f0a664"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ad37c786ea92825534466052f5f22f1f29983b1d00ca71ad43e256715a86bba3"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:c82216864d435ecf6f535798d03e9f6b9025a672a2e815715d629aba4ba70349"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:04284627655629387b63278591e5efd0aded28d3a08432fe8a8765e4daf2d5b2"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4fe627a5248d8e712f5ec3e019420050e61b10600ed241264aefb379a0f1b338"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1d5a9a9664d2f4bdda000652e7febf9ab4545a6391c8f05babd52ecb27d7e03e"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4bf0e421e1ab5f2cd638de44fc903aed3ba4a2fcb19b93c6f528ff0ec63e3a6a"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-win32.whl", hash = "sha256:7b1bd6db403b38addded54866187eba6f9ab9afadf72bb8d0515ed13f0b16c5c"}, + {file = "grpcio_tools-1.83.0-cp313-cp313-win_amd64.whl", hash = "sha256:d654c645af7cf608a30644bffb8d1ef6b14e8846482c3d0131d0dda91f6fb590"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:1ea047ff4bd2bb32fe5268042cb9c9e7bb054e932b52ad756642622f032ef656"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:7a8ac9cb3fbf7a5e4fe59f211e77b4fa4d51279c9f480e6ff98037cf56da1ad8"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a5fa95fb33a600d2491867a1048f47baa27a830eac01f475043b8ccf63a471eb"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:0ca67941524662e01adea91571bb79df1ac9b4b2641812ef8636e21945119bee"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e2a5816a5b6b06b42a6989f02944841c2a8b3daaa7f033dac9267f70078028ef"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f4a83f002895b11c4a862c366d77165825e0064aff14bf2c7453f59f66599b0e"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:1cfee967ae073bc064862971871229248965422f38556096aec76db19d8a8c79"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f7e82ee718ae09f879cb832e4517a56691de24383a2da673be184ad8b18e452f"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-win32.whl", hash = "sha256:846fd211ebb72f50d39d3874cc0d616c2b9bcb71db51121ca86af29eec013c74"}, + {file = "grpcio_tools-1.83.0-cp314-cp314-win_amd64.whl", hash = "sha256:b8c9686b0c19f70b63d8d6cfeff5ad3480bdedecd60f14711fe43950f5397253"}, + {file = "grpcio_tools-1.83.0.tar.gz", hash = "sha256:515907265d14fa9975d0c7723f95a9da01463d7ac607546a03f8741f86a1bb07"}, ] [package.dependencies] -grpcio = ">=1.80.0" -protobuf = ">=6.31.1,<7.0.0" +grpcio = ">=1.83.0" +protobuf = ">=7.35.1,<8.0.0" setuptools = ">=77.0.1" [[package]] @@ -2767,22 +2747,20 @@ markers = {main = "extra == \"all\" or extra == \"async\" or extra == \"asynchtt [[package]] name = "protobuf" -version = "6.33.6" +version = "7.35.1" description = "" optional = false -python-versions = ">=3.9" +python-versions = ">=3.10" groups = ["main"] files = [ - {file = "protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3"}, - {file = "protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326"}, - {file = "protobuf-6.33.6-cp39-abi3-macosx_10_9_universal2.whl", hash = "sha256:9720e6961b251bde64edfdab7d500725a2af5280f3f4c87e57c0208376aa8c3a"}, - {file = "protobuf-6.33.6-cp39-abi3-manylinux2014_aarch64.whl", hash = "sha256:e2afbae9b8e1825e3529f88d514754e094278bb95eadc0e199751cdd9a2e82a2"}, - {file = "protobuf-6.33.6-cp39-abi3-manylinux2014_s390x.whl", hash = "sha256:c96c37eec15086b79762ed265d59ab204dabc53056e3443e702d2681f4b39ce3"}, - {file = "protobuf-6.33.6-cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:e9db7e292e0ab79dd108d7f1a94fe31601ce1ee3f7b79e0692043423020b0593"}, - {file = "protobuf-6.33.6-cp39-cp39-win32.whl", hash = "sha256:bd56799fb262994b2c2faa1799693c95cc2e22c62f56fb43af311cae45d26f0e"}, - {file = "protobuf-6.33.6-cp39-cp39-win_amd64.whl", hash = "sha256:f443a394af5ed23672bc6c486be138628fbe5c651ccbc536873d7da23d1868cf"}, - {file = "protobuf-6.33.6-py3-none-any.whl", hash = "sha256:77179e006c476e69bf8e8ce866640091ec42e1beb80b213c3900006ecfba6901"}, - {file = "protobuf-6.33.6.tar.gz", hash = "sha256:a6768d25248312c297558af96a9f9c929e8c4cee0659cb07e780731095f38135"}, + {file = "protobuf-7.35.1-cp310-abi3-macosx_10_9_universal2.whl", hash = "sha256:24f857477359a85c0c235261b8ba905fd51b2562f4a64ca1df5473f29850cbf6"}, + {file = "protobuf-7.35.1-cp310-abi3-manylinux2014_aarch64.whl", hash = "sha256:11d6b0ec246892d85215b0a13ca6e0233cf5284b68f0ac02646427f4ff88a799"}, + {file = "protobuf-7.35.1-cp310-abi3-manylinux2014_s390x.whl", hash = "sha256:b73f9489a4b8b1c9cb1f8ed951c736392592edb24b9d6819f36d2e10b171d5b4"}, + {file = "protobuf-7.35.1-cp310-abi3-manylinux2014_x86_64.whl", hash = "sha256:74758715c53d7158fb76caf4f0cfdacc5329a4b1bb994f865d6cf302d413a1c4"}, + {file = "protobuf-7.35.1-cp310-abi3-win32.whl", hash = "sha256:353652e4efd0bca5b5fc2656abf8307ef351f0cf938c9eba09f0e09c20a25c30"}, + {file = "protobuf-7.35.1-cp310-abi3-win_amd64.whl", hash = "sha256:230a75ddfc2de4806e56696ce9640c1cdfdb6543b7cfce98d42a4c0a0e7bdb87"}, + {file = "protobuf-7.35.1-py3-none-any.whl", hash = "sha256:4bc97768d8fe4ad6743c8a19403e314511ed9f6d13205b687e52421c023ac1b9"}, + {file = "protobuf-7.35.1.tar.gz", hash = "sha256:ce115a26fe0c39a2c29973d914d327e516a6455464489fe3cd1e51a1b354f81a"}, ] [[package]] @@ -4863,4 +4841,4 @@ sync = ["kafka-python", "requests"] [metadata] lock-version = "2.1" python-versions = ">=3.10, <3.15" -content-hash = "991fc7d4d7ab6cdc03e354c10b6bf6f22452229a68a4f07c78d71fc3b0a89eff" +content-hash = "6627e8707379b8ce1c786dbb516cba83a18acf431eca52e5e17a9969113fd2ec" diff --git a/pyproject.toml b/pyproject.toml index cb5c66e2..8a8d473a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -71,8 +71,8 @@ sw-python = 'skywalking.bootstrap.cli.sw_python:start' [tool.poetry.dependencies] python = ">=3.10, <3.15" -grpcio = '*' -grpcio-tools = '*' +grpcio = '>=1.83' +grpcio-tools = '>=1.83' packaging = '*' wrapt = '>=1.14' psutil = '*' diff --git a/skywalking/agent/__init__.py b/skywalking/agent/__init__.py index b93f620f..4e208eb3 100644 --- a/skywalking/agent/__init__.py +++ b/skywalking/agent/__init__.py @@ -112,6 +112,10 @@ def __init__(self): self.started_pid = None self.__protocol: Optional[Protocol] = None self._finished: Optional[Event] = None + # True only after __bootstrap() in the current process; stays False in a pre-fork master + self.__reporting: bool = False + self.__at_fork_registered: bool = False + self.__fini_registered: bool = False def __bootstrap(self): # when forking, already instrumented modules must not be instrumented again @@ -133,6 +137,8 @@ def __bootstrap(self): # Start reporter threads and register queues self.__init_threading() + self.__reporting = True + def __init_queues(self) -> None: """ This method initializes all the queues for the agent and reporters. @@ -252,8 +258,7 @@ def start(self) -> None: # since 3.6 is EOL, we will not officially support it logger.warning('SkyWalking Python agent does not support Python 3.6 and below, ' 'please upgrade to Python 3.7 or above.') - # Below is required for grpcio to work with fork() - # https://github.com/grpc/grpc/blob/master/doc/fork_support.md + # Required for grpcio to work with fork(), see grpc/grpc doc/fork_support.md. if config.agent_protocol == 'grpc' and config.agent_experimental_fork_support: python_major_version: tuple = sys.version_info[:2] if python_major_version == (3, 7): @@ -264,22 +269,21 @@ def start(self) -> None: 'or use HTTP/Kafka protocol, or disable experimental fork support ' 'if your application did not start successfully.') - os.environ['GRPC_ENABLE_FORK_SUPPORT'] = 'true' - os.environ['GRPC_POLL_STRATEGY'] = 'poll' + # GRPC_POLL_STRATEGY=poll must NOT be set: the legacy poller is broken across + # fork() on grpcio >= 1.80 (the agent requires >= 1.83), + # see https://github.com/apache/skywalking/issues/13958 + os.environ['GRPC_ENABLE_FORK_SUPPORT'] = 'true' # must precede `import grpc` + + if not os.getenv('prefork'): # Gunicorn prefork creates channels only after fork() and is safe + logger.warning('Explicit os.fork() with a live gRPC channel is unreliable on ' + 'grpcio >= 1.80 (see grpc/grpc#43055) and may silently break ' + 'reporting in either process; prefer SW_AGENT_PROTOCOL=http or ' + 'kafka for forking applications.') if not self.__started: # if not already started, start the agent - config.finalize() # Must be finalized exactly once - - self.__started = True logger.info(f'SkyWalking sync agent instance {config.agent_instance_name} starting in pid-{os.getpid()}.') - - # Install log reporter core - if config.agent_log_reporter_active: - from skywalking import log - log.install() - # Here we install all other lib plugins on first time start (parent process) - plugins.install() + self.__init_instrumentation() elif self.__started and os.getpid() == self.started_pid: # if already started, and this is the same process, raise an error raise RuntimeError('SkyWalking Python agent has already been started in this process, ' @@ -308,18 +312,70 @@ def start(self) -> None: self.__bootstrap() # calls init_threading - atexit.register(self.__fini) + # atexit registrations are fork-inherited; register once per lineage so a + # fork-restarted child does not stack a duplicate __fini + if not self.__fini_registered: + self.__fini_registered = True + atexit.register(self.__fini) + + if config.agent_experimental_fork_support: + self.__register_fork_hooks() + + def __init_instrumentation(self) -> None: + """ + Install instrumentation once per process lineage; forked children inherit + the patches and the __started flag, so they skip this. + """ + config.finalize() # Must be finalized exactly once + + self.__started = True + + # Install log reporter core + if config.agent_log_reporter_active: + from skywalking import log + log.install() + # Here we install all other lib plugins on first time start (parent process) + plugins.install() + + def __register_fork_hooks(self) -> None: + # at-fork registrations cannot be removed and are fork-inherited; never register twice + if self.__at_fork_registered: + return + if hasattr(os, 'register_at_fork'): + self.__at_fork_registered = True + os.register_at_fork(before=self.__fork_before, after_in_parent=self.__fork_after_in_parent, + after_in_child=self.__fork_after_in_child) + + def start_prefork_master(self) -> None: + """ + Prepare the agent in a pre-forking server master (Gunicorn): install instrumentation + and arm the fork hooks only. The full agent — queues, gRPC channel, reporter threads — + starts in each forked worker; a channel living across fork() is unsafe with + grpcio >= 1.80, see https://github.com/apache/skywalking/issues/13958. + """ + loggings.init() + + if config.agent_protocol == 'grpc' and config.agent_experimental_fork_support: + # Must be exported before the first `import grpc`; plugins.install() below imports grpc + os.environ['GRPC_ENABLE_FORK_SUPPORT'] = 'true' + + if not self.__started: + self.__init_instrumentation() + logger.info(f'SkyWalking Python agent instrumented pre-fork master pid-{os.getpid()}, ' + f'reporters will start in forked worker processes.') + + self.started_pid = os.getpid() if config.agent_experimental_fork_support: - if hasattr(os, 'register_at_fork'): - os.register_at_fork(before=self.__fork_before, after_in_parent=self.__fork_after_in_parent, - after_in_child=self.__fork_after_in_child) + self.__register_fork_hooks() def __fini(self): """ This method is called when the agent is shutting down. Clean up all the queues and threads. """ + if not self.__reporting: # never bootstrapped in this process (e.g. pre-fork master) + return self.__protocol.report_segment(self.__segment_queue, False) self.__segment_queue.join() @@ -342,7 +398,9 @@ def stop(self) -> None: Stops the agent and reset the started flag. """ atexit.unregister(self.__fini) + self.__fini_registered = False self.__fini() + self.__reporting = False self.__started = False @report_with_backoff(reporter_name='heartbeat', init_wait=config.agent_collector_heartbeat_period) @@ -388,28 +446,45 @@ def __command_dispatch() -> None: # command dispatch will stuck when there are no commands command_service.dispatch() + def started(self) -> bool: + """ + Whether reporting (queues, protocol clients, reporter threads) is active in this process. + False in a pre-forking server master, where only instrumentation is installed. + """ + return self.__reporting + def is_segment_queue_full(self): + if not self.__reporting: + return True # treated as full so span creation short-circuits to NoopSpan return self.__segment_queue.full() def archive_segment(self, segment: 'Segment'): + if not self.__reporting: + return try: # unlike checking __queue.full() then inserting, this is atomic self.__segment_queue.put(segment, block=False) except Full: logger.warning('the queue is full, the segment will be abandoned') def archive_log(self, log_data: 'LogData'): + if not self.__reporting: + return try: self.__log_queue.put(log_data, block=False) except Full: logger.warning('the queue is full, the log will be abandoned') def archive_meter(self, meter_data: 'MeterData'): + if not self.__reporting: + return try: self.__meter_queue.put(meter_data, block=False) except Full: logger.warning('the queue is full, the meter will be abandoned') def add_profiling_snapshot(self, snapshot: TracingThreadSnapshot): + if not self.__reporting: + return try: self.__snapshot_queue.put(snapshot) except Full: diff --git a/skywalking/bootstrap/cli/utility/runner.py b/skywalking/bootstrap/cli/utility/runner.py index d78b7d7a..2d75a3eb 100644 --- a/skywalking/bootstrap/cli/utility/runner.py +++ b/skywalking/bootstrap/cli/utility/runner.py @@ -39,8 +39,8 @@ def prefork_handler(command: List[str]) -> None: if command[0] == 'gunicorn': # Maybe should also check 1: since there could be a command before gunicorn cli_logger.info('We noticed you are using Gunicorn, ' - 'agent will automatically start the SkyWalking Python Agent' - 'in all child (worker) processes and the master.') + 'agent will automatically start the SkyWalking Python Agent ' + 'in all child (worker) processes; the master is instrumented only.') os.environ['prefork'] = 'gunicorn' elif command[0] == 'uwsgi': cli_logger.info('We noticed you are using uWSGI, ' diff --git a/skywalking/bootstrap/loader/sitecustomize.py b/skywalking/bootstrap/loader/sitecustomize.py index bf3ba74b..b9fdc8ea 100644 --- a/skywalking/bootstrap/loader/sitecustomize.py +++ b/skywalking/bootstrap/loader/sitecustomize.py @@ -163,8 +163,16 @@ def _get_sw_loader_logger(): if prefork_server_detected == 'gunicorn': # We need to enable experimental fork support for Gunicorn config.agent_experimental_fork_support = True - # Luckily Gunicorn is based on os.fork and can be safely forked with experimental fork support - agent.start() + # Instrument the master only; the full agent (incl. gRPC channel) starts in each + # forked worker — a pre-fork channel is unsafe with grpcio >= 1.80, see + # https://github.com/apache/skywalking/issues/13958 + if config.agent_asyncio_enhancement: + _sw_loader_logger.error('SW_AGENT_ASYNCIO_ENHANCEMENT does not support pre-forking ' + 'servers, agent NOT started: a gRPC channel created in the ' + 'Gunicorn master breaks forked workers. Remove the asyncio ' + 'enhancement option, or run Gunicorn without sw-python -p.') + else: + agent.start_prefork_master() else: # Either there's some prefix like supervisor, or it simply doesn't use uwsgi/gunicorn agent.start() # CHECK: Not sure what happens when supervisor + gunicorn is used? Will it even work? diff --git a/skywalking/config.py b/skywalking/config.py index 5a2b1ca7..461ccd5c 100644 --- a/skywalking/config.py +++ b/skywalking/config.py @@ -95,15 +95,15 @@ agent_instance_properties_json: str = os.getenv('SW_AGENT_INSTANCE_PROPERTIES_JSON', '') # The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for # short-lived processes as each one will create a new instance in SkyWalking dashboard -# in format of `service_instance-child(pid)`. -# This feature may not work when a precise combination of gRPC + Python 3.7 + subprocess (not fork) is used together. -# The agent will output a warning log when using on Python 3.7 for such a reason. +# in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server +# (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. agent_experimental_fork_support: bool = os.getenv('SW_AGENT_EXPERIMENTAL_FORK_SUPPORT', '').lower() == 'true' # DANGEROUS - This option controls the interval of each bulk report from telemetry data queues # Do not modify unless you have evaluated its impact given your service load. agent_queue_timeout: int = int(os.getenv('SW_AGENT_QUEUE_TIMEOUT', '1')) # Replace the threads to asyncio coroutines to report telemetry data to the OAP. -# This option is experimental and may not work as expected. +# This option is experimental and may not work as expected. Not compatible with pre-forking +# servers (`sw-python run -p`): the agent refuses to start under a Gunicorn master. agent_asyncio_enhancement: bool = os.getenv('SW_AGENT_ASYNCIO_ENHANCEMENT', '').lower() == 'true' # BEGIN: SW_PYTHON Auto Instrumentation CLI diff --git a/skywalking/plugins/sw_grpc.py b/skywalking/plugins/sw_grpc.py index 12b03cb2..d2d671d2 100644 --- a/skywalking/plugins/sw_grpc.py +++ b/skywalking/plugins/sw_grpc.py @@ -27,7 +27,8 @@ link_vector = ['https://grpc.io/docs/languages/python'] support_matrix = {'grpcio': {'>=3.8': ['1.*']}} -note = """""" +note = """The agent package itself depends on grpcio >= 1.83, which is therefore the +effective minimum version of the instrumented library as well.""" def _get_factory_and_method(rpc_handler: Any) -> Tuple[Callable[..., Any], Callable[..., Any]]: @@ -448,25 +449,32 @@ async def intercept_stream_stream( ) -> grpc.aio._call.StreamStreamCall: return await self._intercept(continuation, client_call_details, request_iterator) - def _sw_grpc_aio_channel_factory( - target: str, - options: grpc.aio.ChannelArgumentType, - credentials: Optional[grpc.ChannelCredentials], - compression: Optional[grpc.Compression], - interceptors: Optional[Sequence[grpc.aio.ClientInterceptor]], - ): - if target == config.agent_collector_backend_services: - return _aio_grpc_channel(target, options, credentials, compression, interceptors) - _sw_interceptors: List[grpc.aio.ClientInterceptor] = [ - _AioClientUnaryUnaryInterceptor(target), - _AioClientUnaryStreamInterceptor(target), - _AioClientStreamUnaryInterceptor(target), - _AioClientStreamStreamInterceptor(target), - ] - _sw_interceptors.extend(interceptors or []) - return _aio_grpc_channel(target, options, credentials, compression, _sw_interceptors) - - _aio_channel.Channel = _sw_grpc_aio_channel_factory + # Must remain a class assigned to the module-global name `Channel`: + # since grpcio 1.83.0, grpc.aio._channel._BaseMultiCallable.__init__ validates + # `isinstance(self._references[0], Channel)` against this module global, so + # rebinding it to a plain factory function raises + # `TypeError: isinstance() arg 2 must be a type, ...` on every stub creation. + class _SWAioChannel(_aio_grpc_channel): + def __init__( + self, + target: str, + options: grpc.aio.ChannelArgumentType, + credentials: Optional[grpc.ChannelCredentials], + compression: Optional[grpc.Compression], + interceptors: Optional[Sequence[grpc.aio.ClientInterceptor]], + ): + if target != config.agent_collector_backend_services: + _sw_interceptors: List[grpc.aio.ClientInterceptor] = [ + _AioClientUnaryUnaryInterceptor(target), + _AioClientUnaryStreamInterceptor(target), + _AioClientStreamUnaryInterceptor(target), + _AioClientStreamStreamInterceptor(target), + ] + _sw_interceptors.extend(interceptors or []) + interceptors = _sw_interceptors + super().__init__(target, options, credentials, compression, interceptors) + + _aio_channel.Channel = _SWAioChannel install_async_client() install_async_server() diff --git a/skywalking/plugins/sw_websockets.py b/skywalking/plugins/sw_websockets.py index 128fde3f..711f6c7c 100644 --- a/skywalking/plugins/sw_websockets.py +++ b/skywalking/plugins/sw_websockets.py @@ -21,10 +21,13 @@ link_vector = ['https://websockets.readthedocs.io'] support_matrix = { 'websockets': { - '>=3.7': ['10.3', '10.4'] + '>=3.11': ['10.3', '10.4', '13.1', '17.0.1'], + '>=3.7': ['10.3', '10.4', '13.1'] # websockets >= 14 requires Python >= 3.11 } } -note = """The websocket instrumentation only traces client side connection handshake, +note = """Both the legacy (websockets.legacy, websockets <= 13) and the new asyncio +(websockets.asyncio, websockets >= 13) client implementations are instrumented. +The websocket instrumentation only traces client side connection handshake, the actual message exchange (send/recv) is not traced since injecting headers to socket message body is the only way to propagate the trace context, which requires customization of message structure and extreme care. (Feel free to add this feature by instrumenting the send/recv methods commented out in the code @@ -33,7 +36,22 @@ def install(): - from websockets.legacy.client import WebSocketClientProtocol + import websockets # noqa: F401 -- absence is reported by the plugin loader + + try: + from websockets.legacy.client import WebSocketClientProtocol + _install_legacy_client(WebSocketClientProtocol) + except ImportError: # websockets.legacy is deprecated since 14.0 and will be removed + pass + + try: + from websockets.asyncio.client import ClientConnection + _install_new_client(ClientConnection) + except ImportError: # websockets < 13 has no websockets.asyncio + pass + + +def _install_legacy_client(WebSocketClientProtocol): # noqa _protocol_handshake_client = WebSocketClientProtocol.handshake async def _sw_protocol_handshake_client(self, wsuri, @@ -75,6 +93,46 @@ async def _sw_protocol_handshake_client(self, wsuri, WebSocketClientProtocol.handshake = _sw_protocol_handshake_client + +def _install_new_client(ClientConnection): # noqa + """websockets >= 13 asyncio implementation: inject sw8 via handshake additional_headers""" + _connection_handshake = ClientConnection.handshake + + async def _sw_connection_handshake(self, *args, **kwargs): + # the sans-io ClientProtocol renamed the attribute wsuri -> uri over time + uri = getattr(self.protocol, 'uri', None) or self.protocol.wsuri + span = get_context().new_exit_span(op=uri.path or '/', peer=f'{uri.host}:{uri.port}', + component=Component.Websockets) + with span: + carrier = span.inject() + span.layer = Layer.Http + # connect() passes (additional_headers, user_agent_header) positionally + headers = args[0] if args else kwargs.get('additional_headers') + headers = dict(headers) if headers else {} + for item in carrier: + headers[item.key] = item.val + if args: + args = (headers,) + args[1:] + else: + kwargs['additional_headers'] = headers + + span.tag(TagHttpMethod('websocket.connect')) + + scheme = 'wss' if uri.secure else 'ws' + span.tag(TagHttpURL(f'{scheme}://{uri.host}:{uri.port}{uri.path}')) + status_msg = 'connection open' + try: + await _connection_handshake(self, *args, **kwargs) + except Exception as e: + span.error_occurred = True + span.log(e) + status_msg = 'invalid handshake' + raise e + finally: + span.tag(TagHttpStatusMsg(status_msg)) + + ClientConnection.handshake = _sw_connection_handshake + # To trace per message transactions # _send = WebSocketCommonProtocol.send # _recv = WebSocketCommonProtocol.recv diff --git a/tests/plugin/http/sw_websockets/docker-compose.yml b/tests/plugin/http/sw_websockets/docker-compose.yml index 9ad00cb5..14b45608 100644 --- a/tests/plugin/http/sw_websockets/docker-compose.yml +++ b/tests/plugin/http/sw_websockets/docker-compose.yml @@ -34,7 +34,7 @@ services: depends_on: collector: condition: service_healthy - command: ['bash', '-c', 'pip install fastapi uvicorn && pip install -r /app/requirements.txt && sw-python run python3 /app/services/provider.py'] + command: ['bash', '-c', 'pip install fastapi "uvicorn<0.50" && pip install -r /app/requirements.txt && sw-python run python3 /app/services/provider.py'] healthcheck: test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9091"] interval: 5s @@ -52,7 +52,7 @@ services: - 9090:9090 volumes: - .:/app - command: ['bash', '-c', 'pip install fastapi uvicorn && pip install -r /app/requirements.txt && sw-python run python3 /app/services/consumer.py'] + command: ['bash', '-c', 'pip install fastapi "uvicorn<0.50" && pip install -r /app/requirements.txt && sw-python run python3 /app/services/consumer.py'] depends_on: collector: condition: service_healthy diff --git a/tests/plugin/http/sw_websockets/services/consumer.py b/tests/plugin/http/sw_websockets/services/consumer.py index 204d7f7a..00a01a18 100644 --- a/tests/plugin/http/sw_websockets/services/consumer.py +++ b/tests/plugin/http/sw_websockets/services/consumer.py @@ -14,7 +14,11 @@ # See the License for the specific language governing permissions and # limitations under the License. # -from websockets.client import connect +try: + # new asyncio implementation, websockets >= 13 (the default since 14) + from websockets.asyncio.client import connect +except ImportError: + from websockets.client import connect import asyncio @@ -26,7 +30,7 @@ @app.get('/ws') async def websocket_ping(): - async with connect('ws://provider:9091/ws', extra_headers=None) as websocket: + async with connect('ws://provider:9091/ws') as websocket: await websocket.send('Ping') response = await websocket.recv() diff --git a/tests/plugin/web/sw_fork_support/__init__.py b/tests/plugin/web/sw_fork_support/__init__.py new file mode 100644 index 00000000..b1312a09 --- /dev/null +++ b/tests/plugin/web/sw_fork_support/__init__.py @@ -0,0 +1,16 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# diff --git a/tests/plugin/web/sw_fork_support/docker-compose.yml b/tests/plugin/web/sw_fork_support/docker-compose.yml new file mode 100644 index 00000000..28f54ff1 --- /dev/null +++ b/tests/plugin/web/sw_fork_support/docker-compose.yml @@ -0,0 +1,53 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: '2.1' + +services: + collector: + extends: + service: collector + file: ../../docker-compose.base.yml + + provider: + extends: + service: agent + file: ../../docker-compose.base.yml + ports: + - 9090:9090 + volumes: + - .:/app + command: ['bash', '-c', 'pip install flask requests && pip install -r /app/requirements.txt && sw-python run python3 /app/services/app.py'] + depends_on: + collector: + condition: service_healthy + healthcheck: + test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9090 && cat < /dev/null > /dev/tcp/127.0.0.1/9091"] + interval: 5s + timeout: 60s + retries: 120 + environment: + SW_AGENT_NAME: provider + SW_AGENT_LOGGING_LEVEL: INFO + SW_AGENT_EXPERIMENTAL_FORK_SUPPORT: 'true' + # HTTP reporter keeps this test deterministic: forking with a live gRPC channel + # is subject to upstream at-fork races (grpc/grpc#43055) that can silently drop + # either side's segments; the gRPC transport is covered by sw_gunicorn + SW_AGENT_PROTOCOL: http + SW_AGENT_COLLECTOR_BACKEND_SERVICES: collector:12800 +networks: + beyond: diff --git a/tests/plugin/web/sw_fork_support/expected.data.yml b/tests/plugin/web/sw_fork_support/expected.data.yml new file mode 100644 index 00000000..6608c16a --- /dev/null +++ b/tests/plugin/web/sw_fork_support/expected.data.yml @@ -0,0 +1,104 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +segmentItems: + - serviceName: provider + segmentSize: 3 + segments: + - segmentId: not null + spans: + - operationName: /ping + parentSpanId: -1 + spanId: 0 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://0.0.0.0:9090/ping + - key: http.status_code + value: '200' + startTime: gt 0 + endTime: gt 0 + componentId: 7001 + spanType: Entry + peer: not null + skipAnalysis: false + - segmentId: not null + spans: + - operationName: /users + parentSpanId: -1 + spanId: 0 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://127.0.0.1:9091/users + - key: http.status_code + value: '200' + refs: + - parentEndpoint: /users + networkAddress: '127.0.0.1:9091' + refType: CrossProcess + parentSpanId: 1 + parentTraceSegmentId: not null + parentServiceInstance: not null + parentService: provider + traceId: not null + startTime: gt 0 + endTime: gt 0 + componentId: 7001 + spanType: Entry + peer: not null + skipAnalysis: false + - segmentId: not null + spans: + - operationName: /users + parentSpanId: 0 + spanId: 1 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://127.0.0.1:9091/users + - key: http.status_code + value: '200' + startTime: gt 0 + endTime: gt 0 + componentId: 7002 + spanType: Exit + peer: 127.0.0.1:9091 + skipAnalysis: false + - operationName: /users + parentSpanId: -1 + spanId: 0 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://0.0.0.0:9090/users + - key: http.status_code + value: '200' + startTime: gt 0 + endTime: gt 0 + componentId: 7001 + spanType: Entry + peer: not null + skipAnalysis: false diff --git a/tests/plugin/web/sw_fork_support/services/__init__.py b/tests/plugin/web/sw_fork_support/services/__init__.py new file mode 100644 index 00000000..b1312a09 --- /dev/null +++ b/tests/plugin/web/sw_fork_support/services/__init__.py @@ -0,0 +1,16 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# diff --git a/tests/plugin/web/sw_fork_support/services/app.py b/tests/plugin/web/sw_fork_support/services/app.py new file mode 100644 index 00000000..17d023b6 --- /dev/null +++ b/tests/plugin/web/sw_fork_support/services/app.py @@ -0,0 +1,66 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import os +import time + +if __name__ == '__main__': + from flask import Flask, jsonify + + # explicit os.fork() with SW_AGENT_EXPERIMENTAL_FORK_SUPPORT=true: + # the child must restart the agent as a `-child(pid)` instance and keep tracing. + # Fork outside the agent's active-RPC window (right after start the agent registers + # with the collector; forking mid-RPC trips gRPC's own at-fork races, grpc/grpc#43055) + time.sleep(2) + pid = os.fork() + + if pid == 0: + backend = Flask('backend') + + @backend.route('/users', methods=['GET']) + def users(): + return jsonify({'song': 'Despacito'}) + + backend.run(host='0.0.0.0', port=9091) + else: + import socket + import requests + + # serve only once the forked child's backend is reachable, so early readiness + # probes cannot produce error spans through an instrumented parent + for _ in range(120): + try: + socket.create_connection(('127.0.0.1', 9091), timeout=1).close() + break + except OSError: + time.sleep(1) + + frontend = Flask('frontend') + + # readiness/warm-up endpoint: its segment arrives alone and seeds the service name + # in the mock collector, whose first-insert for a service is not concurrency-safe + # (SegmentItems.addSegmentItem check-then-act) — the parent and child later report + # their /users segments concurrently under the same service name + @frontend.route('/ping', methods=['GET']) + def ping(): + return jsonify({'song': 'Despacito'}) + + @frontend.route('/users', methods=['GET']) + def call_backend(): + res = requests.get('http://127.0.0.1:9091/users', timeout=5) + return jsonify(res.json()) + + frontend.run(host='0.0.0.0', port=9090) diff --git a/tests/plugin/web/sw_fork_support/test_fork_support.py b/tests/plugin/web/sw_fork_support/test_fork_support.py new file mode 100644 index 00000000..b246c94f --- /dev/null +++ b/tests/plugin/web/sw_fork_support/test_fork_support.py @@ -0,0 +1,69 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import time +from typing import Callable + +import pytest +import requests + +from tests.plugin.base import TestPluginBase + + +@pytest.fixture +def prepare(): + # type: () -> Callable + # /ping (parent-only) also seeds the service name in the mock collector before the + # parent and child post their /users segments concurrently: the collector's very first + # insert for a service name is not concurrency-safe and can silently drop one segment + return lambda *_: requests.get('http://0.0.0.0:9090/ping', timeout=5).raise_for_status() + + +class TestPlugin(TestPluginBase): + """ + Explicit os.fork() with SW_AGENT_EXPERIMENTAL_FORK_SUPPORT: the parent keeps its + agent, the forked child restarts one as a `-child(pid)` instance, and the trace + stays continuous across the fork (parent entry/exit -> child entry with ref). + Runs over the HTTP reporter: forking with a live gRPC channel is subject to + upstream at-fork races (grpc/grpc#43055) that can silently drop either side's + segments; the gRPC transport path is covered deterministically by sw_gunicorn. + """ + + @pytest.mark.parametrize('version', ['grpcio>=1.83']) + def test_plugin(self, docker_compose, version): + # the /ping seed segment must be REGISTERED by the collector before /users makes + # the parent and child report concurrently, otherwise all three segments can be + # in flight together and the collector's first-insert race still drops one + for _ in range(30): + if '/ping' in requests.get('http://localhost:12800/receiveData', timeout=5).text: + break + time.sleep(1) + else: + raise Exception('the /ping seed segment never reached the collector') + + response = requests.get('http://0.0.0.0:9090/users', timeout=5) + assert response.status_code == 200 + + self.validate() + + stdout, stderr = docker_compose.get_logs() + stdout = stdout.decode('utf-8') if isinstance(stdout, bytes) else stdout + stderr = stderr.decode('utf-8') if isinstance(stderr, bytes) else stderr + logs = f'{stdout}\n{stderr}' + + # parent started one full agent, the forked child restarted one + assert logs.count('starting in pid-') == 1 + assert logs.count('Agent spawned as') == 1 diff --git a/tests/plugin/web/sw_gunicorn/__init__.py b/tests/plugin/web/sw_gunicorn/__init__.py new file mode 100644 index 00000000..b1312a09 --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/__init__.py @@ -0,0 +1,16 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# diff --git a/tests/plugin/web/sw_gunicorn/docker-compose.yml b/tests/plugin/web/sw_gunicorn/docker-compose.yml new file mode 100644 index 00000000..9d6516c0 --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/docker-compose.yml @@ -0,0 +1,86 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +version: '2.1' + +services: + collector: + extends: + service: collector + file: ../../docker-compose.base.yml + + provider: + extends: + service: agent + file: ../../docker-compose.base.yml + volumes: + - .:/app + command: ['bash', '-c', 'pip install flask gunicorn && pip install -r /app/requirements.txt && cd /app/services && sw-python run -p gunicorn provider:app --preload --workers 2 --threads 2 --bind 0.0.0.0:9091'] + depends_on: + collector: + condition: service_healthy + healthcheck: + test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9091"] + interval: 5s + timeout: 60s + retries: 120 + environment: + SW_AGENT_NAME: provider + SW_AGENT_LOGGING_LEVEL: INFO + + # asyncio enhancement + prefork is rejected: the agent must NOT start, the app still serves + provider-async: + extends: + service: agent + file: ../../docker-compose.base.yml + volumes: + - .:/app + command: ['bash', '-c', 'pip install flask gunicorn && pip install -r /app/requirements.txt && cd /app/services && sw-python run -p gunicorn provider:app --workers 1 --bind 0.0.0.0:9092'] + depends_on: + collector: + condition: service_healthy + healthcheck: + test: ["CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/9092"] + interval: 5s + timeout: 60s + retries: 120 + environment: + SW_AGENT_NAME: provider-async + SW_AGENT_LOGGING_LEVEL: INFO + SW_AGENT_ASYNCIO_ENHANCEMENT: 'true' + + consumer: + extends: + service: agent + file: ../../docker-compose.base.yml + ports: + - 9090:9090 + volumes: + - .:/app + command: ['bash', '-c', 'pip install flask requests && pip install -r /app/requirements.txt && sw-python run python3 /app/services/consumer.py'] + depends_on: + collector: + condition: service_healthy + provider: + condition: service_healthy + provider-async: + condition: service_healthy + environment: + SW_AGENT_NAME: consumer + SW_AGENT_LOGGING_LEVEL: INFO +networks: + beyond: diff --git a/tests/plugin/web/sw_gunicorn/expected.data.yml b/tests/plugin/web/sw_gunicorn/expected.data.yml new file mode 100644 index 00000000..191d39fd --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/expected.data.yml @@ -0,0 +1,88 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +segmentItems: + - serviceName: provider + segmentSize: 1 + segments: + - segmentId: not null + spans: + - operationName: /users + parentSpanId: -1 + spanId: 0 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://provider:9091/users + - key: http.status_code + value: '200' + refs: + - parentEndpoint: /users + networkAddress: 'provider:9091' + refType: CrossProcess + parentSpanId: 1 + parentTraceSegmentId: not null + parentServiceInstance: not null + parentService: consumer + traceId: not null + startTime: gt 0 + endTime: gt 0 + componentId: 7001 + spanType: Entry + peer: not null + skipAnalysis: false + - serviceName: consumer + segmentSize: 1 + segments: + - segmentId: not null + spans: + - operationName: /users + parentSpanId: 0 + spanId: 1 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://provider:9091/users + - key: http.status_code + value: '200' + startTime: gt 0 + endTime: gt 0 + componentId: 7002 + spanType: Exit + peer: provider:9091 + skipAnalysis: false + - operationName: /users + parentSpanId: -1 + spanId: 0 + spanLayer: Http + tags: + - key: http.method + value: GET + - key: http.url + value: http://0.0.0.0:9090/users + - key: http.status_code + value: '200' + startTime: gt 0 + endTime: gt 0 + componentId: 7001 + spanType: Entry + peer: not null + skipAnalysis: false diff --git a/tests/plugin/web/sw_gunicorn/services/__init__.py b/tests/plugin/web/sw_gunicorn/services/__init__.py new file mode 100644 index 00000000..b1312a09 --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/services/__init__.py @@ -0,0 +1,16 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# diff --git a/tests/plugin/web/sw_gunicorn/services/consumer.py b/tests/plugin/web/sw_gunicorn/services/consumer.py new file mode 100644 index 00000000..4030da83 --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/services/consumer.py @@ -0,0 +1,30 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import requests + +if __name__ == '__main__': + from flask import Flask, jsonify + + app = Flask(__name__) + + @app.route('/users', methods=['GET']) + def application(): + res = requests.get('http://provider:9091/users', timeout=5) + return jsonify(res.json()) + + PORT = 9090 + app.run(host='0.0.0.0', port=PORT) diff --git a/tests/plugin/web/sw_gunicorn/services/provider.py b/tests/plugin/web/sw_gunicorn/services/provider.py new file mode 100644 index 00000000..3c9f9c2a --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/services/provider.py @@ -0,0 +1,33 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +import requests +from flask import Flask, jsonify + +try: + # under gunicorn --preload this module imports in the PRE-FORK master, where the + # agent is instrumentation-only: this instrumented call must silently noop + # (NoopSpan via agent.started() guard) instead of crashing the master + requests.get('http://collector:12800/receiveData', timeout=5) +except Exception: # noqa + pass + +app = Flask(__name__) + + +@app.route('/users', methods=['GET']) +def users(): + return jsonify({'song': 'Despacito'}) diff --git a/tests/plugin/web/sw_gunicorn/test_gunicorn.py b/tests/plugin/web/sw_gunicorn/test_gunicorn.py new file mode 100644 index 00000000..b8058792 --- /dev/null +++ b/tests/plugin/web/sw_gunicorn/test_gunicorn.py @@ -0,0 +1,64 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +from typing import Callable + +import pytest +import requests + +from tests.plugin.base import TestPluginBase + + +@pytest.fixture +def prepare(): + # type: () -> Callable + return lambda *_: requests.get('http://0.0.0.0:9090/users', timeout=5).raise_for_status() + + +class TestPlugin(TestPluginBase): + """ + Regression test for Gunicorn prefork over the gRPC reporter, apache/skywalking#13958. + Validates a complete cross-process trace (consumer entry/exit -> gunicorn provider entry) + plus: the master runs no gRPC channel before fork, every worker boots and reports, and + grpcio >= 1.80 produces no poll-engine fork errors. + """ + + @pytest.mark.parametrize('version', ['grpcio>=1.83']) + def test_plugin(self, docker_compose, version): + self.validate() + + stdout, stderr = docker_compose.get_logs() + stdout = stdout.decode('utf-8') if isinstance(stdout, bytes) else stdout + stderr = stderr.decode('utf-8') if isinstance(stderr, bytes) else stderr + logs = f'{stdout}\n{stderr}' + + # every worker booted exactly once (2 provider + 1 provider-async); a worker + # deadlocked at fork would be killed by the arbiter and respawned -> extra boot lines + assert logs.count('Booting worker with pid') == 3 + assert 'WORKER TIMEOUT' not in logs + + # full agent started in each provider worker, instrumentation-only in the master; + # the single full start belongs to the consumer + assert logs.count('Agent spawned as') == 2 + assert 'instrumented pre-fork master' in logs + assert logs.count('starting in pid-') == 1 + + # asyncio enhancement + prefork must be rejected, not started unsafely + assert 'does not support pre-forking' in logs + + # grpcio >= 1.80 EventEngine fork regression signature + assert 'Kick Failure' not in logs + assert 'pollset_kick' not in logs