fix: gRPC fork safety for Gunicorn prefork (grpcio >= 1.83), sw_grpc aio compatibility, websockets >= 13 support

Makefile

@@ -50,14 +50,16 @@ else
 	-curl -sSL https://install.python-poetry.org | python3 -
 endif
 
+# grpcio-tools pinned: generated stubs refuse grpcio older than the tools version,
+# keep in sync with the grpcio floor declared in pyproject.toml
 .PHONY: gen
 gen:
-	poetry run pip install 'grpcio-tools>=1.68.0' packaging
+	poetry run pip install 'grpcio-tools~=1.83.0' packaging
 	poetry run python3 tools/grpc_code_gen.py
 
 .PHONY: gen-basic
 gen-basic:
-	python3 -m pip install 'grpcio-tools>=1.68.0' packaging
+	python3 -m pip install 'grpcio-tools~=1.83.0' packaging
 	python3 tools/grpc_code_gen.py
 
 .PHONY: install

docs/en/setup/CLI.md

@@ -53,7 +53,7 @@ or
 
 `uwsgi --die-on-term --http 0.0.0.0:5000 --http-manage-expect --master --workers 3 --enable-threads --threads 3 --manage-script-name --mount /=main:app`
 
-Please change it to (**the `-p` option starts one agent in each process, which is the correct behavior**):
+Please change it to (**the `-p` option starts one agent in each worker process and none in the master, which is the correct behavior**):
 
 **Important:** if the call to uwsgi/gunicorn is prefixed with other commands, this approach will fail 
 since agent currently looks for the command line input at index 0 for safety as an experimental feature.
@@ -72,6 +72,13 @@ Note that `sw-python` also work with spawned subprocess (os.exec*/subprocess) as
 Additionally, `sw-python` started agent works well with `os.fork` when your application forks workers, 
 as long as the `SW_AGENT_EXPERIMENTAL_FORK_SUPPORT` is turned on. (It will be automatically turned on when gunicorn is detected)
 
+**Important**: with the default gRPC reporter, explicit `os.fork()` is NOT reliable on grpcio >= 1.80: the agent's
+background reporters enter gRPC at any time (heartbeat, segment flush) independent of application requests, and a
+fork while a channel is live is subject to open upstream races (grpc/grpc#43055, grpc/grpc#43062) that can silently
+break reporting in either process. Applications that fork should use `SW_AGENT_PROTOCOL=http` (or `kafka`).
+Gunicorn via `sw-python run -p` is NOT affected: there the agent creates its channels only after the fork,
+which is gRPC's supported model.
+
 ## Configuring the agent 
 
 You would normally want to provide additional configurations other than the default ones.

docs/en/setup/Configuration.md

@@ -38,9 +38,9 @@ export SW_AGENT_YourConfiguration=YourValue
 | agent_collector_heartbeat_period | SW_AGENT_COLLECTOR_HEARTBEAT_PERIOD | <class 'int'> | 30 | The agent will exchange heartbeat message with SkyWalking OAP backend every `period` seconds |
 | agent_collector_properties_report_period_factor | SW_AGENT_COLLECTOR_PROPERTIES_REPORT_PERIOD_FACTOR | <class 'int'> | 10 | The agent will report service instance properties every `factor * heartbeat period` seconds default: 10*30 = 300 seconds |
 | agent_instance_properties_json | SW_AGENT_INSTANCE_PROPERTIES_JSON | <class 'str'> |  | A custom JSON string to be reported as service instance properties, e.g. `{"key": "value"}` |
-| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. This feature may not work when a precise combination of gRPC + Python 3.7 + subprocess (not fork) is used together. The agent will output a warning log when using on Python 3.7 for such a reason. |
+| agent_experimental_fork_support | SW_AGENT_EXPERIMENTAL_FORK_SUPPORT | <class 'bool'> | False | The agent will restart itself in any os.fork()-ed child process. Important Note: it's not suitable for short-lived processes as each one will create a new instance in SkyWalking dashboard in format of `service_instance-child(pid)`. When the sw-python CLI detects a pre-forking server (Gunicorn), only worker processes run a full agent; the master installs instrumentation only. |
 | agent_queue_timeout | SW_AGENT_QUEUE_TIMEOUT | <class 'int'> | 1 | DANGEROUS - This option controls the interval of each bulk report from telemetry data queues Do not modify unless you have evaluated its impact given your service load. |
-| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. |
+| agent_asyncio_enhancement | SW_AGENT_ASYNCIO_ENHANCEMENT | <class 'bool'> | False | Replace the threads to asyncio coroutines to report telemetry data to the OAP. This option is experimental and may not work as expected. Not compatible with pre-forking servers (`sw-python run -p`): the agent refuses to start under a Gunicorn master. |
 ###  SW_PYTHON Auto Instrumentation CLI
 | Configuration | Environment Variable | Type | Default Value | Description |
 | :------------ | :------------ | :------------ | :------------ | :------------ |

docs/en/setup/Plugins.md

@@ -51,7 +51,7 @@ or a limitation of SkyWalking auto-instrumentation (welcome to contribute!)
 | [urllib3](https://urllib3.readthedocs.io/en/latest/) | Python >=3.12 - NOT SUPPORTED YET; Python >=3.10 - ['1.26', '1.25'];  | `sw_urllib3` |
 | [urllib3](https://urllib3.readthedocs.io/en/latest/) | Python >=3.12 - ['2.3', '2.0'];  | `sw_urllib3_v2` |
 | [urllib_request](https://docs.python.org/3/library/urllib.request.html) | Python >=3.7 - ['*'];  | `sw_urllib_request` |
-| [websockets](https://websockets.readthedocs.io) | Python >=3.7 - ['10.3', '10.4'];  | `sw_websockets` |
+| [websockets](https://websockets.readthedocs.io) | Python >=3.11 - ['10.3', '10.4', '13.1', '17.0.1']; Python >=3.7 - ['10.3', '10.4', '13.1'];  | `sw_websockets` |
 ### Notes
 - The celery server running with "celery -A ..." should be run with the HTTP protocol
 as it uses multiprocessing by default which is not compatible with the gRPC protocol implementation
@@ -60,6 +60,8 @@ in SkyWalking currently. Celery clients can use whatever protocol they want.
 Hug is believed to be abandoned project, use this plugin with a bit more caution.
 Instead of Hug, plugin test should move to test actual Falcon.
 - Falcon 3.x/4.x plugin. For legacy hug-based instrumentation, see sw_falcon.
+- The agent package itself depends on grpcio >= 1.83, which is therefore the
+effective minimum version of the instrumented library as well.
 - The Neo4j plugin integrates neo4j python driver 5.x.x versions which
 support both Neo4j 5 and 4.4 DBMS.
 - Sanic 21.9+ plugin using signal listeners.
@@ -68,7 +70,9 @@ Note: Sanic's touchup system recompiles handle_request at startup,
 so we use signal listeners instead of monkey-patching handle_request.
 - urllib3 1.x plugin. For urllib3 2.x, see sw_urllib3_v2.
 - urllib3 2.x plugin. For urllib3 1.x, see sw_urllib3.
-- The websocket instrumentation only traces client side connection handshake,
+- Both the legacy (websockets.legacy, websockets <= 13) and the new asyncio
+(websockets.asyncio, websockets >= 13) client implementations are instrumented.
+The websocket instrumentation only traces client side connection handshake,
 the actual message exchange (send/recv) is not traced since injecting headers to socket message
 body is the only way to propagate the trace context, which requires customization of message structure
 and extreme care. (Feel free to add this feature by instrumenting the send/recv methods commented out in the code

docs/en/setup/advanced/AsyncEnhancement.md

@@ -6,6 +6,10 @@ Since `1.1.0`, the Python agent supports asynchronous reporting of ALL telemetry
 export SW_AGENT_ASYNCIO_ENHANCEMENT=true
 ```
 
+> Limitation: this option is incompatible with pre-forking servers via `sw-python run -p` (e.g. Gunicorn) —
+> the asyncio agent has no fork support, so the agent refuses to start and the application runs without
+> observability. See the [Gunicorn FAQ](../faq/How-to-use-with-gunicorn.md).
+
 ## Why we need this feature
 
 Before version `1.1.0`, SkyWalking Python agent had only an implementation with the Threading module to provide data reporters. Yet with the growth of the Python agent, it is now fully capable and requires more resources than when only tracing was supported (we start many threads and gRPC itself creates even more threads when streaming).

docs/en/setup/faq/How-to-use-with-gunicorn.md

@@ -14,7 +14,7 @@ serves requests.
 
 > Note: Python 3.10+ is required. Earlier versions (3.7-3.9) are no longer supported.
 
-**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and all Gunicorn workers and master will get their own working agent.
+**TL;DR:** specify `-p` or `--prefork` in `sw-python run -p` and every Gunicorn worker will get its own working agent (the master is instrumented only and runs no agent).
 
 **Important:** if the call to gunicorn is prefixed with other commands, this approach will fail 
 since agent currently looks for the command line input at index 0 for safety as an experimental feature.
@@ -29,17 +29,34 @@ sw-python run -p gunicorn gunicorn_consumer_prefork:app --workers 2 --worker-cla
 By specifying the -p or --prefork option in sw-python CLI, the `agent_experimental_fork_support` agent option will be turned on automatically. 
 
 Startup flow:
-sw-python -> gunicorn -> master process (agent starts) -> fork -> worker process (agent restarts due to os.register_at_fork)
+sw-python -> gunicorn -> master process (instrumentation only) -> fork -> worker process (full agent starts due to os.register_at_fork)
 
-The master process will get its own agent, although it won't report any trace, since obviously it doesn't take requests, 
-it still reports metrics that is useful for debugging
+The master process does not run a full agent: it only installs instrumentation, and the reporters plus the
+gRPC channel are created in each forked worker. Therefore the master does not appear as a service instance
+(it takes no requests anyway). A gRPC channel created before fork() is unsafe with grpcio >= 1.80, see
+[apache/skywalking#13958](https://github.com/apache/skywalking/issues/13958).
 
 > A runnable example can be found in the demo folder of skywalking-python GitHub repository
 
+### Known issue with agent <= 1.2.0 and grpcio >= 1.80
+
+Agent versions up to 1.2.0 started a full agent (including a gRPC channel) in the Gunicorn master before forking.
+With grpcio >= 1.80 this produces continuous `Kick Failure (eventfd_write: Bad file descriptor)` errors
+and can silently hang workers. Tracing generally keeps working — the errors come from the gRPC client polling
+engine and are unrelated to the OAP version. Workarounds on old agents: pin `grpcio<1.80` or use `SW_AGENT_PROTOCOL=http`.
+Fixed agent versions require `grpcio >= 1.83` and never create a gRPC channel in the master.
+
+### Incompatible with the asyncio enhancement
+
+`SW_AGENT_ASYNCIO_ENHANCEMENT=true` is incompatible with `sw-python run -p gunicorn`: the asyncio agent has no
+fork support, so the agent refuses to start (an error is logged) and the application serves WITHOUT observability.
+Remove the asyncio enhancement option, or run Gunicorn without `-p`.
+
 ## Manual Approach (only use when sw-python doesn't work)
 
-**Limitation**: Using normal postfork hook will not add observability to the master process, you could also define a prefork hook to
-start an agent in the master process, with a instance name like `instance-name-master(<pid>)`
+**Limitation**: Using normal postfork hook will not add observability to the master process.
+Do NOT start an agent in the master process (e.g. from a prefork hook): a gRPC channel created before
+fork() is unsafe with grpcio >= 1.80 and can deadlock workers.
 
 The following is just an example, since Gunicorn's automatic injection approach is likely to work in many situations.
 

docs/en/setup/faq/How-to-use-with-uwsgi.md

@@ -13,8 +13,8 @@ Some of the original discussion can be found here:
 > You can always fall back to the manual approach. 
 > (although it's also possible to pass postfork hook without changing code, which is essentially how sw-python is implemented)
 
-> Limitation: regardless of the approach used, uWSGI master process cannot be safely monitored. Since it doesn't take any requests, it is generally acceptable.
-> Alternatively, you could switch to Gunicorn, where its master process can be monitored properly along with all child workers.
+> Limitation: regardless of the approach used, the uWSGI master process is not monitored. Since it doesn't take any requests, it is generally acceptable.
+> Gunicorn behaves the same way — only worker processes are monitored, which is the fork-safe design.
 
 **Important**: The `--enable-threads` and `--master` option must be given to allow the usage of post_fork hooks and threading in workers. 
 In the `sw-python` CLI, these two options will be automatically injected for you in addition to the post_fork hook.