Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions context/jvm-opts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# JVM_OPTS 启动参数

Dinky 进程的堆参数通过环境变量 `JVM_OPTS` 注入,`script/bin/auto.sh` 不再写死覆盖。

## 解析规则

启动脚本在 sourcing `/etc/profile.d/dinky_env`、`/etc/profile.d/dinky_db`、`/etc/profile` 之后执行:

```
JVM_OPTS="${JVM_OPTS:-${DEFAULT_JVM_OPTS}}"
```

- 已设置且非空:原样使用(整段替换,不是追加)。只想加 GC/诊断参数时,必须把堆参数一并写进 `JVM_OPTS`
- 未设置或为空:回退到脚本默认值 `-Xms512M -Xmx2048M`(空字符串会盖掉镜像 `ENV`,容器里也会落到这组物理机堆参数)
- 启动/重启命令会打印实际生效值

不要再设置 `-XX:PermSize` / `-XX:MaxPermSize`:PermGen 在 JDK 8 已移除,HotSpot 会忽略并告警。也不在默认值里加 `-XX:MaxMetaspaceSize`,避免给原先无 Metaspace 上限的进程引入新的 OOM。

## 部署默认值

| 场景 | 默认 `JVM_OPTS` | 覆盖方式 |
| --- | --- | --- |
| 物理机 / 虚拟机 | `-Xms512M -Xmx2048M` | `export JVM_OPTS=...` 或写入 `/etc/profile.d/dinky_env` |
| Docker 镜像 | `-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0` | `docker run -e JVM_OPTS=...` |
| Helm | 使用镜像默认;未设置则不注入该环境变量 | `spec.extraEnv.jvmOpts` |

容器场景不要再对 `auto.sh` 做 `sed` 替换。镜像用 `ENV JVM_OPTS` 提供默认值,运行时覆盖即可。

## 约束

- `deploy/docker/.env` 会同时注入 Dinky 与 Flink JM/TM,不要在该文件里设置 `JVM_OPTS`。
- docker-compose 的 dinky 服务不要写空的 `environment.JVM_OPTS`:空字符串会覆盖镜像 `ENV`,脚本会回退到物理机堆参数。
- 使用 `MaxRAMPercentage` 时必须给容器/Pod 设置 memory limit,否则百分比按宿主机可见内存计算,可能过大。
- 容器默认不设 `InitialRAMPercentage`,避免启动时按 70% 提交堆,把 Metaspace/线程/native 挤出 cgroup。
- `java ${JVM_OPTS}` 按空格拆分参数,参数值本身不要含未转义空格。
3 changes: 3 additions & 0 deletions deploy/docker/.env
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ CUSTOM_JAR_PATH=/opt/dinky/extends/
FLINK_PROPERTIES="jobmanager.rpc.address: jobmanager"
TZ=Asia/Shanghai

# 不要在此共享 env 中设置 JVM_OPTS(会同时注入 Flink JM/TM)。
# 仅覆盖 Dinky 时:docker compose run -e JVM_OPTS="-Xms1g -Xmx2g" ...

# mysql,h2,pgsql
DB_ACTIVE=h2

Expand Down
3 changes: 2 additions & 1 deletion deploy/docker/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,15 @@ RUN cp /opt/flink/opt/flink-python*.jar /opt/flink/lib/ && cp /opt/flink/opt/fli
ENV FLINK_VERSION=${FLINK_VERSION}
ENV DINKY_HOME=/opt/dinky/
ENV H2_DB=./tmp/db/h2
ENV JVM_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"


USER root

WORKDIR /opt/dinky/


RUN mkdir /opt/dinky/customJar && chmod -R 777 /opt/dinky/ && mkdir /dinky && chmod -R 777 /dinky/ && sed -i 's/-Xms512M -Xmx2048M -XX:ermSize=512M/-XX:+UseContainerSupport -XX:InitialRAMPercentage=70.0 -XX:MaxRAMPercentage=70.0/g' ./bin/auto.sh
RUN mkdir /opt/dinky/customJar && chmod -R 777 /opt/dinky/ && mkdir /dinky && chmod -R 777 /dinky/

EXPOSE 8888

Expand Down
3 changes: 3 additions & 0 deletions deploy/docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@ services:
image: "dinkydocker/dinky-standalone-server:${DINKY_VERSION}-flink${FLINK_VERSION}"
ports:
- "8888:8888"
# JVM_OPTS is not listed here on purpose: an empty value would override the
# image default (container RAM percentage). Set it via `docker run -e` or
# add it only to this service when you need a custom heap.
env_file: .env
networks:
- dinky
Expand Down
4 changes: 4 additions & 0 deletions deploy/kubernetes/helm/dinky/templates/dinky.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,10 @@ spec:
{{- end }}
{{- end }}
{{- include "dinky.dbActive" . | nindent 12 }}
{{- with .Values.spec.extraEnv.jvmOpts }}
- name: JVM_OPTS
value: {{ . | quote }}
{{- end }}
envFrom:
{{- if .Values.mysql.enabled }}
- secretRef:
Expand Down
5 changes: 5 additions & 0 deletions deploy/kubernetes/helm/dinky/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -69,11 +69,16 @@ spec:
name: rest
extraEnv:
flinkVersion: "1.17"
# Override Dinky JVM options. Leave unset to use the image default
# (UseContainerSupport + 70% of container memory).
# jvmOpts: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"

affinity: {}
nodeSelector: {}
tolerations: []
resources: {}
# Set memory limits when using the image default MaxRAMPercentage (70%).
# Without a limit, the percentage is taken from node-visible memory.
# resources:
# limits:
# memory: "2Gi"
Expand Down
14 changes: 14 additions & 0 deletions docs/docs/deploy_guide/docker_deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,20 @@ docker run --restart=always -p 8888:8888 \
-v /opt/lib:/opt/dinky/customJar/ \
dinkydocker/dinky-standalone-server:1.1.0-flink1.17
```

### 自定义 JVM 内存

镜像默认使用容器内存的 70% 作为堆(`UseContainerSupport` + `MaxRAMPercentage`),无需修改启动脚本。通过环境变量 `JVM_OPTS` 覆盖:

```bash
docker run --restart=always -p 8888:8888 \
--name dinky \
-e JVM_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0" \
dinkydocker/dinky-standalone-server:1.1.0-flink1.17
```

建议同时为容器设置 memory limit(例如 `--memory=2g`),否则百分比会按宿主机可见内存计算。不要在共享的 `deploy/docker/.env` 中设置 `JVM_OPTS`(该文件会同时注入 Flink JM/TM)。不要只把 GC 参数写进 `JVM_OPTS`:该变量是整段替换,不是追加。

---
### 使用docker-compose
docker-compose可快速帮你搭建起来dinky与flink集群环境,
Expand Down
8 changes: 8 additions & 0 deletions docs/docs/deploy_guide/kubernetes_deploy.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,11 @@ spec:
name: rest
extraEnv:
flinkVersion: "1.17"
# 可选。不设置则使用镜像默认:容器内存的 70%
# jvmOpts: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"
```
自定义 Dinky 进程 JVM 参数时设置 `spec.extraEnv.jvmOpts`,并建议同时配置 `spec.resources.limits.memory`,否则 `MaxRAMPercentage` 会按节点/cgroup 可见内存计算。

例如,mysql数据源完整配置如下:
完整配置如下:
```yaml
Expand Down Expand Up @@ -129,6 +133,8 @@ spec:
name: rest
extraEnv:
flinkVersion: "1.17"
# 可选。不设置则使用镜像默认:容器内存的 70%
# jvmOpts: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"

affinity: {}
nodeSelector: {}
Expand Down Expand Up @@ -285,6 +291,8 @@ spec:
name: rest
extraEnv:
flinkVersion: "1.17"
# 可选。不设置则使用镜像默认:容器内存的 70%
# jvmOpts: "-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"

affinity: {}
nodeSelector: {}
Expand Down
11 changes: 11 additions & 0 deletions docs/docs/deploy_guide/normal_deploy.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,17 @@ sh auto.sh startOnPending 1.20

```

#### JVM 内存

启动脚本读取环境变量 `JVM_OPTS`,未设置时默认为 `-Xms512M -Xmx2048M`。

```shell
export JVM_OPTS="-Xms1g -Xmx4g"
sh ./bin/auto.sh start 1.20
```

也可写入 `/etc/profile.d/dinky_env`。JDK 8 已移除 PermGen,请使用 `-Xms` / `-Xmx`,不要再设置 `-XX:PermSize` / `-XX:MaxPermSize`。

#### 版本适配
启动指令的第二个参数则是版本选择,支持 【1.14, 1.15, 1.16, 1.17, 1.18, 1.19】,参考指令如下:
> 自 1.0.2 版本开始加入了自动检测 Flink 版本的功能,如果不传入版本号,则会自动检测 Flink 版本作为启动版本(即检测 extends 目录下的 flink${FLINK_VERSION} 文件夹)
Expand Down
3 changes: 2 additions & 1 deletion e2e_test/docker-compose-env/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ ARG FLINK_VERSION
ENV FLINK_VERSION=${FLINK_VERSION}
ENV DINKY_HOME=/opt/dinky/
ENV H2_DB=./tmp/db/h2
ENV JVM_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=70.0"
ADD build/dinky-release*.tar.gz /opt
RUN ls /opt && mv /opt/dinky-release* /opt/dinky

Expand All @@ -20,7 +21,7 @@ COPY --from=flink-base /opt/flink/lib/*.jar /opt/dinky/extends/flink${FLINK_VE

COPY --from=flink-base /opt/flink/opt/flink-table-planner*.jar /opt/dinky/extends/flink${FLINK_VERSION}/flink/

RUN mkdir /opt/dinky/customJar && chmod -R 777 /opt/dinky/ && sed -i 's/-Xms512M -Xmx2048M -XX:PermSize=512M/-XX:+UseContainerSupport -XX:InitialRAMPercentage=70.0 -XX:MaxRAMPercentage=70.0/g' ./bin/auto.sh
RUN mkdir /opt/dinky/customJar && chmod -R 777 /opt/dinky/

EXPOSE 8888

Expand Down
11 changes: 9 additions & 2 deletions script/bin/auto.sh
Original file line number Diff line number Diff line change
Expand Up @@ -179,8 +179,15 @@ PARAMS_OPT="-Ddinky.logs.path=${DINKY_LOG_PATH} -Ddinky.root.path=${APP_HOME} -D
JAR_PARAMS_OPT="--logging.config=${LOG_CONFIG}"
# JMX path
JMX="-javaagent:$APP_HOME/lib/jmx_prometheus_javaagent-0.20.0.jar=10087:$APP_HOME/config/jmx/jmx_exporter_config.yaml"
#JVM OPTS
JVM_OPTS="-Xms512M -Xmx2048M -XX:PermSize=512M -XX:MaxPermSize=1024M"
# JVM options. Override via JVM_OPTS (Docker ENV / K8s env / systemd / /etc/profile.d/dinky_env).
# PermGen (-XX:PermSize / -XX:MaxPermSize) was removed in JDK 8; do not set them here.
DEFAULT_JVM_OPTS="-Xms512M -Xmx2048M"
JVM_OPTS="${JVM_OPTS:-${DEFAULT_JVM_OPTS}}"
case "$1" in
start|startOnPending|startWithJmx|restart|restartWithJmx)
echo -e "${GREEN}JVM_OPTS : ${JVM_OPTS}${RESET}"
;;
esac

# Check whether the pid path exists
PID_PATH="${APP_HOME}/run"
Expand Down
Loading