[Bug] TaskManager Metaspace never reclaimed: ~18.2 MB leaked per finished SQL job that ships the Paimon connector as a user jar (Flink 1.20.5 session cluster)

[wangmingzhou1986]

Search before asking

• I searched in the issues and found nothing similar.

Closest existing reports, both still open, neither matching this measurement set: #2780 (Flink S3 cause TaskManager Metaspace OOM, Paimon 0.6.0 / Flink 1.18, attributes it to an unclosed S3AFileSystem) and #5998 (paimon-s3 vs flink-s3-fs-hadoop plugin conflict). On the Flink side: FLINK-28248 (open, no fix version, but the reproducer and root cause there are Beam-specific), FLINK-19005 (closed by documenting JDBC drivers as a cause — excluded here, see below), FLINK-32203 (log4j monitorInterval — excluded here, see below).

Paimon version

Paimon Flink connector 1.2.0 / 1.3.1 (paimon-flink-1.20, 52.8 MB shaded jar), paimon-s3-1.3.1.jar.

Compute Engine

Apache Flink 1.20.5, standalone session cluster on Kubernetes.
JVM: Temurin 17.0.19 (the official Flink image ships a JRE), G1GC.
TaskManager -XX:MaxMetaspaceSize=768m, JobManager 2048m.
Storage: S3 (MinIO), catalog declared as s3://warehouse/paimon with s3.endpoint / s3.path.style.access (Paimon-native S3, deliberately not s3a:// + fs.s3a.*).

Minimal reproduce step

In a long-running session cluster, submit this repeatedly (each invocation is a separate short batch job; the Paimon jar is shipped as a user jar via -j):

-- q.sql
SET 'pipeline.name' = 'probe';
SET 'execution.runtime-mode' = 'batch';
CREATE CATALOG lake WITH (
  'type'='paimon',
  'warehouse'='s3://warehouse/paimon',
  's3.endpoint'='http://minio:9000',
  's3.path.style.access'='true',
  's3.access-key'='...','s3.secret-key'='...');
SELECT COUNT(*) FROM lake.`mydb`.`small_pk_table`;

/opt/flink/bin/sql-client.sh -j /path/paimon-sql.jar -f q.sql

Table under test: Paimon primary-key table, parquet + zstd, bucket=4, 24 columns, 11 rows. All jobs finish normally (FINISHED, correct result).

What doesn't meet your expectations?

Each finished job permanently costs ~18.2 MB of TaskManager Metaspace, and it is never returned.

Controlled measurement (cluster-wide sum of Status.JVM.Memory.Metaspace.Used across all 9 TaskManagers):

step	metaspace
baseline	3262.6 MB
after 3 identical probe jobs	3317.3 MB (+54.7 MB, 18.2 MB per job)
after 90 s idle	3317.3 MB (0 MB reclaimed)

Long-run behaviour, from 15-minute polling — 3,008 samples across 48 distinct TaskManager instances:

• several TaskManagers climb monotonically from ~5 % to 86–92 % of the 768 MB cap and never recover;
• across all 48 instances only 9 self-reclamations were ever observed, each just 1.4–4.8 pp;
• the only thing that returns memory is restarting the TaskManager.

The decisive contrast — same cluster, same JVM, same GC settings:

	ClassesLoaded	ClassesUnloaded	unload ratio
JobManager	359,055	241,361	67 %
TaskManager #1	93,989	7,030	7.5 %
TaskManager #2	93,181	9,815	10.5 %
TaskManager #3	80,126	4,568	5.7 %
TaskManager #4	76,509	2,475	3.2 %
TaskManager #5	68,729	687	1.0 %
TaskManager #6	25,755	0	0 %
TaskManager #7	26,465	0	0 %
TaskManager #8	23,460	0	0 %

The JobManager in the same cluster unloads two thirds of everything it loads, and was observed reclaiming 150–180 MB at a time without any restart (restartCount=0, uptime 3 days). The TaskManagers essentially never unload. So this is not GC tuning, not the JVM, and not the G1 configuration — it is user-classloader retention that is specific to the TaskManager side.

Causes we were able to exclude (so nobody repeats the work)

• log4j monitorInterval (FLINK-32203). The log4j-console.properties actually used by JM/TM contains no monitorInterval, and a SIGQUIT thread dump shows no log4j configuration-watch thread on the TaskManager. (monitorInterval=30 exists only in log4j-cli/session.properties, i.e. the short-lived client JVM.)
• JDBC drivers (FLINK-19005). Reproduces with the Paimon jar alone; no JDBC connector or driver is involved in the query above.
• Per-job lingering threads. A TaskManager holding ~570 MB of Metaspace (≈31 retained classloaders at 18.2 MB each) has only 287 JVM threads in total, with 4 distinct s3a-* thread pools and 1 java-sdk-http-connection-reaper. Thread counts do not scale with the number of retained classloaders, so the mechanism described in [Bug] Flink S3 cause TaskManager Metaspace OOM #2780 (one unclosed S3AFileSystem per job) does not by itself account for what we see here — although an unclosed filesystem would still be a real leak and may explain part of it.
• "GC simply never ran". Worth stating explicitly because it is an easy wrong turn: Status.JVM.GarbageCollector.G1_Old_Generation.Count is 0 on every TaskManager and on the JobManager, so that metric cannot be used to argue either way. The JobManager nevertheless unloads 67 % of its classes, which proves reclamation does happen; the TaskManagers just don't benefit from it.

What we could not do, and why

We cannot name the object that retains the classloader, because the official Flink image ships a JRE — there is no jcmd, jmap or jstack in the TaskManager container, so we cannot take a heap dump or run VM.classloader_stats.

If a maintainer wants it, we are happy to run one TaskManager on a JDK-based image and attach a heap dump / VM.classloader_stats output, or to test a patch. Please just say which you'd prefer.

Why we think Paimon is the right place for this

The retained classloader is the one carrying the Paimon Flink connector jar, and the workload is nothing but "create a Paimon catalog, read one small table, exit". Everything else in the job is stock Flink. #2780 already points at Paimon's S3 filesystem lifecycle (S3AFileSystem created but never close()d), which would be exactly this class of problem; this report adds a current-version (Paimon 1.2/1.3, Flink 1.20.5) reproducer with a quantified per-job cost and the JM-vs-TM unload contrast, in case that helps narrow it down.

Impact and current mitigation

In a session cluster used for periodic small Paimon queries this is effectively unbounded: our TaskManagers reach the 768 MB cap and have to be rotated. We currently automate a rolling TaskManager restart at 85 % Metaspace, which takes ~28 s and is transparent to running CDC jobs — but it is obviously a workaround, not a fix.

Are you willing to submit a PR?

• I'm willing to submit a PR! (happy to help with reproduction, measurements, or testing a fix; we can also run the JDK-image heap dump described above)

[wangmingzhou1986]

Follow-up with the heap-level evidence I offered in the report. The retained classloaders survive forced full GCs unchanged, so this is strong reachability, not deferred collection.

How the data was obtained

The official Flink image ships a JRE, so I streamed a JDK 17 into the running TaskManager container (tar pod→pod, no restart, so the already-retained classloaders were preserved) and ran jcmd against the live JVM.

One gotcha worth recording for anyone repeating this: the HotSpot attach mechanism requires the same uid as the target JVM. kubectl exec lands as root while the TaskManager runs as flink, and attaching as root fails with target process 1 doesn't respond within 10500ms. You have to su to the JVM's owner first.

VM.classloader_stats on a TaskManager holding 570.7 MB Metaspace

classloader type	live instances	classes	ChunkSz
org.apache.flink.util.ChildFirstClassLoader	23	68,461	50.7 MB
FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader	23	0	–
org.apache.paimon.plugin.ComponentClassLoader	19	7,521	9.3 MB
org.apache.paimon.shade.org.codehaus.janino.ByteArrayClassLoader	55	55	–
jdk.internal.reflect.DelegatingClassLoader	564	564	0.3 MB
org.codehaus.commons.compiler.util.reflect.ByteArrayClassLoader	13	13	–
(total)	703 CLDs	93,193	71.6 MB

SafetyNetWrapperClassLoader is created 1:1 with a job's user-code classloader, and there are exactly 23 of them alongside 23 ChildFirstClassLoader — i.e. 23 finished jobs whose classloaders are still alive.

Forced full GC — the decisive test

Two consecutive jcmd <pid> GC.run (both reported Command executed successfully), 20 s apart:

	before GC	after 1st	after 2nd
ChildFirstClassLoader	23	23	23
SafetyNetWrapperClassLoader	23	23	23
ComponentClassLoader (Paimon)	19	19	19
shaded-janino ByteArrayClassLoader	55	41	35
total CLDs	703	412	406
Metaspace (Flink metric)	570.7 MB	–	570.7 MB

GC clearly ran and clearly collected things — total CLDs dropped 703 → 406 and the janino loaders shrank 55 → 35. But the user classloaders and the Paimon ComponentClassLoaders did not move at all, and Metaspace did not drop by a single byte.

This rules out "the JVM just hasn't collected yet" and confirms a strong-reference retention.

Two concrete suspects

1. ComponentClassLoader is rooted at the platform loader, not at a job.

jcmd VM.classloaders shows the 19 instances hanging directly off PlatformClassLoader, siblings of Flink's own PluginLoader$PluginClassLoader:

+-- "platform", jdk.internal.loader.ClassLoaders$PlatformClassLoader
      |
      +-- org.apache.paimon.plugin.ComponentClassLoader
      +-- org.apache.flink.core.plugin.PluginLoader$PluginClassLoader
      +-- org.apache.paimon.plugin.ComponentClassLoader
      +-- org.apache.paimon.plugin.ComponentClassLoader
      ...        (19 in total)

They are not children of any user classloader, they accumulate roughly 1:1 with jobs (19 vs 23), and they survive full GC — so something static is holding them. If a ComponentClassLoader (or anything it loaded) also references the job's ChildFirstClassLoader, that alone would pin the whole chain.

2. The shaded janino reflection cache holds Class objects from the user loader.

In GC.class_histogram, org.apache.paimon.shade.org.codehaus.janino.ReflectionIClass$ReflectionIMethod appears as eight separate entries (997 instances ×6, 938 ×2, …). A class name appearing multiple times in a histogram means it was loaded once per classloader and all copies are still live. Janino keeps static reflection/IClass caches, which is a textbook classloader pin when the cached Class objects come from a child loader.

One number I cannot yet explain

VM.classloader_stats accounts for 71.6 MB of chunks in total, while the Metaspace MXBean reports 570.7 MB used on the same JVM at the same moment. I'm reporting the discrepancy rather than explaining it away — possibly freed-but-unreturned chunks, or a different accounting boundary. It does not affect the conclusion above (counts, not bytes, are what stayed flat across GC).

Environment recap

Flink 1.20.5 session cluster, Temurin 17.0.19, G1, TM -XX:MaxMetaspaceSize=768m.
Workload: repeated short batch jobs, each shipping paimon-flink-1.20 (1.2.0/1.3.1) as a user jar via sql-client -j, doing CREATE CATALOG ... 'type'='paimon' on s3:// + SELECT COUNT(*) against a small primary-key table. ~18.2 MB Metaspace per finished job, never returned.

I can still produce a full heap dump with dominator/GC-root analysis if that would be more useful than the above — just say the word. Happy to test a patch as well.

[wangmingzhou1986]

I took the heap dump I offered, and it corrects the attribution in my original report. Paimon is not retaining the classloader through its own caches — but there is still a concrete, Paimon-side trigger, and I can now name the exact retaining field.

TL;DR

The dominant retainer is Hadoop's ReflectionUtils.CONSTRUCTOR_CACHE, a static final ConcurrentHashMap<Class<?>, Constructor<?>> with strong keys, living in the shared AppClassLoader (/opt/flink/lib/hadoop-common). It holds the Class object of org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec, which is loaded by the job's ChildFirstClassLoader. One entry per job, forever.

class org.apache.hadoop.util.ReflectionUtils          <- AppClassLoader (shared, /opt/flink/lib)
  .static CONSTRUCTOR_CACHE      (ConcurrentHashMap, strong keys)
    -> Node.key = class org.apache.paimon.shade.org.apache.parquet.hadoop.codec.ZstandardCodec
      -> .<classloader> = org.apache.flink.util.ChildFirstClassLoader     <- pinned forever

Hadoop's own javadoc on that field states it "pins the classes so they can't be garbage collected until ReflectionUtils can be collected". clearCache() exists but is package-private and test-only; nothing evicts in production.

Why this is reachable from Paimon

Paimon shades parquet but (correctly) does not shade Hadoop. So the shaded CodecFactory hands a user-classloader-loaded class into the shared Hadoop utility:

// parquet-hadoop CodecFactory.getCodec()
codec = (CompressionCodec) ReflectionUtils.newInstance(codecClass, ConfigurationUtil.createHadoopConfiguration(conf));
CODEC_BY_NAME.put(codecCacheKey, codec);

// hadoop-common ReflectionUtils.newInstance()
Constructor<T> meth = (Constructor<T>) CONSTRUCTOR_CACHE.get(theClass);
if (meth == null) { meth = theClass.getDeclaredConstructor(EMPTY_ARRAY); ...; CONSTRUCTOR_CACHE.put(theClass, meth); }

CodecFactory.release() clears compressors/decompressors only — it touches neither CODEC_BY_NAME nor Hadoop's CONSTRUCTOR_CACHE. So correct usage does not release it either; any job that writes/reads a zstd parquet table permanently pins its own classloader.

Measured breakdown on one TaskManager

19 live ChildFirstClassLoader instances, dumped with jcmd GC.heap_dump (1.5 s, no downtime — I used a TaskManager with 0 allocated slots):

retaining mechanism	loaders pinned
Hadoop ReflectionUtils.CONSTRUCTOR_CACHE -> shaded ZstandardCodec	9
lingering threads via Thread.inheritedAccessControlContext -> ProtectionDomain.classloader	8
java.io.ClassCache$CacheRef (SoftReference, ObjectStreamClass cache)	1

The ReflectionUtils copy in AppClassLoader holds 11 cache entries, 9 of them keyed by user-classloader classes, all of them ZstandardCodec — one per job. (There is a second ReflectionUtils copy inside PluginLoader$PluginClassLoader from flink-s3-fs-hadoop; its cache has 5 entries and pins nothing. If you reproduce this, enumerate all copies of the class — I first sampled only one and got the wrong answer.)

The lingering-thread group is mostly not Paimon: 4 × connection-pool-<mysql-host>:3306 housekeeper (HikariCP, from the Flink CDC MySQL source), 1 × java-sdk-progress-listener-callback-thread (AWS SDK), 2 × Flink's own IOManagerAsync / FileChannelManagerImpl shutdown hooks. I'm reporting those to the relevant projects separately.

A hypothesis I had, and which the heap disproves

Before dumping, I traced the source and suspected CodeGenLoader / S3Loader:

private static PluginLoader loader;   // no cleanup anywhere
private static synchronized PluginLoader getLoader() { if (loader == null) loader = new PluginLoader(...); return loader; }

combined with ComponentClassLoader holding ownerClassLoader. It looks exactly like a classloader pin. It is not one here — neither appears on any retention path. The static field lives in a class loaded by the user classloader itself, so it cannot pin that loader from outside. I'm stating this explicitly so nobody spends time on it: the 19 ComponentClassLoader instances are a consequence of the retained loaders, not the cause.

Possible fixes on the Paimon side

Since the pinned class is Paimon's own shaded class, Paimon can break the chain without waiting for Hadoop:

1. Don't route shaded codec classes through shared ReflectionUtils. ReflectionUtils.newInstance is just getDeclaredConstructor().newInstance() plus setConf; instantiating directly in Paimon's shaded CodecFactory avoids the shared strong-keyed cache entirely.
2. Or relocate org.apache.hadoop.util.ReflectionUtils into the shaded namespace as well, so the cache lives and dies with the job classloader.

Option 1 looks like a small, contained change. I'm happy to prepare and test a PR if you agree with the direction — I can verify with the same heap-dump method (the count of user-loaded keys in CONSTRUCTOR_CACHE should go to 0).

Method notes for anyone reproducing

• The container had no CAP_SYS_PTRACE and hostPID=false, so the Serviceability Agent (jhsdb revptrs) was unusable. jcmd GC.heap_dump goes through the HotSpot attach mechanism and needs no ptrace — that is the right tool here. Attach must run as the JVM's own uid (su flink), not root.
• When computing retention paths, two filters are mandatory or the answer is wrong: (a) drop referent edges of java.lang.ref.Reference subclasses (48,977 of them here) — weak/soft/phantom refs do not retain; keep FinalReference; (b) ignore ROOT_STICKY_CLASS self-rooting, since HotSpot marks every loaded class as a root, which would make any live loader trivially "reachable" through its own classes.
• One thing I cannot explain and am not going to explain away: 7 of the 19 loaders are unreachable after those filters, even though GC.heap_dump runs a full GC first. VM.classloader_stats reported 26 CLDs against 19 loader objects in the heap, so classloader-data teardown lagging object collection is consistent with it, but I have not proven that.

[zhang-arvin]

I will work on this.