[Proposal] Remove gpcontrib/gpmapreduce starting from Cloudberry 3.0

[tuhaihe]

Proposers

Dianjin Wang (@tuhaihe)

Proposal Status

Under Discussion

Abstract

This proposal suggests removing gpmapreduce from Apache Cloudberry, starting
with the 3.0 development line.

Concretely, on main (currently 3.0.0-devel) we would delete:

• the gpcontrib/gpmapreduce/ directory (44 files, ~13,500 lines),
• the gpmapreduce client binary that is currently installed into $GPHOME/bin/,
• the --enable-mapreduce configure option and the unused USE_MAPREDUCE define,
• the gpmapreduce_help file shipped by gpMgmt/doc/.

The 2.x line (REL_2_STABLE, currently 2.2.0-incubating) would not change.
gpmapreduce keeps building and shipping there for the entire supported life of
2.x, so users who still depend on it have a supported place to stay. The removal
takes effect in 3.0 and remains in effect for 3.x.

Motivation

1. This functionality is no longer needed according to the commit 4d926ede502

Commit 4d926ede502 (2024-09-19) disabled the gpmapreduce regression test in
gpcontrib/Makefile, with the following commit message:

As gpmapreduce is implemented for compatibility of execution of mapreduce,
which is no longer need now, we disable gpmapreduce regress test in this commit.

However, only the test was turned off. The build was not. Our release build
script still passes --enable-mapreduce
(devops/build/automation/cloudberry/scripts/configure-cloudberry.sh), and
gpcontrib/gpmapreduce/Makefile still installs the binary into
$GPHOME/bin/gpmapreduce.

2. Upstream removed it three years ago

Greenplum deleted the same code in 6bc9129c90b (2023-07-13):

It's decided that this functionality will not be provided for GPDB7 and
forward. Hence, deleting the code for the same.

3. The code is effectively frozen

git log gpcontrib/gpmapreduce shows that every commit since the initial
Cloudberry code dump has been mechanical: rebranding, typo fixes, Python 3
migration, and PostgreSQL merge fallout. The last functional change dates back
to the Greenplum era.

4. Users have better alternatives today

The workloads gpmapreduce was designed for in 2008 are well covered by plain SQL
plus UDFs (PL/pgSQL, PL/Python, PL/Perl, PL/Container), external tables /
gpfdist, and FDWs. We are not aware of active users of the YAML job-spec
interface, and this proposal is partly a call for such users to speak up.

Implementation

The change is confined to build glue and one self-contained directory. Proposed
as a single PR with atomic commits:

1. Stop building and shipping gpmapreduce — drop --enable-mapreduce from
   devops/build/automation/cloudberry/scripts/configure-cloudberry.sh (and
   update the feature comment at the top of that file), and remove the
   mapreduce target plus the install / clean / distclean branches from
   gpcontrib/Makefile.
2. Delete gpcontrib/gpmapreduce/ — 44 files, ~13,500 lines.
3. Remove the configure surface — --enable-mapreduce, USE_MAPREDUCE, the
   libyaml/yaml.h checks that are specific to mapreduce, and the
   mapreduce-specific Perl requirement, in configure.ac; regenerate
   configure; drop #undef USE_MAPREDUCE from src/include/pg_config.h.in.
4. Remove the help file — gpMgmt/doc/gpmapreduce_help and its entry in
   gpMgmt/doc/Makefile.
5. Drop the RAT exclusion — remove <exclude>gpcontrib/gpmapreduce/**</exclude>
   from pom.xml.

Rollout/Adoption Plan

Version plan

Line	Behaviour
2.x (REL_2_STABLE, 2.2.0-incubating and later 2.x releases)	No change. gpmapreduce continues to be built and shipped. Users who still depend on it stay on 2.x for the supported life of that line.
3.0 (main, 3.0.0-devel)	Removed.
3.x	Stays removed; not re-introduced.

This gives users an explicit, supported landing spot rather than an abrupt
removal, while keeping the 3.x tree clean from the start of the line.

Open questions for the community

1. Is anyone still running gpmapreduce in production? If so, on which
   Cloudberry version, and what is the migration blocker?

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[jianlirong]

I agree with this proposal.

From our perspective, gpmapreduce is a legacy component with very limited usage today, and we are not aware of any significant active user base relying on it.

More importantly, the data ecosystem has changed considerably since gpmapreduce was originally introduced. With the growing adoption of the Lakehouse architecture, there are now many more mature approaches for data interoperability between MPP databases and big data platforms, including external tables, FDWs, open table formats, and various compute/storage integration solutions.

Compared with these newer approaches, gpmapreduce does not have any obvious advantage, either in terms of performance or user adoption. Its original use cases can generally be addressed more effectively through modern data integration and interoperability mechanisms.

I also think this is an important part of keeping Apache Cloudberry moving forward. As an actively evolving open-source project, we need to not only continuously add support for new technologies and emerging use cases, but also periodically remove legacy components that no longer provide sufficient value. Otherwise, the codebase will inevitably become more complex and the maintenance burden will continue to grow.

This is a common pattern in mature open-source infrastructure projects. For example, Linux continues to add support for new hardware architectures and platforms while also removing support for obsolete hardware and platforms when they are no longer widely used. This allows the project to evolve without carrying unnecessary legacy code indefinitely.

In that sense, removing gpmapreduce is not just about removing an old feature; it is part of keeping Cloudberry's architecture and codebase focused on technologies that are relevant to its current and future users.

Therefore, I support removing gpcontrib/gpmapreduce starting from Cloudberry 3.0. Keeping it in the 2.x line for existing users, as proposed, also seems like a reasonable migration path.

Lirong