[kgeisz]

https://issues.apache.org/jira/browse/HBASE-30087

AI Usage

Some changes were generated using Claude Opus 4.6.

Summary

HBase's Read-Replica feature was merged into master in PR #8044. This pull request adds Jenkins integration tests for Read-Replica.

The typical HBase mini-cluster integration test framework cannot be used to test the Read-Replica feature because each cluster in a read-replica setup must have a unique meta table name. These mini-clusters run in the same JVM and end up sharing the same static META_TABLE_NAME variable. (PR #7730 is an open pull request to make the meta table name non-static.)

Instead, these integration tests use two Docker containers running HBase that are part of a read-replica setup. An HBase Docker client uses Python to send HBase commands to the HBase shell within each container. This allows us to run each HBase cluster in its own isolated JVM.

The Read-Replica integration tests reside in dev-support/read-replica. The tests have their own Jenkinsfile that runs the hbase_nightly_read_replica_test.sh driver script, which does the following:

1. Builds a Docker test environment image using dev-support/docker/Dockerfile and starts this container. The container helps with portability for running tests either locally or in Jenkins.

Once the test environment container is up and running, the driver script runs run_read_replica_integration_tests.sh inside of this container, which does the following:

2. Sets up the test environment - creates Python env, compile Protobuf, etc.
3. Builds the hbase-docker image, which is used to establish a read-replica setup between two Docker containers running HBase.
4. Starts the hbase-docker containers using docker compose. These containers are run alongside the test env container by bind-mounting the host's Docker socket (a Docker-outside-of-Docker (DooD) setup)
5. Uses pytest to run several integration tests
6. Creates a test report to be published by Jenkins (similar to the Nightly Test's console report)

These integration tests have been run in Jenkins against this feature branch (HBASE-30087). We now would like to merge this feature branch to the master so the Read-Replica feature can continue to be tested against new changes to HBase.

Results Examples

Previous runs against the HBASE-30087 branch, including successful runs, can be found here:

https://ci-hbase.apache.org/job/HBase-Read-Replica-Tests/job/HBASE-30087/

Here is an example of a console report from a successful test run:

Here is an example of a failed run. When a test fails, the test's logs are linked along with the HBase logs from each read-replica Docker container:

[anmolnar]

lgtm. 🚢

[taklwu]

the changes have been reviewed in https://github.com/apache/hbase/tree/HBASE-30087

although the dev email has been asking if we should have nightly migrated to github action, IMO we could have a separate story / JIRA for evaluation.

[anmolnar]

Co-authored by Claude (Anthropic) on behalf of @anmolnar

Took a closer look after my earlier approval. The overall approach is solid: running each cluster in its own container is a reasonable way around the static META_TABLE_NAME until #7730 lands, and the failure report with per-test HBase logs is really useful. There are a few things I'd like addressed before merge, plus some that could be follow-ups.

Before merge

1. Tests modify tracked files and strip their license headers. set_hbase_conf_property_value rewrites cluster{1,2}/conf/hbase-site.xml in the checkout. ElementTree.parse() drops XML comments by default, so the first run removes the ASF header from both files and leaves the tree dirty. A failed run also leaves the read-only flags flipped for the next run. Could we copy the configs into a scratch dir (e.g. under tmp-read-replica-data) and mount those instead?

2. Fixed container and compose project names. hbase-docker / hbase-docker-2 and the default compose project are global on the Docker host. disableConcurrentBuilds() only applies within a single job, so the master and branch-3 jobs, or any other job on the same agent, can collide. If the 2h timeout kills the outer docker run, the inner trap cleanup never runs, and cleanup_host only removes the image. The containers then leak and the next build fails on a name conflict. Suggest deriving COMPOSE_PROJECT_NAME and the container names from BUILD_TAG, and having the outer cleanup also run docker compose down or remove containers by label.

3. Duplicate JIRA comments. The post block reuses the main nightly's jiraComment logic with the same "Results for branch master … +1/-1 overall" header. Every JIRA committed to master would get a second comment that's indistinguishable from the real nightly's. Can we drop the JIRA commenting, or at least label it clearly as Read-Replica checks?

Can be follow-ups

• read-replica/Dockerfile builds HBase twice. Stage 0 runs mvn clean install on the full source, and stage 1 runs mvn clean package … assembly:single on the same source. The cache stage gets invalidated by COPY ./hbase anyway. Also, because docker build runs on the host daemon, the -m/--m2 mount never reaches it, so every run re-downloads all dependencies. A single build stage with a BuildKit --mount=type=cache,target=/root/.m2 would roughly halve build time.
• -j/--java-version has no effect on what's tested. It only sets JAVA_HOME for the mvn clean step; the build and runtime always use ubi8/openjdk-17. Either wire it through as a build arg or remove it.
• Sleeps and races. major_compact_and_wait can see NONE before the compaction starts. The split assertions rely on time.sleep(5), and the dual-active test sleeps a fixed 60s per iteration. Combined with the class-level @pytest.mark.flaky(reruns=2), a real race in the feature can pass on the third attempt and show green. I'd prefer polling for the expected state and narrowing the reruns.
• Shared CI image. Adding docker.io / docker-compose-v2 to dev-support/docker/Dockerfile puts them in every nightly image. The docker.io='20.10.*' pin also selects jammy's original 20.10.12 rather than the patched version in jammy-updates. Might be worth a separate stage or image.
• Docker socket mount. Mounting /var/run/docker.sock on ASF agents is probably worth a quick check with Infra.
• Smaller robustness items:
  • The exec timeout abandons the thread, but hbase shell keeps running in the container.
  • hbase shell -n <<< "{cmd}" breaks on ", $ or backticks.
  • list_tables uses ast.literal_eval on the first [...] in the output, which breaks on bracketed log lines.
  • bulkload.sh lacks set -e, so a failed ImportTsv still runs completebulkload.
  • The mvn clean in build-images.sh is redundant after rsync --exclude target.
• Nits:
  • It's called "nightly" but uses the every-3-days cron.
  • The getCronParams cases for branch-2 and other branches are unreachable given the when clause.
  • The client mixes logging.info and logger.info.
  • The docstring path in test_dual_active_cluster_startup.py points to scripts/ instead of test/.
  • num_regions/2 passes a float.
  • Help text typo: "by the refreshing meta".

Happy to re-approve once 1–3 are sorted. The rest could go into a follow-up JIRA if you'd rather not hold this up.

[kgeisz]

Hey @anmolnar, I made some changes based on your review. I addressed the 3 major "Before merge" points, as well as the minor points in the "Smaller robustness items" and "Nits" sections. Some notes on the changes:

• Cluster conf files are now moved to the data store root directory, where they have their hostnames modified and are used by their respective Docker container.
  • Conf files are moved and set up by Python before starting the Docker containers.
  • Conf files are deleted along with the data store root directory after the containers are stopped.
  • Addresses 1 and 2 in "Before Merge" section of review.
• Added ORIG_ACTIVE_CLUSTER_CONF_DIR and ORIG_REPLICA_CLUSTER_CONF_DIR config values to the .env file.
  • The ACTIVE_CLUSTER_CONF_DIR and REPLICA_CLUSTER_CONF_DIR config values still exist, but now point to the data store root directory.
• Updated test results Jira comment with "Read-Replica Jenkins Test Results for branch..."
  • Addresses 3 in "Before Merge" section of review.
• Removed branch-2 from getCronParams() in the read-replica Jenkinsfile since the feature will not be back-ported to this branch.
  • Also, branch-3 is now tested on a different day than master.

[anmolnar]

Co-authored by Claude (Anthropic) on behalf of @anmolnar

Thanks @kgeisz, this looks much better. Moving the configs to a scratch dir and naming everything from BUILD_TAG takes care of the two things I was most worried about, and the quoted heredoc for hbase shell is a nice fix. A few things I noticed in the update:

1. _kill_orphaned_processes won't match anything. The image is a binary install (lib/ruby exists), so bin/hbase starts the shell as org.jruby.JarBootstrapMain, not org.jruby.Main. The pkill -f 'org.jruby.Main.*shell' pattern finds nothing and the timed-out shell keeps running. Matching on JarBootstrapMain should fix it.

2. list_tables returning [] when it can't parse the output. This makes assert_table_does_not_exist pass whenever the shell output is unexpected (an error, a startup failure, a format change). I'd raise an exception that includes the raw output instead, so a bad result fails loudly.

3. zoo.cfg is no longer read. _replace_hostnames_in_configs only rewrites hbase-site.xml, so in CI the copied zoo.cfg still says hbase-docker. That's harmless, because master doesn't read zoo.cfg at all (nothing in the Java sources refers to it). But that means cluster{1,2}/conf/zoo.cfg are dead files that make it look like they matter. Can we delete them, or rewrite them too if I've missed somewhere they're used?

4. The README says local runs use hbase-docker / hbase-docker-2, but run_read_replica_integration_tests.sh sets HBASE_CONTAINER_NAME=hbase-local when BUILD_TAG is unset. Only running pytest directly gets the .env default. Small doc fix.

5. Hostname length (minor). Container names and hostnames come from the full BUILD_TAG, e.g. hbase-jenkins-HBase-Read-Replica-Tests-<branch>-<n>-2. A long branch name can push that past the 63-character limit for a hostname. Truncating it, or using a short hash of BUILD_TAG, would avoid that.

6. Leftover stacks (optional). Because each build now gets a unique project name, a stack leaked by an earlier SIGKILLed build won't collide with later ones, but nothing removes it either. Removing leftover containers whose name starts with rr-jenkins- at the start of each run would keep the agent clean.

Items 1 and 2 are the ones I'd want fixed in this PR. The rest, plus the earlier follow-ups (single-stage Dockerfile and .m2 cache, -j doing nothing, sleeps and blanket reruns, the docker packages in the shared CI image), are fine as a follow-up JIRA if you can link it here. With that I'm happy to re-approve.

[kgeisz]

Hey @anmolnar, I have addressed the first three points in your most recent review

For 1 and 2, I addressed them as suggested - Kill org.jruby.JarBootstrapMain rather than org.jruby.Main.*shell, and throw an error if the list of tables cannot be parsed.

For 3, I chose to keep the zoo.cfg file and have the hostname get modified when hbase-site.xml gets modified.

I will make a Jira ticket and address the other comments in another PR.