8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit #26401

sendaoYan · 2025-07-19T02:08:59Z

Hi all,

The test runtime/Thread/ThreadCountLimit.java was observed fails when run with other tests. The test start subprocess with shell prefix command ulimit -u 4096 which intend to limite the usage of thread number. But this will cause test fails when this test run with other tests. I create a demo to demonstrate that.

I start a java process which will create 5k threads, and then I can not start new java process with prefix ulimit -u 4096 on the same machine.

ManyThreads.java.txt

So it's necessary to make this test run sperately to make this test success.
Change has been verified locally, test-fix only, risk is low.

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue

Issue

JDK-8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit (Enhancement - P4)

Reviewers

David Holmes (@dholmes-ora - Reviewer)

Contributors

David Holmes <[email protected]>

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/26401/head:pull/26401
$ git checkout pull/26401

Update a local copy of the PR:
$ git checkout pull/26401
$ git pull https://git.openjdk.org/jdk.git pull/26401/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 26401

View PR using the GUI difftool:
$ git pr show -t 26401

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/26401.diff

Using Webrev

Link to Webrev Comment

…vely

bridgekeeper · 2025-07-19T02:09:46Z

👋 Welcome back syan! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2025-07-19T02:10:33Z

@sendaoYan This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit

Co-authored-by: David Holmes <[email protected]>
Reviewed-by: dholmes

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 50 new commits pushed to the master branch:

ad510fb: 8338977: Parallel: Improve heap resizing heuristics
2292246: 8350621: Code cache stops scheduling GC
03e9ea1: 8358530: Properties#list should warn against non-String values
... and 47 more: https://git.openjdk.org/jdk/compare/d83346dcff0824575d580ec421476c0ea5c6e783...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

openjdk · 2025-07-19T02:11:12Z

@sendaoYan The following label will be automatically applied to this pull request:

hotspot-runtime

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2025-07-19T02:15:36Z

Webrevs

dholmes-ora · 2025-07-21T02:23:39Z

@sendaoYan exclusiveAccess.dirs does not work the way you expect/require. It simply indicates that only one test at a time may run from the given directory. It does not mean no other tests from any other directories may run.

dholmes-ora · 2025-07-21T02:29:37Z

FWIW we see no issue running this test, but we ensure we already have a high ulimit setting available in our test machines by default.

sendaoYan · 2025-07-21T09:00:03Z

FWIW we see no issue running this test, but we ensure we already have a high ulimit setting available in our test machines by default.

Maybe this test has been excluded by TEST.groups
The error reported in this testcase should not be related to the ulimit configuration of the test environment, but may be related to the number of CPU cores of the machine. On a machine with a large number of CPU cores, each testcase will start more gc threads and JIT threads, and the number of jtreg concurrency will also be relatively large, causing the total number of threads of all testcases to easily exceed 4096. For example, in the example below, my environment configuration ulimit -u is unlimited. I first start a background java process, which will start 5000 threads and will not exit; then I use shell predix ulimit -u to start the java process (similar to the test situation of this testcase), and then I cannot start java.

ManyThreads.java.txt

…t/hotspot/jtreg/resourcehogs/runtime/Thread/

sendaoYan · 2025-07-21T09:20:13Z

@sendaoYan exclusiveAccess.dirs does not work the way you expect/require. It simply indicates that only one test at a time may run from the given directory. It does not mean no other tests from any other directories may run.

Thanks your correction @dholmes-ora.
I have move this test to test/hotspot/jtreg/resourcehogs, similar to JDK-8227645.

dholmes-ora · 2025-07-21T11:59:00Z

I first start a background java process, which will start 5000 threads and will not exit; then I use shell predix ulimit -u to start the java process (similar to the test situation of this testcase), and then I cannot start java.

Okay, but in that scenario what is it you are actually running out of?

You are changing the test to suit the way you need to run it, but I'm not aware of anyone else reporting issues running this test.

sendaoYan · 2025-07-21T13:30:47Z

Okay, but in that scenario what is it you are actually running out of?

I think it's running out of "user processes" which limit by ulimit -u 4096.

I think it is the user processes set by ulimit -u are exhausted that Java cannot start.
I created a small example in C language to illustrate this problem. The create-thread program will try to create threads continuously until it can no longer create threads, or the number of threads created exceeds 5000.

Use bash -c "ulimit -u 1000 && ./create-thread" command shows that the number of threads that the create-thread program can create is about 580;

First directly start the create-thread program (background running mode), and then use bash -c "ulimit -u 4096 && ./create-thread" command to test the number of threads that the second create-thread process can create. It will be found that the second create-thread process cannot create any thread. Explain that the number of max user processes limited by ulimit -u 4096 includes the number of threads created by the first create-thread process.
This C language example shows that this testcase is not suitable for concurrent running with other test cases, otherwise we may encounter the failure described by the issue

#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>

void *thread_func() {
    while (1) {
        sleep(1);
    }
    return NULL;
}

int main() {
    pthread_t tid;
    int thread_count = 0;
    int ret;

    while (thread_count < 5000) {
        ret = pthread_create(&tid, NULL, thread_func, NULL);
        if (ret != 0) {
            if (ret == EAGAIN) {
                printf("can not create thread(EAGAIN) anymore: %d\n", thread_count);
                break;
            } else {
                printf("pthread_create error: %s\n", strerror(ret));
                break;
            }
        }
        thread_count++;
        if (thread_count % 1000 == 0) {
            printf("already create thread number %d\n", thread_count);
        }
    }
    printf("total created thread number: %d\n", thread_count);

    while (1) {
        sleep(1);
    }

    return 0;
}

You are changing the test to suit the way you need to run it, but I'm not aware of anyone else reporting issues running this test.

I think the failure descripted by issue, only appearance on huge CPU core number machine.

dholmes-ora · 2025-07-21T21:20:19Z

Explain that the number of max user processes limited by ulimit -u 4096 includes the number of threads created by the first create-thread process.

That's not the way ulimit should work in different sub-shells. What is the ulimit in the parent shell? I think the subshells are limited by the parent.

sendaoYan · 2025-07-22T01:59:30Z

That's not the way ulimit should work in different sub-shells.

I initially thought that ulimit shouldn't work like that in different sub-shells. But actually ulimit works in different sub-shells as unexpectedly.
The testcase runtime/Thread/ThreadCountLimit.java attempts to limited the number user processes of 4096 by adding the prefix "bash -c ulimit -u 4096" to start the child process, but the actual situation is that ulimit does not work as expected. If this testcase run with other tests simultaneously, the number of threads can created maybe be zero, at least the number always less than 4096, it depends how many user processes has been created in the test machine.

What is the ulimit in the parent shell? I think the subshells are limited by the parent.

The ulimit in the parent shell is unlimited. The first process "./create-thread" can create 5k threads shows that the parent shell has no limit.

dholmes-ora · 2025-07-22T05:28:28Z

There is definitely something unexpected/odd about the behaviour of ulimit when used in this way, though I do not observe the exact problems you describe unless I run a number of test processes concurrently - which is simply demonstrating machine overloading.

First, what does it even mean to use ulimit -u? The manpage says it limits the maximum number of processes the user can create - it doesn't say "per shell" (and setrlimit confirms this). But you can easily demonstrate that the user can create far more processes/threads than have been set by a ulimit command running in another shell. So perhaps there is something else that affects how ulimit works, and that something is different between our systems and yours. ?? (I know there are capabilities that disable the limit but I couldn't see any indication such capabilities were present.)

Second, I observe that with ulimit -u 1024 I can't even run java-version - which makes no sense in terms of number of threads created. Relatedly with a 4096 limit the test typically can only create around 2500 threads - so where did the other 1500+ go?

The use of ulimit was added to the test, for Linux only, because we found we could exhaust other resources that could then cause fatal errors in the VM in unexpected places - rather than the failure of pthread_create that we were trying to induce.

I'm really not sure how to proceed here. The change you propose affects all platforms, but there is only an issue for you on Linux.

sendaoYan · 2025-07-22T09:58:53Z

Hi @dholmes-ora

which is simply demonstrating machine overloading.

I think it's not machine overloading, becasuse the setting of 'ulimit -u' on my machine is 'unlimited'. I can create 5000 threads many times, show as below:

you can easily demonstrate that the user can create far more processes/threads than have been set by a ulimit command running in another shell

I think the 'ulimit -u' in sub-shell take effect in the sub-shell only, it's temporary setting, it will not affect the parent shell.

Relatedly with a 4096 limit the test typically can only create around 2500 threads - so where did the other 1500+ go?

It seems that the sub-shell with 'ulimit -u 4096' prefix will count all the user processes number. It's just my speculatation. That's why this test not suitable run with other tests simultaneous

Anyway, I change this PR to use docker run --pids-limit 4096 to instead the original 'ulimit -u 4096'. It will make this test more complict but more elegant and more robustness.

… 4096'

dholmes-ora · 2025-07-22T12:07:31Z

You can't just change the test to use docker! This is not a container test. We use special test tasks to run container tests in an environment where containers are enabled.

This reverts commit 374c297.

…limit -u 4096'" This reverts commit f993419.

dholmes-ora · 2025-07-22T12:13:44Z

I think the 'ulimit -u' in sub-shell take effect in the sub-shell only, it's temporary setting, it will not affect the parent shell.

I'm finding some of these statements to be contradictory to the problem being stated. If the ulimit setting only affects the sub-shell then it can't cause other concurrent tests to hit the limit and fail to create threads!

It seems that the sub-shell with 'ulimit -u 4096' prefix will count all the user processes number. It's just my speculatation. That's why this test not suitable run with other tests simultaneous

If the sub-shell counts all processes/threads belonging to the user and applies the new ulimit then that would make some sense. But again how does that then cause any problem in a different shell?

sendaoYan · 2025-07-22T12:14:47Z

You can't just change the test to use docker! This is not a container test. We use special test tasks to run container tests in an environment where containers are enabled.

Okey, I have revert the docker commit.

sendaoYan · 2025-07-22T12:39:42Z

If the ulimit setting only affects the sub-shell then it can't cause other concurrent tests to hit the limit and fail to create threads!

Maybe some of my previous statements have caused some misunderstandings.
The usage of ulimit in this testcase will not cause other concurrent tests to hit the limit, but will cause this test itself do not have enough user processes to start the java.
On the huge core number machine, every test will create more JIT compiler threads and more GC work threads. So when this test run with other tests simultancely, we can see this test can not start subprocess java with prefix "ulimit -u", the subprocess java report Failed to start thread "GC Thread#0", because the subprocess has limited by "ulimit -u 4096", and the user processes resources maybe has been occupied by other tests which run simultancely. And the other tests run normally, because they do not have 'ulimit -u' explicitly.

dholmes-ora · 2025-07-22T21:41:44Z

The usage of ulimit in this testcase will not cause other concurrent tests to hit the limit, but will cause this test itself do not have enough user processes to start the java.

Sorry - right I get it now. So basically with enough other activity on the machine all the 4096 process/thread capacity may have already been used up before the test can run. It isn't this test that is the "resource hog" as such. What we really want to do is more like ulimit -u <current process/thread count> + 4096 but getting the current value is tricky.

One possible solution would be to loop so that if we get exitcode 1 then we retry with 4096*2 etc.

      if (Platform.isLinux()) {
        // On Linux this test sometimes hits the limit for the maximum number of memory mappings,
        // which leads to various other failure modes. Run this test with a limit on how many
        // threads the process is allowed to create, so we hit that limit first. What we want is
        // for another "limit" processes to be available, but ulimit doesn't work that way and
        // if there are already many running processes we could fail to even start the JVM properly.
        // So we loop increasing the limit until we get a successful run. This is not foolproof.
        int pLimit = 4096;
        final String ULIMIT_CMD = "ulimit -u ";
        ProcessBuilder pb = ProcessTools.createTestJavaProcessBuilder(ThreadCountLimit.class.getName());
        String javaCmd = ProcessTools.getCommandLine(pb);
        for (int i = 1; i <= 10; i++) {
            // Relaunch the test with args.length > 0, and the ulimit set
            String cmd = ULIMIT_CMD + Integer.toString(pLimit * i) + " && " + javaCmd + " dummy";
            System.out.println("Trying: bash -c " + cmd);
            OutputAnalyzer oa = ProcessTools.executeCommand("bash", "-c", cmd);
            int exitValue = oa.getExitValue();
            switch (exitValue) {
              case 0: System.out.println("Success!"); return;
              case 1: System.out.println("Retry ..."); continue;
              default: oa.shouldHaveExitValue(0); // generate error report
            }
        }
        throw new Error("Failed to perform a successful run!");
      } else {
        // Not Linux so run directly.
        test();
      }

Took 5 loops to run on my system (ulimit -u 24576) when I had an unrestricted version of the test running which has about 20700 live threads.

sendaoYan · 2025-07-23T02:28:08Z

Thanks @dholmes-ora very much.

I will verified this patch by ours CI.

/contributor add @dholmes-ora

openjdk · 2025-07-23T02:29:04Z

@sendaoYan
Contributor David Holmes <[email protected]> successfully added.

dholmes-ora

Looks good to me :)

I will run this through our CI as well.

dholmes-ora · 2025-07-23T07:28:03Z

Testing in our CI was fine - test passes on first iteration with 4096 in all case. I ran this standalone and within tier5.

sendaoYan · 2025-07-23T11:35:16Z

Testing in our CI was fine - test passes on third iteration with 12288 in all case. I run this within all the jtreg tests.

ThreadCountLimit_id0.jtr.txt

sendaoYan · 2025-07-23T11:43:35Z

/issue JDK-8359827

openjdk · 2025-07-23T11:43:41Z

@sendaoYan This issue is referenced in the PR title - it will now be updated.

sendaoYan · 2025-07-24T01:47:00Z

Thanks your patient reviews and suggestions @dholmes-ora.

/integrate

openjdk · 2025-07-24T01:48:01Z

Going to push as commit fc80384.
Since your change was applied there have been 52 commits pushed to the master branch:

b746701: 8363898: RISC-V: TestRangeCheckHoistingScaledIV.java fails after JDK-8355293 when running without RVV
2da0cda: 8363895: Minimal build fails with slowdebug builds after JDK-8354887
ad510fb: 8338977: Parallel: Improve heap resizing heuristics
... and 49 more: https://git.openjdk.org/jdk/compare/d83346dcff0824575d580ec421476c0ea5c6e783...master

Your commit was automatically rebased without conflicts.

openjdk · 2025-07-24T01:48:10Z

@sendaoYan Pushed as commit fc80384.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

dholmes-ora · 2025-07-24T06:43:41Z

@sendaoYan any hotspot related changes, including tests, require two reviews before integration.

It is probably prudent to wait for any feedback from other CI maintainers before backporting this change.

sendaoYan · 2025-07-24T07:26:34Z

@sendaoYan any hotspot related changes, including tests, require two reviews before integration.

It is probably prudent to wait for any feedback from other CI maintainers before backporting this change.

Sorry, I will pay attention next time.

8359827: Test runtime/Thread/ThreadCountLimit.java should run exclusi…

2b42b7c

…vely

openjdk bot added the rfr Pull request is ready for review label Jul 19, 2025

openjdk bot added the hotspot-runtime [email protected] label Jul 19, 2025

add test/hotspot/jtreg/runtime/Thread/stress/TEST.properties

12a1b4c

update TEST.groups

3c3c697

mv test/hotspot/jtreg/runtime/Thread/stress/ThreadCountLimit.java tes…

5d6c69b

…t/hotspot/jtreg/resourcehogs/runtime/Thread/

Use 'docker run --pids-limit 4096' to instead the original 'ulimit -u…

f993419

… 4096'

openjdk bot removed the rfr Pull request is ready for review label Jul 22, 2025

Remove extra whitespaces

374c297

openjdk bot added the rfr Pull request is ready for review label Jul 22, 2025

SendaoYan added 2 commits July 22, 2025 20:13

Revert "Remove extra whitespaces"

809d3b1

This reverts commit 374c297.

Revert "Use 'docker run --pids-limit 4096' to instead the original 'u…

77884c2

…limit -u 4096'" This reverts commit f993419.

Increase ulimit -u value to try 10 times

7b86e5b

SendaoYan added 2 commits July 23, 2025 10:31

Revert change to TEST.groups

f49733d

Revert change to TEST.groups

00cd0ab

dholmes-ora reviewed Jul 23, 2025

View reviewed changes

openjdk bot changed the title ~~8359827: Test runtime/Thread/ThreadCountLimit.java should run exclusively~~ 8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit Jul 23, 2025

dholmes-ora approved these changes Jul 23, 2025

View reviewed changes

openjdk bot added the ready Pull request is ready to be integrated label Jul 23, 2025

openjdk bot added the integrated Pull request has been integrated label Jul 24, 2025

openjdk bot closed this Jul 24, 2025

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Jul 24, 2025

sendaoYan deleted the jbs8359827 branch July 24, 2025 01:56

8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit #26401

8359827: Test runtime/Thread/ThreadCountLimit.java need loop increasing the limit #26401

Conversation

sendaoYan commented Jul 19, 2025 • edited by openjdk bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress

Issue

Reviewers

Contributors

Reviewing

Uh oh!

bridgekeeper bot commented Jul 19, 2025

Uh oh!

openjdk bot commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openjdk bot commented Jul 19, 2025

Uh oh!

mlbridge bot commented Jul 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Webrevs

Uh oh!

dholmes-ora commented Jul 21, 2025

Uh oh!

dholmes-ora commented Jul 21, 2025

Uh oh!

sendaoYan commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sendaoYan commented Jul 21, 2025

Uh oh!

dholmes-ora commented Jul 21, 2025

Uh oh!

sendaoYan commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora commented Jul 21, 2025

Uh oh!

sendaoYan commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora commented Jul 22, 2025

Uh oh!

sendaoYan commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora commented Jul 22, 2025

Uh oh!

dholmes-ora commented Jul 22, 2025

Uh oh!

sendaoYan commented Jul 22, 2025

Uh oh!

sendaoYan commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dholmes-ora commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sendaoYan commented Jul 23, 2025

Uh oh!

openjdk bot commented Jul 23, 2025

Uh oh!

dholmes-ora left a comment

Choose a reason for hiding this comment

Uh oh!

dholmes-ora commented Jul 23, 2025

Uh oh!

sendaoYan commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sendaoYan commented Jul 23, 2025

Uh oh!

openjdk bot commented Jul 23, 2025

Uh oh!

sendaoYan commented Jul 24, 2025

Uh oh!

openjdk bot commented Jul 24, 2025

Uh oh!

openjdk bot commented Jul 24, 2025

Uh oh!

sendaoYan commented Jul 19, 2025 •

edited by openjdk bot

Loading

openjdk bot commented Jul 19, 2025 •

edited

Loading

mlbridge bot commented Jul 19, 2025 •

edited

Loading

sendaoYan commented Jul 21, 2025 •

edited

Loading

sendaoYan commented Jul 21, 2025 •

edited

Loading

sendaoYan commented Jul 22, 2025 •

edited

Loading

sendaoYan commented Jul 22, 2025 •

edited

Loading

sendaoYan commented Jul 22, 2025 •

edited

Loading

dholmes-ora commented Jul 22, 2025 •

edited

Loading

sendaoYan commented Jul 23, 2025 •

edited

Loading