ConcurrencyError: No message available

[fedenusy]

I use futures to load data from my DB in parallel. Then I wait for all futures to complete. Most of the time this works just fine, but occasionally I see the following error:

ConcurrencyError: No message available
File "com/concurrent_ruby/ext/SynchronizationLibrary.java" line 198 in ns_wait
File "/home/app/vendor/bundle/jruby/2.2.0/gems/concurrent-ruby-1.0.1-java/lib/concurrent/synchronization/abstract_lockable_object.rb" line 43 in ns_wait_until
File "/home/app/vendor/bundle/jruby/2.2.0/gems/concurrent-ruby-1.0.1-java/lib/concurrent/atomic/event.rb" line 67 in block in wait
File "com/concurrent_ruby/ext/SynchronizationLibrary.java" line 174 in synchronize
File "/home/app/vendor/bundle/jruby/2.2.0/gems/concurrent-ruby-1.0.1-java/lib/concurrent/atomic/event.rb" line 64 in wait
File "/home/app/vendor/bundle/jruby/2.2.0/gems/concurrent-ruby-1.0.1-java/lib/concurrent/concern/obligation.rb" line 75 in wait
File "/home/app/vendor/bundle/jruby/2.2.0/gems/concurrent-ruby-1.0.1-java/lib/concurrent/concern/obligation.rb" line 66 in value

What does this error mean, and how can I fix it?

---

Tasks

• fix edge futures - @pitr-ch
• audit rest of the gem - @pitr-ch
• when source of interruption is identified, handle ConcurrencyError properly in LockableObject - @pitr-ch

[pitr-ch]

Could you share whole backtrace and possibly a reproducer? Which version of JRuby are you using?

[fedenusy]

@pitr-ch I'm using JRuby 9.0.1.0. In case it's helpful, this is happening in a Rails app, running atop torquebox-web and openjdk-8-jre-headless.

I'm not able to reproduce this reliably. It only seems to happen in production, when my servers are under heavy load.

Regarding the backtrace, my original comment includes all relevant lines. I could go back further but that probably wouldn't be helpful to you (it's just my app code). Probably more helpful if I give you a rough sketch of what I'm doing:

futures = [
  Concurrent::Future.execute { blocking_db_call_1 },
  Concurrent::Future.execute { blocking_db_call_2 },
  Concurrent::Future.execute { blocking_db_call_3 },
  Concurrent::Future.execute { blocking_db_call_4 }
]

futures.each do |future|
  future.value # block until the future completes
  raise(future.reason) if future.rejected?
end

It's one of the calls to future.value that comes before the other lines in my original comment's backtrace.

[pitr-ch]

@fedenusy Thank you, I'll have a look.

[pitr-ch]

Update: Which version of torquebox-web? Never-mind.

[pitr-ch]

please try to use master with following commit d634acf, the issue should disappear.

[fedenusy]

Tysm Petr. I'll try that patch in production tomorrow and let you know how
it goes.

Currently using torquebox-web 4.0.0.beta1.

On Sun, Jun 19, 2016 at 11:47 AM Petr Chalupa notifications@github.com
wrote:

please try to use master with following commit d634acf
d634acf,
the issue should disappear.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#539 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ABAfC2T5HJu_LZvlMjTakiNV8ROgeUZFks5qNY7YgaJpZM4I2gYO
.

[jdantonio]

If this fixes the issue we can publish a new release next week. We have the C extension fix in master that needs to go out, too.

[fedenusy]

I tried the following line in my Gemfile:

gem 'concurrent-ruby', github: 'ruby-concurrency/concurrent-ruby', branch: 'master', ref: 'd634acf', require: 'concurrent'

And received the following error:

Could not find gem 'concurrent-ruby' in
git://github.com/ruby-concurrency/concurrent-ruby.git (at master@ff06e51).
Source contains 'concurrent-ruby' at: 1.0.3.pre1 java

I'm able to install without problems on MRI, but jruby-9.0.1.0 and jruby-9.1.0.0 both throw the error above. Any tips?

I'll keep trying to see if I can figure something out.

[fedenusy]

Figured it out, had to delete Gemfile.lock before bundle install for some reason.

[fedenusy]

Now concurrent-ruby is failing to load... @pitr-ch any clue where I'm going wrong?

Bundler::GemRequireError: There was an error while trying to load the gem 'concurrent'.
Gem Load Error is: On JRuby but Java extensions failed to load.
Backtrace for gem load error is:
/home/app/vendor/bundle/jruby/2.2.0/bundler/gems/concurrent-ruby-d634acfb0bfa/lib/concurrent/utility/native_extension_loader.rb:63:in `load_native_extensions'
/home/app/vendor/bundle/jruby/2.2.0/bundler/gems/concurrent-ruby-d634acfb0bfa/lib/concurrent/synchronization.rb:5:in `<top>'
org/jruby/RubyKernel.java:939:in `require'
/home/app/vendor/bundle/jruby/2.2.0/gems/activesupport-4.2.0/lib/active_support/dependencies.rb:1:in `block in (root)'
...

[pitr-ch]

ah, I've forgot to warn you that gem ... git: does not compile the native extensions. Please clone the master branch run rake compile and then use gem ... path:.

[fedenusy]

My deploy pipeline makes it difficult to use the gem ... path: approach.

I ended up ssh'ing into a single server, with the idea of installing this patch manually and then running a canary test.

Turns out my production machine was missing rake-compiler and javac. I spent a good hour installing them -- and then my server crashed. Not sure if the machine ran OOM or if I just messed up the installation. Either way, the canary release is probably a no-go.

@pitr-ch, are you and @jdantonio comfortable releasing 1.0.3 as is? If you publish 1.0.3 (or 1.0.3.pre1), it'll be much easier for me to test. I'll also be able to run this at scale rather than on a single server.

[pitr-ch]

I'll create a pre version for you to try.