Use Bundler.with_original_env so that BUNDLER_APP_CONFIG is honored

[mattbrictson]

Fixes #545.

Bundler's with_clean_env completely erases all Bundler-related environment variables, even if those variables existed prior to loading Bundler. That means that custom Bundler settings that we legitimately want to pass down to child processes - namely, BUNDLE_APP_CONFIG - are erased, leading to errors when the child process loads bundler/setup.

Note that clean_env (used by with_clean_env) is actually deprecated by Bundler. The method recommended by Bundler going forward, and the one used in this commit, is with_original_env. This clears any variables that were set by Bundler itself, but retains any user-supplied settings that existed prior.

This fixes a bug where spring binstubs would fail if BUNDLE_APP_CONFIG was set to a custom location (i.e. other than the default ./.bundle) and bundle install --path was used. The binstub would not be able to load the bundle because without BUNDLE_APP_CONFIG, it could not know the path where the gems were installed.

[rails-bot]

r? @rafaelfranca

(@rails-bot has picked a reviewer for you, use r? to override)

[cmwall]

Our builds were failing at the test suite stage in Circle CI because the rake gem was not being found in any of our sources.

We updated our Gemfile to use this branch for spring and the problem was solved.

Thanks!

The error we received was:

#!/bin/bash -eo pipefail
xvfb-run -a \
./bin/rspec --profile 10 \
--format RspecJunitFormatter \
--out ./test-results/rspec.xml \
--format progress \
$(circleci tests glob "spec/**/*_spec.rb" | circleci tests split --split-by=timings)
Could not find rake-12.2.1 in any of the sources
Run `bundle install` to install missing gems.
Exited with code 1

[mattbrictson]

@rafaelfranca in your opinion is this fix worth pursuing? It seems that at least a handful of other people have found that this branch fixes their problems using spring binstubs on CircleCI.

I can spend some more time on this PR to try and fix the Travis failures, but first I want to make sure it has a good chance of being accepted. Thanks!

[anvox]

The same problem to me, CircleCI 2.0. I must disable spring on CI :( https://github.com/rails/spring#temporarily-disabling-spring
Under company policy, I dont want to put a dev branch gem to app.

[sponomarev]

@mattbrictson Do you know what introduced that problem? We haven't experienced this problem about a month ago, but now it's a common problem for local and Docker environment

[matiasleidemer]

I have the same issue in a Docker environment. Updating bundler to 1.16.1causes the error, but after monkeypatching spring to use Bundler.with_original_env it works like a charm. Downgrading bundler to 1.15.4 also fixes the issue.

I'm not sure how this change would impact other projects, however. As a side note, we don't have the BUNDLE_APP_CONFIG env variable.

EDIT: my rubygems version is 2.5.2.2

[rmm5t]

Confirmed that I am experiencing the same underlying problem and that this change fixes it.

What's the likelihood of something like this making it into a new release? I guess I'm not sure I fully understand the benefits of .with_clean_env vs .with_original_env.

[sponomarev]

with_clean_env was legacy and have a lot of caveats - not all the proper ENV variables are cleaned. It will be deprecated in Bundler 2.0. Recent Bundler major version changed the logic a little bit (to correct it), but, unfortunately, Spring relies on that incorrect behaviour in some limited cases.

As a temporary solution, we set GEM_HOME and BUNDLE_APP_CONFIG to our own location if we want to use custom BUNDLE_PATH.

Not actually related, but could be useful. Our part of Dockerfile

ENV GEM_HOME /bundle
ENV BUNDLE_PATH=$GEM_HOME \
  BUNDLE_APP_CONFIG=$BUNDLE_PATH \
  BUNDLE_BIN=$BUNDLE_PATH/bin
ENV PATH /app/bin:$BUNDLE_BIN:$PATH

[rmm5t]

with_clean_env was legacy and have a lot of caveats - not all the proper ENV variables are cleaned. It will be deprecated in Bundler 2.0

Thanks for that explanation. Assuming this is all true, what's holding up this PR from being applied?
Obviously the build failed, but it looks like the build is failing on the master branch for the same reasons.

Where is help needed to move this forward? I have some incentive and willingness to help here where I can.

[matthewd]

What are the version/dependency implications of this change / when did Bundler introduce with_original_env?

Is it safe to just switch outright, or might there be a need to support both methods?

[rmm5t]

@matthewd with_original_env was introduced 7 years ago, so around bundler v1.0.x.

[sponomarev]

Here is the proof from Bundler's source code

https://github.com/bundler/bundler/blob/23dfadcd8d561d48e89c7287cbbdd816e0a10ab8/lib/bundler.rb#L281-L284

[rmm5t]

@matthewd Any chance we might see some progress on this issue? Where do you need help to move this forward. I'm willing to help if needed.

[davidstosik]

Interested in seeing this fixed too! 👍
The issue goes a bit further down: setting BUNDLE_PATH only seems to lead to the same problem.
See this issue I originally created on the Rails project, thinking the problem was located there: rails/rails#32587.

Using @mattbrictson's fix branch for Spring (gem 'spring', github: 'mattbrictson/spring', branch: 'honor-bundle-app-config' in my Gemfile) fixes everything and I can run bundle exec rails test now! 🚀

[pdx91]

Facing the same issue in GitLab CI as well.

Since I'm just starting out with my app and don't have a ton of tests, I'm disabling spring till this gets fixed.

DISABLE_SPRING=true bin/rails test

[ajacksified]

I wasted my morning trying to figure out why rails s worked but rails g didn't; it was this issue. Commenting spring out of the Gemfile immediately solved my issue - we use a virtualenv for projects in a dev vm, and .bundle/config points at vendor/bundle.

Can we merge this one-line change that's been open for half a year? It's especially bad because it's in the default Gemfile for new rails projects, and the error is pretty difficult to track down.

[rmm5t]

Thank you, @matthewd!!

Will there be a new versioned release soon, or should we plan to work against the master branch for the near term?

[matthewd]

I'd like to get master's CI properly green before we cut a release.

For this change: I'm happy that only Bundler >= 1.0 < 1.1 had with_clean_env and not with_original_env, and that substantially predates when we started using it.

[mtsmfm]

@matthewd Can I ask you to cut new release?

master's CI was fixed via #562

[mattwelke]

Just wanted to mention that I needed to resort to using the DISABLE_SPRING=true bundle exec rails test command in my CircleCI config for it to work, despite this branch being merged in. Without disabling Spring, it did not find the Rake gem:

#!/bin/bash -eo pipefail
bundle exec rails test
Could not find rake-12.3.1 in any of the sources
Run `bundle install` to install missing gems.
Exited with code 1

[davidstosik]

@Welkie The fix was merged, but no new version of Spring was released.

I confirmed the fix works for me by using gem 'spring', github: 'rails/spring' in my Gemfile.

Thanks to anyone involved in fixing this! 👏

[mattwelke]

@davidstosik Awesome. I've been out of the loop with Rails for like 2 years but I wanted to update a project's dependencies and throw it on CircleCI. So this threw me off. Just wanted to comment so if anyone else comes here from googling, they know a fix is still needed.

@literally_everyone Ty for the fix. :)

[ypresto]

FYI:
binstub (bin/spring) won't work with gem 'spring', github: 'rails/spring'.
It assumes spring gem to be installed on standard gem path (vendor/bundle/ruby/2.5.0/gems) , not bundler's git repo directory (vendor/bundle/ruby/2.5.0/bundler/gems).

Part of binstub (gem is rubygems' one):

if spring
  Gem.use_paths Gem.dir, Bundler.bundle_path.to_s, *Gem.path
  gem 'spring', spring.version
  require 'spring/binstub'
end

Workaround not to stop using bin/spring:

Replace contents of bin/spring with below snippet.

#!/bin/bash
exec bundle exec spring "$@"

[ypresto]

Is there ETA to release this? I got the same issue in another project 😢

[rmm5t]

@matthewd I recognize and appreciate that you're waiting to get master fully green before pushing out a new release, but spring is kind of dead in the water without this fix released. It's not really possible to use the master branch as a gem dependency, because the spring binstub doesn't know how to find the path to spring if it's installed via a bundler gem "spring", github: "rails/spring" declaration, because the spring binstub only searches in Bundler.bundle_path and not also Bundler.install_path

spring/lib/spring/client/binstub.rb

Lines 39 to 41 in 6f1203a

	Gem.use_paths Gem.dir, Bundler.bundle_path.to_s, *Gem.path
	gem 'spring', spring.version
	require 'spring/binstub'

[bobbytables]

Can we please release this?

[ypresto]

Today I faced to this issue, again.
I know you are busy, but, just a release, not a review or writing code.
Please, please release it..! It has been 1.5 year after the PR..!! 😂 😂 🙏 🙏

[ypresto]

Please mention me if there is any blocker for release and perhaps I can help solving it.

[rafaelfranca]

The CI is very broken so we can't release in that way. This PR may be the cause why it is broken.

[rmm5t]

This PR may be the cause why it is broken.

@rafaelfranca The build in the master branch has been green multiple times after this PR was merged. And it was red multiple times before this PR was merged.

[rafaelfranca]

Yes, I confirmed this PR is not the cause

[davidstosik]

Yes, I confirmed this PR is not the cause

If it is not, is there any chance we get a release soon? 🤞

[y-yagi]

@rafaelfranca Do you have a plan to release a new gem? I think that this fix is also related to rubygems/bundler#7184. If need to fix CI, I will do it.

[ypresto]

fix-ci branch looks already passing CI..!

[rmacklin]

Since no one mentioned it explicitly in this thread, I'll note that v2.1.0 was eventually released (Jun 14, 2019) which includes this change.