Fixtures error when tests requiring 'spec_helper' are run alongside tests requiring 'rails_helper'

[tmertens]

A failure occurs when some tests require only a very barebones spec_helper for classes that have no dependency on rails, while other tests require rails_helper if they have dependencies on the rails side of the app.

When run individually, both types of tests run fine. When run together, the spec_helper tests will raise an error. It does not appear to matter which order the tests run in.

Failure/Error: Unable to find matching line from backtrace
     NoMethodError:
       undefined method `pre_loaded_fixtures' for SomeClass
     # ruby-2.1.5@avant_basic/gems/attr_encryptor-2.0.0/lib/attr_encryptor.rb:263:in `method_missing'
     # ruby-2.1.5@avant_basic/gems/activesupport-3.2.19/lib/active_support/core_ext/class/attribute.rb:98:in `pre_loaded_fixtures'
     # ruby-2.1.5@avant_basic/gems/activerecord-3.2.19/lib/active_record/fixtures.rb:836:in `setup_fixtures'

I've traced the issue to this line: https://github.com/rspec/rspec-rails/blob/master/lib/rspec/rails/fixture_support.rb#L10

activesupport (3.2.19)
rails (3.2.19)
rspec-core (3.2.1)
rspec-expectations (3.2.0)
rspec-mocks (3.2.1)
rspec-rails (3.2.1)
rspec-support (3.2.2)

[JonRowe]

I think we'll need more information to debug what's going wrong in your case, mixing plain specs and rails specs is a common scenario which we know works so theres something else going on in your case.

[tmertens]

@JonRowe I don't know about rails 4, but this is definitely an issue with rails 3.2.19.

To reproduce:

1. Create a new rails 3.2.19 app.
2. Create two spec files - one requiring rails_helper, and one requiring spec_helper. The type or file location of the specs don't matter, and you do not need to create any actual files in the app to reproduce.
3. Add a single test in each spec file which simply expect(true).to be
4. Run the tests. The test requiring spec_helper will fail with the above referenced error.

The real culprit IMO is this line: https://github.com/rspec/rspec-rails/blob/master/lib/rspec/rails/configuration.rb#L59 which includes RSpec::Rails::FixtureSupport into all specs, rather than only including it into the types of specs governed by rspec rails.

[tmertens]

My proposed solution is to move the inclusion of RSpec::Rails::FixtureSupport out of RSpec::Rails::Configuration and into the base RSpec::Rails::RailsExampleGroup.

[cupakromer]

That LOC you linked should only be loaded if you are loading rspec-rails configuration. That should only happen when you require your rails_helper. Additionally, it should only have an affect if you have ActiveRecord::TestFixtures available: https://github.com/rspec/rspec-rails/blob/master/lib/rspec/rails/fixture_support.rb#L5

If you are running a spec which does not include rails_helper then the rspec-rails configuration should not get loaded. I followed your repro steps and I am not able to reproduce the issue. Here is my repo: https://github.com/cupakromer/issue-1355

Can you please share a repo which demonstrates your issue?

[cupakromer]

BTW your stack trace shows the error coming from gems/attr_encryptor-2.0.0/lib/attr_encryptor.rb. It looks like it is expecting to be in a Rails environment but is not.

[tmertens]

@cupakromer I pushed an example here: https://github.com/tmertens/rspec_rails_1355

Download, bundle install, and run bundle exec rspec from the root directory to reproduce the error.

[tmertens]

@cupakromer Apparently load order of the specs is a problem. If you rename standard_spec.rb to aastandard_spec.rb, the tests will fail in your repo.

[cupakromer]

@tmertens I have reproduced the issue per your adjustment. This is very odd indeed. I will dig into it. Thank you for reporting it ❤️

[tmertens]

Thank you @cupakromer. The problem was resolved by my solution above, moving RSpec::Rails::FixtureSupport out of RSpec::Rails::Configuration and into the base RSpec::Rails::RailsExampleGroup. This also seems to me to be the correct solution, as IMO rspec-rails should not be adding includes to global configuration for all specs.

[cupakromer]

Ok, I thought I knew what is going on, but it seems to run deeper. The repo can be simplified to simply adding require 'rails_helper' at the end of the standard spec file.

@tmertens I appreciate the suggestion for the fix. However, I'm a little concerned about what that will break in the gem ecosystem. There are many projects which have custom configurations to allow testing ActiveRecord models in isolation of loading all of Rails. Without knowing how those projects are setup this change could potentially break those applications.

From what I can find, this seems deal more with how rspec-core handles configuration settings.

Take the following sample:

RSpec.configure do |c|
  c.order = :defined
end

RSpec.describe "Defined before `Test`" do
  it "passes" do
    expect(foobar).to eq "Fooooooooo"
  end
end

module Test
  def foobar
    "Fooooooooo"
  end
end

RSpec.configure do |c|
  c.include Test
  c.before do |ex|
    puts "\nINSIDE BEFORE: #{ex.full_description}"
  end
  c.prepend_before do |ex|
    puts "\nPREPENDED BEFORE: #{ex.full_description}"
  end
end

RSpec.describe "Defined after `Test`" do
  it "passes" do
    expect(foobar).to eq "Fooooooooo"
  end
end

The before and prepend_before are only run for the "Defined after Test" group. However, both specs pass. If you check the ancestors, the Test module is included on the "Defined before Test" singleton class. Where as it's higher in the ancestor chain for the later "Defined after Test" group.

Now both specs include foobar. So I'm a little lost as to why the fixture support methods are not available in the defined hooks. However, this seems to expose a potentially deeper problem with configuration settings that may surprise users.

/cc @rspec/rspec

[myronmarston]

If you check the ancestors, the Test module is included on the "Defined before Test" singleton class. Where as it's higher in the ancestor chain for the later "Defined after Test" group.

This is no longer true as of rspec/rspec-core#1935 -- with that in place, the module inclusion will be applied to both example groups.

I think we should do a similar thing for configured hooks, and any other config that would apply to a group if it was defined after the config was set.

[cupakromer]

I think with the fix from rspec/rspec-core#1935 we get most of the way there. The runtime failure, which caused the spec to fail, is gone. However, we have a 2nd load order issue which appears under the same conditions, but is caused by a load time failure which aborts the spec run. Interestingly, this issue only happens if a spec is already created:

RSpec.describe "This fails badly" do
  it "passes" do
    expect(1).to eq 1
  end
end
require 'rails_helper' # This cause a load time error

/rspec-dev/repos/rspec-rails/lib/rspec/rails/fixture_support.rb:20:in `block (2 levels) in <module:FixtureSupport>': undefined method `fixture_path' for #<RSpec::Core::Configuration:0x007fd2e119b688> (NoMethodError)
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/fixture_support.rb:18:in `class_exec'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/fixture_support.rb:18:in `block in <module:FixtureSupport>'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/concern.rb:121:in `class_eval'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/concern.rb:121:in `append_features'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1279:in `include'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1279:in `safe_include'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1237:in `block in configure_existing_groups'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1235:in `each'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1235:in `configure_existing_groups'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1137:in `include'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/configuration.rb:59:in `initialize_configuration'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/configuration.rb:96:in `<module:Rails>'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/configuration.rb:2:in `<module:RSpec>'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails/configuration.rb:1:in `<top (required)>'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `require'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `block in require'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `require'
    from /rspec-dev/repos/rspec-rails/lib/rspec/rails.rb:14:in `<top (required)>'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `require'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `block in require'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:236:in `load_dependency'
    from /rspec-dev/repos/bundle/ruby/2.2.0/gems/activesupport-3.2.21/lib/active_support/dependencies.rb:251:in `require'
    from /rspec-dev/repos/rspec-rails/tmp/example_app/spec/rails_helper.rb:5:in `<top (required)>'
    from /rspec-dev/repos/rspec-rails/tmp/example_app/spec/astandard_spec.rb:6:in `require'
    from /rspec-dev/repos/rspec-rails/tmp/example_app/spec/astandard_spec.rb:6:in `<top (required)>'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1318:in `load'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1318:in `block in load_spec_files'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1316:in `each'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/configuration.rb:1316:in `load_spec_files'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/runner.rb:102:in `setup'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/runner.rb:88:in `run'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/runner.rb:73:in `run'
    from /rspec-dev/repos/rspec-core/lib/rspec/core/runner.rb:41:in `invoke'
    from /rspec-dev/repos/rspec-core/exe/rspec:4:in `<top (required)>'
    from bin/rspec:16:in `load'
    from bin/rspec:16:in `<main>'

Seems like an easy 1-line change. I'll also add a note to re-consider where we attempt to load fixtures in the next major release

[tmertens]

I'm a little concerned about what that will break in the gem ecosystem.

That is certainly a concern, my earlier suggestion is probably not appropriate as either a minor or patch level version change. But, the current behavior seems at odds with the division of spec/rails_helper, where spec_helper is intended to be as lightweight as possible for tests that require it. rspec-rails is implicitly increasing the footprint for every spec which does not require it when they are run alongside specs that do. I should add though, the performance impact itself is probably insignificant - the biggest concern is the dependency injection which occurs and results in unexpected failures as seen here.

[tmertens]

On another note, thank you for being so responsive on this issue!