Introduce have_reported_error matcher #2849

skatkov · 2025-06-04T22:53:33Z

rspec-rails is missing support for Rails ErrorReporter, this was introduced to rails in v7 and has been evolving ever since. With my client, we have moved to using ErrorReporter as a unified error reporting interface, so we can easily move from one error tracking software to another with minimal code changes. And we had a need to test this interface with rspec, so we implemented our own matcher to handle this.

I'm suggesting our internal implementation as is. This is probably not suitable as is for this gem, but I'd like to open discussion with this starting point.

Example usage

@example Checking for any error
  expect { Rails.error.report(StandardError.new) }.to have_reported_error

@example Checking for specific error class
  expect { Rails.error.report(MyError.new) }.to have_reported_error(MyError)

@example Checking for specific error instance with message
  expect { Rails.error.report(MyError.new("message")) }.to have_reported_error(MyError.new("message"))

@example Checking error attributes
  expect { Rails.error.report(StandardError.new, context: "test") }.to have_reported_error.with(context: "test")

@example Checking error message patterns
  expect { Rails.error.report(StandardError.new("test message")) }.to have_reported_error(/test/)

@example Negation
  expect { "safe code" }.not_to have_reported_error

pirj

Thank you!

spec/rspec/rails/matchers/have_reported_error_spec.rb

pirj · 2025-06-07T21:46:38Z

features/matchers/README.md

+
+```ruby
+# passes if Rails.error.report was called with any error
+expect { Rails.error.report(StandardError.new) }.to have_reported_error


I think this file is supposed to contain just the outline of the matchers defined here (and at a glance it’s rather incomplete). What do you think of keeping just one example of usage of this matcher? The rest will be taken into the same documentation from the feature file.

This is very verbose for us... These lines take up half this file when our primary documentation is the feature file itself...

features/matchers/README.md

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-06-21T07:05:36Z

lib/rspec/rails/matchers/have_reported_error.rb

+          attributes_match?(event_context)
+        end
+
+        def actual_error


Cosmetic. [0] and [1] look cryptic, is it worth using a struct?

sounds good!

lib/rspec/rails/matchers/have_reported_error.rb

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

features/matchers/have_reported_error_matcher.feature

JonRowe · 2025-06-25T17:14:51Z

features/matchers/have_reported_error_matcher.feature

+  Scenario: Using in controller specs
+    Given a file named "spec/controllers/users_controller_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe UsersController, type: :controller do
+        describe "POST #create" do
+          it "reports validation errors" do
+            expect {
+              post :create, params: { user: { email: "invalid" } }
+            }.to have_reported_error(ValidationError)
+          end
+        end
+      end
+      """
+    When I run `rspec spec/controllers/users_controller_spec.rb`
+    Then the examples should all pass
+
+  Scenario: Using in request specs
+    Given a file named "spec/requests/users_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe "Users", type: :request do
+        describe "POST /users" do
+          it "reports processing errors" do
+            expect {
+              post "/users", params: { user: { name: "Test" } }
+            }.to have_reported_error.with(context: "user_creation")
+          end
+        end
+      end
+      """
+    When I run `rspec spec/requests/users_spec.rb`
+    Then the examples should all pass
+


This isn't a controller / request specific matcher so these feel out of place, why these, why not every aspect of Rails... I'd just cut them.

spec/rspec/rails/matchers/have_reported_error_spec.rb

JonRowe · 2025-06-25T17:25:27Z

👋 I think this would be a great addition, sorry for the size of the review its been on my todo list for a while, its mostly just grammar / wording tweaks plus a few "fit" things.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

skatkov · 2025-06-27T11:45:47Z

@JonRowe thanks for review.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Which of these you would rather go with?
.have_reported_error(StandardError, /wrong attributes/).with(param1: 'test')

or

.have_reported_error(StandardError).with(message:/wrong attributes/, params1: 'test1')

??

I assume, that first option is prefered to keep similarity with raised_error matcher.
https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/raise-error/

JonRowe · 2025-06-27T11:59:45Z

Given that you already have with, I think:

.have_reported_error(StandardError).with(/wrong attributes/, params1: 'test1')

But if you insisted on message: in with I could be persuaded

pirj · 2025-06-27T14:31:21Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.
raise_error accepts have_attributes as an argument matcher. It supports with_message. Wdyt if we reserve with for additional args passed to report, but keep it separate from with_message?

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-11T18:55:38Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def failure_message
+          if !@error_subscriber.events.empty? && [email protected]?


Cosmetic: any? or present??

pirj · 2025-07-11T18:57:19Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def failure_message
+          if !@error_subscriber.events.empty? && [email protected]?
+            event_context = @error_subscriber.events.last.attributes[:context]


Won’t it be confusing to use last when several errors were reported?

pirj · 2025-07-11T19:07:30Z

lib/rspec/rails/matchers/have_reported_error.rb

+              return "Expected error message to be '#{@expected_message}', but got: #{reported_errors}"
+            end
+          else
+            if @expected_error && !actual_error.is_a?(@expected_error)


actual_error should be is_a? expected_class, no? Just by looking at the implementation. Or what is the case when they won’t match?

pirj · 2025-07-11T19:22:25Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+          case @expected_message
+          when Regexp
+            error.message&.match(@expected_message)


Cosmetic: is safe nav necessary?

spec/rspec/rails/matchers/have_reported_error_spec.rb

pirj · 2025-07-12T09:56:44Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        private
+
+        def error_matches_expectation?


Is the last condition necessary? We already check if it’s empty? before calling this method.

pirj

A fee minor things and simplifications and it looks good to go.

Let’s leave out matching multiple with chains, severity qualifiers, and all extras for later.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-12T13:08:34Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def matches?(block)
+          if block.nil?
+            raise ArgumentError, "this matcher doesn't work with value expectations"


Does this check have the same effect as defining a method supports_value_expectations? that returns false? It should be.

RSpec is actually allowing the value expectation to reach the matches? method, even with supports_value_expectations? defined as false. This mechanism is NOT preventing the value expectation from reaching matches?. The explicit check was necessary in my testing! But maybe I missed something?

Ha! Nice catch. The RSpec.deprecate call somehow didn't fail the build. I'll have a closer look.

The block is not nil though for expect(:foo).to reports_error, so this check is useless in its current implementation.
I suggest removing it, and relying on the supports_value_expectations?.

I recall now that we kept in 3.x the possibility to pass a lambda to expect, or define subject as a lambda and use is_expected.to with block matchers. We just print a deprecation message now.

Swallowed deprecation messages are due to #2857

After the aftermentioned fix, with

def supports_value_expectations? false end

The following passes:

it "warns when used as a value expectation" do expect { expect(Rails.error.report(StandardError.new("test error"))).to have_reported_error }.to raise_error(/implicit block expectation.+deprecated/) end

We can change the code when that swallowed deprecation fix is merged.

This can be removed now and any test marked as pending with the reason instead.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-12T13:16:43Z

lib/rspec/rails/matchers/have_reported_error.rb

+        def error_message_matches?(error)
+          return true if @expected_message.nil?
+
+          case @expected_message


There should be existing examples amongst our matchers of something like value_matches? that would also allow the use of argument matchers like a_string_including("network").

On a second thought, this may make matching more complicated, as if just one argument is passed, and it’s an argument matcher, what should we match it against, the exception itself, or the exception’s message?

This makes it more compelling to build the with_message qualifier. But it then limits the argument to the matcher to only accept the class. But this is what the Rails’ assertion does anyway.

lib/rspec/rails/matchers/have_reported_error.rb

pirj · 2025-07-12T13:27:05Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), context: { context: "user_processing", severity: :error })


Nested context here, too. Rename inner to topic: or :section?

pirj · 2025-07-12T13:28:03Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), context: { context: "user_processing", severity: :error })


The severity: here is inside the context:, is this correct?

pirj · 2025-07-12T13:35:26Z

features/matchers/have_reported_error_matcher.feature

+          expect {
+            # Safe operation that doesn't report errors
+            "safe code"
+          }.not_to have_reported_error


I wonder if the negated form should warn if qualifiers were passed to avoid mistakenly letting unmatched reports through:

expect { User.process_custom_error }.not_to have_reported_error("Email is TYPO")

Meaning that if the matcher is used in the negated form, no class, message, or attributes qualifiers are acceptable.

See https://github.com/rspec/rspec/blob/95fbf88255de04e4ba8288172292bda494c4f187/rspec-expectations/lib/rspec/matchers/built_in/raise_error.rb#L162

pirj · 2025-08-04T05:30:39Z

Ping

skatkov · 2025-08-04T15:38:18Z

Sorry had a small vacation with relatives and had to format my laptop! I'll be pushing improvements soon..

JonRowe · 2025-08-08T15:53:07Z

features/matchers/README.md

+# passes when error is reported with specific context attributes
+expect { Rails.error.report(StandardError.new, context: { user_id: 123 }) }.to have_reported_error.with_context(user_id: 123)
+
+


Remove these two blank lines, I'd do it in a suggestion but Github isn't co-operating.

JonRowe · 2025-08-08T15:59:22Z

log/development.log

Don't commit this.

JonRowe · 2025-08-08T16:02:02Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def and(_)
+          raise ArgumentError, "Chaining is not supported"


Why? If you include the composition helper this should just work ™ ?

JonRowe · 2025-08-08T16:09:41Z

This is looking really good, I think its mostly just little things left?

features/matchers/README.md

skatkov force-pushed the have-reported-error branch from 8b837a8 to d2d0e51 Compare June 7, 2025 20:44

pirj reviewed Jun 7, 2025

View reviewed changes

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

skatkov force-pushed the have-reported-error branch 2 times, most recently from b38ff1c to 87ebd2a Compare June 22, 2025 20:50

skatkov added 12 commits June 22, 2025 22:59

first pass at have_reported_error matcher

5e6efcc

Adding a feature

5dd2c5c

simplify matcher/readmer.md

ce1a162

Remove mocks out of tests

550c25e

test failure messages better

e074991

rewrite specs

1fe7e91

should require a block matcher

6b41d2b

Create ErrorEvent struct to store errors

3859b04

simplify matcher

8352668

disable matcher chaining

030c27a

improved argument error message in case block is not passed to matcher

2b9bd02

further clean-up

6135d5d

skatkov force-pushed the have-reported-error branch from 71685ad to 6135d5d Compare June 22, 2025 20:59

pirj reviewed Jun 23, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

JonRowe requested changes Jun 25, 2025

View reviewed changes

skatkov and others added 2 commits June 25, 2025 23:21

Apply suggestions from code review

8335664

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

clean-up features

a14f5c7

skatkov added 5 commits July 2, 2025 00:08

rename .with to .with_context

7259a25

have_reported_error to accept class name and a message

1b6deb6

switch to error_report matcher type of api

c77fca5

refactor

f5e427a

Using UndefinedValue as a default attribute

7b3abc5

skatkov force-pushed the have-reported-error branch 3 times, most recently from c1134da to f90fc3b Compare July 1, 2025 23:20

Look through all errors, don't just go with last error

2488a71

skatkov force-pushed the have-reported-error branch from f90fc3b to 2488a71 Compare July 1, 2025 23:28

pirj reviewed Jul 2, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jul 2, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

warn about nil being passed

c1816e2

pirj reviewed Jul 11, 2025

View reviewed changes

pirj reviewed Jul 12, 2025

View reviewed changes

Stanislav (Stas) Katkov added 2 commits August 4, 2025 19:03

Use ErrorCollector from Rails instead of self-written version'

e1ab316

Use defined UNDEFINED value instead of defining our own

8674529

pirj mentioned this pull request Aug 6, 2025

Reduce sandboxing to prevent swallowed deprecations #2857

Open

JonRowe reviewed Aug 8, 2025

View reviewed changes

log/development.log Outdated

Copy link

Member

JonRowe Aug 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't commit this.

JonRowe reviewed Aug 8, 2025

View reviewed changes

pirj reviewed Aug 8, 2025

View reviewed changes

features/matchers/README.md Outdated Show resolved Hide resolved

Update features/matchers/README.md

8b89342

		# passes when error is reported with specific context attributes
		expect { Rails.error.report(StandardError.new, context: { user_id: 123 }) }.to have_reported_error.with_context(user_id: 123)

Uh oh!

Introduce have_reported_error matcher #2849

Are you sure you want to change the base?

Introduce have_reported_error matcher #2849

Uh oh!

Conversation

skatkov commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example usage

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JonRowe commented Jun 25, 2025

Uh oh!

skatkov commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonRowe commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pirj commented Jun 27, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

skatkov commented Jun 4, 2025 •

edited

Loading

skatkov commented Jun 27, 2025 •

edited

Loading

JonRowe commented Jun 27, 2025 •

edited

Loading