MONGOID-4781 Refactor embedded matchers to behave as MongoDB server queries (#4783)

Co-authored-by: Oleg Pudeyev <[email protected]>

Author: p-mongo

source/tutorials/mongoid-queries.txt

@@ -1729,24 +1729,26 @@ The following code illustrates retrieving the statistics:
   from the raw results if multiple statistics are desired.
 
 
-Full Text Search
-================
+.. _text-search:
 
-To perform full text search with Mongoid, follow these steps:
+Text Search
+===========
 
-1. Define a full text search index on a model.
-2. Create the full text search index on the server.
-3. Build a full text search query.
+To perform text search with Mongoid, follow these steps:
 
+1. Define a text index on a model.
+2. Create the text index on the server.
+3. Build a text search query.
 
-Defining Full Text Search Index
--------------------------------
+
+Defining Text Search Index
+--------------------------
 
 Index definition through Mongoid is described in detail on the `indexes
-<mongoid-indexes>`_ page. Full text search indexes are described in detail on
+<mongoid-indexes>`_ page. Text search indexes are described in detail
 under `text indexes <https://docs.mongodb.com/manual/core/index-text/>`_
 in the MongoDB manual. Below is an example definition of a Band model with
-a full text search index specified on the description field:
+a text index utilizing the description field:
 
 .. code-block:: ruby
 
@@ -1762,8 +1764,8 @@ a full text search index specified on the description field:
 Note that the index type (``text``) must be given as a string, not as a symbol.
 
 
-Creating Full Text Search Index
--------------------------------
+Creating Text Index
+-------------------
 
 To create the index, invoke the ``db:mongoid:create_indexes`` Rake task:
 
@@ -1772,8 +1774,8 @@ To create the index, invoke the ``db:mongoid:create_indexes`` Rake task:
   bundle exec rake db:mongoid:create_indexes
 
 
-Querying Using Full Text Search Index
--------------------------------------
+Querying Using Text Index
+-------------------------
 
 To find bands whose description contains "ounces" or its variations, use the
 `$text operator <https://docs.mongodb.com/manual/reference/operator/query/text/#op._S_text>`_:
@@ -1786,6 +1788,6 @@ To find bands whose description contains "ounces" or its variations, use the
 Note that the description contains the word "ounce" even though the search
 query was "ounces".
 
-Note also that when performing a full text search, the name of the field is
-not explicitly specified - ``$text`` operator searches all fields indexed with
-full text search indexes.
+Note also that when performing text search, the name of the field is not
+explicitly specified - ``$text`` operator searches all fields indexed with
+the text index.

source/tutorials/mongoid-relations.txt

@@ -241,6 +241,68 @@ to the other in the form of an array.
     "band_ids" : [ ObjectId("4d3ed089fb60ab534684b7e9") ]
   }
 
+
+Querying Referenced Associations
+--------------------------------
+
+In most cases, efficient queries across referenced associations (and in general
+involving data or conditions or multiple collections) are performed using
+the aggregation pipeline. Mongoid helpers for constructing aggregation pipeline
+queries are described in the :ref:`aggregation pipeline <aggregation-pipeline>`
+section.
+
+For simple queries, the use of aggregation pipeline may be avoided and
+associations may be queried directly. When querying associations directly,
+all conditions must be on that association's collection only (which typically
+means association in question and any associations embedded in it).
+
+For example, given the following models:
+
+.. code-block:: ruby
+
+  class Band
+    include Mongoid::Document
+    
+    has_many :tours
+    has_many :awards
+    
+    field :name, type: String
+  end
+
+  class Tour
+    include Mongoid::Document
+    
+    belongs_to :band
+    
+    field :year, type: Integer
+  end
+
+  class Award
+    include Mongoid::Document
+    
+    belongs_to :band
+    
+    field :name, type: String
+  end
+
+One could retrieve all bands that have toured since 2000 as follows:
+
+.. code-block:: ruby
+
+  band_ids = Tour.where(year: {'$gte' => 2000}).pluck(:band_id)
+  bands = Band.find(band_ids)
+
+The conditions on ``Tour`` can be arbitrarily complex, but they must all
+be on the same ``Tour`` document (or documents embedded in ``Tour``).
+
+To find awards for bands that have toured since 2000:
+
+.. code-block:: ruby
+
+  band_ids = Tour.where(year: {'$gte' => 2000}).pluck(:band_id)
+  awards = Award.where(band_id: {'$in' => band_ids})
+
+
 Embedded Associations
 =====================
 
@@ -442,6 +504,92 @@ help of MongoDB projection operation:
   Band.where(started_on: {'$gt' => Time.now - 1.year}).only(:label).map(&:label).compact.uniq
 
 
+Querying Embedded Associations
+------------------------------
+
+When querying top-level documents, conditions can be specified on documents
+in embedded associations using the dot notation. For example, given the
+following models:
+
+.. code-block:: ruby
+
+  class Band
+    include Mongoid::Document
+    embeds_many :tours
+    embeds_many :awards
+    field :name, type: String
+  end
+
+  class Tour
+    include Mongoid::Document
+    embedded_in :band
+    field :year, type: Integer
+  end
+
+  class Award
+    include Mongoid::Document
+    embedded_in :band
+    field :name, type: String
+  end
+
+To retrieve bands based on tour attributes, use the dot notation as follows:
+
+.. code-block:: ruby
+
+  # Get all bands that have toured since 2000
+  Band.where('tours.year' => {'$gte' => 2000})
+
+To retrieve only documents of embedded associations, without retrieving
+top-level documents, use the ``pluck`` projection method:
+
+.. code-block:: ruby
+
+  # Get awards for bands that have toured since 2000
+  Band.where('tours.year' => {'$gte' => 2000}).pluck(:awards)
+
+.. _embedded-matching:
+
+Querying Loaded Associations
+````````````````````````````
+
+Mongoid query methods can be used on embedded associations of documents which
+are already loaded in the application. This mechanism is sometimes called
+"embedded matching" or "embedded document matching" and it is implemented
+entirely in Mongoid - the queries are NOT sent to the server.
+
+Embedded matching is supported for most general-purpose query operators. It
+is not implemented for :ref:`text search <text-search>`, :manual:`geospatial query
+operators </reference/operator/query-geospatial/>`,
+operators that execute JavaScript code (:manual:`$where </reference/operator/query/where/>`)
+and operators that are implemented via other server functionality such as
+:manual:`$expr <https://docs.mongodb.com/manual/reference/operator/query/expr/>`
+and :manual:`$jsonSchema <https://docs.mongodb.com/manual/reference/operator/query/jsonSchema/>`.
+The following operators are supported:
+
+- :manual:`Comparison operators </reference/operator/query-comparison/>`.
+- :manual:`Logical operators </reference/operator/query-logical/>`.
+- :manual:`$exists </reference/operator/query/exists/>`
+  (:manual:`$type </reference/operator/query/type/>` is not currently supported).
+- :manual:`$regex </reference/operator/query/regex/>` (``$options`` field
+  is not currently supported, the options must be given as part of the
+  regular expression object).
+- :manual:`Array query operators </reference/operator/query-array/>`.
+
+For example, using the model definitions just given, we could query
+tours on a loaded band:
+
+.. code-block:: ruby
+
+  band = Band.where(name: 'Astral Projection').first
+  tours = band.tours.where(year: {'$gte' => 2000})
+
+.. note::
+
+  Mongoid aims to provide the same semantics when performing embedded matching
+  as those of MongoDB server. This means, for example, that embedded matchers
+  generally support arguments of the same types as ``Criteria`` methods do.
+
+
 Common Behavior
 ===============
 
@@ -996,104 +1144,10 @@ with, and is useful for developers of extensions to Mongoid.
      - Returns whether the association has an associated validation.
 
 
-Querying Associations
-=====================
-
-Mongoid supports several forms of efficient querying of documents based on
-associations.
-
-
-Embedded Associations
----------------------
-
-Given the following models:
-
-.. code-block:: ruby
-
-  class Band
-    include Mongoid::Document
-    embeds_many :tours
-    embeds_many :awards
-    field :name, type: String
-  end
-
-  class Tour
-    include Mongoid::Document
-    embedded_in :band
-    field :year, type: Integer
-  end
-
-  class Award
-    include Mongoid::Document
-    embedded_in :band
-    field :name, type: String
-  end
-
-Mongoid allows retrieving bands whose tours have certain attributes via the
-dot notation, as follows:
-
-.. code-block:: ruby
-
-  # Get all bands that have toured since 2000
-  Band.where('tours.year' => {'$gte' => 2000})
-
-Mongoid also can retrieve embedded documents only, without retrieving top-level
-documents, using projection:
-
-.. code-block:: ruby
-
-  # Get awards for bands that have toured since 2000
-  Band.where('tours.year' => {'$gte' => 2000}).pluck(:awards)
-
-
-Referenced Associations
------------------------
-
-If the associations are referenced rather than embedded, performing queries
-through them takes a bit more work. Given the following models modified from
-the previous example:
-
-.. code-block:: ruby
-
-  class Band
-    include Mongoid::Document
-    has_many :tours
-    has_many :awards
-    field :name, type: String
-  end
-
-  class Tour
-    include Mongoid::Document
-    belongs_to :band
-    field :year, type: Integer
-  end
-
-  class Award
-    include Mongoid::Document
-    belongs_to :band
-    field :name, type: String
-  end
-
-One could retrieve all bands that have toured since 2000 as follows:
-
-.. code-block:: ruby
-
-  band_ids = Tour.where(year: {'$gte' => 2000}).pluck(:band_id)
-  bands = Band.find(band_ids)
-
-The conditions on ``Tour`` can be arbitrarily complex, but they must all
-be on the same ``Tour`` document (or documents embedded in ``Tour``).
-
-To find awards for bands that have toured since 2000:
-
-.. code-block:: ruby
-
-  band_ids = Tour.where(year: {'$gte' => 2000}).pluck(:band_id)
-  awards = Award.where(band_id: {'$in' => band_ids})
-
+.. _aggregation-pipeline:
 
 Aggregation Pipeline
---------------------
+====================
 
 Mongoid exposes MongoDB's aggregation pipeline for queries involving multiple
 referenced associations at the same time. Given the same setup as before with
@@ -1156,8 +1210,8 @@ having to send the list of document ids to Mongoid in the second query
 
 .. _aggregation-pipeline-builder-dsl:
 
-Aggregation Pipeline Builder DSL
---------------------------------
+Builder DSL
+-----------
 
 Mongoid provides limited support for constructing the aggregation pipeline
 itself using a high-level DSL. The following aggregation pipeline operators