Improve performance on metadata computation #12785

charlesBochet · 2025-06-23T11:18:41Z

In this PR:

Improve recompute metadata cache performance. We are aiming for ~100ms

Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as typeorm is suboptimizing

Remove caching lock

As recomputing the metadata cache is lighter, we try to stop preventing multiple concurrent computations. This also simplifies interfaces

Introduce self recovery mecanisms to recompute cache automatically if corrupted

Aka getFreshObjectMetadataMaps

custom object resolver performance improvement: 1sec to 200ms

Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap

reduce objectMetadataMaps to 500kb

We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName). While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap and introduce two otherMaps to idByName, idByJoinColumnName to make the bridge

Add dataloader on IndexMetadata (aka indexMetadataList in the API)

Improve field resolver performances too

Deprecate ClientConfig

greptile-apps

PR Summary

Major refactoring to improve metadata computation performance by removing the caching lock mechanism and simplifying cache operations across the application.

Removed ignoreLock parameter and associated locking logic from all cache recomputation methods, suggesting a shift towards a more efficient caching strategy without manual locks
Added getFreshObjectMetadataMaps as the main entry point for metadata retrieval in WorkspaceMetadataCacheService
Introduced composite index on workspaceId and createdAt columns in DataSourceEntity to optimize query performance
Removed relationMetadata table and its constraints through migration, though migration strategy needs improvement
Enabled comprehensive TypeORM logging in core datasource, which could impact production performance

_{23 files reviewed, 6 comments}
_{Edit PR Review Bot Settings | Greptile}

greptile-apps · 2025-06-23T11:19:15Z

packages/twenty-server/src/engine/twenty-orm/factories/workspace-datasource.factory.ts

+          const { objectMetadataMaps: cachedObjectMetadataMaps } =
+            await this.workspaceMetadataCacheService.getFreshObjectMetadataMaps(
+              {
+                workspaceId,
+              },
            );


style: this could create unnecessary recomputation overhead since getFreshObjectMetadataMaps() is not using cached values. Consider using getObjectMetadataMaps() first and only call getFreshObjectMetadataMaps() if cache is missing

packages/twenty-server/src/engine/twenty-orm/twenty-orm-global.manager.ts

...server/src/database/typeorm/core/migrations/common/1750673748111-remove-relation-metadata.ts

greptile-apps · 2025-06-23T11:19:18Z

packages/twenty-server/src/engine/workspace-cache-storage/workspace-cache-storage.service.ts

+  async flush(workspaceId: string, metadataVersion?: number): Promise<void> {
+    await this.flushVersionedMetadata(workspaceId, metadataVersion);


style: consider awaiting all cache deletions in parallel using Promise.all for better performance

greptile-apps · 2025-06-23T11:19:20Z

.../metadata-modules/workspace-permissions-cache/workspace-permissions-cache-storage.service.ts

  removeUserWorkspaceRoleMap(workspaceId: string) {
    return this.cacheStorageService.del(
      `${WorkspaceCacheKeys.MetadataPermissionsUserWorkspaceRoleMap}:${workspaceId}`,
    );
  }


logic: Should also remove the version when removing the map to prevent stale version references

...c/engine/metadata-modules/workspace-permissions-cache/workspace-permissions-cache.service.ts

github-actions · 2025-06-23T11:32:39Z

🚀 Preview Environment Ready!

Your preview environment is available at: http://bore.pub:27435

This environment will automatically shut down when the PR is closed or after 5 hours.

charlesBochet · 2025-06-23T12:33:32Z

...server/src/database/typeorm/core/migrations/common/1750673748111-remove-relation-metadata.ts

+    await queryRunner.query(
+      `CREATE INDEX "IDX_DATA_SOURCE_WORKSPACE_ID_CREATED_AT" ON "core"."dataSource" ("workspaceId", "createdAt") `,
+    );
+    await queryRunner.query(


also removing this

packages/twenty-server/src/engine/metadata-modules/data-source/data-source.entity.ts

Weiko

"also introduce self recovery mecanisms to recompute cache automatically if corrupted" I don't see it? You mean you do that by comparing versions in cache VS DB? Or did you introduce something new?
~~Do we really want to remove ignoreLock? What about race conditions?~~
It seems we were already using ignoreLock: true almost everywhere so I guess it's fine
CC @ijreilly for permissions change :)

packages/twenty-server/src/engine/metadata-modules/object-metadata/object-metadata.service.ts

Weiko · 2025-06-23T12:59:30Z

packages/twenty-server/src/engine/metadata-modules/object-metadata/object-metadata.service.ts

@@ -89,6 +94,11 @@ export class ObjectMetadataService extends TypeOrmQueryService<ObjectMetadataEnt
  override async createOne(
    objectMetadataInput: CreateObjectInput,
  ): Promise<ObjectMetadataEntity> {
+    const { objectMetadataMaps } =


Weiko · 2025-06-23T13:01:02Z

...engine/metadata-modules/object-metadata/utils/build-default-fields-for-custom-object.util.ts

@@ -10,6 +11,7 @@ export const buildDefaultFieldsForCustomObject = (
  workspaceId: string,
 ): Partial<FieldMetadataEntity>[] => [
  {
+    id: v4(),


why do we need this?

because I want to avoid making multiple calls to db if not needed

...server/src/database/typeorm/core/migrations/common/1750673748111-remove-relation-metadata.ts

charlesBochet · 2025-06-23T13:53:27Z

"also introduce self recovery mecanisms to recompute cache automatically if corrupted" I don't see it? You mean you do that by comparing versions in cache VS DB? Or did you introduce something new?

~~Do we really want to remove ignoreLock? What about race conditions?~~
It seems we were already using ignoreLock: true almost everywhere so I guess it's fine

CC @ijreilly for permissions change :)

I've introduced 'getFreshMetadata'

Weiko

LGTM

ijreilly · 2025-06-23T13:57:50Z

...server/src/database/typeorm/core/migrations/common/1750673748111-remove-relation-metadata.ts

+    await queryRunner.query(
+      `CREATE INDEX "IDX_DATA_SOURCE_WORKSPACE_ID_CREATED_AT" ON "core"."dataSource" ("workspaceId", "createdAt") `,
+    );
+    await queryRunner.query(


...ine/api/graphql/graphql-query-runner/resolvers/graphql-query-create-many-resolver.service.ts

...ngine/metadata-modules/workspace-metadata-cache/services/workspace-metadata-cache.service.ts

packages/twenty-server/src/engine/api/graphql/workspace-schema.factory.ts

github-actions · 2025-06-23T18:58:55Z

📊 API Changes Report

REST API Changes

Summary

🔄 Changed Operations (63)

/apiKeys/duplicates: Modified operation
/attachments/duplicates: Modified operation
/blocklists/duplicates: Modified operation
/calendarChannelEventAssociations/duplicates: Modified operation
/calendarChannels/duplicates: Modified operation
/calendarEventParticipants/duplicates: Modified operation
/calendarEvents/duplicates: Modified operation
/companies/duplicates: Modified operation
/connectedAccounts/duplicates: Modified operation
/favoriteFolders/duplicates: Modified operation
/favorites/duplicates: Modified operation
/messageChannelMessageAssociations/duplicates: Modified operation
/messageChannels/duplicates: Modified operation
/messageFolders/duplicates: Modified operation
/messageParticipants/duplicates: Modified operation
/messages/duplicates: Modified operation
/messageThreads/duplicates: Modified operation
/notes/duplicates: Modified operation
/noteTargets/duplicates: Modified operation
/opportunities/duplicates: Modified operation
/people/duplicates: Modified operation
/pets/duplicates: Modified operation
/rockets/duplicates: Modified operation
/surveyResults/duplicates: Modified operation
/tasks/duplicates: Modified operation
/taskTargets/duplicates: Modified operation
/timelineActivities/duplicates: Modified operation
/viewFields/duplicates: Modified operation
/viewFilterGroups/duplicates: Modified operation
/viewFilters/duplicates: Modified operation
/viewGroups/duplicates: Modified operation
/views/duplicates: Modified operation
/viewSorts/duplicates: Modified operation
/webhooks/duplicates: Modified operation
/workflowAutomatedTriggers/duplicates: Modified operation
/workflowRuns/duplicates: Modified operation
/workflows/duplicates: Modified operation
/workflowVersions/duplicates: Modified operation
/workspaceMembers/duplicates: Modified operation
/apiKeys: Modified operation
/apiKeys/{id}: Modified operation
/calendarEvents: Modified operation
/calendarEvents/{id}: Modified operation
/companies: Modified operation
/companies/{id}: Modified operation
/opportunities: Modified operation
/opportunities/{id}: Modified operation
/people: Modified operation
/people/{id}: Modified operation
/pets: Modified operation
/pets/{id}: Modified operation
/viewFields: Modified operation
/viewFields/{id}: Modified operation
/viewFilters: Modified operation
/viewFilters/{id}: Modified operation
/viewGroups: Modified operation
/viewGroups/{id}: Modified operation
/views: Modified operation
/views/{id}: Modified operation
/viewSorts: Modified operation
/viewSorts/{id}: Modified operation
/workspaceMembers: Modified operation
/workspaceMembers/{id}: Modified operation

⚠️ Please review these API changes carefully before merging.

⚠️ Breaking Change Protocol

Breaking changes detected but PR title does not contain "breaking" - CI will pass but action needed.

🔄 Options:

If this IS a breaking change: Add "breaking" to your PR title and add BREAKING CHANGE: to your commit message
If this is NOT a breaking change: The API diff tool may have false positives - please review carefully

For breaking changes, add to commit message:

feat: add new API endpoint

BREAKING CHANGE: removed deprecated field from User schema

sentry-io · 2025-06-23T19:25:27Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ TypeError: Cannot read properties of undefined (re... in staging
‼️ TypeError: Cannot read properties of undefined (re... in staging
‼️ TypeError: Cannot read properties of undefined (re... in prod
‼️ TypeError: Cannot read properties of undefined (re... in staging
‼️ TypeError: Cannot read properties of undefined (re... in staging

_{Did you find this useful? React with a 👍 or 👎}

…n front (#12886) Context : - IndexFieldMetadata was no longer available on 'objects' gql query ([since this PR](#12785)). Then, unicity checks on import do not work anymore. Fix : - Add a dataloader logic in indexFieldMetadata - Add extra check in unicity hook on import

Improve performance on metadata computation

4212c33

greptile-apps bot reviewed Jun 23, 2025

View reviewed changes

Fix

6d007fa

charlesBochet commented Jun 23, 2025

View reviewed changes

packages/twenty-server/src/engine/metadata-modules/data-source/data-source.entity.ts Show resolved Hide resolved

Fix

5a62bb6

Weiko reviewed Jun 23, 2025

View reviewed changes

charlesBochet added 3 commits June 23, 2025 15:37

Fix

cfd9e21

Fix

11a0a75

Fix

621e58e

charlesBochet added 2 commits June 23, 2025 16:06

Fix tests

ea6f3b9

Fix tests

9c6d176

Weiko approved these changes Jun 23, 2025

View reviewed changes

charlesBochet added 5 commits June 23, 2025 17:05

More fix

493c08a

Fix

4c38624

Fix

7c03b89

Merge branch 'main' into improve-performance-data-model

ad06503

Fix

907867f

ijreilly approved these changes Jun 23, 2025

View reviewed changes

charlesBochet added 9 commits June 23, 2025 17:48

Fix

d2b0420

Improve performances on fields

75b6413

Fix

dda22e9

Fix

799ec55

Fix

d040993

Fix

3dc4243

Fix tests

bf24dff

Fix

5f3a50e

Remove client config

7039e32

charlesBochet added 2 commits June 23, 2025 20:51

Fix metdata

012a202

Fix testss

0104c18

charlesBochet merged commit d5c9740 into main Jun 23, 2025
57 checks passed

charlesBochet deleted the improve-performance-data-model branch June 23, 2025 19:06

etiennejouan mentioned this pull request Jun 25, 2025

fix IndexFieldMetadata availability in IndexMetadata/ObjectMetadata in front #12886

Merged

		async flush(workspaceId: string, metadataVersion?: number): Promise<void> {
		await this.flushVersionedMetadata(workspaceId, metadataVersion);

Improve performance on metadata computation #12785

Improve performance on metadata computation #12785

Uh oh!

Conversation

charlesBochet commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Improve recompute metadata cache performance. We are aiming for ~100ms

Remove caching lock

Introduce self recovery mecanisms to recompute cache automatically if corrupted

custom object resolver performance improvement: 1sec to 200ms

reduce objectMetadataMaps to 500kb

Add dataloader on IndexMetadata (aka indexMetadataList in the API)

Improve field resolver performances too

Deprecate ClientConfig

Uh oh!

greptile-apps bot left a comment

Choose a reason for hiding this comment

PR Summary

Uh oh!

greptile-apps bot Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps bot Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

greptile-apps bot Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

charlesBochet Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

ijreilly Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Weiko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Weiko Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Weiko Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

charlesBochet Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

charlesBochet commented Jun 23, 2025

Uh oh!

Weiko left a comment

Choose a reason for hiding this comment

Uh oh!

ijreilly Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jun 23, 2025

📊 API Changes Report

REST API Changes

Summary

🔄 Changed Operations (63)

⚠️ Breaking Change Protocol

Uh oh!

Uh oh!

sentry-io bot commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Suspect Issues

Uh oh!

Uh oh!

charlesBochet commented Jun 23, 2025 •

edited

Loading

github-actions bot commented Jun 23, 2025 •

edited

Loading

sentry-io bot commented Jun 23, 2025 •

edited

Loading