DB models for custom taxonomies #6421

galvana · 2025-08-05T20:08:17Z

Closes ENG-1073

Description Of Changes

Introduces first-class DB models to support custom taxonomies and cross-taxonomy tagging:
- Taxonomy: defines a taxonomy type and its allowed targets via applies_to
- TaxonomyAllowedUsage: declares which target types/taxonomies a taxonomy can apply to
- TaxonomyElement: hierarchical elements within a taxonomy (supports parent/child)
- TaxonomyUsage: records applications of one taxonomy’s element to another element
Adds Alembic migration to create new tables, indexes, FKs, and seed system-managed legacy taxonomies (data_categories, data_uses, data_subjects, system_groups).
Adds a TaxonomyService with centralized validation and hierarchy activation/deactivation behavior, and a base handler abstraction with a legacy handler implementation.
Routes legacy taxonomy endpoints (data_category, data_use, data_subject) through the new TaxonomyService to unify validation and reactivation behavior.
Comprehensive tests for the new models, service behavior, and constraints.

Potential caveats:

Removing an applies_to target that’s currently referenced by taxonomy_usage will fail due to FK constraints; usages must be removed first.
Deleting a parent TaxonomyElement with existing children is restricted at the DB level; delete or move children first.
The service layer currently supports legacy taxonomies only for API operations; CRUD for fully custom taxonomies/elements/usages is not yet exposed via public endpoints.

Code Changes

New models:
- src/fides/api/models/taxonomy.py (adds Taxonomy, TaxonomyAllowedUsage, TaxonomyElement, TaxonomyUsage)
Migration:
- src/fides/api/alembic/migrations/versions/3baf42d251a6_add_generic_taxonomy_models.py
API changes:
- src/fides/api/api/v1/endpoints/generic_overrides.py (create/update for data uses/categories/subjects now use TaxonomyService; legacy inline validation removed)
DI and model registration:
- src/fides/api/service/deps.py (new get_taxonomy_service)
- src/fides/api/db/base.py (imports new models)
New service layer:
- src/fides/service/taxonomy/__init__.py
- src/fides/service/taxonomy/handlers/base.py
- src/fides/service/taxonomy/handlers/legacy_handler.py
- src/fides/service/taxonomy/taxonomy_service.py
- src/fides/service/taxonomy/utils.py
Dataset updates:
- .fides/db_dataset.yml (alignments for taxonomy-related filtering)
Tests:
- tests/ops/models/test_taxonomy.py
- tests/service/test_taxonomy_service.py

Steps to Confirm

Run targeted tests for models and service:
- docker exec fides bash -c "pytest -q tests/ops/models/test_taxonomy.py"
- docker exec fides bash -c "pytest -q tests/service/test_taxonomy_service.py"
Exercise legacy taxonomy endpoints (requires backend running and auth):
- POST /api/v1/data_category with a new category; expect 201 and generated/validated key behavior.
- PUT /api/v1/data_category to toggle active and verify parent/child cascade semantics via subsequent GETs.
- Repeat for /api/v1/data_use and /api/v1/data_subject.
Database checks after migration:
- Verify tables exist: taxonomy, taxonomy_allowed_usage, taxonomy_element, taxonomy_usage.
- Confirm seeded rows exist for the legacy taxonomies in taxonomy.
Confirm FK behavior:
- Attempt to remove an applies_to target that’s in use (should fail).
- Attempt to delete a TaxonomyElement with children (should be restricted).

Pre-Merge Checklist

vercel · 2025-08-05T20:08:22Z

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Preview	Comments	Updated (UTC)
fides-plus-nightly	Ignored	Preview		Aug 19, 2025 9:45pm
fides-privacy-center	Ignored			Aug 19, 2025 9:45pm

codecov · 2025-08-05T23:04:34Z

Codecov Report

❌ Patch coverage is 93.40278% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.65%. Comparing base (5a6c9fd) to head (daa6282).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/fides/api/api/v1/endpoints/generic_overrides.py	75.00%	4 Missing and 3 partials ⚠️
src/fides/service/taxonomy/handlers/base.py	76.92%	6 Missing ⚠️
src/fides/service/taxonomy/utils.py	90.90%	2 Missing and 3 partials ⚠️
src/fides/api/models/taxonomy.py	98.70%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6421      +/-   ##
==========================================
+ Coverage   87.61%   87.65%   +0.04%     
==========================================
  Files         459      465       +6     
  Lines       29494    29723     +229     
  Branches     3280     3303      +23     
==========================================
+ Hits        25841    26054     +213     
- Misses       2948     2960      +12     
- Partials      705      709       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

galvana · 2025-08-07T23:32:32Z

@greptileai

galvana · 2025-08-09T05:10:06Z

@greptileai

erosselli · 2025-08-13T13:09:28Z

src/fides/api/api/v1/endpoints/generic_overrides.py

@@ -354,133 +346,7 @@ def clean_datasets(
    )


-def activate_taxonomy_parents(


thanks for moving all these to a service class! 😌

erosselli · 2025-08-13T13:09:55Z

src/fides/api/api/v1/endpoints/generic_overrides.py

-        return validate_and_create_taxonomy(db, model, validation_schema, data)
-
-    return validate_and_create_taxonomy(db, model, validation_schema, data)
+# Legacy validation functions removed - functionality moved to TaxonomyService


I don't think we need this comment -- it reads like cursor explaining its changes lol

You right 😄

erosselli · 2025-08-13T13:11:30Z

src/fides/api/models/taxonomy.py

+
+
+class Taxonomy(Base, FidesBase):
+    """The SQL model for taxonomy resources."""


can we maybe add some info to the docstring about how the out-of-the-box taxonomies relate to this model? so that new devs (or us in 2 years lol) running into this code don't have to search around for an explanation

erosselli · 2025-08-13T13:11:58Z

src/fides/api/models/taxonomy.py

+    id = Column(
+        String(255), nullable=False, index=False, default=FidesBase.generate_uuid
+    )


usually we inherit the id column from FidesBase right? is there no issue with re-defining the id column?

also don't we need a name column?

This is fine for the id column, I'm just overriding the base definition so it's not treated as a primary key. As for name, it's inherited from FidesBase.

okay I totally missed that on my first read. So what's the primary key of this table?

src/fides/api/models/taxonomy.py

erosselli · 2025-08-13T13:23:56Z

src/fides/api/models/taxonomy.py

+    __table_args__ = (
+        # Simple two-column index for queries
+        Index("ix_allowed_usage_lookup", "source_taxonomy_key", "target_type"),
+    )


do you need this index given that you've defined (source_taxonomy_key, target_type) as a composite PK with index=True ?

Nice catch! I got rid of the extra index definition

erosselli · 2025-08-14T14:18:23Z

src/fides/service/taxonomy/handlers/base.py

+    @abstractmethod
+    def get_model(self, taxonomy_type: str) -> Type:
+        pass
+


I think we should be able to type this more strictly; maybe we can define a TaxonomyType union type or a base class that other types inherit from?

Actually what I'd really like to know is if we can actually use overloads to do something like the following,(ab?)using the fact that a string literal is actually a type:

@overloads def get_model(self, taxonomy_type: Literal['data_category']) -> DataCategory: @overloads def get_model(self, taxonomy_type: Literal['data_use'])->DataUse: .... other overloads

so if you call get_model('data_category') the type checker will actually know the return type is DataCategory

idk if this would be done here or below in the concrete class

This is nice! I didn't know about @overloads in Python

I too didn't know they existed but whenever I find myself thinking "In Typescript I could do this way cleaner" I go and look up if there's a way to do it in Python 😆 and there was!

src/fides/service/taxonomy/handlers/legacy_handler.py

src/fides/service/taxonomy/utils.py

galvana

Thanks for the recommendations, I think it's a lot cleaner now!

galvana · 2025-08-18T01:29:30Z

src/fides/api/models/taxonomy.py

+    id = Column(
+        String(255), nullable=False, index=False, default=FidesBase.generate_uuid
+    )


This is fine for the id column, I'm just overriding the base definition so it's not treated as a primary key. As for name, it's inherited from FidesBase.

src/fides/api/models/taxonomy.py

galvana · 2025-08-18T01:55:28Z

src/fides/api/models/taxonomy.py

+
+        # Create TaxonomyAllowedUsage objects for each applies_to value
+        allowed_usages = [
+            TaxonomyAllowedUsage(  # type: ignore


I was getting this error from mypy

src/fides/api/models/taxonomy.py:111: error: Unexpected column "source_taxonomy_key" for model "TaxonomyAllowedUsage" [misc] TaxonomyAllowedUsage( ^ src/fides/api/models/taxonomy.py:111: error: Unexpected column "target_type" for model "TaxonomyAllowedUsage" [misc] TaxonomyAllowedUsage( ^

But I just found out you could do this instead (I was missing the Column[str] annotations)

source_taxonomy_key: Column[str] = Column( String, ForeignKey("taxonomy.fides_key", ondelete="CASCADE"), primary_key=True, index=True, ) target_type: Column[str] = Column( String, primary_key=True ) # Can be "system", "dataset", OR a taxonomy key like "data_categories"

src/fides/api/models/taxonomy.py

galvana · 2025-08-18T02:06:45Z

src/fides/api/models/taxonomy.py

+    __table_args__ = (
+        # Simple two-column index for queries
+        Index("ix_allowed_usage_lookup", "source_taxonomy_key", "target_type"),
+    )


Nice catch! I got rid of the extra index definition

galvana · 2025-08-18T03:14:59Z

src/fides/service/taxonomy/handlers/base.py

+    @abstractmethod
+    def get_model(self, taxonomy_type: str) -> Type:
+        pass
+


This is nice! I didn't know about @overloads in Python

src/fides/service/taxonomy/handlers/legacy_handler.py

src/fides/service/taxonomy/utils.py

galvana · 2025-08-18T03:46:39Z

src/fides/api/api/v1/endpoints/generic_overrides.py

-        return validate_and_create_taxonomy(db, model, validation_schema, data)
-
-    return validate_and_create_taxonomy(db, model, validation_schema, data)
+# Legacy validation functions removed - functionality moved to TaxonomyService


You right 😄

cypress · 2025-08-19T22:08:18Z

fides Run #13252

Run Properties: Passed #13252 • 047dc431c1: DB models for custom taxonomies (#6421)

Project	`fides`
Branch Review	`main`
Run status	`Passed #13252`
Run duration	`00m 54s`
Commit	`047dc431c1: DB models for custom taxonomies (#6421)`
Committer	`Adrian Galvan`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`5`
View all changes introduced in this branch ↗︎

DB models for custom taxonomies

dbf620b

This comment was marked as outdated.

Sign in to view

Fixing migrations

8b668fa

galvana and others added 5 commits August 5, 2025 20:42

Simplifying models and adding tests

691a562

Fixing tests

8dd29c8

Updating dataset

69b579d

Updating dataset

8edd68f

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

1efeeec

This comment was marked as outdated.

Sign in to view

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

d6a9b8a

vercel bot deployed to Preview – fides-plus-nightly August 9, 2025 00:33 View deployment

galvana added 3 commits August 8, 2025 18:03

Misc migration cleanup

7f65775

Adding tests

c4b2152

Removing inline commit

c359c79

This comment was marked as outdated.

Sign in to view

galvana added 7 commits August 11, 2025 08:49

Adding more tests

4b5b7dc

Fixing migration and adding tests

be47d01

Fixing types

990e318

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

4c2314d

Fixing validation for update_element

4fc6f56

Fixing tests and formatting

1bed4f7

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

192164d

galvana requested a review from erosselli August 12, 2025 00:09

erosselli reviewed Aug 14, 2025

View reviewed changes

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

91527bc

vercel bot deployed to Preview – fides-plus-nightly August 14, 2025 15:53 View deployment

galvana added 2 commits August 17, 2025 21:08

Changes based on PR feedback

409abee

Loosening type

cdc2f42

galvana commented Aug 18, 2025

View reviewed changes

galvana requested a review from erosselli August 18, 2025 05:27

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

4c15c61

vercel bot deployed to Preview – fides-plus-nightly August 18, 2025 05:31 View deployment

galvana added 6 commits August 18, 2025 13:04

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

6079b9e

Bumping downrev

6d32169

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

46a2362

Updating taxonomy fides_keys

fe4ae15

Additional changes based on PR feedback

ce8616e

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

60a2085

erosselli approved these changes Aug 19, 2025

View reviewed changes

vercel bot deployed to Preview – fides-plus-nightly August 19, 2025 19:54 View deployment

Merge branch 'main' into ENG-1073-db-models-for-custom-taxonomies

daa6282

galvana merged commit 047dc43 into main Aug 19, 2025
17 checks passed

galvana deleted the ENG-1073-db-models-for-custom-taxonomies branch August 19, 2025 21:51

		@@ -354,133 +346,7 @@ def clean_datasets(
		)


		def activate_taxonomy_parents(



		class Taxonomy(Base, FidesBase):
		"""The SQL model for taxonomy resources."""

DB models for custom taxonomies #6421

DB models for custom taxonomies #6421

Uh oh!

Conversation

galvana commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description Of Changes

Code Changes

Steps to Confirm

Pre-Merge Checklist

Uh oh!

vercel bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

codecov bot commented Aug 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

galvana commented Aug 7, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

galvana commented Aug 9, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

erosselli Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

galvana left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

galvana commented Aug 5, 2025 •

edited

Loading

vercel bot commented Aug 5, 2025 •

edited

Loading

codecov bot commented Aug 5, 2025 •

edited

Loading

erosselli Aug 18, 2025 •

edited

Loading