Semantic Search

ai:true My search now uses AI! tag:clothing

What is it?

Semantic search takes a search queryString and uses an LLM to create an additional query that can help the user find what they're looking for.

Let's say I'm looking for a donut. So I type in food. I get no hits because Donut is an asset in my system but it's named Donut, not food. Maybe the ML tagging kicked in and we can do a tag:food and get a hit that way. Maybe however it didn't or it's not exactly a good tag but semantically it could be a good match. This is when ai:true can help!

How does it work?

We use a hybrid approach combining vector embeddings and LLM reasoning for efficient and accurate semantic search:

Step 1: Vector Similarity Search

Generate an embedding for the user's search terms and perform vector similarity search against pre-computed embeddings of asset names and descriptions. This quickly identifies the most semantically relevant assets.

Step 2: LLM Query Expansion

Send only the top matching assets (typically 10-20) to the LLM for intelligent query expansion:

You are a semantic search assistant. Based on these semantically similar assets, help expand the search query to find related items.

User's search query: "food"

Top semantically similar assets:

• Asset ID: 123, Name: "Donut", Description: "Sweet glazed pastry", Tags: ["bakery", "sweet"], Attributes: {type: "food", category: "pastry"}, Similarity: 0.89
• Asset ID: 456, Name: "Cake", Description: "Birthday celebration dessert", Tags: ["dessert", "celebration"], Attributes: {type: "food", occasion: "party"}, Similarity: 0.85
• Asset ID: 789, Name: "Apple", Description: "Fresh red fruit", Tags: ["fruit", "healthy"], Attributes: {type: "food", category: "produce"}, Similarity: 0.78

Task: Analyze these relevant assets and suggest QueryFilter DSL terms that would help find similar items. Focus on patterns you see in names, tags, and attributes.

Your response should include:

1. Direct name matches: Items whose names are semantically related
2. Tag matches: Tags that commonly appear in similar assets
3. Attribute matches: Attribute patterns that indicate relevance

Example response: +name:donut +name:cake +name:apple +tag:bakery +tag:dessert +tag:fruit +attribute:type:_eq:food

Query Preprocessing Flow

1. Parse Original Query: Extract the base search terms and detect ai:true modifier
2. Generate Query Embedding: Create vector embedding for the search terms
3. Vector Similarity Search: Query against pre-computed asset name/description embeddings
4. Filter Top Matches: Select top 10-20 most similar assets (above similarity threshold, e.g., 0.7)
5. LLM Enhancement: Send only the relevant assets to LLM for semantic expansion
6. Query Reconstruction: Combine original query with LLM suggestions using OR logic
7. Execute Enhanced Query: Run the expanded query through normal QueryFilter DSL processing
8. Return Results: Users see improved results without knowing the technical details

Implementation Examples

Basic Enhancement with Vector Search

Input:  "food ai:true"

Vector Search Results (top 3):
- Donut (similarity: 0.89, tags: ["bakery", "sweet"])
- Cake (similarity: 0.85, tags: ["dessert", "celebration"])
- Apple (similarity: 0.78, tags: ["fruit", "healthy"])

LLM Enhancement: "+name:donut +name:cake +name:apple +tag:bakery +tag:dessert +tag:fruit"
Final Query: "food +name:donut +name:cake +name:apple +tag:bakery +tag:dessert +tag:fruit"

Complex Query Enhancement

Input:  "medieval building ai:true tag:architecture"

Vector Search Results (top 3):
- Castle (similarity: 0.92, tags: ["fortress", "stone"])
- Cathedral (similarity: 0.87, tags: ["gothic", "religious"])
- Manor (similarity: 0.81, tags: ["estate", "noble"])

LLM Enhancement: "+name:castle +name:cathedral +name:manor +tag:fortress +tag:gothic +tag:stone"
Final Query: "medieval building +name:castle +name:cathedral +name:manor +tag:fortress +tag:gothic +tag:stone tag:architecture"

Multi-term Enhancement

Input:  "ai:true transportation space"

Vector Search Results (top 3):
- Spaceship (similarity: 0.94, tags: ["vehicle", "sci-fi"])
- Rocket (similarity: 0.89, tags: ["launch", "propulsion"])
- Space Station (similarity: 0.85, tags: ["orbital", "habitat"])

LLM Enhancement: "+name:spaceship +name:rocket +name:'space station' +tag:vehicle +tag:sci-fi +tag:orbital"
Final Query: "transportation space +name:spaceship +name:rocket +name:'space station' +tag:vehicle +tag:sci-fi +tag:orbital"

User Experience

From the user's perspective, semantic search is completely invisible:

• They add ai:true to their query
• Results magically include more relevant items
• No additional UI complexity or result categorization needed
• They can always remove ai:true to see the difference if curious

Benefits

• Highly Efficient: Vector search quickly filters to relevant assets before LLM processing
• Cost Effective: LLM only processes 10-20 relevant assets instead of entire catalog
• Accurate Results: Embeddings ensure LLM works with genuinely similar assets
• Invisible Enhancement: Better results without UI complexity
• Leverages Existing DSL: Works seamlessly with current QueryFilter system
• Scalable: Performance stays consistent even with massive asset catalogs
• Debuggable: Can log both vector similarities and expanded queries for analysis
• Flexible: Works with any combination of existing DSL modifiers

Technical Considerations

• Embedding Quality: Ensure embeddings capture semantic meaning of asset names/descriptions
• Similarity Threshold: Tune threshold (e.g., 0.7) to balance relevance vs. recall
• Result Count: Optimize number of assets sent to LLM (10-20 typically works well)
• Vector Search Performance: Use appropriate vector database/indexing for your scale
• LLM Context Size: Monitor token usage with asset metadata
• Fallback Strategy: Always execute original query if vector search or LLM enhancement fails
• Caching: Consider caching embeddings and popular query expansions
• Embedding Updates: Plan for re-embedding when asset metadata changes significantly