balanceboard/FILTER_PIPELINE.md

Filter Pipeline Documentation

Overview

BalanceBoard's Filter Pipeline is a plugin-based content filtering system that provides intelligent categorization, moderation, and ranking of posts using AI-powered analysis with aggressive caching for cost efficiency.

Architecture

Three-Level Caching System

1. Level 1: Memory Cache (5-minute TTL)

   • In-memory cache for fast repeated access
   • Cleared on application restart

2. Level 2: AI Analysis Cache (Permanent, content-hash based)

   • Stores AI results (categorization, moderation, quality scores)
   • Keyed by SHA-256 hash of content
   • Never expires - same content always returns cached results
   • Huge cost savings: Never re-analyze the same content

3. Level 3: Filterset Results Cache (24-hour TTL)

   • Stores final filter results per filterset
   • Invalidated when filterset definition changes
   • Enables instant filterset switching

Pipeline Stages

Posts flow through 4 sequential stages:

Raw Post
  ↓
1. Categorizer  (AI: topic detection, tags)
  ↓
2. Moderator    (AI: safety, quality, sentiment)
  ↓
3. Filter       (Rules: apply filterset conditions)
  ↓
4. Ranker       (Score: quality + recency + source + engagement)
  ↓
Filtered & Scored Post

Stage 1: Categorizer

• Purpose: Detect topics and assign categories
• AI Model: Llama 70B (cheap model)
• Caching: Permanent (by content hash)
• Output: Categories, category scores, tags

Stage 2: Moderator

• Purpose: Safety and quality analysis
• AI Model: Llama 70B (cheap model)
• Caching: Permanent (by content hash)
• Metrics:
  • Violence score (0.0-1.0)
  • Sexual content score (0.0-1.0)
  • Hate speech score (0.0-1.0)
  • Harassment score (0.0-1.0)
  • Quality score (0.0-1.0)
  • Sentiment (positive/neutral/negative)

Stage 3: Filter

• Purpose: Apply filterset rules
• AI: None (fast rule evaluation)
• Rules Supported:
  • equals, not_equals
  • in, not_in
  • min, max
  • includes_any, excludes

Stage 4: Ranker

• Purpose: Calculate relevance scores
• Scoring Factors:
  • Quality (30%): From Moderator stage
  • Recency (25%): Age-based decay
  • Source Tier (25%): Platform reputation
  • Engagement (20%): Upvotes + comments

Configuration

filter_config.json

{
  "ai": {
    "enabled": false,
    "openrouter_key_file": "openrouter_key.txt",
    "models": {
      "cheap": "meta-llama/llama-3.3-70b-instruct",
      "smart": "meta-llama/llama-3.3-70b-instruct"
    },
    "parallel_workers": 10,
    "timeout_seconds": 60
  },
  "cache": {
    "enabled": true,
    "ai_cache_dir": "data/filter_cache",
    "filterset_cache_ttl_hours": 24
  },
  "pipeline": {
    "default_stages": ["categorizer", "moderator", "filter", "ranker"],
    "batch_size": 50,
    "enable_parallel": true
  }
}

filtersets.json

Each filterset defines filtering rules:

{
  "safe_content": {
    "description": "Filter for safe, family-friendly content",
    "post_rules": {
      "moderation.flags.is_safe": {"equals": true},
      "moderation.content_safety.violence": {"max": 0.3},
      "moderation.content_safety.sexual_content": {"max": 0.2},
      "moderation.content_safety.hate_speech": {"max": 0.1}
    },
    "comment_rules": {
      "moderation.flags.is_safe": {"equals": true}
    }
  }
}

Usage

User Perspective

1. Navigate to Settings → Filters
2. Select a filterset from the dropdown
3. Save preferences
4. Feed automatically applies your filterset
5. Posts sorted by relevance score (highest first)

Developer Perspective

from filter_pipeline import FilterEngine

# Get singleton instance
engine = FilterEngine.get_instance()

# Apply filterset to posts
filtered_posts = engine.apply_filterset(
    posts=raw_posts,
    filterset_name='safe_content',
    use_cache=True
)

# Access filter metadata
for post in filtered_posts:
    score = post['_filter_score']  # 0.0-1.0
    categories = post['_filter_categories']  # ['technology', 'programming']
    tags = post['_filter_tags']  # ['reddit', 'python']

AI Integration

Enabling AI

1. Get OpenRouter API Key:

   • Sign up at https://openrouter.ai
   • Generate API key

2. Configure:

   echo "your-api-key-here" > openrouter_key.txt
   

3. Enable in config:

   {
     "ai": {
       "enabled": true
     }
   }
   

4. Restart application

Cost Efficiency

• Model: Llama 70B only (~$0.0003/1K tokens)
• Caching: Permanent AI result cache
• Estimate: ~$0.001 per post (first time), $0 (cached)
• 10,000 posts: ~$10 first time, ~$0 cached

Performance

Benchmarks

• With Cache Hit: < 10ms per post
• With Cache Miss (AI): ~500ms per post
• Parallel Processing: 10 workers (configurable)
• Typical Feed Load: 100 posts in < 1 second (cached)

Cache Hit Rates

After initial processing:

• AI Cache: ~95% hit rate (content rarely changes)
• Filterset Cache: ~80% hit rate (depends on TTL)
• Memory Cache: ~60% hit rate (5min TTL)

Monitoring

Cache Statistics

stats = filter_engine.get_cache_stats()
# {
#   'memory_cache_size': 150,
#   'ai_cache_size': 5000,
#   'filterset_cache_size': 8,
#   'ai_cache_dir': '/app/data/filter_cache',
#   'filterset_cache_dir': '/app/data/filter_cache/filtersets'
# }

Logs

Filter pipeline logs to app.log:

INFO - FilterEngine initialized with 5 filtersets
DEBUG - Categorizer: Cache hit for a3f5c2e8...
DEBUG - Moderator: Analyzed b7d9e1f3... (quality: 0.75)
DEBUG - Filter: Post passed filterset 'safe_content'
DEBUG - Ranker: Post score=0.82 (q:0.75, r:0.90, s:0.70, e:0.85)

Filtersets

no_filter

• Description: No filtering - all content passes
• Use Case: Default, unfiltered feed
• Rules: None
• AI: Disabled

safe_content

• Description: Family-friendly content only
• Use Case: Safe browsing
• Rules:
  • Violence < 0.3
  • Sexual content < 0.2
  • Hate speech < 0.1
• AI: Required

tech_only

• Description: Technology and programming content
• Use Case: Tech professionals
• Rules:
  • Platform: hackernews, reddit, lobsters, stackoverflow
  • Topics: technology, programming, software (confidence > 0.5)
• AI: Required

high_quality

• Description: High quality posts only
• Use Case: Curated feed
• Rules:
  • Score ≥ 10
  • Quality ≥ 0.6
  • Readability grade ≤ 14
• AI: Required

Plugin System

Creating Custom Plugins

from filter_pipeline.plugins import BaseFilterPlugin

class MyCustomPlugin(BaseFilterPlugin):
    def get_name(self) -> str:
        return "MyCustomFilter"

    def should_filter(self, post: dict, context: dict = None) -> bool:
        # Return True to filter OUT (reject) post
        title = post.get('title', '').lower()
        return 'spam' in title

    def score(self, post: dict, context: dict = None) -> float:
        # Return score 0.0-1.0
        return 0.5

Built-in Plugins

• KeywordFilterPlugin: Blocklist/allowlist filtering
• QualityFilterPlugin: Length, caps, clickbait detection

Troubleshooting

Issue: AI Not Working

Check:

1. filter_config.json: "enabled": true
2. OpenRouter API key file exists
3. Logs for API errors

Solution:

# Test API key
curl -H "Authorization: Bearer $(cat openrouter_key.txt)" \
  https://openrouter.ai/api/v1/models

Issue: Posts Not Filtered

Check:

1. User has filterset selected in settings
2. Filterset exists in filtersets.json
3. Posts match filter rules

Solution:

# Check user settings
user_settings = json.loads(current_user.settings)
print(user_settings.get('filter_set'))  # Should not be 'no_filter'

Issue: Slow Performance

Check:

1. Cache enabled in config
2. Cache hit rates
3. Parallel processing enabled

Solution:

{
  "cache": {"enabled": true},
  "pipeline": {"enable_parallel": true, "parallel_workers": 10}
}

Future Enhancements

• Database persistence for FilterResults
• Filter statistics dashboard
• Custom user-defined filtersets
• A/B testing different filter configurations
• Real-time filter updates without restart
• Multi-language support
• Advanced ML models for categorization

Contributing

When adding new filtersets:

1. Define in filtersets.json
2. Test with sample posts
3. Document rules and use case
4. Consider AI requirements

When adding new stages:

1. Extend BaseStage
2. Implement process() method
3. Use caching where appropriate
4. Add to pipeline_config.json

License

AGPL-3.0 with commercial licensing option (see LICENSE file)