MCP Documentation Assistant

Smart Content Scraping

• Automatically detects and scrapes documentation pages and related content
• Intelligent filtering of relevant documentation links
• Context-aware scraping based on query complexity
• Efficient extraction of main content while filtering out navigation and irrelevant sections

AI-Powered Answers

• Uses Google's Gemini AI to generate accurate, contextual responses
• Carefully crafted prompts to ensure high-quality answers
• Handles complex technical questions with detailed explanations
• Maintains focus on provided content to ensure accuracy

Multi-Page Context

• Intelligently gathers information from related documentation pages
• Automatically identifies and follows relevant documentation links
• Combines content from multiple sources for comprehensive answers
• Adapts search depth based on query complexity

Caching System

• Implements KV storage to cache responses and reduce API calls
• Separate caching for content and responses with appropriate TTLs
• Content cached for one week to minimize scraping operations
• Responses cached for 24 hours to balance freshness and performance

Citation Support

• Provides answers with specific citations to source content
• Clear attribution using square bracket notation
• Source URLs included for reference
• Distinguishes between main and related page content

Error Handling

• Graceful fallbacks for scraping issues
• User-friendly error messages
• Automatic handling of rate limits and API failures
• Robust error recovery strategies

Worker Entrypoint (src/index.ts)

• Main Cloudflare Worker implementation using MCP framework
• Handles request routing and environment configuration
• Implements caching strategy with KV storage
• Provides user-friendly error handling

Content Scraper (src/utils.ts)

• Utilizes FireCrawl SDK for efficient web scraping
• Implements intelligent related page detection
• Filters and processes content for optimal AI consumption
• Handles batch scraping for related pages

AI Integration (src/utils.ts)

• Integrates with Google's Gemini AI
• Formats prompts for optimal response quality
• Ensures proper citation and content focus
• Handles AI response processing

Caching System

• Implements dual-layer caching strategy
• Separate caches for content and responses
• Optimized TTLs for different cache types
• Efficient cache key generation

Content Scraping Logic

The content scraping system intelligently adapts to the complexity of the query and the type of website being analyzed:

export async function scrapeContent(url: string, apiKey: string, isComplex: boolean = false): Promise {
  const app = new FirecrawlApp({ apiKey });
  
  // Always scrape the main URL first
  const mainResult = await app.scrapeUrl(url, { 
    formats: ['markdown', 'links'],
    onlyMainContent: true
  }) as ScrapeResponse;
  
  if (!mainResult.success || !mainResult.markdown) {
    throw new Error(`Failed to scrape: ${mainResult.error || 'Unknown error'}`);
  }
  
  // For simple queries or non-doc sites, return just the main content
  if (!isComplex || !url.match(/docs|documentation|github\.com|api|guide/)) {
    return mainResult.markdown;
  }
  
  // For complex queries on doc sites, try to add related content
  try {
    // Get up to 2 related pages from the links in the main page
    const relatedUrls = (mainResult.links || [])
      .filter(linkedUrl => isRelevantDocUrl(linkedUrl, url))
      .slice(0, 2);
    
    if (relatedUrls.length === 0) {
      return mainResult.markdown;
    }
    
    // Use batch scraping for efficiency
    const batchResult = await app.batchScrapeUrls(relatedUrls, {
      formats: ['markdown'],
      onlyMainContent: true
    });
    
    // Combine all content with source labels
    const allContent = [`## Main Page: ${url}\n\n${mainResult.markdown}`];
    
    if (batchResult.success && batchResult.data) {
      batchResult.data.forEach((result: any) => {
        if (result.success && result.markdown) {
          allContent.push(`## Related Page: ${result.metadata.url}\n\n${result.markdown}`);
        }
      });
    }
    
    return allContent.join('\n\n');
  } catch (error) {
    // If related page scraping fails, just return the main content
    console.error('Error getting related pages:', error);
    return mainResult.markdown;
  }
}

AI Prompt Formatting

The system uses carefully crafted prompts to ensure high-quality, focused responses:

export function formatContentQuery(content: string, question: string): string {
  return `You are a helpful AI assistant that provides accurate, concise answers based on the given content.
Your task is to answer questions using ONLY the information provided.
If the answer cannot be found in the content, say so clearly.
Always include specific citations in [square brackets] and present the most relevant information first.

Content:
${content}

Question: ${question}

Answer: `;
}

Caching Implementation

The caching system uses Cloudflare KV storage with separate caches for content and responses:

async askAboutUrl(url: string, question: string): Promise {
  // Use two distinct cache keys
  const responseCacheKey = `r:${url}:${question}`;
  const contentCacheKey = `c:${url}`;
  
  // Try to get the AI response from cache first
  const cachedResponse = await this.env.CONTENT_CACHE.get(responseCacheKey);
  if (cachedResponse) {
    return cachedResponse;
  }

  try {
    // Determine complexity of the query
    const isComplexQuery = question.length > 50 || 
                         question.includes('how') ||
                         question.includes('explain') ||
                         question.includes('compare');
    
    // Try to get scraped content from cache
    let content = await this.env.CONTENT_CACHE.get(contentCacheKey);
    
    // If no cached content, scrape it
    if (!content) {
      content = await scrapeContent(url, this.env.FIRE_CRAWL_API_KEY, isComplexQuery);
      
      // Cache scraped content for 1 week
      await this.env.CONTENT_CACHE.put(contentCacheKey, content, { expirationTtl: 604800 });
    }
    
    // Format prompt and get AI response
    const prompt = formatContentQuery(content, question);
    const response = await ask(prompt, this.env.GEMINI_API_KEY);
    
    // Cache the response for 24 hours
    await this.env.CONTENT_CACHE.put(responseCacheKey, response, { expirationTtl: 86400 });
    
    return response;
  } catch (error) {
    // Error handling...
  }
}

Basic Usage

The MCP Documentation Assistant exposes a single endpoint through the MCP framework:

async askAboutUrl(url: string, question: string): Promise

Example usage:

const worker = new MyWorker();
const answer = await worker.askAboutUrl(
  'https://docs.example.com/api',
  'What are the authentication methods?'
);

Response Format

Authorization: Bearer YOUR_API_KEY

Prerequisites

• Node.js and npm installed
• Cloudflare Workers account
• FireCrawl API key
• Google Gemini API key

Environment Configuration

The worker requires the following environment variables:

• SHARED_SECRET: Secret key for MCP authentication
• FIRE_CRAWL_API_KEY: API key for FireCrawl service
• GEMINI_API_KEY: API key for Google's Gemini AI
• CONTENT_CACHE: KV namespace binding for caching

Deployment

Deploy to Cloudflare Workers with:

npm run deploy