Advanced Topics

Performance Optimization

Index Size Considerations

The in-memory search index scales linearly with page count. Each record stores:

• Heading text (average 40-80 characters)
• Body text snippet (extracted from page content)
• URL and breadcrumb (40-100 characters)
• Selector (for actions and fields)

Estimated memory usage:

Parallel Fetching

Reef Search fetches pages in parallel batches to optimize loading time. The default concurrency is 6 requests:

// In search.ts - fetchPagesParallel()
const concurrency = 6;
const results: (IndexRecord[] | null)[] = new Array(urls.length);

let idx = 0;
const fetchBatch = async () => {
  while (idx < urls.length) {
    const i = idx++;
    // Fetch page and extract content
  }
};

// Launch concurrent batch fetchers
await Promise.all([...Array(concurrency)].map(() => fetchBatch()));

To reduce server load, you can lower this by modifying the source, though this is not currently exposed as a configuration option.

Debouncing and Idle Processing

The search input uses requestAnimationFrame debouncing for responsive queries:

this.input?.addEventListener('input', () => {
  this.currentQuery = this.input?.value ?? '';
  this.selectedIndex = 0;
  if (this.searchDebounce) cancelAnimationFrame(this.searchDebounce);
  this.searchDebounce = requestAnimationFrame(() => this.renderResults());
});

Initial indexing runs during browser idle time via requestIdleCallback:

if ('requestIdleCallback' in window) {
  requestIdleCallback(() => this.boot());
} else {
  // Fallback: immediate initialization
  this.boot();
}

Query Performance

Median query time is approximately 38ms for typical indexes. Search uses:

• Prefix index for heading text matching (O(prefix length) lookup)
• Word index for body text matching (O(1) per word lookup)
• Memory-based scoring with sorted results

Security Model

Same-Origin Policy

Reef Search enforces strict same-origin restrictions:

// Only fetch pages from the same origin
const isExternal = !resolvedUrl.startsWith(window.location.origin);
if (isExternal) {
  // Skip external URLs during indexing
  continue;
}

This prevents:

• Data exfiltration from external sites
• Cross-site scripting through fetched content
• Unauthorized access to protected resources

Content Sanitization

All fetched content is processed through multiple sanitization layers:

// In extractSections()
const cleanHtml = html
  .replace(/<script[\s\S]*?<\/script>/gi, ' ')  // Remove scripts
  .replace(/<style[\s\S]*?<\/style>/gi, ' ')    // Remove styles  
  .replace(/<noscript[\s\S]*?<\/noscript>/gi, ' ') // Remove noscript
  .replace(/<!--[\s\S]*?-->/g, ' ');             // Remove comments

Content is parsed with DOMParser and never injected into the live DOM.

Action Safety

The action execution system has multiple safety checks:

Destructive Action Detection

function isDestructiveAction(label: string): boolean {
  const destructiveVerbs = [
    'delete', 'remove', 'cancel subscription', 'unsubscribe',
    'pay', 'checkout', 'submit order', 'confirm'
  ];
  const lowerLabel = label.toLowerCase();
  return destructiveVerbs.some(verb => lowerLabel.includes(verb));
}

Same-Page Execution Check

const isSamePage = result.url === window.location.href.split('#')[0];
const canExecuteHere = isAction && isSamePage && !result.destructive;

Deferred Action Handling

Cross-page actions use sessionStorage for safe deferred execution:

// Store action for next page load
sessionStorage.setItem('reef-deferred-action', JSON.stringify({
  selector: result.selector,
  type: result.type,
  destructive: result.destructive
}));

// Execute on page load
const deferredActionStr = sessionStorage.getItem('reef-deferred-action');
if (deferredActionStr) {
  // Validate and execute the action
}

Content Extraction Internals

Heading Extraction

Headings are extracted using a regex-based parser:

const headingRegexGlobal = /<(h[1-6])[^>]*>([\s\S]*?)<\/h[1-6]>/gi;

while ((match = headingRegexGlobal.exec(cleanHtml)) !== null) {
  const [, tag, text] = match;
  const headingText = stripTags(text);
  const level = parseInt(tag[1], 10);
  // Create section record for each heading
}

Breadcrumb Generation

Breadcrumbs are built by traversing heading hierarchy:

let breadcrumb = '';
for (let j = 0; j <= i; j++) {
  if (j > 0) breadcrumb += ' › ';
  breadcrumb += matches[j].text;
}

This creates a path like: Installation › Advanced Options › Custom Builds

Selector Generation

The selector generator creates robust CSS selectors for elements:

function generateSelector(element: Element): string {
  // Priority: ID > Class + nth-child > Tag + nth-child
  if (element.id) {
    return `tagName#${element.id}`;
  }
  if (element.className) {
    const classes = element.className.trim().split(/\s+/);
    // Include nth-child for specificity
  }
  // ...
}

Search Algorithm Deep Dive

Tokenization

Content is tokenized into three indices:

interface SearchIndex {
  headingIds: Map<string, IndexRecord[]>;  // Exact heading text
  headingIndex: Map<string, IndexRecord[]>;  // Heading prefixes
  bodyIndex: Map<string, IndexRecord[]>;      // Body words
  allSections: IndexRecord[];                // Unfiltered list
}

Scoring Weights

Search results are ranked using weighted scoring:

Fuzzy Matching

Levenshtein distance provides "Did you mean?" suggestions:

function levenshteinDistance(a: string, b: string): number {
  // O(m*n) dynamic programming algorithm
  // Returns edit distance between two strings
}

function findClosestWord(query: string, index: SearchIndex, maxDistance = 2): string | null {
  // Find index terms within edit distance 2
  // Used for typo correction suggestions
}

Implementing min.js Into Your Site

File Size Optimization

The minified bundle is approximately 23KB gzipped. To minimize impact:

• Load with defer or async attribute
• Place before closing </body> tag
• Use a CDN with compression (jsDelivr, unpkg both gzip)

Building From Source

To create a custom build:

# Clone and install
git clone https://github.com/reef-search/reef-search.git
cd reef-search
npm install

# Development mode (with source maps)
npm run dev

# Production build
npm run build

Build Configuration

The build process uses esbuild (see package.json):

// package.json scripts
{
  "scripts": {
    "build": "esbuild src/browser.ts --bundle --minify --outfile=dist/reef.min.js",
    "dev": "esbuild src/browser.ts --bundle --outfile=dist/reef.min.js --serve"
  }
}

Rollup for Advanced Builds

For tree-shaking or custom configurations:

import { rollup } from 'rollup';
import typescript from '@rollup/plugin-typescript';
import { terser } from 'rollup-plugin-terser';

export default {
  input: 'src/browser.ts',
  plugins: [typescript(), terser()],
  output: { file: 'dist/reef.min.js', format: 'iife' }
};

Known Limitations

Client-Side Rendering

Reef Search indexes static HTML only. Client-rendered content (React, Vue, Svelte apps) cannot be indexed from other pages because:

• Fetched HTML doesn't execute JavaScript
• Dynamic content is generated at runtime
• Hydration state is lost in fetch operations
• SPA routes may not exist in sitemap.xml

Large Site Performance

Sites with 10,000+ pages may experience:

• Longer initial indexing time (10-30 seconds)
• Increased memory usage (10+ MB)
• Potential browser tab slowdown during indexing

Mitigations for large sites:

• Reduce data-max-pages to prioritize important content
• Use data-scope to limit content extraction
• Consider pre-generated index files (future feature)

Browser Compatibility

Requires ES6+ features:

• fetch API (IE not supported)
• DOMParser for HTML parsing
• Shadow DOM for UI isolation
• requestIdleCallback (fallback available)

IndexedDB Caching (Planned)

The current implementation indexes in-memory only. Persistent caching via IndexedDB is planned for a future release to:

• Avoid re-indexing on page navigation
• Enable offline search capability
• Respect lastmod timestamps for incremental updates