eBay Data Collection Engine

Jan 2025 - Feb 2025

Built a production-ready C# library for collecting eBay product data using the official eBay Browse API. The engine provides automatic OAuth 2.0 authentication, comprehensive data extraction including full seller business information, intelligent caching, and rate limiting — all through a simple, developer-friendly interface with one-line dependency injection setup.

Key Features

  • Automatic OAuth 2.0 Token Management: Client credentials flow with token caching and automatic refresh, eliminating manual token updates.
  • Multi-Marketplace Support: Configurable marketplace ID enables searching across different eBay country sites (UK, US, Germany, etc.) with correct currency and locale.
  • Complete Data Extraction: Collects all product images (1-16 per listing), full seller business information (company name, email, phone, VAT ID, legal address, contacts), product details, and metadata.
  • Variant Expansion & Grouping: Expands variant listings (colour, size, model) into individual items and builds hierarchical item groups with aggregated options for UI/catalogue use.
  • Intelligent Rate Limiting: Configurable request throttling via a custom HTTP handler pipeline to respect API limits and prevent 429 errors.
  • Duplicate Detection: Multi-key deduplication using item ID, URL, and GUID to prevent collecting the same listing twice.
  • Regex Filtering: Client-side filtering of search results by pattern matching against item titles.

Architecture

The solution follows a clean, layered architecture with three projects:

  • Ebay.DataCollection.Engine: The core reusable library containing configuration, models, service abstractions, HTTP infrastructure, and orchestration logic. Consumers integrate this into any .NET application with a single line: builder.Services.AddEbayEngine(builder.Configuration).
  • Ebay.DataCollection.Console: A reference implementation demonstrating both raw API access and full pipeline orchestration with timing metrics and data export.
  • Ebay.DataCollection.Tests: Comprehensive unit test suite using xUnit, Moq, and FluentAssertions covering all services, DTOs, mappings, and HTTP handlers.

Technical Highlights

  • HTTP Handler Pipeline: Three-layer delegating handler chain (BearerAuth, MarketplaceHeader, Throttling) automatically enhances every API request.
  • Refit Integration: Type-safe HTTP client generation from interface definitions for the eBay Browse API.
  • Smart Seller Caching: Progressive enrichment — basic seller data from search results is updated with comprehensive business details from item detail calls, always retaining the most complete information.
  • Full Test Coverage: 30+ unit tests validating authentication, search, enrichment, caching, deduplication, throttling, and DTO mapping with edge cases.
  • Dual Export Formats: Outputs both flat enriched items (database-ready) and hierarchical variant groups (UI/catalogue-ready) as JSON.
LinkedIn