Dibya Sarothi
Dibya Sarothi
Developer ToolsProduction ReadyAugust 2026

Autonomous Web & Data Scraping Suite

High-Throughput Selenium & Python Workflow Automation

Industrial-grade web automation pipelines handling dynamic SPAs, authentication states, and tabular synchronization.

Autonomous Web & Data Scraping Suite overview
Project Overview

A suite of resilient data ingestion and browser automation tools built with Python, Selenium, and BeautifulSoup. Engineered to bypass brittle layout changes, handle dynamic infinite-scroll interfaces, and reliably synchronize parsed data into structured relational tables and Google Sheets.

Problem Statement

Ad-hoc web scraping scripts break frequently due to asynchronous DOM updates, aggressive rate limits, and fragile CSS selectors.

Engineering Rationale

Automated business operations demand reliable data ingestion pipelines that log anomalies, retry gracefully, and validate extracted data types.

Engineering Objectives

Build resilient selectors with multi-attribute fallbacks.
Implement exponential backoff and randomized human-like delays.
Automate data entry into Google Forms, Sheets, and SQL backends.
Architecture Blueprint

System Architecture

Modular Python pipeline with decoupled extraction, validation, and storage layers.

01

Self-Healing Selector Logic

Uses semantic DOM proximity searches when primary class selectors undergo obfuscation.

Core & Backend Implementation

  • Selenium WebDriver orchestrator with headless execution configurations.
  • BeautifulSoup parsing pipelines with schema-level data type coercion.
  • Google Sheets API v4 integration with OAuth2 client credential rotation.

Client & Interface Layer

  • CLI status visualizer and lightweight logging dashboard.

Data & Schema Design

Engine / Tier: SQLite / Google Sheets API

Normalized extraction records with deduplication hashes.

Core Models & Entities
ListingRecordScrapeSessionLogSyncAudit

Authentication & Security Controls

Auth Model: Encrypted environment secrets and OAuth2 token refresh
  • Zero hard-coded credentials and safe proxy rotation.

Engineering Challenges & Technical Decisions

Key Challenges Solved

Challenge: Handling infinite scroll lists that dynamically remove off-screen DOM nodes.
Engineering Solution: Implemented incremental scroll-and-capture loops with memory deduplication sets.

Trade-Off Decisions

Decision: Pydantic Schemas for Extracted Records
Architectural Rationale: Guarantees no malformed or truncated fields reach downstream database sinks.

Testing & Verification Methodology

Frameworks & Approach: Pytest, Mock Server
  • Synthetic test page scrapers testing pagination edge cases.

Results & Engineering Takeaways

Mastered asynchronous DOM mechanics, headless browser performance, and resilient data sanitization.