Secure Web Scraping From Leasing & Ev Sites

By Author: Den Rediant
Total Articles: 105
Comment this article

Automating Data Collection from Leasing & EV Websites with iWeb Data Scraping
A client approached iWeb Data Scraping with a clear but technically challenging request: extract structured, up-to-date data from three industry-specific websites—a vehicle leasing group, a car manufacturer’s leasing portal, and an EV charging station network—each protected by login authentication. The end goal was to consolidate the data into clean CSV files for analytics and business decision-making. The project involved securely managing client-provided credentials while navigating complex login-based authentication flows, including sessions, cookies, and CSRF tokens. Our team scraped both structured and semi-structured data from protected areas across three industry-specific websites. The extracted information was then normalized, deduplicated, and delivered as clean CSV files ready for analytics and business decision-making.

Goals and Deliverables
Primary objectives

Access login-protected portals using provided authentication credentials.
Extract structured records:
Leasing Group Website: ...
... vehicle model, lease terms, monthly payment, mileage limits, special offers
Car Company Portal: available leasing inventory, VIN, trim, color, availability date, location
Charging Station Network: station name, address, charger type, availability status, cost per kWh, network ID
Normalize data into a standard schema for CSV export.
Ensure data compliance—no bypassing of technical protections outside approved credentials.
Challenges
Authentication handling

Each portal had different login flows—some required multi-step authentication with CSRF token handling, while others used session cookies that expired quickly.

Dynamic content

Leasing portals often used React/Angular frameworks, requiring headless browser automation to capture data.
Pagination and filtering

Data was split across multiple pages or required applying search filters before results appeared.
Rate limits

Scraping too fast risked temporary account suspension. We implemented rate-aware crawlers.
Data consistency

Vehicle leasing offers could change daily; charging station availability could change hourly.

Our Approach
Step 1: Secure Authentication Handling

Used Selenium and Playwright for sites with heavy JavaScript rendering.
Stored credentials securely in encrypted variables; never hardcoded.
Managed cookies and tokens to maintain authenticated sessions.
Logged session flows for troubleshooting but excluded sensitive details from logs.
Step 2: Dynamic Content Extraction

Used DOM parsing for table-based data.
For infinite scroll pages, simulated scroll events until all content loaded.
Captured hidden API endpoints (when legally allowed) to fetch structured JSON responses.
Step 3: Data Normalization

Converted date formats to ISO 8601.
Unified currency formats (USD).
Standardized vehicle naming (Make, Model, Trim).
Mapped charging station connector types (CCS, CHAdeMO, Tesla) to a standard list.
Step 4: Quality Assurance

Validated VIN formats with checksum.
Verified addresses with Google Maps API for geocoding.
Checked for duplicate leasing offers and merged where necessary.
Technical Stack
Playwright / Selenium for login handling and rendering
Python (BeautifulSoup, Pandas) for parsing and data wrangling
PostgreSQL temporary staging before CSV export
Docker container for isolated execution
AWS S3 for secure CSV delivery

Sample Data (Illustrative)
Leasing Group CSV

Make Model Trim Monthly Payment Lease Term (Months) Mileage Limit Offer Expiry State Dealer Name
Toyota Camry XSE V6 $399 36 12,000 2025-09-30 CA LA Toyota
BMW X5 xDrive40i $799 36 10,000 2025-09-15 NY BMW Manhattan
Car Company Portal CSV

VIN Make Model Trim Color Availability Date Location
1HGCM82633A1 Honda Civic EX-L Red 2025-08-25 Houston, TX
3VWDP7AJ9DM1 VW Jetta SEL White 2025-08-28 Miami, FL
Charging Station Network CSV

Station Name Address Charger Type Availability Cost per kWh Network ID
EV FastCharge #102 123 Main St, Austin, TX CCS Available $0.25 EVNET123
GreenCharge 204 45 Oak Ave, Sacramento, CA CHAdeMO In Use $0.30 EVNET456

Results
Total records extracted:
Leasing Group: 4,200 active offers
Car Company Portal: 7,800 vehicles
Charging Station Network: 1,300 stations with live status
Accuracy rate: 98% after QA pass
Delivery format: CSV with clean headers, ready for analytics tools or CRM imports
Turnaround time: 2.5 weeks from kickoff to delivery
Compliance and Security
Only accessed authenticated areas with client-provided credentials.
Respected each site’s terms of service and relevant laws.
Implemented IP rotation only when approved and within allowed request volumes.
No password storage—credentials provided at runtime.

Client Impact
The client now runs:

Lease offer trend analysis across brands and regions
Inventory monitoring for high-demand vehicle trims
Charging station uptime tracking for network partnerships
By automating this process, manual effort dropped from 40+ hours per week to under 2 hours for periodic reviews.