Data pipelines that
keep running.

Price monitoring, catalogue collection, competitor tracking, feeds that have to land in your database every morning. Built to survive the day a source changes its markup, and built within robots.txt, terms of service and sane rate limits. Prepaid in Hour Blocks of 6, 12 or 24 hours.

Price monitoring Catalogue collection Change detection Scheduled pipelines Robots & rate limits

Within two business days

Your message goes straight to the studio. No sales queue, no account manager in between.

Prepaid Hour Blocks

Buy 6, 12, or 24 hours up front at $30/hr. Unused hours never expire.

10 live products

We run our own price-comparison and aggregation platforms, so the failure modes here are ones we have personally been woken up by.

You own the code

Full source, in your repository, with the reasoning written down. No lock-in to us or anyone else.

What we will and will not build.

This section is first on purpose. Scraping work goes wrong through carelessness far more often than through malice, and the constraints shape the engineering - so they belong before the feature list, not buried in a footnote.

Will

What we do build

Collection of publicly reachable pages, at polite request rates, honouring robots.txt and any declared crawl delay, with an honest user agent that says who the crawler is and how to contact its owner. Official APIs, product feeds and sitemaps used first wherever they exist, because they are cheaper, more reliable and unambiguous. Conditional requests and caching so we are not re-downloading pages that have not changed.

Will not

What we decline

Anything that logs into an account you are not entitled to use, defeats a paywall, or works around a bot-protection challenge. Request volumes that would degrade a small site. Collecting personal data without a lawful basis you can point to. Copying a database wholesale where database rights apply. If your project needs one of these, we are the wrong studio, and we will say so in the first reply rather than after you have paid.

On personal data: publicly visible does not mean free to collect. Under the GDPR you still need a lawful basis, individuals keep their access and deletion rights, and you owe them notice even when you did not collect the data from them directly. Our default design collects no personal data at all. If your use case genuinely needs it, that is a decision for your own legal advice, and we will want to see that it has been taken before building the collection side. We are developers, not your lawyers - this is where we draw our own line, not legal advice.

The pipeline, end to end.

Fetching pages is the easy quarter of this work. The other three quarters are what decides whether the data is trustworthy six months from now.

Collection that does not hammer anyone

Plain HTTP first, because it is an order of magnitude cheaper and steadier than a browser. Many pages that look JavaScript-driven are calling a JSON endpoint you can call directly, or already carry structured product data in the HTML. Where a real browser is genuinely required, it is used deliberately and sparingly. Requests are rate-limited per host with exponential backoff, and conditional requests avoid re-downloading pages that have not changed.

  • Sitemaps, feeds and official APIs first
  • Structured data extracted where present
  • Headless browser only where required
  • Per-host rate limiting and backoff
  • Conditional requests and response caching
  • Honest user agent with contact details

Extraction and validation

Extraction rules live in configuration rather than scattered through code, so adjusting to a redesign is a config change and a test, not an archaeology expedition. Every extracted field is validated against an expected type and range: a price that parses as zero, a name that comes back empty, a currency that changed - each is a rejected record with a reason attached rather than a silent row in your database.

  • Per-source extraction config, not hardcoding
  • Type, range and format validation
  • Units, currency and decimal normalisation
  • Rejected records kept with a reason
  • Raw payloads retained for re-parsing
  • Re-running history after a rule fix

Matching the same thing across sources

The genuinely hard part of price comparison. Two retailers describe the same product with different names, different pack sizes and different capitalisation, and one of them has a typo. Barcodes and manufacturer part numbers first where they exist, then normalisation, then fuzzy matching with a confidence score. Anything below the threshold waits for a human instead of being merged, because one wrong match costs more trust than fifty missing rows.

  • Barcode and part-number matching first
  • Name, unit and pack-size normalisation
  • Fuzzy matching with confidence scoring
  • Human review queue for weak matches
  • Manual overrides that survive re-runs
  • Duplicate detection within a single source

Storage, history and change detection

How you store it decides what you can ask later. Price history as an append-only series rather than an overwritten column, so a chart is possible a year from now. Delta detection so you know a value changed and when, not merely what it is today. Indexes designed for the queries you will actually run, and a retention policy decided early rather than when the table hits a hundred million rows.

  • Append-only history, not overwritten fields
  • Change detection with timestamps
  • Postgres, MySQL or SQLite as fits the scale
  • Indexes for your real query patterns
  • Retention and archival decided up front
  • Export to CSV, JSON or your warehouse

Running unattended

A pipeline nobody is watching will fail quietly, and quiet failure is the expensive kind - you keep making decisions on data that stopped updating three weeks ago. Every run records what it fetched, what it rejected and how long it took. Alerts fire on the shape of the data, not only on crashes: a source that returns half its usual row count is a problem even when every request returned a clean response.

  • Scheduled runs with per-run audit records
  • Alerts on volume and field-fill anomalies
  • Retries with backoff, then a dead-letter path
  • Partial failure that does not poison the batch
  • A dashboard or digest showing source health
  • Deployment as systemd or container jobs

Getting the data somewhere useful

Collected data is only worth what you can do with it. The last mile is usually an API your own application can query, a scheduled export into the warehouse or spreadsheet a team already lives in, or an alert when a competitor crosses a price threshold. That part gets built by the same studio that built the collection, which is why the schema and the consumer actually agree with each other.

  • A queryable API over the collected data
  • Scheduled exports and warehouse loads
  • Threshold and change alerting
  • Comparison and history views
  • Access control on who sees what
  • Documentation of every field meaning

Pipelines we run ourselves.

Client work stays under NDA, so what we show is what we run ourselves: our products, where the collection breaking is our problem at seven in the morning rather than somebody else.

Price monitoring

CijeneHR

Price comparison across Croatian retailers: automated collection from multiple chains with different site structures, normalisation into one comparable catalogue, price history, and a search that stays fast over thousands of products.

Read the case study
Aggregation

NajNajNovije

A news aggregator pulling stories from multiple media portals into one fast feed. Different publishers, different structures, continuous collection, and deduplication so the same story does not appear four times.

Read the case study
Internal tool · Python

SEO Wander

An internal tool on Python with Celery and Redis running generation and submission queues in the background. The scheduling, retry and queue-health engineering here is the same machinery a collection pipeline needs.

Read the case study

There is also SlashKeys, which combines collection with a self-hosted mail pipeline for alerts - relevant if your project ends in notifying people rather than filling a table.

Sounds like your project? Prepaid Hour Blocks · reply within two business days
Start a project

Four steps, no discovery call
you have to sit through.

Tell us the sources and the cadence. Our first reply includes an honest read on whether those sources are fair game, before any hours are spent.

1

Tell us the problem

A few sentences on what you have and what you need. Rough is fine - we ask the questions that matter and tell you honestly if it is not a fit.

2

Pick an Hour Block

Prepay a block of hours - 6, 12, or 24. The block is the budget: we work until it is used, or you top up.

3

Work in the open

A single source end to end first, so you can see real rows in your own database early, then the remaining sources against that proven shape.

4

Handover and support

You get the pipeline, the extraction configs, the runbook for adding a source, and a short support window after handover.

Bought in
prepaid Hour Blocks.

No volume games and no agency margin on top. You buy the block of hours that fits the size of the job, so you know the number before we start rather than after.

Starter 6 hrs
$180

Typically enough for a focused fix, a small feature, or an integration you are stuck on.

Start with Starter
Pro 24 hrs
$720

Typically enough for a larger build with integrations, auth, or billing wired in.

Start with Pro

Every Hour Block is hours at $30/hr with no discount. It is a bucket of time, not a fixed-scope deliverable. Unused hours do not expire - roll them into the same project or a later top-up. If the work runs long, you buy another block; nothing is billed without you agreeing to it first.

Frequently asked.

Is web scraping legal?

It depends entirely on what you collect and how, and we are developers rather than your lawyers. What we can tell you is where we draw our own line. We collect publicly reachable pages, honour robots.txt and any crawl delay, keep request rates polite, identify the crawler honestly, and read the terms of service of the sources first. We do not build systems that log into an account you are not entitled to use, defeat a paywall, or break a bot-protection challenge.

What is an Hour Block, and what if the job takes more or fewer hours?

An Hour Block is a prepaid block of hours: Starter is 6 hours for $180, Standard is 12 hours for $360, Pro is 24 hours for $720. It is a bucket of time, not a fixed-scope deliverable. Unused hours do not expire and roll into the same project or a top-up. If the work runs over, you buy another block, and nothing gets billed without you agreeing to it first.

What about personal data and GDPR?

Personal data changes the job from an engineering question into a compliance one. Being publicly visible does not make personal data free to collect: you still need a lawful basis, people still have information and deletion rights, and you owe them notice even when the data did not come from them directly. Our default is to design the pipeline so it does not collect personal data at all, and if your use case genuinely requires it, that needs your own legal sign-off before we build it.

Will the scraper break when a site changes?

Eventually, yes, and any developer who says otherwise is selling you something. The engineering question is whether it fails loudly or quietly. Our pipelines validate every extraction against expected shapes and alert when a field suddenly goes empty or a page count collapses, because silent partial data is far more expensive than an obvious outage.

Do you use a headless browser?

Only when the page genuinely needs one. Plain HTTP requests are an order of magnitude cheaper and more reliable, and a surprising number of sites that look JavaScript-heavy expose the same data through a JSON endpoint the page itself calls, or through structured data already embedded in the HTML. We check the cheap routes first and reach for a headless browser when they are not there.

How do you match the same product across different sites?

That is usually the hardest part of a price-comparison project, and far harder than the fetching. Barcodes and manufacturer part numbers where they exist, then normalisation of names, units and pack sizes, then fuzzy matching with a confidence score, with low-confidence matches held for human review rather than silently merged. Wrong matches destroy trust in the data faster than missing rows.

Can you run and maintain it, or only build it?

Either. We can hand you a documented pipeline that your own team runs, deploy it to your infrastructure and hand over the runbook, or stay on as the studio that fixes it when a source changes. Maintenance is billed from the same prepaid Hour Blocks, so there is no separate contract to negotiate.

How do we start?

Send a few sentences through the form on this page: which sources, how often you need the data, and what happens to it downstream. You get a reply from us directly, usually within two business days, including an honest opinion on whether those sources are fair game.

Tell us what data you need.

Which sources, how often you need the data refreshed, and what you will do with it downstream. A few sentences is plenty - we read every message and reply within two business days with an honest read, including whether we think the sources are fair game.

Prefer email?

[email protected]
Please enter your name
Please enter a valid email
Which Hour Block?
Tell us a little about the project
Thanks - we have got it.

Expect a reply from us within two business days.

Hour Blocks from $180reply within two business days Start a project