Using Chirai · Chapter 18

Data feeds

A website can't crawl other websites by itself. Data feeds bring records from a real source into your site's own tables, on a schedule you choose. Examples: news about a topic, companies from a list, items from an RSS feed, rows from a Google Sheet.

Open Manage site → Settings → Data feeds.

Sources

  • News and press releases for any words (Google News).
  • Y Combinator companies (a public list, updated daily).
  • Any RSS or Atom feed.
  • Any JSON API. Name the list inside the answer, e.g. data.items.
  • A CSV file or a published Google Sheet.
  • Paid providers (Crunchbase, PitchBook and others): add your own API key in Connections, then choose it in the feed. Chirai doesn't copy their websites.

Adding a feed

  1. Choose the source and press Read it once. Chirai fetches it and shows its fields.
  2. Choose the table, and which field goes into each column. Matching names are filled in for you (title → title, link → source_url).
  3. Choose the column that identifies a record (e.g. slug or name), so a record is updated, not added twice.
  4. Choose how often: every 1, 6, 12 or 24 hours, or weekly. Then press Save and run now.

Each run adds new records and updates existing ones (up to 500 a run). Manage site shows when each feed last ran, how many records it brought, and any error.

What visitors see

A site with feeds shows a small note: "Data updated 3 h ago · 2 sources". A site whose request needs outside data, but has no feed yet, says "Sample data · live sources not connected yet". That way starter rows are never presented as real.