the research engine
from prompt to dataset.
'research beachfront homes under $150k in Baja.' the agent creates the dataset, defines columns, schedules a scraping workflow, and keeps the table fresh. query it from the UI, the API, or the agent.
structured datasets
when the goal isn't promotion but data gathering — competitor prices, lead lists, market research — start a research initiative. it creates a structured table the agent populates.
typed columns
define columns as text, number, checkbox, date, URL, or select. the agent adds missing columns automatically when it discovers new data fields.
automatic deduplication
records dedup on source URL. re-running a scraping workflow updates existing rows (e.g. a new price) instead of creating duplicates.
queryable with predicates
filter datasets with column predicates — price < $150k, sqft > 1000, beachfront = true. the same query works from the UI, the API, and the agent.
scheduled scraping
pair research with a workflow that runs on a schedule. the agent scrapes, parses, and upserts records automatically — your dataset stays fresh without manual work.
multiple views
view the same dataset as a table, gallery, or chart. create saved views with different filters, sorts, and column visibility for different stakeholders.
the research dataset
a self-updating table, powered by agents.
typed columns (text, number, checkbox, date, URL), automatic deduplication on source URL, and multiple views — table, gallery, and chart. scheduled workflows keep every dataset current.
faq
questions about research
what's the difference between research and a mission?
a mission promotes a subject on social media. research gathers structured data. they share the same base layers (projects, knowledge, workflows) but serve different goals. you might research competitors, then use that data to inform a mission's content strategy.
how does deduplication work?
records dedup on sourceUrl. when the agent upserts a record with a sourceUrl that already exists in the dataset, it updates the existing row instead of creating a duplicate. this means re-running a scraping workflow updates prices, availability, etc. without piling up copies.
can I export the data?
yes. datasets are queryable through the UI, the API, and the agent. use records.query with column predicates to filter, page, and export exactly the data you need.
under the hood
pgvector, hybrid search, explainable scoring.
research isn't just scraping. every record is embedded, indexed with HNSW, and searchable with a three-signal hybrid ranking engine. content dedup uses cosine distance to prevent the agent from re-surfacing ideas you've already published.
pgvector semantic search
every record, document, contact, and content draft is embedded as a 1536-dimension vector and indexed with HNSW for sub-millisecond nearest-neighbour queries across your entire workspace.
hybrid ranking (RRF)
search combines three independent signals — full-text (tsvector + GIN), fuzzy trigram (pg_trgm word_similarity), and vector cosine — fused with Reciprocal Rank Fusion. no single signal dominates.
semantic content dedup
before generating new content, the agent runs a pgvector cosine lookup against every existing draft and published post in the mission. if the nearest neighbour is too close, the idea is rejected.
explainable content scoring
every generated draft gets a 0–1 score from five real signals: pillar fit, novelty (vector distance), voice fit, constraint fit, and historical engagement. no black-box scoring.