A source plugin is a Python package that defines tables as dataclasses and extracts data from an API. dlt handles incremental loading and schema management. You focus on the API.
[project]
name = "shenas-source-myapi"
dependencies = ["dlt[duckdb]>=1.24.0", "shenas-source-core"]
[project.entry-points."shenas.sources"]
myapi = "shenas_sources.myapi.source:MyApiSource"from shenas_sources.core.table import EventTable, Field
from typing import Annotated, Any, Iterator
class Activities(EventTable):
class _Meta:
name = "activities"
display_name = "Activities"
pk = ("id",)
id: Annotated[str, Field(db_type="VARCHAR")] = ""
name: Annotated[str, Field(db_type="VARCHAR")] = ""
duration_s: Annotated[int, Field(db_type="INTEGER", unit="s")] = 0
time_at: Annotated[str, Field(db_type="TIMESTAMP")] = ""
@classmethod
def extract(cls, client: Any, **ctx) -> Iterator[dict]:
yield from client.get_activities()from shenas_sources.core.source import Source
class MyApiSource(Source):
name = "myapi"
display_name = "My API"
primary_table = "activities"
def build_client(self):
return MyApiClient(self.get_config_value("api_key"))
def resources(self, client):
from .tables import TABLES
return [t.to_resource(client) for t in TABLES]mkdir -p plugins/sources/myapi/shenas_sources/myapi
cd plugins/sources/myapi
uv sync # installs your plugin as a workspace member
shenasctl source myapi syncCreate the directory tree under plugins/sources/myapi/, then add pyproject.toml with the shenas.sourcesentry point pointing at your source class.
Each table is a dataclass extending EventTable. Theextract classmethod yields raw dicts from your API client. dlt handles schema inference and incremental loading.
The source class wires together your client and tables. Implementbuild_client to construct the API client from config, and resources to return the list of tables.
Run uv sync to install the plugin as a workspace member, then shenasctl source myapi sync to pull the first batch. Your data lands in a local DuckDB database, queryable from surfaces and transforms.