A source plugin is a Python package that defines tables as dataclasses and extracts data from an API. dlt handles incremental loading and schema management. You focus on the API.
[project]
name = "shenas-source-myapi"
dependencies = ["dlt[duckdb]>=1.24.0", "shenas-source-core"]
[project.entry-points."shenas.sources"]
myapi = "shenas_sources.myapi.source:MyApiSource"from shenas_sources.core.table import EventTable, Field
from typing import Annotated, Any, Iterator
class Activities(EventTable):
class _Meta:
name = "activities"
display_name = "Activities"
pk = ("id",)
id: Annotated[str, Field(db_type="VARCHAR")] = ""
name: Annotated[str, Field(db_type="VARCHAR")] = ""
duration_s: Annotated[int, Field(db_type="INTEGER", unit="s")] = 0
time_at: Annotated[str, Field(db_type="TIMESTAMP")] = ""
@classmethod
def extract(cls, client: Any, **ctx) -> Iterator[dict]:
yield from client.get_activities()from shenas_sources.core.source import Source
class MyApiSource(Source):
name = "myapi"
display_name = "My API"
primary_table = "activities"
def build_client(self):
return MyApiClient(self.get_config_value("api_key"))
def resources(self, client):
from .tables import TABLES
return [t.to_resource(client) for t in TABLES]mkdir -p plugins/sources/myapi/shenas_sources/myapi
cd plugins/sources/myapi
uv sync # installs your plugin as a workspace member
shenasctl source myapi sync
Create the directory tree under plugins/sources/myapi/,
then add pyproject.toml with the shenas.sources
entry point pointing at your source class.
Each table is a dataclass extending EventTable. The
extract classmethod yields raw dicts from your API
client. dlt handles schema inference and incremental loading.
The source class wires together your client and tables. Implement
build_client to construct the API client from config,
and resources to return the list of tables.
Run uv sync to install the plugin as a workspace member,
then shenasctl source myapi sync to pull the first batch.
Your data lands in a local DuckDB database, queryable from dashboards
and transforms.