Incorporator, Turn any API/File into typed Python graph with pipeline



Changed

• Unified null-handling across calc / calc_all / pluck /
  link_to / link_to_list / split_and_get. Aligned with the
  null contract inc() has always provided: when input values are
  garbage (None, "", "N/A", "null", "unknown",
  "nan", "undefined"), the user-supplied callable
  (func / chain / extractor / cast_type) is no
  longer invoked. calc returns default (or None for the
  extractors) silently — no warning emitted. Warnings still fire
  when the callable raises on real data, separating the
  "missing data" case from the "function exploded" case.

  Migration: explicit null guards in user lambdas are no longer
  necessary. Use stdlib callables directly:

  # before — defensive null guard inside the lambda
  calc(lambda v: v.lower() if v else "", "title", default="", target_type=str)
  pluck("data.title", chain=lambda v: v.lower() if v else "")
  link_to(books, extractor=lambda v: v.upper() if v else None)
  
  # after — same behaviour, no log noise, no lambda
  calc(str.lower, "title", default="", target_type=str)
  pluck("data.title", chain=str.lower)
  link_to(books, extractor=str.upper)
  

  Performance: net win. is_garbage_value pre-checks cost
  ~50 ns per row but eliminate the Python exception raise (~30 µs)

  • logger.warning call (~10 µs) that previously fired on every
    garbage row. On garbage-heavy datasets the dispatch path is now
    ~95% faster; on garbage-free datasets the overhead is <0.5%.
    split_and_get's narrow null check (None/"") is
    widened to the full garbage set for consistency.

Added

• RejectEntry structured reject list. IncorporatorList
  now carries a rejects: List[RejectEntry] property with structured
  failure records (source, error_kind, message, retry_after,
  wave_index). HTTP error sites in incorporator/io/fetch.py
  build entries with error_kind from the exception type and
  retry_after parsed from any Retry-After header. ETL practice
  calls failed-load rows rejects rather than the messaging-system
  dead-letter queue term — the rename follows that convention.

  The legacy failed_sources: List[str] attribute remains as a
  derived view ([entry.source for entry in rejects]) so existing
  user code, tests, and tutorials continue to work unchanged.
  Reach for rejects when you need structured access to the
  exception type or retry hint:

  result = await Coin.incorp(inc_url=["...", "https://broken/"])
  for entry in result.rejects:
      if entry.error_kind == "HTTPStatusError" and entry.retry_after:
          schedule_retry(entry.source, after=entry.retry_after)
  

  Sidecar pipeline write sites (chunked.py, _outflow.py,
  _stateful_shim.py) still route through the back-compat
  failed_sources=[...] constructor kwarg and are auto-wrapped into
  entries with error_kind="Unknown".

• SourceRef value type for source dispatch. A new
  incorporator.io.SourceRef frozen dataclass consolidates the
  "what kind of source is this?" classification used by
  incorp(), architect(), and other source-consuming verbs. Five
  factories (from_url, from_file, from_parent, from_payload,
  from_kwargs) plus an auto-detect parse() classmethod. Public
  verb signatures unchanged; SourceRef is internal scaffolding plus
  an opt-in public type for callers that want explicit source typing.

Breaking

• _EdgeState now composes a FlowState field. The Tideweaver
  scheduler's per-edge bookkeeping (_EdgeState) used to declare four
  fields — last_consumed_at, bucket_tokens, bucket_last_refill_at,
  window_log — directly. They're now grouped under
  _EdgeState.flow_state: FlowState (the canal-toolkit type from
  incorporator.io.penstock). Built-in Penstocks were updated; only
  third-party Penstock subclasses that override
  consume_reason(edge_state, flow, now) and read the old top-level
  fields are affected.

  Migration:

  # Before:
  if edge_state.last_consumed_at is None:
      ...
  edge_state.bucket_tokens = float(self.burst)
  edge_state.window_log.append(now)
  
  # After:
  if edge_state.flow_state.last_consumed_at is None:
      ...
  edge_state.flow_state.bucket_tokens = float(self.burst)
  edge_state.flow_state.window_log.append(now)
  

  Finishes Phase A2's intent (i14): the dead Penstock-specific fields
  no longer live at the top of _EdgeState for subclasses that don't
  use them — they're behind the flow_state namespace owned by the
  edge's Penstock.

• Removed implicit per-host throttling for api.coingecko.com,
  pokeapi.co, and vpic.nhtsa.dot.gov. The framework now ships
  throttle-agnostic — calls to these hosts that previously auto-paced
  at 0.2 / 1.5 / 1.5 req/sec respectively will hit the
  DEFAULT_RPS=15 fallback unless you explicitly opt in.

  Migration — pick one of:

  # Option A: per-call kwarg (most local, easiest to discover).
  await Coin.incorp(
      inc_url="https://api.coingecko.com/api/v3/coins/markets",
      requests_per_second=0.2,
  )
  
  # Option B: register once at startup; every subsequent call against
  # the host respects the rate.
  from incorporator import register_host_penstock
  from incorporator.io.penstock import SustainedPenstock
  
  register_host_penstock("api.coingecko.com", SustainedPenstock(rate_per_sec=0.2))
  register_host_penstock("pokeapi.co", SustainedPenstock(rate_per_sec=1.5))
  register_host_penstock("vpic.nhtsa.dot.gov", SustainedPenstock(rate_per_sec=1.5))
  

  Per-host rationale (rates from the previous registry — re-verify
  against the provider's published docs when you copy):

  • api.coingecko.com → 0.2 r/s (12/min, comfortably under the 5-15/min anon ceiling).
  • pokeapi.co → 1.5 r/s (90/min, under the 100/min documented ceiling).
  • vpic.nhtsa.dot.gov → 1.5 r/s (under the 100-200/min documented ceiling).

• Dropped incorporator.io.fetch._KNOWN_API_RATE_LIMITS and
  _resolve_host_safe_rate — the backward-compat shims have no
  remaining callers. Use incorporator.io.penstock.known_host_rates()
  for the live registry view.

• Unified rate-limit primitive: Penstock replaces ThrottleStrategy.
  The HTTP throttle layer and the Tideweaver edge layer now share one
  canal-toolkit primitive — Penstock is the structural gate, the
  throttle settings (rate_per_sec, burst, window_sec) configure
  it, and the rate is the computed output. io/throttle.py is gone;
  the new home is io/penstock.py. JSON config shapes (watershed.json)
  are unchanged — the same {"type": "burst", "rate_per_sec": ..., "burst": ...}
  payload works at both layers.

  Migration:

  | Before | After |
  | --- | --- |
  | from incorporator.io.throttle import FixedIntervalThrottle | from incorporator.io.penstock import SustainedPenstock |
  | FixedIntervalThrottle(0.2) | SustainedPenstock(rate_per_sec=0.2) |
  | BurstThrottle(2.0, 10) | BurstPenstock(rate_per_sec=2.0, burst=10) |
  | NullThrottle() | NullPenstock() |
  | register_host_throttle("h", lambda: FixedIntervalThrottle(0.2)) | register_host_penstock("h", SustainedPenstock(rate_per_sec=0.2)) |
  | from incorporator import register_host_throttle | from incorporator import register_host_penstock |
  | ThrottleStrategy (Protocol) | Penstock (Pydantic BaseModel) |
  | resolve_throttle(...) | resolve_penstock(...) |

  The legacy factory-callable form still works on
  register_host_penstock (it accepts either a Penstock instance or
  a zero-arg callable returning one), so lambda: FixedIntervalThrottle(...)
  ports cleanly by changing the inner class name.

  Bug fix included: BurstPenstock's refill logic now does an explicit
  None check on bucket_last_refill_at instead of or now, so a
  legitimate 0.0 watermark no longer silently erases the refill
  window (latent in the previous BurstThrottle since v1.0).

• Removed watershed.json legacy aliases — the v1.2.0
  dependency_mode (top-level) and "mode" (per-edge) aliases for
  gate_mode are gone. Passing them now raises ValueError with
  inline migration guidance instead of silently warning.

  Migration:

  // Before v1.3.0:
  {"shape": "chain", "dependency_mode": "hard", "currents": [...]}
  
  // After v1.3.0:
  {"shape": "chain", "gate_mode": "hard", "currents": [...]}
  
  // Per-edge:
  {"shape": "custom", "edges": [{"from": "a", "to": "b", "mode": "hard"}]}
  // becomes:
  {"shape": "custom", "edges": [{"from": "a", "to": "b", "gate_mode": "hard"}]}
  

  The valid values ("hard" / "soft" / "weir") are unchanged.

• Narrowed Gate / SurgeBarrier / Penstock / Spillway method
  signatures. Strategies no longer accept the full Tideweaver
  scheduler as their first argument — they read narrow GateContext
  / SurgeContext value types instead. Tightens the FlowControl ↔
  scheduler boundary so subclasses can be unit-tested without a real
  scheduler. Affected:

  • Gate.gate_reason(scheduler, dependent, up_name, now) →
    Gate.gate_reason(ctx: GateContext)
  • SurgeBarrier.is_tripped(scheduler, dependent, up_name, now) →
    SurgeBarrier.is_tripped(ctx: SurgeContext)
  • Penstock.consume_reason(scheduler, edge_state, flow, now) →
    Penstock.consume_reason(edge_state, flow, now)
  • Spillway.overflow(scheduler, edge, displaced_wave) →
    Spillway.overflow(edge, displaced_wave, overflow_count)
  • SignalPenstock.rate_fn(scheduler, edge_state, now) -> float →
    rate_fn(edge_state, now) -> float

  Most users never override these; the change is invisible. Users
  with custom Gate/Penstock/Spillway subclasses or rate_fn
  callables update their signatures (drop the first scheduler arg).

Added

• register_host_throttle promoted to package top-level.
  from incorporator import register_host_throttle works; the
  submodule path incorporator.io.throttle.register_host_throttle
  continues to work and is the same callable. New entry in
  docs/api_atlas.md walks the registration API
  side-by-side with the existing resolve_throttle resolver.

Internal

• incorporator/observability/tideweaver/architect.py routes
  Penstock tier-1 (host-aware) recommendations through the live
  known_host_rates() view rather than the import-time
  _KNOWN_API_RATE_LIMITS shim. Behavior unchanged for users who
  register hosts; tier-1 falls silent for users who don't.