mstar.engine.resources.sampler.config#
What a model declares about sampling: its spec, its per-request config, its step.
Kept free of the resource and its Triton kernels so a submodule can declare a step without pulling them in behind it.
Functions
|
|
|
Raise |
|
Raise |
|
A seed for one named stream of a request, as a non-negative int64. |
Classes
|
Per-request state a decode loop carries from step to step on the device, beside its last sampled token: one row of |
|
|
|
|
|
- class mstar.engine.resources.sampler.config.LoopbackState(width, dtype, default)[source]#
Bases:
objectPer-request state a decode loop carries from step to step on the device, beside its last sampled token: one row of
widthvalues per request, kept by sampler slot. A row starts (and restarts, when its slot is reused) atdefault.
- class mstar.engine.resources.sampler.config.SamplerSpec(resource_key: str, nodes: set[str], vocab_size: int | None, enable_repetition_penalty: bool = True, enable_min_p: bool = False, loopback_state: dict[str, mstar.engine.resources.sampler.config.LoopbackState] = <factory>, max_repetition_window: int = 0, min_tokens_stop_ids: tuple[int, ...] = (), enable_top_p_first: bool = False, enable_logprobs: bool = False, enable_processed_logprobs: bool = False, greedy_capture: bool | None = None)[source]#
Bases:
NodeResourceSpec- Parameters:
- apply_yaml_overrides(enable_logprobs=None, enable_processed_logprobs=None, greedy_capture=None)[source]#
Patch declared parameters from this resource’s YAML block.
The model declares shapes that suit the model; a deployment tunes what suits the box it runs on. The block is scoped to this spec’s
resource_key(seeapply_yaml_overridesbelow), so an unrecognized key here is a typo, not another resource’s setting — subclasses name exactly what they accept and let the rest raise.TODO: generalize. Matching YAML keys against the spec’s (and its config’s) dataclass fields would remove these per-resource implementations, at the cost of silently accepting anything named alike.
- loopback_state: dict[str, LoopbackState]#
- class mstar.engine.resources.sampler.config.SamplerStep(segments: tuple[mstar.engine.resources.step.Segment, ...] | None = None, apply_penalty: bool = True, apply_filters: bool = True, prefill_tracked_tokens: dict[str, torch.Tensor] = <factory>, kept_rids: frozenset[int] | None = None, loopback_writes: frozenset[str] = frozenset(), *, combined_labels: dict[tuple[str, ...], str] = <factory>)[source]#
Bases:
ResourceStep- Parameters:
- class mstar.engine.resources.sampler.config.SamplingReqConfig(temperature: float = 0.6, top_k: int = 0, top_p: float = 1, ignore_eos: bool = False, repetition_penalty: float = 1, min_p: float = 0.0, repetition_window: int = 0, min_tokens: int = 0, top_p_first: bool = False, top_p_min_keep: int = 1, penalize_prompt: bool = True, return_logprobs: bool = False, logprobs_mode: str = 'raw', _seed: int = 0)[source]#
Bases:
ResourceReqConfig- Parameters:
- property seed#
- validate(spec=None)[source]#
Raise
ValueErroron a value outside the contract, or onespec’s node cannot honour. None means the sampler’s default.- Parameters:
spec (SamplerSpec | None)
- Return type:
None
- mstar.engine.resources.sampler.config.check_bool_knob(key, value)[source]#
- Parameters:
key (str)
- Return type:
None
- mstar.engine.resources.sampler.config.check_int_knob(key, value, rule, ok)[source]#
Raise
ValueErrorunlessvalueis an integer satisfyingok.
- mstar.engine.resources.sampler.config.check_real_knob(key, value, rule, ok)[source]#
Raise
ValueErrorunlessvalueis a finite number satisfyingok.