Skip to content

yival.schemas.model_configs

ModelProvider Objects

@dataclass
class ModelProvider()

Represents the details required to interact with a GebAI model.

Arguments:

  • api_key str - The API key used for authenticating requests to the model provider.
  • provider_name str - Name of the model provider.

Request Objects

@dataclass
class Request()

Represents the information needed to make an inference request to a GenAI model.

Arguments:

  • model_name str - The name of the machine learning model to be used for inference.
  • params Dict[str, Any] - Additional parameters to configure the model for inference.
  • prompt Union[str, List[Dict[str, str]]] - The input prompt or set of prompts to use for the inference. Can be a simple string or a list of dictionaries, each containing key-value pairs for more complex prompt structures.

Response Objects

@dataclass
class Response()

Represents the outcome of an inference request.

Arguments:

  • output Any - The result generated by the GenAI model, which can be of any type depending on the model and task.

CallOption Objects

@dataclass
class CallOption()

Represents call options with llm

Arguments:

  • temperature float - hyperparameter that controls the randomness of predictions by scaling the logits before applying softmax.

presence_penalty (float)

yival.schemas.varation_generator_configs

BaseVariationGeneratorConfig Objects

@dataclass
class BaseVariationGeneratorConfig()

Base configuration class for all variation generators.

OpenAIPromptBasedVariationGeneratorConfig Objects

@dataclass
class OpenAIPromptBasedVariationGeneratorConfig(BaseVariationGeneratorConfig)

Generate variation using chatgpt. Currently only support openai models.

yival.schemas.experiment_config

Module for experiment configuration structures.

This module provides data structures to capture configurations required to run an experiment.

WrapperVariation Objects

@dataclass
class WrapperVariation()

Represents a variation within a wrapper. The value can be any type, but typical usages might include strings, numbers, configuration dictionaries, or even custom class configurations.

value_type

e.g., "string", "int", "float", "ClassA", ...

value

The actual value or parameters to initialize a value

instantiate

def instantiate() -> Any

Returns an instantiated value based on value_type and params.

WrapperConfig Objects

@dataclass
class WrapperConfig()

Configuration for each individual wrapper used in the experiment.

OutputConfig Objects

@dataclass
class OutputConfig()

Configuration for experiment output.

Attributes:

  • path (str): Path where the experiment output should be saved.
  • formatter (Callable): Function to format the output.

ComparisonOutput Objects

@dataclass
class ComparisonOutput()

Result of a comparison evaluation.

Attributes:

  • better_output (str): Name of the wrapper that produced the better output.
  • reason (str): Reason or metric based on which the decision was made.

HumanRating Objects

@dataclass
class HumanRating()

Human rating for an output.

Attributes:

  • aspect (str): Aspect being rated.
  • rating (float): Rating value.
  • scale (Tuple[float, float]): Minimum and maximum value of the rating scale.

scale

Default scale from 1 to 5

Metric Objects

@dataclass
class Metric()

Represents a metric calculated from evaluator outputs.

Attributes:

  • name (str): Name of the metric (e.g., "accuracy").
  • value (float): Calculated value of the metric.
  • description (Optional[str]): Description or details about the metric.

ExperimentSummary Objects

@dataclass
class ExperimentSummary()

Represents the summary of an entire experiment.

Attributes:

  • aggregated_metrics (Dict[str, Dict[str, Metric]]): A dictionary where keys are evaluator names and values are dictionaries mapping metric names to their values.
  • ... (other summary attributes)

Context Objects

@dataclass
class Context()

Custom function context that will be used for evlaution

MultimodalOutput Objects

@dataclass
class MultimodalOutput()

Multimodal output that can include a string, a PIL Image, or both.

Attributes:

  • text_output (str): Text output for this example.
  • image_output (PIL.Image.Image): Image output for this example.

ExperimentResult Objects

@dataclass
class ExperimentResult()

Result for a single example based on a specific combination of active variations across wrappers.

Attributes:

  • combination (Dict[str, str]): The combination of wrapper names and their active variation_ids for this example.
  • raw_output (Any): Raw output for this example. Support str and PILimage
  • latency (float): Latency for producing the output for this example (in milliseconds or appropriate unit).
  • token_usage (int): Number of tokens used for this example.
  • evaluator_outputs (List[EvaluatorOutput]): Evaluator outputs for this example.
  • human_rating (Optional[HumanRating]): Human rating for this example.
  • intermediate_logs (List[str]): Logs captured during the experiment.

FunctionMetadata Objects

@dataclass
class FunctionMetadata()

parameters

[(param_name, description), ...]

EnhancerOutput Objects

@dataclass
class EnhancerOutput()

Represents the outputs related to the "enhancer" component of the experiment.

The enhancer's role is to enhance or optimize certain aspects of the experiment. This dataclass captures the results, metrics, and decisions made by the enhancer.

Attributes:

  • group_experiment_results List[GroupedExperimentResult] - List of grouped results after enhancement. combination_aggregated_metrics (List[CombinationAggregatedMetrics]): Aggregated metrics post-enhancement.
  • original_best_combo_key str - The best combination key before the enhancer made optimizations.
  • selection_output Optional[SelectionOutput] - Output from the selection

Experiment Objects

@dataclass
class Experiment()

Represents the entirety of an experiment run.

This dataclass encapsulates the results, metrics, and configurations used and generated during the experiment. It is a comprehensive view of everything related to a specific experiment run.

Attributes:

  • group_experiment_results List[GroupedExperimentResult] - List of results grouped by test cases. combination_aggregated_metrics (List[CombinationAggregatedMetrics]): Metrics aggregated for specific combinations.
  • selection_output Optional[SelectionOutput] - Output from the selection strategy. enhancer_output (Optional[enhancerOutput]): Output from the enhancer component, if used.

yival.schemas.data_generator_configs

BaseDataGeneratorConfig Objects

@dataclass
class BaseDataGeneratorConfig()

Base configuration class for all data generators.

OpenAIPromptBasedGeneratorConfig Objects

@dataclass
class OpenAIPromptBasedGeneratorConfig(BaseDataGeneratorConfig)

Generate test cases from prompt. Currently only support openai models.

DocumentDataGeneratorConfig Objects

@dataclass
class DocumentDataGeneratorConfig(BaseDataGeneratorConfig)

Generate question data from documents.

yival.schemas.common_structures

InputData Objects

@dataclass
class InputData()

Represents the input data for an experiment example.

Attributes:

  • example_id Optional[str]: A unique identifier for the example.
  • content (Dict[str, Any]): A dictionary that contains all the necessary input parameters for the custom function.
  • expected_result (Optional[Any]): The expected result given the input.

yival.schemas.evaluator_config

MethodCalculationMethod Objects

class MethodCalculationMethod(Enum)

Configuration for metric calculation method.

MetricCalculatorConfig Objects

@dataclass
class MetricCalculatorConfig()

Configuration for metric calculation.

BaseEvaluatorConfig Objects

@dataclass
class BaseEvaluatorConfig()

Base configuration for evaluators.

EvaluatorConfig Objects

@dataclass
class EvaluatorConfig(BaseEvaluatorConfig)

Configuration for custom evaluator.

ComparisonEvaluatorConfig Objects

@dataclass
class ComparisonEvaluatorConfig(BaseEvaluatorConfig)

Configuration for evaluators that compare different outputs.

GlobalEvaluatorConfig Objects

@dataclass
class GlobalEvaluatorConfig(BaseEvaluatorConfig)

Configuration for evaluators that compare based on all outputs.

EvaluatorOutput Objects

@dataclass
class EvaluatorOutput()

Result of an evaluator.

BertScoreEvaluatorConfig Objects

@dataclass
class BertScoreEvaluatorConfig(EvaluatorConfig)

indicator

p,r,f

yival.schemas.combination_enhancer_configs

BaseCombinationEnhancerConfig Objects

@dataclass
class BaseCombinationEnhancerConfig()

Base configuration class for all combination enhancers.

yival.schemas.trainer_configs

BaseTrainerConfig Objects

@dataclass
class BaseTrainerConfig()

Base configuration class for all trainers

LoRAConfig Objects

@dataclass
class LoRAConfig()

LoRA config for SFT Trainer

TrainArguments Objects

@dataclass
class TrainArguments()

Train Arguments in trl trainer Parameters for training arguments details => https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L158

SFTTrainerConfig Objects

@dataclass
class SFTTrainerConfig(BaseTrainerConfig)

Supervised Fine-tuning trainer config

yival.schemas.dataset_config

DatasetSourceType Objects

class DatasetSourceType(Enum)

Enum to specify the source of dataset: USER, DATASET, or MACHINE_GENERATED.

DatasetConfig Objects

@dataclass
class DatasetConfig()

Configuration for the dataset used in the experiment.

Attributes:

  • source_type (DatasetSourceType): Source of dataset, either directly from the user, from a dataset, or machine-generated.
  • file_path (Union[str, None]): Path to the dataset file. Relevant only if source_type is DATASET.
  • reader (Union[str, None]): Class name to process the dataset file. Relevant only if source_type is DATASET.
  • reader_config (Union[BaseReaderConfig, None]): Configuration for the reader.
  • output_path (Union[str, None]): Path to store the machine-generated data. Relevant only if source_type is MACHINE_GENERATED.
  • data_generators (Optional[Dict[str, BaseDataGeneratorConfig]]): List of data_generators to generate data. Relevant only if source_type is MACHINE_GENERATED.

yival.schemas.wrapper_configs

BaseWrapperConfig Objects

@dataclass
class BaseWrapperConfig()

Base configuration class for wrappers.

StringWrapperConfig Objects

@dataclass
class StringWrapperConfig(BaseWrapperConfig)

Configuration specific to the StringWrapper.

yival.schemas.selector_strategies

yival.schemas.reader_configs

BaseReaderConfig Objects

@dataclass
class BaseReaderConfig()

Base configuration class for all readers.

CSVReaderConfig Objects

@dataclass
class CSVReaderConfig(BaseReaderConfig)

Configuration specific to the CSV reader.