Pharia Studio SDK
  • Quick Start
    • Installation
    • Example Usage
    • Next Steps
  • Core Concepts
    • Evaluation
    • Dataset
    • Evaluation Process
    • Data Storage
  • SDK Reference
    • Connectors Module
      • AggregationLogicIdentifier
      • ArgillaClient
        • ArgillaClient.add_record()
        • ArgillaClient.add_records()
        • ArgillaClient.create_dataset()
        • ArgillaClient.ensure_dataset_exists()
        • ArgillaClient.evaluations()
        • ArgillaClient.split_dataset()
      • ArgillaEvaluation
        • ArgillaEvaluation.example_id
        • ArgillaEvaluation.record_id
        • ArgillaEvaluation.responses
        • ArgillaEvaluation.metadata
      • BenchmarkLineage
      • EvaluationLogicIdentifier
      • GetBenchmarkLineageResponse
      • GetBenchmarkResponse
        • GetBenchmarkResponse.transform_id_to_str()
      • GetDatasetExamplesResponse
      • PostBenchmarkExecution
      • PostBenchmarkLineagesRequest
        • PostBenchmarkLineagesRequest.model_construct()
      • PostBenchmarkLineagesResponse
        • PostBenchmarkLineagesResponse.model_construct()
      • PostBenchmarkRequest
      • Record
        • Record.id
      • RecordData
        • RecordData.content
        • RecordData.example_id
        • RecordData.metadata
      • StudioClient
        • StudioClient.project_id
        • StudioClient.url
        • StudioClient.create_project()
        • StudioClient.get_benchmark()
        • StudioClient.get_benchmark_lineage()
        • StudioClient.get_dataset_examples()
        • StudioClient.get_headers()
        • StudioClient.get_url()
        • StudioClient.submit_benchmark()
        • StudioClient.submit_benchmark_execution()
        • StudioClient.submit_benchmark_lineages()
        • StudioClient.submit_dataset()
        • StudioClient.submit_from_tracer()
        • StudioClient.submit_trace()
      • StudioDataset
        • StudioDataset.id
        • StudioDataset.name
        • StudioDataset.label
        • StudioDataset.metadata
      • StudioExample
        • StudioExample.input
        • StudioExample.expected_output
        • StudioExample.id
        • StudioExample.metadata
      • StudioProject
    • Evaluation Module
      • AggregatedComparison
      • AggregationLogic
        • AggregationLogic.aggregate()
      • AggregationOverview
        • AggregationOverview.evaluation_overviews
        • AggregationOverview.id
        • AggregationOverview.start
        • AggregationOverview.end
        • AggregationOverview.end
        • AggregationOverview.successful_evaluation_count
        • AggregationOverview.crashed_during_evaluation_count
        • AggregationOverview.failed_evaluation_count
        • AggregationOverview.run_ids
        • AggregationOverview.description
        • AggregationOverview.statistics
        • AggregationOverview.labels
        • AggregationOverview.metadata
        • AggregationOverview.raise_on_evaluation_failure()
        • AggregationOverview.run_overviews()
      • AggregationRepository
        • AggregationRepository.aggregation_overview()
        • AggregationRepository.aggregation_overview_ids()
        • AggregationRepository.aggregation_overviews()
        • AggregationRepository.store_aggregation_overview()
      • Aggregator
        • Aggregator.aggregate_evaluation()
        • Aggregator.aggregated_evaluation_type()
        • Aggregator.evaluation_type()
      • ArgillaEvaluationLogic
        • ArgillaEvaluationLogic.from_record()
        • ArgillaEvaluationLogic.to_record()
      • ArgillaEvaluator
        • ArgillaEvaluator.evaluation_lineage()
        • ArgillaEvaluator.evaluation_lineages()
        • ArgillaEvaluator.evaluation_type()
        • ArgillaEvaluator.expected_output_type()
        • ArgillaEvaluator.failed_evaluations()
        • ArgillaEvaluator.input_type()
        • ArgillaEvaluator.output_type()
        • ArgillaEvaluator.retrieve()
        • ArgillaEvaluator.submit()
      • AsyncEvaluationRepository
        • AsyncEvaluationRepository.evaluation_overview()
        • AsyncEvaluationRepository.evaluation_overview_ids()
        • AsyncEvaluationRepository.evaluation_overviews()
        • AsyncEvaluationRepository.example_evaluation()
        • AsyncEvaluationRepository.example_evaluations()
        • AsyncEvaluationRepository.failed_example_evaluations()
        • AsyncEvaluationRepository.initialize_evaluation()
        • AsyncEvaluationRepository.partial_evaluation_overview()
        • AsyncEvaluationRepository.partial_evaluation_overview_ids()
        • AsyncEvaluationRepository.partial_evaluation_overviews()
        • AsyncEvaluationRepository.store_evaluation_overview()
        • AsyncEvaluationRepository.store_example_evaluation()
        • AsyncEvaluationRepository.store_partial_evaluation_overview()
        • AsyncEvaluationRepository.successful_example_evaluations()
      • AsyncFileEvaluationRepository
        • AsyncFileEvaluationRepository.evaluation_overview()
        • AsyncFileEvaluationRepository.evaluation_overview_ids()
        • AsyncFileEvaluationRepository.evaluation_overviews()
        • AsyncFileEvaluationRepository.example_evaluation()
        • AsyncFileEvaluationRepository.example_evaluations()
        • AsyncFileEvaluationRepository.exists()
        • AsyncFileEvaluationRepository.failed_example_evaluations()
        • AsyncFileEvaluationRepository.file_names()
        • AsyncFileEvaluationRepository.initialize_evaluation()
        • AsyncFileEvaluationRepository.mkdir()
        • AsyncFileEvaluationRepository.partial_evaluation_overview()
        • AsyncFileEvaluationRepository.partial_evaluation_overview_ids()
        • AsyncFileEvaluationRepository.partial_evaluation_overviews()
        • AsyncFileEvaluationRepository.path_to_str()
        • AsyncFileEvaluationRepository.read_utf8()
        • AsyncFileEvaluationRepository.remove_file()
        • AsyncFileEvaluationRepository.store_evaluation_overview()
        • AsyncFileEvaluationRepository.store_example_evaluation()
        • AsyncFileEvaluationRepository.store_partial_evaluation_overview()
        • AsyncFileEvaluationRepository.successful_example_evaluations()
        • AsyncFileEvaluationRepository.write_utf8()
      • AsyncInMemoryEvaluationRepository
        • AsyncInMemoryEvaluationRepository.evaluation_overview()
        • AsyncInMemoryEvaluationRepository.evaluation_overview_ids()
        • AsyncInMemoryEvaluationRepository.evaluation_overviews()
        • AsyncInMemoryEvaluationRepository.example_evaluation()
        • AsyncInMemoryEvaluationRepository.example_evaluations()
        • AsyncInMemoryEvaluationRepository.failed_example_evaluations()
        • AsyncInMemoryEvaluationRepository.initialize_evaluation()
        • AsyncInMemoryEvaluationRepository.partial_evaluation_overview()
        • AsyncInMemoryEvaluationRepository.partial_evaluation_overview_ids()
        • AsyncInMemoryEvaluationRepository.partial_evaluation_overviews()
        • AsyncInMemoryEvaluationRepository.store_evaluation_overview()
        • AsyncInMemoryEvaluationRepository.store_example_evaluation()
        • AsyncInMemoryEvaluationRepository.store_partial_evaluation_overview()
        • AsyncInMemoryEvaluationRepository.successful_example_evaluations()
      • BleuGrader
        • BleuGrader.calculate_bleu()
      • ComparisonEvaluation
      • ComparisonEvaluationAggregationLogic
        • ComparisonEvaluationAggregationLogic.aggregate()
      • Dataset
        • Dataset.id
        • Dataset.name
        • Dataset.label
        • Dataset.metadata
      • DatasetRepository
        • DatasetRepository.create_dataset()
        • DatasetRepository.dataset()
        • DatasetRepository.dataset_ids()
        • DatasetRepository.datasets()
        • DatasetRepository.delete_dataset()
        • DatasetRepository.example()
        • DatasetRepository.examples()
      • EloCalculator
        • EloCalculator.calculate()
      • EloEvaluationLogic
        • EloEvaluationLogic.do_evaluate()
        • EloEvaluationLogic.do_incremental_evaluate()
        • EloEvaluationLogic.grade()
        • EloEvaluationLogic.set_previous_run_output_ids()
      • EloGradingInput
      • EvaluationFailed
        • EvaluationFailed.add_note()
        • EvaluationFailed.args
        • EvaluationFailed.description
        • EvaluationFailed.end
        • EvaluationFailed.failed_example_count
        • EvaluationFailed.successful_example_count
        • EvaluationFailed.with_traceback()
      • EvaluationLogic
        • EvaluationLogic.do_evaluate()
      • EvaluationOverview
        • EvaluationOverview.run_overviews
        • EvaluationOverview.id
        • EvaluationOverview.start_date
        • EvaluationOverview.end_date
        • EvaluationOverview.successful_evaluation_count
        • EvaluationOverview.failed_evaluation_count
        • EvaluationOverview.description
        • EvaluationOverview.labels
        • EvaluationOverview.metadata
      • EvaluationRepository
        • EvaluationRepository.evaluation_overview()
        • EvaluationRepository.evaluation_overview_ids()
        • EvaluationRepository.evaluation_overviews()
        • EvaluationRepository.example_evaluation()
        • EvaluationRepository.example_evaluations()
        • EvaluationRepository.failed_example_evaluations()
        • EvaluationRepository.initialize_evaluation()
        • EvaluationRepository.store_evaluation_overview()
        • EvaluationRepository.store_example_evaluation()
        • EvaluationRepository.successful_example_evaluations()
      • Evaluator
        • Evaluator.evaluate()
        • Evaluator.evaluate_runs()
        • Evaluator.evaluation_lineage()
        • Evaluator.evaluation_lineages()
        • Evaluator.evaluation_type()
        • Evaluator.expected_output_type()
        • Evaluator.failed_evaluations()
        • Evaluator.input_type()
        • Evaluator.output_type()
      • Example
        • Example.input
        • Example.expected_output
        • Example.id
        • Example.metadata
      • ExampleEvaluation
        • ExampleEvaluation.evaluation_id
        • ExampleEvaluation.example_id
        • ExampleEvaluation.result
      • ExampleOutput
        • ExampleOutput.run_id
        • ExampleOutput.example_id
        • ExampleOutput.output
      • FScores
      • FailedExampleEvaluation
        • FailedExampleEvaluation.error_message
        • FailedExampleEvaluation.from_exception()
      • FileAggregationRepository
        • FileAggregationRepository.aggregation_overview()
        • FileAggregationRepository.aggregation_overview_ids()
        • FileAggregationRepository.aggregation_overviews()
        • FileAggregationRepository.exists()
        • FileAggregationRepository.file_names()
        • FileAggregationRepository.mkdir()
        • FileAggregationRepository.path_to_str()
        • FileAggregationRepository.read_utf8()
        • FileAggregationRepository.remove_file()
        • FileAggregationRepository.store_aggregation_overview()
        • FileAggregationRepository.write_utf8()
      • FileDatasetRepository
        • FileDatasetRepository.create_dataset()
        • FileDatasetRepository.dataset()
        • FileDatasetRepository.dataset_ids()
        • FileDatasetRepository.datasets()
        • FileDatasetRepository.delete_dataset()
        • FileDatasetRepository.example()
        • FileDatasetRepository.examples()
        • FileDatasetRepository.exists()
        • FileDatasetRepository.file_names()
        • FileDatasetRepository.mkdir()
        • FileDatasetRepository.path_to_str()
        • FileDatasetRepository.read_utf8()
        • FileDatasetRepository.remove_file()
        • FileDatasetRepository.write_utf8()
      • FileEvaluationRepository
        • FileEvaluationRepository.evaluation_overview()
        • FileEvaluationRepository.evaluation_overview_ids()
        • FileEvaluationRepository.evaluation_overviews()
        • FileEvaluationRepository.example_evaluation()
        • FileEvaluationRepository.example_evaluations()
        • FileEvaluationRepository.exists()
        • FileEvaluationRepository.failed_example_evaluations()
        • FileEvaluationRepository.file_names()
        • FileEvaluationRepository.initialize_evaluation()
        • FileEvaluationRepository.mkdir()
        • FileEvaluationRepository.path_to_str()
        • FileEvaluationRepository.read_utf8()
        • FileEvaluationRepository.remove_file()
        • FileEvaluationRepository.store_evaluation_overview()
        • FileEvaluationRepository.store_example_evaluation()
        • FileEvaluationRepository.successful_example_evaluations()
        • FileEvaluationRepository.write_utf8()
      • FileRunRepository
        • FileRunRepository.create_temporary_run_data()
        • FileRunRepository.create_tracer_for_example()
        • FileRunRepository.delete_temporary_run_data()
        • FileRunRepository.example_output()
        • FileRunRepository.example_output_ids()
        • FileRunRepository.example_outputs()
        • FileRunRepository.example_tracer()
        • FileRunRepository.exists()
        • FileRunRepository.failed_example_outputs()
        • FileRunRepository.file_names()
        • FileRunRepository.finished_examples()
        • FileRunRepository.mkdir()
        • FileRunRepository.path_to_str()
        • FileRunRepository.read_utf8()
        • FileRunRepository.remove_file()
        • FileRunRepository.run_overview()
        • FileRunRepository.run_overview_ids()
        • FileRunRepository.run_overviews()
        • FileRunRepository.store_example_output()
        • FileRunRepository.store_example_output_parallel()
        • FileRunRepository.store_run_overview()
        • FileRunRepository.successful_example_outputs()
        • FileRunRepository.temp_store_finished_example()
        • FileRunRepository.write_utf8()
      • HighlightCoverageGrader
        • HighlightCoverageGrader.compute_fscores()
      • HuggingFaceAggregationRepository
        • HuggingFaceAggregationRepository.aggregation_overview()
        • HuggingFaceAggregationRepository.aggregation_overview_ids()
        • HuggingFaceAggregationRepository.aggregation_overviews()
        • HuggingFaceAggregationRepository.create_repository()
        • HuggingFaceAggregationRepository.delete_repository()
        • HuggingFaceAggregationRepository.exists()
        • HuggingFaceAggregationRepository.file_names()
        • HuggingFaceAggregationRepository.mkdir()
        • HuggingFaceAggregationRepository.path_to_str()
        • HuggingFaceAggregationRepository.read_utf8()
        • HuggingFaceAggregationRepository.remove_file()
        • HuggingFaceAggregationRepository.store_aggregation_overview()
        • HuggingFaceAggregationRepository.write_utf8()
      • HuggingFaceDatasetRepository
        • HuggingFaceDatasetRepository.create_dataset()
        • HuggingFaceDatasetRepository.create_repository()
        • HuggingFaceDatasetRepository.dataset()
        • HuggingFaceDatasetRepository.dataset_ids()
        • HuggingFaceDatasetRepository.datasets()
        • HuggingFaceDatasetRepository.delete_dataset()
        • HuggingFaceDatasetRepository.delete_repository()
        • HuggingFaceDatasetRepository.example()
        • HuggingFaceDatasetRepository.examples()
        • HuggingFaceDatasetRepository.exists()
        • HuggingFaceDatasetRepository.file_names()
        • HuggingFaceDatasetRepository.mkdir()
        • HuggingFaceDatasetRepository.path_to_str()
        • HuggingFaceDatasetRepository.read_utf8()
        • HuggingFaceDatasetRepository.remove_file()
        • HuggingFaceDatasetRepository.write_utf8()
      • HuggingFaceRepository
        • HuggingFaceRepository.create_repository()
        • HuggingFaceRepository.delete_repository()
        • HuggingFaceRepository.exists()
        • HuggingFaceRepository.file_names()
        • HuggingFaceRepository.mkdir()
        • HuggingFaceRepository.path_to_str()
        • HuggingFaceRepository.read_utf8()
        • HuggingFaceRepository.remove_file()
        • HuggingFaceRepository.write_utf8()
      • InMemoryAggregationRepository
        • InMemoryAggregationRepository.aggregation_overview()
        • InMemoryAggregationRepository.aggregation_overview_ids()
        • InMemoryAggregationRepository.aggregation_overviews()
        • InMemoryAggregationRepository.store_aggregation_overview()
      • InMemoryDatasetRepository
        • InMemoryDatasetRepository.create_dataset()
        • InMemoryDatasetRepository.dataset()
        • InMemoryDatasetRepository.dataset_ids()
        • InMemoryDatasetRepository.datasets()
        • InMemoryDatasetRepository.delete_dataset()
        • InMemoryDatasetRepository.example()
        • InMemoryDatasetRepository.examples()
      • InMemoryEvaluationRepository
        • InMemoryEvaluationRepository.evaluation_overview()
        • InMemoryEvaluationRepository.evaluation_overview_ids()
        • InMemoryEvaluationRepository.evaluation_overviews()
        • InMemoryEvaluationRepository.example_evaluation()
        • InMemoryEvaluationRepository.example_evaluations()
        • InMemoryEvaluationRepository.failed_example_evaluations()
        • InMemoryEvaluationRepository.initialize_evaluation()
        • InMemoryEvaluationRepository.store_evaluation_overview()
        • InMemoryEvaluationRepository.store_example_evaluation()
        • InMemoryEvaluationRepository.successful_example_evaluations()
      • InMemoryRunRepository
        • InMemoryRunRepository.create_temporary_run_data()
        • InMemoryRunRepository.create_tracer_for_example()
        • InMemoryRunRepository.delete_temporary_run_data()
        • InMemoryRunRepository.example_output()
        • InMemoryRunRepository.example_output_ids()
        • InMemoryRunRepository.example_outputs()
        • InMemoryRunRepository.example_tracer()
        • InMemoryRunRepository.failed_example_outputs()
        • InMemoryRunRepository.finished_examples()
        • InMemoryRunRepository.run_overview()
        • InMemoryRunRepository.run_overview_ids()
        • InMemoryRunRepository.run_overviews()
        • InMemoryRunRepository.store_example_output()
        • InMemoryRunRepository.store_example_output_parallel()
        • InMemoryRunRepository.store_run_overview()
        • InMemoryRunRepository.successful_example_outputs()
        • InMemoryRunRepository.temp_store_finished_example()
      • IncrementalEvaluationLogic
        • IncrementalEvaluationLogic.do_evaluate()
        • IncrementalEvaluationLogic.do_incremental_evaluate()
        • IncrementalEvaluationLogic.set_previous_run_output_ids()
      • IncrementalEvaluator
        • IncrementalEvaluator.evaluate()
        • IncrementalEvaluator.evaluate_additional_runs()
        • IncrementalEvaluator.evaluate_runs()
        • IncrementalEvaluator.evaluation_lineage()
        • IncrementalEvaluator.evaluation_lineages()
        • IncrementalEvaluator.evaluation_type()
        • IncrementalEvaluator.expected_output_type()
        • IncrementalEvaluator.failed_evaluations()
        • IncrementalEvaluator.input_type()
        • IncrementalEvaluator.output_type()
      • InstructComparisonArgillaEvaluationLogic
        • InstructComparisonArgillaEvaluationLogic.from_record()
        • InstructComparisonArgillaEvaluationLogic.to_record()
      • LanguageMatchesGrader
        • LanguageMatchesGrader.languages_match()
      • MatchOutcome
        • MatchOutcome.from_rank_literal()
      • Matches
      • MatchesAggregationLogic
        • MatchesAggregationLogic.aggregate()
      • MeanAccumulator
        • MeanAccumulator.add()
        • MeanAccumulator.extract()
        • MeanAccumulator.standard_deviation()
        • MeanAccumulator.standard_error()
      • MultipleChoiceInput
      • RecordDataSequence
      • RepositoryNavigator
        • RepositoryNavigator.evaluation_lineage()
        • RepositoryNavigator.evaluation_lineages()
        • RepositoryNavigator.run_lineage()
        • RepositoryNavigator.run_lineages()
      • RougeGrader
        • RougeGrader.calculate_rouge()
      • RunOverview
        • RunOverview.dataset_id
        • RunOverview.id
        • RunOverview.start
        • RunOverview.end
        • RunOverview.failed_example_count
        • RunOverview.successful_example_count
        • RunOverview.description
        • RunOverview.labels
        • RunOverview.metadata
      • RunRepository
        • RunRepository.create_temporary_run_data()
        • RunRepository.create_tracer_for_example()
        • RunRepository.delete_temporary_run_data()
        • RunRepository.example_output()
        • RunRepository.example_output_ids()
        • RunRepository.example_outputs()
        • RunRepository.example_tracer()
        • RunRepository.failed_example_outputs()
        • RunRepository.finished_examples()
        • RunRepository.run_overview()
        • RunRepository.run_overview_ids()
        • RunRepository.run_overviews()
        • RunRepository.store_example_output()
        • RunRepository.store_example_output_parallel()
        • RunRepository.store_run_overview()
        • RunRepository.successful_example_outputs()
        • RunRepository.temp_store_finished_example()
      • Runner
        • Runner.failed_runs()
        • Runner.input_type()
        • Runner.output_type()
        • Runner.run_dataset()
        • Runner.run_is_already_computed()
        • Runner.run_lineage()
        • Runner.run_lineages()
      • SingleHuggingfaceDatasetRepository
        • SingleHuggingfaceDatasetRepository.create_dataset()
        • SingleHuggingfaceDatasetRepository.dataset()
        • SingleHuggingfaceDatasetRepository.dataset_ids()
        • SingleHuggingfaceDatasetRepository.datasets()
        • SingleHuggingfaceDatasetRepository.delete_dataset()
        • SingleHuggingfaceDatasetRepository.example()
        • SingleHuggingfaceDatasetRepository.examples()
      • SingleOutputEvaluationLogic
        • SingleOutputEvaluationLogic.do_evaluate()
        • SingleOutputEvaluationLogic.do_evaluate_single_output()
      • StudioBenchmark
        • StudioBenchmark.execute()
      • StudioBenchmarkRepository
        • StudioBenchmarkRepository.create_benchmark()
        • StudioBenchmarkRepository.get_benchmark()
      • StudioDatasetRepository
        • StudioDatasetRepository.create_dataset()
        • StudioDatasetRepository.dataset()
        • StudioDatasetRepository.dataset_ids()
        • StudioDatasetRepository.datasets()
        • StudioDatasetRepository.delete_dataset()
        • StudioDatasetRepository.example()
        • StudioDatasetRepository.examples()
        • StudioDatasetRepository.map_to_example()
        • StudioDatasetRepository.map_to_many_example()
        • StudioDatasetRepository.map_to_many_studio_example()
        • StudioDatasetRepository.map_to_studio_dataset()
        • StudioDatasetRepository.map_to_studio_example()
      • SuccessfulExampleOutput
        • SuccessfulExampleOutput.run_id
        • SuccessfulExampleOutput.example_id
        • SuccessfulExampleOutput.output
      • WinRateCalculator
        • WinRateCalculator.calculate()
      • aggregation_overviews_to_pandas()
      • evaluation_lineages_to_pandas()
      • run_lineages_to_pandas()
Pharia Studio SDK
  • Search

© Copyright 2025, Aleph Alpha.

Built with Sphinx using a theme provided by Read the Docs.