Metadata-Version: 2.5
Name: axelera-evalsuite
Version: 0.2.1
Summary: Axelera Evaluation Suite
Author-email: Gian Guido Parenza <gianguido.parenza@axelera.ai>
License-Expression: LicenseRef-Proprietary
License-File: LICENSE.txt
Requires-Python: <3.14,>=3.10
Requires-Dist: anthropic<1,>=0.40.0
Requires-Dist: datasets<5.0.0,>=4.0.0
Requires-Dist: fastapi<1,>=0.100.0
Requires-Dist: hf-transfer<1.0.0,>=0.1.9
Requires-Dist: httpx<1,>=0.28.1
Requires-Dist: immutabledict<5,>=4.2.0
Requires-Dist: inspect-ai==0.3.99
Requires-Dist: jsonschema<5,>=4.0.0
Requires-Dist: langdetect<2,>=1.0.9
Requires-Dist: lm-eval<0.5.0,>=0.4.9.2
Requires-Dist: numpy<3.0.0,>=1.26.0
Requires-Dist: openai<2,>=1.0.0
Requires-Dist: opencv-python<5,>=4.11.0.86
Requires-Dist: optimum<2.0.0,>=1.0.0
Requires-Dist: pycocotools<3,>=2.0.8
Requires-Dist: pydantic-settings<3,>=2.0.0
Requires-Dist: pydantic<3.0.0,>=2.0.0
Requires-Dist: python-multipart<0.1.0,>=0.0.6
Requires-Dist: requests<3.0.0,>=2.32.0
Requires-Dist: starlette<0.41.0,>=0.40.0
Requires-Dist: tenacity<10,>=8.0.0
Requires-Dist: tiktoken<1.0,>=0.11.0
Requires-Dist: torch<2.10.0,>=2.8.0
Requires-Dist: tqdm<5.0.0,>=4.66.0
Requires-Dist: transformers<5,>=4.30.0
Requires-Dist: types-requests<3.0.0,>=2.32.0
Requires-Dist: ultralytics<9.0.0,>=8.3.191
Requires-Dist: uvicorn<0.24.0,>=0.23.0
Description-Content-Type: text/markdown

# Axelera Evaluation Suite

[![Python Versions](https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12-blue)](https://github.com/axelera-ai/Axelera-ML/tree/dev/packages/axelera-evalsuite)

A comprehensive toolkit for evaluating machine learning models, particularly focused on natural language
processing (NLP) and computer vision tasks.

## Overview

The Axelera Evaluation Suite provides a flexible and extensible framework for evaluating various
types of machine learning models across different frameworks. It features:

- **Type-Safe Components**: Comprehensive type safety with typed result interfaces
- **Robust Error Handling**: Graceful degradation and comprehensive validation
- **Unified Registry System**: Consistent component registration and discovery
- **Extensible Architecture**: Plugin-like component system with lazy loading

### Supported Evaluation Types

- Language models (via lm-eval-harness integration)
- Computer vision models (via Ultralytics integration)
  - **Native YOLO models**: Direct evaluation of YOLO models for detection, classification, segmentation,
  pose estimation, and OBB tasks
  - **Arbitrary PyTorch models**: Support for ResNet, ViT, and other custom models through automatic wrapping
  - **Custom preprocessing/postprocessing**: User-defined functions for data transformation
- Custom evaluation workflows with user-defined metrics
- Model serving via OpenAI-compatible APIs

## Security Features

The server implements secure-by-default model injection controls to prevent potential code execution vulnerabilities.

### Security Overview

- **Model Injection Protection**: Server blocks external model objects by default
- **Authentication Controls**: Configurable API key authentication
- **Type Validation**: Ensures only valid PreTrainedModel objects are accepted
- **Audit Logging**: Security events are logged for monitoring

### Quick Security Setup

**For internal engineering use:**

```python
from axelera.evalsuite.engine.server import APIServer
from axelera.evalsuite.engine.server.config import ServerConfig

config = ServerConfig(
    ALLOW_MODEL_INJECTION=True,  # Explicit opt-in for engineering use
    HOST="localhost",
    PORT=8000,
    API_KEYS=["your-secure-key"],  # Required for security
    MAX_INPUT_LENGTH=8192,        # Optional: limit input to 8K tokens
)

# Start server with model injection (for benchmarking/testing)
server = APIServer(config=config, model=your_model, tokenizer=your_tokenizer)
server.start_server()
```

⚠️ **Security Note**: Model injection is disabled by default. Only enable `ALLOW_MODEL_INJECTION=True` for trusted
internal engineering workflows.

For comprehensive security documentation see: [`src/axelera/evalsuite/engine/server/README.md`](src/axelera/evalsuite/engine/server/README.md)

## Core API

### Main Evaluation Function

The primary entry point for all evaluations is the `evaluate` function:

```python
from axelera.evalsuite import evaluate

def evaluate(
    eval_target: Any,
    eval_config: BaseEvalConfig,
) -> Optional[dict[str, Any]]:
    """
    Run evaluation on a model with the specified configuration.

    Args:
        eval_target: Model to evaluate
        eval_config: Configuration object specifying evaluation parameters

    Returns:
        Dictionary of evaluation results, or None if evaluation failed
    """
```

**Key Features:**

- **Configurable Logging**: Verbosity controlled via configuration
- **Error Recovery**: Graceful handling of evaluation failures
- **Handler Dispatch**: Automatic selection of evaluation handler based on config type

### Configuration System

All evaluations are configured using Pydantic-based configuration classes:

#### BaseEvalConfig

```python
from axelera.evalsuite.engine.configurations import BaseEvalConfig

class BaseEvalConfig(BaseModel):
    """Base configuration with common evaluation settings."""

    verbosity: Literal["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"] = "INFO"
```

#### CustomEvalConfig

```python
from axelera.evalsuite.engine.configurations import CustomEvalConfig, MetricInstanceConfig, AggregatorInstanceConfig

class CustomEvalConfig(BaseEvalConfig):
    """Configuration for custom evaluation with metrics and aggregators."""

    # Device specification (required)
    device: str

    # Data loading
    dataloader_name: str
    dataloader_args: dict[str, Any] = Field(default_factory=dict)

    # Model interface
    input_keys: list[str] = Field(default_factory=list)
    output_keys: list[str] = Field(default_factory=list)

    # Metrics configuration
    metric_configs: dict[str, MetricInstanceConfig]
    aggregator_configs: dict[str, AggregatorInstanceConfig]
```

#### UltralyticsConfig

```python
from axelera.evalsuite.engine.configurations import UltralyticsConfig

class UltralyticsConfig(BaseEvalConfig):
    """Configuration for Ultralytics-based computer vision evaluation."""

    # Task configuration
    task: Literal["detect", "segment", "classify", "pose", "obb"] = "detect"
    data: str  # Dataset YAML path, dataset name, or directory path
    split: Literal["train", "val", "test"] = "val"

    # Model configuration
    imgsz: int = 640  # Input image size
    batch_size: int = 16  # Batch size for evaluation

    # Evaluation thresholds
    conf_thres: float = 0.001  # Confidence threshold
    iou_thres: float = 0.7     # IoU threshold for NMS

    # Performance options
    workers: int = 8        # Number of data loading workers
    save_json: bool = False # Save results in COCO JSON format
    plots: bool = True      # Generate evaluation plots

    # Custom preprocessing/postprocessing functions (optional)
    preprocessing_fn: Optional[Callable[[Any], Any]] = None
    postprocessing_fn: Optional[Callable[[Any], Any]] = None
```

## Architecture

The evaluation suite is organized into these main components:

### Core Components

- **`benchmarks/`**: Standard benchmark tools and API wrappers for external evaluation frameworks
- **`custom_evaluation/`**: Custom metrics, aggregators, dataloaders, and evaluation logic
  - **`dataloaders/`**: Standardized data loading with structured outputs and validation
  - **`metrics/`**: Type-safe batch-level metric computation with error recovery
  - **`metric_aggregators/`**: Numerically stable result aggregation across batches
  - **`third_party/`**: Integration wrappers for external evaluation libraries on custom datasets
- **`engine/`**: Core evaluation engine, configuration management, and orchestration
  - **`configurations/`**: Pydantic-based configuration classes with validation
  - **`server/`**: OpenAI-compatible API server for model serving
- **`registry/`**: Unified registration system with lazy loading and type validation

### Key Features

#### Handler Registry System

The evaluation engine uses a specialized handler registry that automatically dispatches to the appropriate
evaluation logic based on configuration type:

```python
from axelera.evalsuite import register_eval_handler, EVAL_HANDLERS

@register_eval_handler(CustomEvalConfig)
def handle_custom_eval(eval_target, eval_config):
    """Handler for custom evaluation configurations."""
    # Implementation details...
    return results

# Automatic dispatch in evaluate() function
config_type = type(eval_config)
handler = EVAL_HANDLERS.get(config_type)
results = handler(eval_target, eval_config)
```

#### Unified Registry System

Components use a simple, efficient registration system:

- **Built-in Components**: Static lazy loading from predefined dictionaries
- **Custom Components**: Direct registration via `@register_*` decorators
- **Type Safety**: Runtime validation with clear error messages

#### Type Safety & Validation

- **Typed result interfaces** for clear contracts between components
- **Configuration validation** before computation
- **Result validation** after computation
- **Comprehensive error reporting** with detailed context

#### Robust Error Handling

- **Graceful degradation** when components fail
- **Safe defaults** with NaN values for failed computations
- **Comprehensive logging** with error context
- **Evaluation continues** even with partial failures

## Installation

```bash
# Clone the repository
git clone https://github.com/yourusername/axelera-ml.git
cd axelera-ml

# Install with uv
uv sync --all-groups
```

## Quick Start

### Model Serving with Security

For internal engineering and benchmarking, you can serve models with built-in security controls:

```python
from axelera.evalsuite.engine.server import APIServer
from axelera.evalsuite.engine.server.config import ServerConfig

# Secure server configuration (required)
config = ServerConfig(
    API_KEYS=["your-secure-key"],  # Required for authentication
    ALLOW_MODEL_INJECTION=True,     # Explicitly enable for engineering
    HOST="localhost",
    PORT=8000,
)

# Inject model directly (required approach)
server = APIServer(
    model=your_model,           # Required
    tokenizer=your_tokenizer,   # Required
    model_id="your-model-id",   # Optional, auto-detected if not provided
    server_config=config,       # Required
)
server.start_server()
```

**⚠️ Security Note**: Model injection is disabled by default. Only enable `ALLOW_MODEL_INJECTION=True`
for trusted internal engineering workflows.

## CLI Tools

### HuggingFace Model Server

Serve any HuggingFace model through a REST API compatible with OpenAI's format:

```bash
# Basic usage - serve GPT-2 model
python scripts/serve_hf_model.py --model gpt2

# Serve a chat model with custom settings
python scripts/serve_hf_model.py --model microsoft/Phi-3-mini-4k-instruct --port 8766 --api-key my-secret-key

# Serve without authentication (not recommended for production)
python scripts/serve_hf_model.py --model gpt2 --no-auth

# Serve on specific GPU
python scripts/serve_hf_model.py --model gpt2 --device cuda:1
```

The server provides OpenAI-compatible endpoints:

- `GET /v1/health` - Health check
- `GET /v1/models` - List available models
- `POST /v1/completions` - Text completion
- `POST /v1/chat/completions` - Chat completion with function calling support
- `GET /v1/metrics` - Server metrics

**Example requests:**

```bash
# Health check
curl -X GET "http://localhost:8765/health"

# Text completion
curl -X POST "http://localhost:8765/v1/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer test-key" \
    -d '{
        "model": "gpt2",
        "prompt": "The future of AI is",
        "max_tokens": 50,
        "temperature": 0.7
    }'

# Chat completion
curl -X POST "http://localhost:8765/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer test-key" \
    -d '{
        "model": "gpt2",
        "messages": [
            {"role": "user", "content": "Hello! How are you?"}
        ],
        "max_tokens": 50,
        "temperature": 0.7
    }'
```

**Function Calling Support:**
The server supports OpenAI-compatible function calling, allowing models to interact with external tools:

```bash
# Function calling example
curl -X POST "http://localhost:8765/v1/chat/completions" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer test-key" \
    -d '{
        "model": "gpt2",
        "messages": [
            {"role": "user", "content": "What is the weather like in Paris?"}
        ],
        "tools": [
            {
                "type": "function",
                "function": {
                    "name": "get_weather",
                    "description": "Get weather information for a location",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "location": {"type": "string", "description": "The city and country"}
                        },
                        "required": ["location"]
                    }
                }
            }
        ],
        "tool_choice": "auto"
    }'
```

**Features:**

- **Tools parameter** - Define functions the model can call
- **Tool choice control** - "auto", "none", or force specific functions
- **Multiple function calls** - Model can call multiple functions in one response
- **Function execution** - Execute functions and continue conversations
- **OpenAI compatibility** - Full compatibility with OpenAI's function calling API

**Stopping the server:**

- Press `Ctrl+C` in the terminal where the script is running
- Or send a `SIGTERM` signal to the process: `kill <process_id>`

For complete examples with programmatic client code, see:

- [`examples/axelera-evalsuite/hf_model_server_example.py`](examples/axelera-evalsuite/hf_model_server_example.py)
- [`examples/axelera-evalsuite/function_calling_example.py`](examples/axelera-evalsuite/function_calling_example.py)

### COCO Object Detection Evaluation

```python
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import CustomEvalConfig, MetricInstanceConfig, AggregatorInstanceConfig

# Define COCO evaluation configuration
config = CustomEvalConfig(
    device="cuda",
    dataloader_name="my_detection_dataloader",
    dataloader_args={"batch_size": 1},  # Detection typically uses batch_size=1
    input_keys=["images"],
    output_keys=["predictions"],

    metric_configs={
        "coco_data": MetricInstanceConfig(
            name="coco_detection",
            args={
                "num_classes": 80,  # COCO dataset has 80 classes
                "score_threshold": 0.05,
                "max_detections": 100,
            }
        )
    },

    aggregator_configs={
        "mAP_primary": AggregatorInstanceConfig(
            aggregator="coco_mAP",
            source_metric="coco_data",
            args={
                "num_classes": 80,
                "threshold": None,  # Primary COCO mAP@0.5:0.95
                "max_detections": 100,
                "iou_type": "bbox",
            }
        ),
        "mAP_50": AggregatorInstanceConfig(
            aggregator="coco_mAP",
            source_metric="coco_data",
            args={
                "num_classes": 80,
                "threshold": 0.5,  # mAP@0.5
                "max_detections": 100,
                "iou_type": "bbox",
            }
        ),
    }
)

# Run evaluation
results = evaluate(model, config)
print(f"COCO mAP: {results['mAP_primary']:.4f}")
print(f"mAP@0.5: {results['mAP_50']:.4f}")
```

### Basic Custom Evaluation

```python
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import CustomEvalConfig, MetricInstanceConfig, AggregatorInstanceConfig

# Define evaluation configuration
config = CustomEvalConfig(
    device="cuda",
    dataloader_name="my_dataloader",
    dataloader_args={"batch_size": 32},
    input_keys=["input_ids"],
    output_keys=["logits"],
    metric_configs={
        "accuracy": MetricInstanceConfig(
            name="topk_accuracy",
            args={"k": 1}
        )
    },
    aggregator_configs={
        "final_accuracy": AggregatorInstanceConfig(
            aggregator="average",
            source_metric="accuracy",
            args={"result_key": "value"}
        )
    }
)

# Run evaluation
results = evaluate(model, config)
print(f"Final accuracy: {results['final_accuracy']:.4f}")
```

### Computer Vision with Ultralytics

The Ultralytics integration supports both native YOLO models and arbitrary PyTorch models for various computer vision tasks.

#### YOLO Model Evaluation

```python
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import UltralyticsConfig
from ultralytics import YOLO

# Load a YOLO model
model = YOLO("yolov8n.pt")  # or "yolov8s.pt", "yolov8m.pt", etc.

# Configure evaluation for object detection
config = UltralyticsConfig(
    task="detect",
    data="coco8.yaml",  # Built-in dataset or custom YAML
    split="val",
    imgsz=640,
    batch_size=16,
    conf_thres=0.001,
    iou_thres=0.7,
    verbosity="INFO"
)

# Run evaluation
results = evaluate(model, config, device_hint="cuda")
print(f"mAP@0.5: {results['metrics/mAP50(B)']:.4f}")
print(f"mAP@0.5:0.95: {results['metrics/mAP50-95(B)']:.4f}")
```

#### Custom PyTorch Model with Preprocessing

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import models
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import UltralyticsConfig

# Define a custom ResNet-50 model for classification
class CustomResNet(nn.Module):
    def __init__(self, num_classes=10):
        super().__init__()
        self.backbone = models.resnet50(weights=models.ResNet50_Weights.IMAGENET1K_V2)
        self.backbone.fc = nn.Linear(2048, num_classes)

    def forward(self, x):
        return self.backbone(x)

model = CustomResNet(num_classes=10)

# Define custom preprocessing function
def imagenet_preprocessing(batch, augment=False):
    """Apply ImageNet normalization."""
    # Handle different input formats
    if isinstance(batch, dict) and "img" in batch:
        img = batch["img"]
    elif isinstance(batch, tuple):
        img, _ = batch
    else:
        img = batch

    # Normalize to [0, 1] range if needed
    if img.max() > 1.0:
        img = img / 255.0

    # Apply ImageNet normalization
    mean = torch.tensor([0.485, 0.456, 0.406]).view(1, 3, 1, 1)
    std = torch.tensor([0.229, 0.224, 0.225]).view(1, 3, 1, 1)

    if img.device.type == 'cuda':
        mean = mean.cuda()
        std = std.cuda()

    return (img - mean) / std

# Define custom postprocessing function
def softmax_postprocessing(output, augment=False):
    """Apply softmax to model outputs."""
    return F.softmax(output, dim=1)

# Configure evaluation with custom functions
config = UltralyticsConfig(
    task="classify",
    data="cifar10",  # Built-in dataset
    split="val",
    imgsz=224,
    batch_size=32,
    preprocessing_fn=imagenet_preprocessing,
    postprocessing_fn=softmax_postprocessing,
    verbosity="INFO"
)

# Run evaluation
results = evaluate(model, config, device_hint="cuda")
print(f"Top-1 Accuracy: {results['metrics/accuracy_top1']:.4f}")
print(f"Top-5 Accuracy: {results['metrics/accuracy_top5']:.4f}")
```

#### Multi-Task Computer Vision

```python
from ultralytics import YOLO
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import UltralyticsConfig

# Segmentation task
segmentation_model = YOLO("yolov8n-seg.pt")
seg_config = UltralyticsConfig(
    task="segment",
    data="coco8-seg.yaml",
    split="val",
    imgsz=640
)
seg_results = evaluate(segmentation_model, seg_config)

# Pose estimation task
pose_model = YOLO("yolov8n-pose.pt")
pose_config = UltralyticsConfig(
    task="pose",
    data="coco8-pose.yaml",
    split="val",
    imgsz=640
)
pose_results = evaluate(pose_model, pose_config)

# Classification task
classify_model = YOLO("yolov8n-cls.pt")
cls_config = UltralyticsConfig(
    task="classify",
    data="imagenet100",
    split="val",
    imgsz=224
)
cls_results = evaluate(classify_model, cls_config)

print("Segmentation mAP:", seg_results['metrics/mAP50-95(M)'])
print("Pose mAP:", pose_results['metrics/mAP50-95(P)'])
print("Classification Accuracy:", cls_results['metrics/accuracy_top1'])
```

#### Custom Dataset Configuration

```python
from axelera.evalsuite import evaluate
from axelera.evalsuite.engine.configurations import UltralyticsConfig

# Using custom dataset YAML
config = UltralyticsConfig(
    task="detect",
    data="/path/to/custom_dataset.yaml",  # Custom dataset configuration
    split="val",
    imgsz=640,
    batch_size=8,
    save_json=True,  # Save results in COCO format for analysis
)

# Using dataset directory (classification only)
config_dir = UltralyticsConfig(
    task="classify",
    data="/path/to/dataset",  # Directory with train/val/test subdirs
    split="val",
    imgsz=224
)

# Using built-in dataset names
config_builtin = UltralyticsConfig(
    task="classify",
    data="imagenet10",  # Built-in Ultralytics dataset
    split="val"
)
```

### Component Registration

#### Register a Custom Metric

```python
from axelera.evalsuite import register_metric
from axelera.evalsuite.custom_evaluation.metrics.base_metric import BaseMetric
from axelera.evalsuite.custom_evaluation.metrics.result_types import AccuracyResult
import torch

@register_metric("custom_accuracy")
class CustomAccuracyMetric(BaseMetric):
    """Custom accuracy metric with additional validation."""

    REQUIRED_MODEL_OUTPUT_KEYS = {"logits"}
    REQUIRED_BATCH_KEYS = {"labels"}

    def _compute_batch(self, model_outputs, batch) -> tuple[AccuracyResult, int]:
        logits = model_outputs["logits"]
        labels = batch["labels"]

        predictions = torch.argmax(logits, dim=-1)
        correct = (predictions == labels).sum()
        total = labels.size(0)
        accuracy = correct.float() / total

        return {
            "correct": correct,
            "total": torch.tensor(total),
            "value": accuracy,
            "count": total
        }, total
```

#### Register a Custom Aggregator

```python
from axelera.evalsuite import register_aggregator
from axelera.evalsuite.custom_evaluation.metric_aggregators.base_aggregator import BaseAggregator
import torch

@register_aggregator("weighted_average")
class WeightedAverageAggregator(BaseAggregator):
    """Weighted average aggregator with custom weighting."""

    NAME = "weighted_average"

    def __init__(self, weight_key: str = "count"):
        self.weight_key = weight_key
        self.weighted_sum = torch.tensor(0.0)
        self.total_weight = torch.tensor(0.0)

    def update(self, result: dict[str, torch.Tensor], count: int) -> None:
        if "value" in result and self.weight_key in result:
            value = result["value"]
            weight = result[self.weight_key]
            self.weighted_sum += value * weight
            self.total_weight += weight

    def compute(self) -> torch.Tensor:
        if self.total_weight == 0:
            return torch.tensor(0.0)
        return self.weighted_sum / self.total_weight

    def reset(self) -> None:
        self.weighted_sum = torch.tensor(0.0)
        self.total_weight = torch.tensor(0.0)
```

#### Register a Custom Dataloader

```python
from axelera.evalsuite import register_dataloader
from axelera.evalsuite.custom_evaluation.dataloaders.base_dataloader import BaseDataLoader
from torch.utils.data import Dataset
import torch

class CustomDataset(Dataset):
    def __init__(self, data_path: str):
        # Load your data
        self.data = self._load_data(data_path)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        return self.data[idx]

@register_dataloader("custom_dataloader")
class CustomDataLoader(BaseDataLoader):
    """Custom dataloader for specific data format."""

    REQUIRED_KEYS = ["inputs", "targets"]

    def __init__(self, data_path: str, **kwargs):
        dataset = CustomDataset(data_path)
        super().__init__(dataset=dataset, **kwargs)

    def standardize_batch(self, batch):
        """Convert batch to standardized tensor format."""
        return {
            "inputs": torch.stack([item["input"] for item in batch]),
            "targets": torch.tensor([item["target"] for item in batch])
        }
```

#### Register a Custom Evaluation Handler

```python
from axelera.evalsuite import register_eval_handler
from axelera.evalsuite.engine.configurations import BaseEvalConfig

class MyCustomConfig(BaseEvalConfig):
    """Custom configuration for specialized evaluation."""
    model_type: str
    dataset_path: str
    custom_threshold: float = 0.5

@register_eval_handler(MyCustomConfig)
def handle_my_custom_evaluation(eval_target, eval_config):
    """
    Custom evaluation handler for MyCustomConfig.

    Args:
        eval_target: Model to evaluate
        eval_config: MyCustomConfig instance

    Returns:
        Dictionary of evaluation results
    """
    # Implement custom evaluation logic
    results = {"custom_metric": 0.95}
    return results
```

### Component Discovery

```python
from axelera.evalsuite import METRIC_REGISTRY, AGGREGATOR_REGISTRY, DATALOADER_REGISTRY, EVAL_HANDLERS

# List available components (includes lazy-loadable built-ins)
print("Available metrics:", METRIC_REGISTRY.list_available())
print("Available aggregators:", AGGREGATOR_REGISTRY.list_available())
print("Available dataloaders:", DATALOADER_REGISTRY.list_available())
print("Available handlers:", EVAL_HANDLERS.list_available())

# Get specific components (triggers lazy loading for built-ins)
MetricClass = METRIC_REGISTRY.get("topk_accuracy")
AggregatorClass = AGGREGATOR_REGISTRY.get("average")
DataLoaderClass = DATALOADER_REGISTRY.get("custom_dataloader")
HandlerFunction = EVAL_HANDLERS.get(CustomEvalConfig)
```

## Advanced Features

### Logging and Verbosity Control

```python
from axelera.evalsuite.engine.configurations import CustomEvalConfig

config = CustomEvalConfig(
    device="cuda",
    verbosity="DEBUG",  # Options: "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"
    # ... other config
)

# The evaluate function automatically configures logging based on verbosity
results = evaluate(model, config)
```

### Type Safety

The framework uses typed interfaces to ensure consistency:

```python
from axelera.evalsuite.custom_evaluation.metrics.result_types import (
    MetricResult, AccuracyResult, RegressionResult, CocoDataResult, ValidationResult
)

# All metric results conform to typed interfaces
def my_metric_computation() -> AccuracyResult:
    return {
        "correct": torch.tensor(95),
        "total": torch.tensor(100),
        "value": torch.tensor(0.95),
        "count": 100
    }
```

### Error Handling & Validation

The framework provides comprehensive error handling:

```python
# Configuration validation before computation
validation_result = MyMetric.validate_config(model_outputs, batch)
if not validation_result["is_valid"]:
    print(f"Validation errors: {validation_result['errors']}")

# Automatic error recovery with safe defaults
# Metrics return NaN values instead of crashing
# Evaluation continues even if some components fail
```

### Lazy Loading

Built-in components are loaded only when needed:

```python
# This triggers lazy loading for built-in components
metric_class = METRIC_REGISTRY.get("topk_accuracy")  # Imports module on first access

# Benefits:
# - Efficient startup - no unnecessary imports
# - Better error isolation - import failures don't break the system
```

## Documentation

- [Custom Evaluation Guide](src/axelera/evalsuite/custom_evaluation/README.md)
- [Metrics Documentation](src/axelera/evalsuite/custom_evaluation/metrics/README.md)
- [Aggregators Documentation](src/axelera/evalsuite/custom_evaluation/metric_aggregators/README.md)
- [Engine Documentation](src/axelera/evalsuite/engine/README.md)
- [Benchmarks Documentation](src/axelera/evalsuite/benchmarks/README.md)

## Contributing

Contributions are welcome! Please see the [Contributing Guide](../../CONTRIBUTING.md) for more details.

The framework is designed to be easily extensible. New metrics, aggregators, dataloaders, and evaluation
handlers can be added by:

1. Inheriting from the appropriate base class
2. Implementing the required methods
3. Registering with the appropriate decorator
4. Adding comprehensive tests

## License

This software is governed by the Axelera AI End User License Agreement, see
[LICENSE.txt](LICENSE.txt) and <https://www.axelera.ai/eula>.
