How the geometry of a recording is computed

Orange Owns Camera-Space Evidence

Orange is the source of truth for:

• raw or diagnostic camera frames used for dish/tank observation
• detected top-rim circle proposals
• operator-confirmed accepted camera-space mask geometry
• review overlays that show what was accepted
• artifact fingerprints/checksums for the source evidence
• recording snapshots that say which Orange mask was active

Orange artifacts must remain usable when no Citrus experiment is involved.

Citrus Owns Spatial Interpretation

Citrus is the source of truth for:

• rig_id, canvas_name, and arena_id
• rig/canvas/arena geometry
• base homography and display/camera calibration
• previewing Orange evidence against Citrus geometry
• operator acceptance/rejection of a Citrus registration
• accepted citrus.dish_pose_registration artifacts
• Citrus session/H5 calibration snapshots

Citrus may reference or copy Orange artifacts, but Citrus is not the source of truth for Orange camera-space evidence.

Stable IDs

• host_id: pancakebatter/system_config.yml -> system_info.hostname
• camera_serial: pancakebatter/system_config.yml -> cameras[*].serial_number
• rig_id: Citrus rig directory name under targets/rigs/<rig_id>/
• canvas_name: Citrus canvas directory name and the canvas_name field in the canvas JSON
• arena_id: key under arenas in the Citrus canvas JSON

Important:

• the YAML map key under system_config.yml:cameras is a local inventory key and may remain the camera MAC address
• the cross-repo camera join key is always camera_serial

Join Invariants

These invariants are required for cross-repo compatibility:

1. orange camera config filename stem must equal camera_serial.
2. citrus camera_calibrations[].camera_id must equal the same camera_serial.
3. rig_id, canvas_name, and arena_id are Citrus-owned identifiers and must not be redefined elsewhere with conflicting meaning.
4. Other repos may reference Citrus IDs, but must not copy Citrus calibration payloads as independent editable sources.
5. Other repos may read hardware facts from Pancakebatter, but must not maintain separate hand-edited full copies of that inventory.

Canonical Frames

• arena_relative_canvas_px
  • local active-arena coordinates
  • origin = top-left of arena region
  • axes = +x right, +y down
• final_display_canvas_px
  • full Citrus display canvas coordinates
  • origin = top-left of fullscreen output canvas
  • axes = +x right, +y down
• camera_view_px
  • raw camera image coordinates
  • origin = top-left of camera image
  • axes = +x right, +y down

Canonical Coordinate Frames

Every payload must name its pixel frame and, where applicable, physical plane.

• camera_view_px

  • Raw camera image coordinates.
  • Origin top-left, x right, y down.

• camera_native_pixels

  • Orange synonym used in existing Orange docs for full native camera image coordinates.
  • Contract payloads should prefer camera_view_px when crossing into Citrus, but Orange-native artifacts may preserve camera_native_pixels.

• final_display_canvas_px

  • Full Citrus display/projector canvas coordinates.
  • Origin top-left, x right, y down.

• arena_relative_canvas_px

  • Local Citrus active-arena coordinates.
  • Origin top-left of active arena region, x right, y down.

Recommended physical plane IDs:

• dish_top_rim
• projected_surface
• tank_bottom_outer_surface
• tank_bottom_inner_surface
• estimated_fish_plane

V0 Orange dish masks target dish_top_rim. They should not claim to describe the projection plane, tank bottom, or fish plane.

Orange-native artifacts may use camera_native_pixels; Citrus-facing payloads may use camera_view_px. A consumer must treat these as equivalent only when the source image is the full native camera frame. If detection runs on a downsampled or cropped array, the artifact must name that array role and image shape explicitly.

Known Citrus implementation aliases:

• Orange coordinate_space maps to Citrus source_frame.
• Orange image_shape_px maps to Citrus image_size_px.
• Orange accepted circle geometry maps to Citrus geometry_camera_px.

Implementations should normalize these aliases deliberately rather than relying on loose string matching.

camera_view_px

The camera image coordinate system used by tracking and homography input points.

• Status: Code-defined
• Units: pixels
• Origin: top-left of the camera image
• Axes: +x = right, +y = down
• Typical data:
  • bounding-box x_min, y_min, width, height
  • object centers passed into homography
• Evidence:
  • Citrus uses incoming bbox image coordinates directly in src/ipc/ShamanIPCReader.cpp
  • bbox centers are transformed with cv::perspectiveTransform in src/ipc/ShamanIPCReader.cpp

final_display_canvas_px

The logical full-canvas composition buffer into which Citrus places arena textures. The established name is retained for compatibility; display in the name does not mean that these indices are already physical projector pixels or top-left-oriented GLFW framebuffer pixels.

• Status: Code-defined
• Units: pixels
• Origin: top-left of the logical CUDA composition buffer
• Axes: +x = right, +y = down
• Typical data:
  • arena destination rectangles
  • full-canvas previews such as arena-definition preview and experimental-area preview
• Evidence:
  • arenas are composited by top-left destination coordinates dest_x, dest_y in src/core/StimulusDisplayManager.cpp
  • full-canvas previews use dest_x = 0, dest_y = 0 in src/core/StimulusDisplayManager.cpp

arena_relative_canvas_px

The coordinate system for arena-local geometry inside the active arena region on the stimulus canvas.

• Status: Code-defined
• Units:
  • center and positions in pixels
  • sizes in pixels and optionally millimeters
• Origin: top-left of the active arena region
• Axes: +x = right, +y = down
• Typical data:
  • experimental area center
  • experimental area width/height/radius
• Evidence:
  • the UI explicitly labels experimental-area position as Arena-Relative Coordinates in src/ui/arena_view_ui.cpp
  • experimental-area center sliders run from 0 .. arena_width/height in src/ui/arena_view_ui.cpp
  • Citrus now saves this explicitly in /calibration_snapshot/arena_geometry

Important distinction:

• experimental_area_center_x_px and experimental_area_center_y_px are arena-relative values in this frame.
• arena_center_x_px and arena_center_y_px in the active camera calibration are full final_display_canvas_px arena placement values.
• A consumer must not inverse-project experimental_area_center_x_px/y_px through the homography directly. Convert arena-relative coordinates to final_display_canvas_px first by adding the arena origin.

Coordinate-space enum

Supported values in v1:

• camera_native_pixels
• layout_mm
• layout_units

Rules:

• camera_native_pixels means raw pixel coordinates in the producer camera's native stream resolution.
• layout_mm is preferred for canonical layout authoring when physical measurements are available.
• layout_units is allowed when a stable rig-level template exists but the layout has not yet been measured in real millimeters.

Current Transform Chain

The current intended interpretation is:

1. Tracking arrives in camera_view_px.
2. Homography maps those points into Citrus stimulus/display coordinates.
3. Arena-local geometry such as the experimental area lives in arena_relative_canvas_px.
4. Arenas are composited into logical final_display_canvas_px.
5. The established OpenGL presentation mapping converts that buffer to the stimulus window, including its vertical reflection.
6. Desktop runtime state for the chosen display output is captured separately in /display_snapshot.
7. Physical projector-body orientation is recorded separately in /rig_snapshot.

The concrete mapping formulas are:

arena_origin_canvas = (
  arena_region_center_in_canvas_x_px - arena_region_width_px / 2,
  arena_region_center_in_canvas_y_px - arena_region_height_px / 2
)

p_final_display_canvas = p_arena_relative_canvas + arena_origin_canvas
p_camera_view ~ inverse(H_camera_to_final_display_canvas) * p_final_display_canvas

p_final_display_canvas ~ H_camera_to_final_display_canvas * p_camera_view
p_arena_relative_canvas = p_final_display_canvas - arena_origin_canvas

For a daily Orange dish/top-rim observation in full camera pixels, Citrus should map the accepted center through H_camera_to_final_display_canvas, then subtract the active arena origin if it needs an arena_relative_canvas_px value. This is the correct path for comparing against experimental_area_center_x_px/y_px.

camera_view_to_canvas_mapping

The effective mapping from camera_view_px into the stimulus canvas, computed from the active homography.

• Status: Calibration-derived
• Nature:
  • point mapping via homography
  • local direction mapping via finite differences around the active arena center
• Evidence:
  • camera points are transformed into stimulus coordinates with cv::perspectiveTransform in src/ipc/ShamanIPCReader.cpp
  • direction basis is derived in src/calibration/direction_frame_mapping.h
  • saved H5 fields live under /calibration_snapshot/<camera_id> and /calibration_snapshot/arena_geometry

The OpenCV sidecar homography_<arena>_<camera>.yml and the H5 /calibration_snapshot/<camera_id>/homography_matrix represent the same direction:

camera_view_px -> final_display_canvas_px

The homography source points are camera-image pixels from the calibration image. For current Orange/Citrus workflows these should be full raw camera-frame pixels when the source image is the native camera frame. If a future calibration is derived from a crop or downsampled image, that source image shape and crop transform must be documented in the artifact before Orange can consume it safely.

Canonical Mapping Rule

For the current square-arena overlay use case, consumers should use:

arena_relative_canvas_px
  -> final_display_canvas_px
  -> camera_view_px

More explicitly:

p_display = p_arena_local + arena_origin_in_canvas
p_camera  ~ inverse(H_camera_to_display) * p_display

Where:

• p_arena_local = (x, y) from chaser_states
• arena_origin_in_canvas = (arena_origin_in_canvas_x_px, arena_origin_in_canvas_y_px)
• H_camera_to_display is the per-camera homography from /calibration_snapshot/<camera_id>/homography_matrix

If a point already exists in final_display_canvas_px, the arena-origin step is not needed.

Direct Answers For Consumers

• Homography is sufficient for the current square-arena use case if it is calibrated correctly and the starting point is already in final_display_canvas_px.
• If the starting point comes from chaser_states, homography alone is not sufficient; the arena-origin translation is required first.
• No extra offset should be applied before or after inverse homography beyond the explicit arena embedding step above.

final_display_canvas_px to presented display

Presentation is a separate mapping from the logical composition buffer. The current implementation uploads the CUDA/PBO rows directly to an OpenGL texture and draws a full-screen quad with texture v=0 at the bottom. Therefore, before desktop or projector transforms, the established render path presents:

x_window = x_final_display_canvas
y_window = (canvas_height - 1) - y_final_display_canvas

This vertical reflection is existing behavior and is a compatibility constraint, not a proposed correction. Homographies and orientation receipts must measure and preserve the effective optical mapping; they must not assume that camera, logical canvas, GLFW window, projector body, and laboratory axes have matching signs.

Arena embedding into full display canvas

Group:

• /calibration_snapshot/arena_geometry

Authoritative attrs:

• arena_region_center_in_canvas_x_px
• arena_region_center_in_canvas_y_px
• arena_region_width_px
• arena_region_height_px
• arena_origin_in_canvas_x_px
• arena_origin_in_canvas_y_px

Meaning:

• These place the active arena region inside final_display_canvas_px.
• Overlay consumers should use arena_origin_in_canvas_x_px and arena_origin_in_canvas_y_px as the authoritative local-to-global embedding.

        float arena_region_center_x_px = static_cast<float>(filtered_config.GetArenaCenterXInCanvasPx());
        float arena_region_center_y_px = static_cast<float>(filtered_config.GetArenaCenterYInCanvasPx());
        float arena_region_width_px = static_cast<float>(filtered_config.GetArenaWidthPx());
        float arena_region_height_px = static_cast<float>(filtered_config.GetArenaHeightPx());
        std::string arena_geometry_source = "legacy_sub_arena_fields";

        if (const auto* active_cam = filtered_config.GetActiveCameraCalibration()) {
            arena_region_center_x_px =
                static_cast<float>(filtered_config.GetArenaCenterXInCanvasPx());
            arena_region_center_y_px =
                static_cast<float>(filtered_config.GetArenaCenterYInCanvasPx());
            arena_region_width_px = static_cast<float>(active_cam->arena_width_px);
            arena_region_height_px = static_cast<float>(active_cam->arena_height_px);
            arena_geometry_source = "active_camera_calibration";
        }

        writeStrAttr(arena_geometry_group, "arena_geometry_source", arena_geometry_source);
        writeFloatAttr(arena_geometry_group, "arena_region_center_in_canvas_x_px",
                       arena_region_center_x_px);
        writeFloatAttr(arena_geometry_group, "arena_region_center_in_canvas_y_px",
                       arena_region_center_y_px);
        writeFloatAttr(arena_geometry_group, "arena_region_width_px",
                       arena_region_width_px);
        writeFloatAttr(arena_geometry_group, "arena_region_height_px",
                       arena_region_height_px);
        writeFloatAttr(arena_geometry_group, "arena_origin_in_canvas_x_px",
                       arena_region_center_x_px - (arena_region_width_px / 2.0f));
        writeFloatAttr(arena_geometry_group, "arena_origin_in_canvas_y_px",
                       arena_region_center_y_px - (arena_region_height_px / 2.0f));
        arena_geometry_group.close();

Per-camera homography

Per-camera group:

• /calibration_snapshot/<camera_id>

Authoritative dataset:

• homography_matrix (3x3, float64)

Compatibility fallback:

• homography_matrix_yml

Required interpretation for both:

• source_frame="camera_view_px"
• dest_frame="final_display_canvas_px"
• coordinate_origin="top_left"
• axes="x_right_y_down"
• image_space="raw"
• camera_id=<camera_id> when present

Meaning:

• The stored homography direction is:

camera_view_px -> final_display_canvas_px

• Consumers that need camera overlay coordinates must invert it.

Arena-local tracked positions

Dataset:

• /tracking_data/chaser_states

Position fields:

• chaser_pos_x
• chaser_pos_y
• target_pos_x
• target_pos_y

Required interpretation:

• These fields are in arena_relative_canvas_px.
• They are not camera-space fields.
• They are not full-display-canvas fields.

Dataset attrs consumers should read:

• coordinate_frame="arena_relative_canvas_px"
• coordinate_origin="top_left_of_active_arena"
• x_axis_direction="right"
• y_axis_direction="down"
• position_fields="chaser_pos_x,chaser_pos_y,target_pos_x,target_pos_y"
• camera_id when present

Raw camera-space tracking

Dataset:

• /tracking_data/bounding_boxes

Fields:

• x_min
• y_min
• width
• height

Meaning:

• These are already in camera_view_px.
• They do not require homography for camera-space rendering.

Crop Metadata CSV (Cam<serial>_crop_meta.csv)

Header (exact):

frame_id,timestamp,timestamp_sys,detection_confidence,crop_x,crop_y,crop_w,crop_h

Field semantics:

• frame_id: recording-frame counter
• timestamp: camera SDK timestamp (uint64, source-native unit)
• timestamp_sys: realtime nanoseconds
• detection_confidence: float
• crop_x,crop_y,crop_w,crop_h: source-frame pixel rect

Behavior note:

• blank crop frames may be encoded when detection is missing
• metadata row is appended only when detection exists

                *writer_.metadata << "frame_id,timestamp,timestamp_sys,detection_confidence,crop_x,crop_y,crop_w,crop_h\n";

                // Write metadata to file
                if (writer_.metadata && writer_.metadata->is_open()) {
                    *writer_.metadata << entry->recording_frame_id << ',' << entry->timestamp << ','
                                      << entry->timestamp_sys << ',' << best_detection.prob << ','
                                      << best_detection.rect.x << ',' << best_detection.rect.y << ','
                                      << best_detection.rect.width << ',' << best_detection.rect.height << '\n';
                }

"""Historical calibration lookup plus direction-neutral homography math.

The lookup helpers in this module are legacy compatibility surfaces.  They do
not prove a transform direction, selected camera, or reference extent and must
not authorize a future-normal coordinate publication.  Canonical callers load
a typed directed-transform record and pass its already validated numerical
matrix to :func:`apply_homography_xy`.

The historical ``projector_to_camera_*`` names remain for explicit legacy
inspection callers.  Their names describe the caller's assertion; they are not
coordinate authority.
"""

def apply_homography_xy(
    points_xy: np.ndarray,
    matrix: np.ndarray,
    *,
    denominator_epsilon: float = 1e-12,
) -> np.ndarray:
    """Apply one already direction-validated 3x3 homography to XY points.

    This function deliberately has no source/destination space names.  Those
    semantics belong to the persisted directed-transform authority used by the
    caller.  Malformed inputs and points whose homogeneous denominator is zero
    fail closed instead of being coerced to a plausible finite coordinate.

    Parameters
    ----------
    points_xy : (N, 2) or (2,) float
        XY coordinates in the transform record's declared source space.
    matrix : (3, 3) float
        Homography payload from that exact directed-transform record.
    denominator_epsilon : float
        Strict positive absolute threshold below which homogeneous ``w`` is
        treated as undefined.

    Returns
    -------
    transformed_xy : same shape as input, float64
        XY coordinates in the transform record's declared destination space.
    """
    pts = np.asarray(points_xy, dtype=np.float64)
    single = pts.ndim == 1
    if single:
        if pts.shape != (2,):
            raise ValueError("A single homography point must have shape (2,).")
        pts = pts.reshape(1, 2)
    elif pts.ndim != 2 or pts.shape[1] != 2:
        raise ValueError("Homography points must have shape (N, 2) or (2,).")

    transform = np.asarray(matrix, dtype=np.float64)
    if transform.shape != (3, 3) or not np.isfinite(transform).all():
        raise ValueError("Homography matrix must be one finite 3x3 array.")
    if (
        isinstance(denominator_epsilon, (bool, np.bool_))
        or not np.isscalar(denominator_epsilon)
        or not np.isfinite(float(denominator_epsilon))
        or float(denominator_epsilon) <= 0.0
    ):
        raise ValueError("denominator_epsilon must be one finite positive scalar.")

    if pts.size == 0:
        return np.empty_like(pts, dtype=np.float64)
    if not np.isfinite(pts).all():
        raise ValueError("Homography points must be finite; select valid rows first.")

    # Homogeneous coordinates.
    ones = np.ones((pts.shape[0], 1), dtype=np.float64)
    homogeneous = np.hstack([pts, ones])  # (N, 3)
    transformed = (transform @ homogeneous.T).T  # (N, 3)

    # Normalize by w.
    w = transformed[:, 2:3]
    if not np.isfinite(transformed).all() or np.any(
        np.abs(w) <= float(denominator_epsilon)
    ):
        raise ValueError(
            "Homography maps at least one point to an undefined point at infinity."
        )
    result = transformed[:, :2] / w

    if single:
        return result.ravel()
    return result

def _arena_origin_px(root: zarr.Group, run_group: zarr.Group) -> tuple[float, float] | None:
    source_path = str(run_group.attrs.get("source_stimulus_path") or "").strip()
    group = _get_group(root, f"{source_path}/calibration/arena_geometry") if source_path else None
    if group is None:
        return None
    x = _optional_float(group.attrs.get("arena_origin_in_canvas_x_px"))
    y = _optional_float(group.attrs.get("arena_origin_in_canvas_y_px"))
    if x is None or y is None:
        return None
    return float(x), float(y)

def detection_frame_size_px(root: zarr.Group) -> tuple[float, float] | None:
    """(width, height) of the frame the detections were produced in."""

    for family in ("detect_runs", "refined_detect_runs"):
        parent = root.get(family)
        if parent is None:
            continue
        for key in parent.keys():
            attrs = parent[key].attrs
            width = _optional_float(attrs.get("source_video_width")) or _optional_float(attrs.get("source_full_width"))
            height = _optional_float(attrs.get("source_video_height")) or _optional_float(attrs.get("source_full_height"))
            if width and height and width > 0 and height > 0:
                return float(width), float(height)
    calibration = _get_group(root, "analysis/calibration")
    if calibration is not None:
        width = _optional_float(calibration.attrs.get("native_width_px"))
        height = _optional_float(calibration.attrs.get("native_height_px"))
        if width and height and width > 0 and height > 0:
            return float(width), float(height)
    return None

Required Arrays (for any selected detect group)

• frame_indices (int32, shape (N,))
• bbox_norm_coords (float64 in current runtime, shape (N, 4))

bbox_norm_coords ordering is:

• [cx, cy, w, h]
• normalized to detector input frame dimensions

Coordinate Conversion

Given frame width W and height H:

• x1 = (cx - w/2) * W
• y1 = (cy - h/2) * H
• x2 = (cx + w/2) * W
• y2 = (cy + h/2) * H

Recommended safety:

• clip normalized values to [0, 1] before conversion
• then clip pixel bounds to [0, W-1] / [0, H-1]