`dataset.preprocessing`¶

preprocessing ¶

JP2/IMG → Cloud-Optimized GeoTIFF conversion for HiRISE rasters.

jp2_to_cog ¶

jp2_to_cog(jp2_path: Path, overwrite: bool = False) -> pathlib.Path | None

Convert a single HiRISE JP2 to a Cloud-Optimized GeoTIFF sidecar.

The COG is written alongside the source JP2 with the same stem and a .tif extension. :meth:~temp.MarsHiRISE.prefer_cog will automatically use it when it exists, bypassing the slower JP2 path.

The conversion proceeds in two passes:

Write a temporary intermediate GeoTIFF so that overviews can be built on a writeable dataset (rasterio requires this).
Copy the intermediate file to the final COG path with copy_src_overviews=True to embed the overviews efficiently.

Parameters:

Name	Type	Description	Default
`jp2_path`	`Path`	Path to the source JPEG2000 file.	required
`overwrite`	`bool`	If `False` (default), skip files that already have a `.tif` sidecar.	`False`

Returns:

Type	Description
`Path \| None`	Path to the output COG on success, or `None` if conversion failed.

Source code in src/dataset/preprocessing/cog_conversion.py

def jp2_to_cog(jp2_path: pathlib.Path, overwrite: bool = False) -> pathlib.Path | None:
    """Convert a single HiRISE JP2 to a Cloud-Optimized GeoTIFF sidecar.

    The COG is written alongside the source JP2 with the same stem and a
    ``.tif`` extension.  :meth:`~temp.MarsHiRISE.prefer_cog` will
    automatically use it when it exists, bypassing the slower JP2 path.

    The conversion proceeds in two passes:

    1. Write a temporary intermediate GeoTIFF so that overviews can be built
       on a writeable dataset (rasterio requires this).
    2. Copy the intermediate file to the final COG path with
       ``copy_src_overviews=True`` to embed the overviews efficiently.

    Args:
        jp2_path: Path to the source JPEG2000 file.
        overwrite: If ``False`` (default), skip files that already have a
            ``.tif`` sidecar.

    Returns:
        Path to the output COG on success, or ``None`` if conversion failed.
    """
    cog = _cog_path(jp2_path)

    if cog.exists() and not overwrite:
        logger.debug("COG sidecar already exists, skipping: %s", cog.name)
        return cog

    tmp = cog.with_suffix(".tmp.tif")
    try:
        # Some HiRISE JP2s carry no embedded geotransform; rasterio emits
        # NotGeoreferencedWarning on open (identity matrix assumed) and again
        # on the intermediate write.  Both are expected — we copy whatever
        # spatial metadata is present — so suppress them for the whole pass.
        with warnings.catch_warnings():
            warnings.filterwarnings(
                "ignore",
                category=UserWarning,
                message=".*geotransform.*|.*identity matrix.*",
            )
            with rasterio.open(jp2_path) as src:
                profile = src.profile.copy()
                profile.update(
                    driver="GTiff",
                    tiled=True,
                    blockxsize=512,
                    blockysize=512,
                    bigtiff="IF_SAFER",  # switch to 64-bit offsets when >4 GiB
                )
                # The intermediate is a scratch file deleted in `finally` —
                # compressing it with deflate is the main conversion bottleneck
                # (CPU-intensive on gigabytes of data that are immediately
                # discarded).  Compression is applied only in the final
                # rasterio.shutil.copy call below.
                for key in ("lossless", "quality", "compress", "predictor", "zstd_level"):
                    profile.pop(key, None)

                logger.debug("Writing intermediate GeoTIFF to RAM for %s …", jp2_path.name)

                with rasterio.MemoryFile() as memfile:
                    with memfile.open(**profile) as dst:
                        for band_idx in src.indexes:
                            band_data = src.read(band_idx)
                            dst.write(band_data, band_idx)
                            del band_data

                        dst.build_overviews(_OVERVIEW_LEVELS, _OVERVIEW_RESAMPLING)
                        dst.update_tags(ns="rio_overview", resampling=_OVERVIEW_RESAMPLING.name)

                        # Second pass: copy to final COG with overviews embedded.
                        # Use _COG_CREATION_OPTIONS directly — it already carries compress,
                        # predictor, tiling, and copy_src_overviews.  Deriving opts from
                        # `profile` would omit compression because we stripped it above.

                        # Second pass: copy directly from RAM to the final NVMe file
                        rasterio.shutil.copy(dst, cog, **_COG_CREATION_OPTIONS)

        # Source JP2 is now closed; release any lingering references before the
        # second pass so the decompressed pixel data can be reclaimed.
        gc.collect()

        logger.info("COG written: %s", cog.name)
        return cog

    except rasterio.errors.RasterioIOError as exc:
        if _is_corrupt_jp2_error(exc):
            logger.warning(
                "Corrupted JP2 detected — deleting so it can be re-downloaded: %s",
                jp2_path.name,
            )
            jp2_path.unlink(missing_ok=True)
        else:
            logger.error("COG conversion failed for %s: %s", jp2_path.name, exc)
        cog.unlink(missing_ok=True)
        return None
    except Exception as exc:
        logger.error("COG conversion failed for %s: %s", jp2_path.name, exc)
        cog.unlink(missing_ok=True)
        return None

    finally:
        tmp.unlink(missing_ok=True)

img_to_cog ¶

img_to_cog(img_path: Path, overwrite: bool = False) -> pathlib.Path | None

Convert a single HiRISE DTM .IMG (PDS3 float32) to a Cloud-Optimized GeoTIFF.

DTM .IMG files are flat binary rasters with attached PDS3 labels. Unlike JP2 orthoimages, they are not compressed, so the conversion is mainly about adding internal 512×512 tiling and overviews for fast random-access reads during ML training.

Float32 elevation data uses predictor=3 (floating-point differencing) for better deflate compression. The nodata sentinel -3.4028226550889045e+38 is preserved in the output GeoTIFF metadata.