GenomeKit

GenomeKit is Deep Genomics’ open-source Python library for fast access to genomic sequence, tracks, and annotations. Its hot paths are written in C++20 and exposed to Python as an extension module, making it nontrivial to package. Until this work it was distributed only through conda-forge.

PR	What
#142	Fix GENCODE parsing that depended on UTR/CDS line order
#150	Build and test manylinux wheels for Python 3.9–3.12 (x86_64)
#156	Extend the wheel pipeline to ARM (aarch64)

1. GENCODE parser relied on UTR/CDS ordering (#142)

Problem

Building annotations from gencode.v47lift37 crashed:

>>> gk.GenomeAnnotation.build_gencode("v47lift37.gff3.gz", "gencode.v47lift37", gk.Genome("hg19"))
Loading v47lift37.basic.gff3.gz...
utr3: chr1:139309-139378:-:hg19
utr3:chr1:137620-138528:-:hg19
utr3:chr1:134900-135801:-:hg19
RuntimeError: src/genome_anno_io.cpp:630:
  Expected unique UTR3 per exon for UTR3 chr1:137620-138528:-:hg19.

Three UTRs, all labelled utr3 where there should be only one per exon. GENCODE GFF3 files describe a transcript as a sequence of child features exon, CDS, and UTR. The file doesn’t say whether a UTR is 5′ or 3′, so GenomeKit has to infer that from where the UTR sits relative to the coding sequence.

The old parser made that decision as each line was read, using only the CDS lines seen so far:

// before: classify at parse time
if (cdss_to_link.empty()) {
    // no CDS seen yet: guess by strand
    if (strand == pos_strand) utr5s_to_link.push_back(utr);
    else                      utr3s_to_link.push_back(utr);
} else {
    // compare against the CDSs seen *so far*
    if (utr.upstream_of(cdss_to_link.front())) utr5s_to_link.push_back(utr);
    if (utr.dnstream_of(cdss_to_link.back()))  utr3s_to_link.push_back(utr);
}

This logic houses the false assumption that a transcript’s lines arrive in ascending genomic-coordinate order, so the first UTR seen before any CDS must be the leftmost UTR. Leftmost is 5′ on a plus strand and 3′ on a minus strand.

However GENCODE writes children in transcript order, exon 1 first. On a plus strand that happens to coincide with ascending coordinates but on a minus strand, transcript order is descending coordinates so the first UTR is the rightmost one (the 5′ UTR) and the guess is wrong.

The failing transcript, ENST00000423372.3, is a two-exon minus-strand gene:

 134901          137621        138530      139310
 |-- exon 2 --|  |--------------- exon 1 --------------|
 |-- 3′ UTR --|  |-- 3′ UTR --|--- CDS ---|-- 5′ UTR --|
         135802          138529      139309       139379

 3′ ◀──────── transcription (− strand) ─────────── 5′

exon 1’s children are listed first, so the 5′ UTR is the very first UTR line before any CDS. Tracing the old parser:

 GFF3 line            actual   CDSs seen   old parser verdict
 ──────────────────── ──────── ─────────── ───────────────────────
 UTR  139310–139379   5′ UTR   none        − strand ⇒ utr3      ✗
 CDS  138530–139309            1
 UTR  137621–138529   3′ UTR   1           downstream ⇒ utr3    ✓
 UTR  134901–135802   3′ UTR   1           downstream ⇒ utr3    ✓

Exon 1 now has two 3′ UTRs attached to it, breaking the “one UTR3 per exon” invariant.

Fix

Don’t classify eagerly. Buffer generic UTRs alongside the exons and CDSs, and decide once the whole transcript is known and the CDSs have been sorted:

// after: classify when the transcript is complete
sort(begin(cdss_to_link), end(cdss_to_link), upstream_ordering);

for (packed_utr& utr : generic_utrs_to_link) {
    const packed_cds first_cds = cdss_to_link.front();
    const packed_cds last_cds  = cdss_to_link.back();
    GK_CHECK(utr.upstream_of(first_cds) || utr.dnstream_of(last_cds), value,
             "Found utr {} in middle of cds", utr);
    if (utr.upstream_of(first_cds)) utr5s_to_link.push_back(utr);
    if (utr.dnstream_of(last_cds))  utr3s_to_link.push_back(utr);
}

Because the CDSs are sorted, front() and back() are now the true start and end of the coding region instead of the first and last lines read. A UTR before the start is 5′, a UTR after the end is 3′.

I added gencode.v47lift37 to the test-annotation builders, extended the mini1 test genome to cover the offending locus (chr1:134900–139381):

def test_out_of_order_utrs(self):
    genome = MiniGenome("gencode.v47lift37")
    tran = genome.transcripts["ENST00000423372.3"]
    # |-----------exon------------||---intron---||-----------exon------------|
    # |--utr5--||--cds--||--utr3--|              |-----------utr3------------|
    self.assertEqual(1, len(tran.utr5s))
    self.assertEqual(2, len(tran.utr3s))
    self.assertEqual(1, len(tran.cdss))
    self.assertTrue(tran.utr5s[0].upstream_of(tran.utr3s[0]))
    self.assertEqual(tran.cdss[0].interval.end, tran.utr5s[0].interval.start)
    ...

2. Building manylinux wheels (#150)

Problem

GenomeKit was only distributed through conda-forge. Because of the C++ extension, pip install meant compiling from source which shut out a lot of users and most lightweight environments.

Fix

A GitHub Actions workflow that builds and tests PyPI-compliant wheels for Python 3.9–3.12:

 ┌────────────────────────────────────────┐
 │  quay.io/pypa/manylinux_2_28_x86_64    │
 │  for py in cp39 cp310 cp311 cp312:     │──▶  wheels/*.whl
 │      pip install cmake numpy==2 …      │
 │      python setup.py bdist_wheel       │
 └────────────────────────────────────────┘
   │
   ▼  test container per version
 ┌────────────────────────────────────────┐
 │  python:3.X-slim                       │
 │  pip install wheels/*cp3X*.whl         │
 │  python -c "import genome_kit"         │
 │  python -m unittest tests.test_*       │
 └────────────────────────────────────────┘

The core of the build step:

- name: Build wheels for Python 3.9–3.12
  run: |
    docker run --rm -v $(pwd):/io -e GK_BUILD_WHEELS=1 \
      quay.io/pypa/manylinux_2_28_x86_64 /bin/bash -c "
        set -euxo pipefail
        for PYTHON in /opt/python/cp3{9,10,11,12}-*/bin/python; do
          \$PYTHON -m pip install -U pip setuptools wheel cmake auditwheel numpy==2
          rm -rf build dist *.egg-info
          \$PYTHON setup.py bdist_wheel
          cp dist/*.whl wheels/
        done
      "

• Use manylinux_2_28. It satisfies PyPI’s manylinux requirement and supports GenomeKit’s C++20 features while manylinux2014 does not.
• Build with NumPy 2, support NumPy 1/2. NumPy 2 broke ABI compatibility, but extensions built against 2.x can run on 1.x, so builds pin NumPy 2 while runtime keeps the conda-compatible numpy<2.0dev0.
• Preserve conda compatibility. Conda-forge manages its own dependencies, so wheel-only runtime dependencies are added to setup.py behind GK_BUILD_WHEELS=1.

install_requires = []
if os.environ.get("GK_BUILD_WHEELS") is not None:
    install_requires = ["appdirs>=1.4.0", "numpy<2.0dev0",
                        "google-cloud-storage>=2.10.0", "boto3", "tqdm", ...]

• Use bdist_wheel. cibuildwheel and python -m build use isolated environments that broke GenomeKit’s local setup/ imports and NumPy access. bdist_wheel builds reliably from the repo root. A later PR migrated to cibuildwheel after simplifying native dependencies.

3. ARM wheels (#156)

An increasing share of users are on aarch64.

Turned the single-arch job into a matrix. Each arch gets its own manylinux image, runner, and output directory:

strategy:
  matrix:
    arch: [x86_64, aarch64]
    include:
      - arch: x86_64
        image: quay.io/pypa/manylinux_2_28_x86_64
        runner: ubuntu-latest
        wheel_dir: wheels
      - arch: aarch64
        image: quay.io/pypa/manylinux_2_28_aarch64
        runner: ubuntu-24.04-arm
        wheel_dir: wheels/arm
runs-on: ${{ matrix.runner }}

              ┌── x86_64 ──▶ manylinux_2_28_x86_64  ──▶ 4 wheels ──▶ tested
 tag v* ──────┤
              └── aarch64 ─▶ manylinux_2_28_aarch64 ──▶ 4 wheels ──▶ tested

My first version emulated ARM with QEMU on the default x86 runner. It worked, but building and testing four wheels under emulation took a couple of hours per run. A reviewer pointed out that GitHub had just made native ubuntu-24.04-arm runners free for public repos; switching to them brought the ARM job down to roughly the same time as x86 and let me delete the emulation setup entirely.