Skip to content

Developer documentation

This document was created to help you set up your development environment and to understand the project structure.

Setting up the development environment

Step 1: (Optional) Install SeaGOAT

This step is recommended in order to make sure that SeaGOAT works properly on your computer.

Follow the official documentation

Step 2: Make sure you have all developer dependencies installed

The following dependencies need to be installed:

Step 3: Clone the repository

Use Git to clone the repository:

git clone git@github.com:kantord/SeaGOAT.git
cd SeaGOAT

Step 4: Run uv sync

uv is used to manage dependencies in this project. uv also manages virtualenvs automatically.

If you have uv installed correctly, automatically setting up a virtualenv and installing all dependencies is as easy as running:

uv sync

uv sync rebuilds the Rust extension whenever its sources change.

Step 5: Run tests

To make sure that your development environment was set up correctly, run tests:

uv run pytest

If all tests pass, you have set up your development environment correctly.

Step 6: (Optional) Set up commit hooks

There are several tools in use in SeaGOAT to make sure that code is ready to merge. Some of these tools will automatically fix issues with the code, such as reformatting the code to enforce code style. Other tools will merely point out issues before you commit your code.

It is recommended to set up commit hooks with prek so that these checks are executed automatically. You can do so by running this command:

uv run prek install

If you do not wish to use commit hooks, you can still execute all checks manually by running:

uv run prek run --all-files

Developing SeaGOAT

Automated testing

Automated testing is done using pytest. Here are some example use cases:

Watch mode

Automatically runs tests for all files when you save your changes:

uv run ptw .

Test changed files

uv run pytest . --testmon

Test all files

uv run pytest .

Snapshot testing

Snapshot testing is used in a few test cases. You can update snapshots by running

uv run pytest  --snapshot-update

Indexing scheduler

Embedding and storing chunks is done by one process-wide scheduler (seagoat/queue/indexing_scheduler.py); each project's TaskQueue only plans, submits, publishes completion and prunes. The module docstring is the reference for the contracts below.

  • Batches are self-contained (ids, text, metadata, project identity, rank). The scheduler never reads a project's repository or counters.
  • Order: lowest absolute file rank first, FIFO for ties. A project served FAIRNESS_BURST times in a row yields to another project with eligible work.
  • Resources: embedding outage, probe and retry state is kept per embedding function (embedding_resource_key), so projects sharing a model back off and probe together. Storage failures pause only the affected project.
  • Completion: after a successful upsert the scheduler sends the project a BatchCompleted event; the project publishes progress and then acks the batch. Replanning and pruning wait until nothing is outstanding.
  • Removal: Projects.remove cancels queued batches and waits for a running one; its late result is dropped.

Manual testing

In order to test your local changes to SeaGOAT manually, you can prefix the command by uv run. For example to run your local version of seagoat-server, just run:

uv run seagoat-server

Similarly, to run your local version of gt/seagoat, you can run:

uv run gt

You can run an ipython in the correct virtualenv also by prefixing it with uv run:

uv run ipython

Automatic checks (linting) and automatic formatting

This repository uses prek to run automatic checks and fixes on the codebase in addition to automatic and manual testing.

If you have set up commit hooks (recommended), then these checks and fixes are automatically executed each time you attempt to make a commit.

If you don't like commit hooks, or if you want to run the checks for all files, not just changed files then you can run the following command:

uv run prek run --all-files

Contributing to SeaGOAT

Welcome to SeaGOAT!

We value your contributions and ask you to please adhere to our Code of Conduct and follow these brief guidelines.

Getting Started

Contribution Process

  1. Fork & Create a Branch: Fork the repo and create a branch with a descriptive name in your fork.
  2. Develop & Test Changes: Make and test your changes, adhering to existing coding standards. If you have configured your development environment correctly, automated tools will help you with this.
  3. Submit a Pull Request: Open a Pull Request targeting the main branch of the original repo.
  4. Address Review Comments: After you submitted a pull request, it will be reviewed by a maintainer as soon as possible. In order to make sure your changes get merged, you have to address any comments in your pull request until it is finally approved. After your changes are approved and merged, they will be released automatically if there are any user facing changes.

By contributing to SeaGOAT you agree to release your contributions under the same license SeaGOAT is licensed to, including future relicensing or double licensing of the codebase

Releasing

Releases are automated with release-plz, see release-plz.toml and .github/workflows/release.yml:

  1. On every push to main, release-plz opens or updates a release PR that bumps the version and updates CHANGELOG.md, based on conventional commits.
  2. Merging the release PR creates the vX.Y.Z tag and GitHub release, then the same workflow builds the wheels and the sdist and publishes them to PyPI.

The version lives in Cargo.toml: pyproject.toml declares it as dynamic and maturin reads it from there, so do not add a version to pyproject.toml.

To build and publish an existing tag again, for example after a failed upload, run the release workflow manually from the Actions tab and enter the tag.