Analysis

The analysis phase runs after the facade phase and performs dependency scanning, libyear calculation, and code complexity analysis. It uses a temporary full clone created from the bare clone.

On-demand full clone design

The analysis phase needs to read file contents (manifest files for dependencies, all files for code complexity). Since the facade phase uses bare clones (no working tree), a temporary full checkout is created.

Workflow

Bare clone (permanent)
    |
    v
git clone /bare.git /tmp/checkout    (local, no network)
    |
    v
Run dependency scanning
Run libyear calculation
Run scc (code complexity)
    |
    v
rm -rf /tmp/checkout                 (deleted after analysis)

Key points

  • The full clone is created locally from the bare clone – no network request is needed.

  • The checkout is deleted immediately after analysis completes.

  • If the analysis phase fails (e.g., disk full), the bare clone is unaffected.

  • Disk usage temporarily doubles during analysis (bare clone + full checkout).

Dependency scanning

The dependency scanner walks the full checkout looking for manifest files across 14 ecosystems (the table below shows the core set; Elixir, Dart, and C/C++ manifests are also covered). It also parses 19 lockfile formats for locked versions, the transitive dependency closure, and parent→child dependency edges (vuln_scan_transitive, default on since v0.27.136).

Supported ecosystems

Manifest File

Ecosystem

Parser

package.json

npm (JavaScript/TypeScript)

JSON parser extracts dependencies + devDependencies

requirements.txt

Python (pip)

Line parser (the name matched in any letter case — Requirements.txt too — since v0.29.68), handles ==, >= and comments; pip’s option lines (-e, -r) are skipped, so an -r include is never followed

go.mod

Go

Parses require block

Cargo.toml

Rust (Cargo)

TOML parser extracts [dependencies] + [dev-dependencies]

Gemfile

Ruby (Bundler)

Parses gem declarations

pom.xml

Java (Maven)

XML parser extracts <dependency> elements

pyproject.toml

Python (PEP 621)

TOML parser extracts [project.dependencies]

setup.py

Python (setuptools)

Regex parser for install_requires

build.gradle

Java (Gradle)

Parses implementation, compile, api declarations

composer.json

PHP (Composer)

JSON parser extracts require + require-dev

Package.swift

Swift (SPM)

Parses .package(url:) declarations

*.csproj

.NET (NuGet)

XML parser extracts <PackageReference> elements

Output

Results are stored in aveloxis_data.repo_dependencies:

Column

Description

repo_id

The repository

dep_name

Dependency name

dep_count

Number of times this dependency appears

dep_language

Language/ecosystem

Multiple manifests

If a repo contains multiple manifest files (e.g., both package.json and requirements.txt), all are scanned. Dependencies from different ecosystems are stored as separate rows.

Libyear calculation

For each versioned dependency found during scanning, the libyear calculator queries its package registry to determine how out-of-date the dependency is.

What is libyear?

Libyear measures the age of a dependency by comparing the release date of the version in use against the release date of the latest available version:

libyear = (latest_release_date - current_release_date) / 365

A libyear of 0 means the dependency is up to date. A libyear of 2.5 means the version in use was released 2.5 years before the latest version.

Supported registries

Twelve registries are queried: npm, PyPI, Go proxy, crates.io, RubyGems, Maven Central, Packagist, Hex.pm, NuGet, pub.dev, Hackage, and SwiftPM (via GitHub releases). Examples:

Registry

URL Pattern

Ecosystems

npm

https://registry.npmjs.org/{package}

JavaScript, TypeScript

PyPI

https://pypi.org/pypi/{package}/json

Python

Go proxy

https://proxy.golang.org/{module}/@latest

Go

crates.io

https://crates.io/api/v1/crates/{crate}

Rust

RubyGems

https://rubygems.org/api/v1/versions/{gem}.json

Ruby

Maven Central

https://repo1.maven.org/maven2/{group path}/{artifact}/maven-metadata.xml

Java, Kotlin, Scala

How requests are made (since v0.29.56):

  • Go module paths and versions are case-encoded. The module proxy replaces every uppercase letter with ! and its lowercase form, so github.com/Masterminds/semver/v3 is requested as github.com/!masterminds/semver/v3. The unencoded spelling returns 404.

  • Maven Central is read from the repository, not the search API. maven-metadata.xml gives the latest release and the version list, and each version’s .pom carries its publication time in the Last-Modified header — which is what makes a Maven libyear value possible at all.

  • Lookups GitHub hosts go through the API key pool: the license of a Go module’s repository, and a SwiftPM package’s releases. They were anonymous before, which GitHub limits to 60 requests an hour per IP.

  • Answers are reused for a day. Successful lookups and definitive “no such package” answers are cached in the process, keyed by package and version (and by repository for licenses). A failure that says nothing — a rate limit, a timeout, a server error — is never cached.

  • Registries with a published request rate are paced across all workers. crates.io allows one request per second, so the collector spaces its requests to it however many repos are being analysed at once, and honours Retry-After on a 429 within the lookup’s own time budget.

  • Dependencies that are not from the registry — a local path or workspace package, a git or URL reference — are never looked up for npm and cargo: the registry package of the same name is a different thing, and the lookup only ever returned 404. One deliberate exception: a Cargo path dependency that ALSO declares a version is a published crate being developed locally (published-member = { path = "../member", version = "2.1.0" }), so that version is what gets published and it is looked up normally. They appear in the dependency inventory (repo_dependencies) but get no libyear row, so they do not enter the vulnerability scan as unscannable dependencies. One line per repository reports how many were skipped. Python splits the two: the libyear and vulnerability parsers drop such requirement lines while parsing — in every entry point (requirements*.txt, pyproject.toml including its Poetry tables, setup.py, setup.cfg, Pipfile and the dev/build variants) — so those dependencies get no libyear row and are never looked up in a registry. The inventory parser skips pip’s option lines in requirements.txt (-e, -r), so an editable install has no inventory row. In the line-shaped entry points it reads (requirements.txt exactly, PEP 621 dependencies, setup.py and setup.cfg’s install_requires) one name rule applies (v0.29.66): a PEP 508 direct reference keeps the name before its @ (requests @ git+https://… is stored as requests), and a line that names no package (a bare URL such as https://example.com/pkg-1.0.tar.gz, or a local path such as ./local-pkg, an scp-style git@github.com:o/r.git, or prose such as Note (optional)) gets no inventory row. The requirements VARIANTS are read only by the libyear and vulnerability path, and only with collection.dev_build_deps on, so they produce no inventory row at all. That arm takes .txt files alone: a requirements variant by name (requirements-dev.txt, test_requirements.txt, matched case-insensitively) or any other .txt directly inside a directory named requirements (that directory name byte-exact), and never requirements.txt itself in any casing. Both walks match requirements.txt in any letter case (one normalizer, canonicalManifestName, v0.29.68 — through v0.29.67 they matched it byte for byte and a Requirements.txt was collected by nothing), so requirements/requirements.txt and Requirements.txt are the ordinary manifest, gate or no gate. Every other manifest name stays byte-exact: the mixed-case keys (Gemfile, Cargo.toml, Package.swift, …) are the real names. The Poetry and Pipfile tables key on the dependency’s name, so those rows keep the clean name too. The other ecosystems (RubyGems, Packagist, Maven, …) do not yet make this distinction: a path:- or git:-sourced gem is still looked up by name.

Version cleaning

Before querying registries, version strings are cleaned:

  • Leading v is stripped (v1.2.3 -> 1.2.3)

  • Constraint operators are stripped (>=1.2.3 -> 1.2.3, ~>1.2 -> 1.2)

  • Whitespace is trimmed

Output

Results are stored in aveloxis_data.repo_deps_libyear:

Column

Description

repo_id

The repository

name

Dependency name

requirement

Version requirement string from the manifest

type

Dependency type (e.g., "runtime", "development")

package_manager

Package manager name

current_version

Version currently in use

latest_version

Latest available version

current_release_date

Release date of current version

latest_release_date

Release date of latest version

libyear

Years between current and latest (float)

Rate limiting

Each stored row also carries the registry-reported license (normalized to SPDX downstream) and a spec-conformant purl — these feed the license table, SBOM generation, and vulnerability scanning. Registry queries are not subject to GitHub/GitLab rate limits. However, some registries (notably crates.io) have their own rate limits. The libyear calculator makes requests sequentially to avoid overwhelming registries.

Code complexity via scc

If scc (Sloc Cloc and Code) is installed, Aveloxis runs it against the full checkout to get per-file code metrics.

Installation

aveloxis install-tools

This installs scc via go install github.com/boyter/scc@latest.

Execution

scc -f json --by-file /path/to/checkout

The --by-file flag produces per-file output (not just per-language summaries). The -f json flag produces machine-readable JSON output.

Output

Results are stored in aveloxis_data.repo_labor:

Column

Description

repo_id

The repository

repo_clone_date

When the repo was cloned

rl_analysis_date

When the analysis was run

programming_language

Language of the file

file_path

Full path within the repo

file_name

File name only

total_lines

Total lines in the file

code_lines

Lines of code (excluding comments and blanks)

comment_lines

Lines of comments

blank_lines

Blank lines

code_complexity

Cyclomatic complexity score

repo_url

Git URL of the repo

Snapshot rotation and history (v0.27.7)

repo_labor is latest-snapshot-only, following the house history pattern (repo_info, repo_deps_scorecard, repo_distribution): the current table holds exactly one per-file snapshot per repo — the most recent successful scc run — and prior snapshots live in aveloxis_data.repo_labor_history.

Each analysis run calls ReplaceRepoLaborSnapshot, which in one transaction rotates the repo’s current rows into repo_labor_history and inserts the fresh per-file rows. Because rotation and insert are fused into a single store method, a caller cannot insert without rotating (the pre-v0.27.7 bug: every run stacked a full new snapshot and nothing was ever rotated or deleted — production reached 2.0M live rows / 29 GB, growing unboundedly). A failed insert rolls the rotation back too, so the previous snapshot stays current; scc failures return before the call, so a failed scan never rotates anything. A successful scan that finds zero source files still replaces — an empty current snapshot is the truth of the last observation.

repo_labor_history is declared LIKE aveloxis_data.repo_labor INCLUDING ALL, keeping the primary key on repo_labor_id (ids come from the shared parent sequence, so rotated rows never collide). Per the v0.25.1 lesson, a parent’s natural-key UNIQUE constraints must not survive into a history table — repo_labor has none (only the BIGSERIAL PK, audited at v0.27.7), and an integration tripwire fails the build if one ever leaks in.

One-shot migration. The first aveloxis migrate after installing v0.27.7 moves every NON-latest snapshot into history, batched by keyset windows over the repo_labor_id primary key (the v0.26.6 bulk-backfill rule — no LIMIT-rescan loops, no per-batch global sorts). For each repo the cohort with the maximum rl_analysis_date stays; everything else moves. Legacy rows with rl_analysis_date IS NULL rotate to history as “oldest” whenever the repo has any dated snapshot; a repo whose rows are all NULL-dated keeps them (with no dates there is exactly one indistinguishable cohort, and it is by definition the latest). The migration is idempotent — a re-run moves nothing.

Operator note — disk reclaim. Moving ~90% of a 29 GB table leaves dead tuples behind in repo_labor. Plain VACUUM makes the space reusable by new rows but does not return it to the operating system. To actually reclaim the disk, run pg_repack (online, preferred) or VACUUM FULL aveloxis_data.repo_labor (takes an ACCESS EXCLUSIVE lock — use a maintenance window) after the migration completes.

Downstream consumers are unchanged. The explorer_repo_files / explorer_repo_languages materialized views and the COCOMO labor_investment snapshot query all filter to the latest rl_analysis_date per repo — exactly the cohort that stays in the current table. Analysts who want time-series questions (“how did this repo’s LOC evolve?”) query repo_labor_history directly.

If scc is not installed

The code complexity phase is silently skipped. No error is logged. The repo_labor table remains empty for repos analyzed without scc.

Materialized view

The explorer_repo_languages materialized view aggregates repo_labor data to provide per-repo language breakdowns for analytics tools.

Disk usage summary

Component

Persistence

Size

Bare clones

Permanent

10 MB - 5+ GB per repo

Full checkouts

Temporary (deleted after analysis)

Roughly equal to bare clone

scc output

In-memory (written to DB)

Negligible

Registry responses

In-memory (written to DB)

Negligible

For a repo with a 500 MB bare clone, the analysis phase temporarily uses an additional 500 MB for the full checkout, then frees it.

Error handling

  • Missing manifest files: Silently skipped. Not all repos have dependencies.

  • Malformed manifest files: A warning is logged, but analysis continues with other manifests.

  • Registry errors: If a registry query fails (timeout, 404, rate limit), the dependency’s libyear is not calculated. Other dependencies are still processed. Failures are counted per ecosystem and surfaced as one libyear resolution failures for ecosystem warning per repository, with a sample error. License lookups that fail without an answer get their own line (libyear license lookups failed without an answer); the row is still stored, without a license.

  • scc failure: If scc crashes or returns invalid JSON, a warning is logged and repo_labor is not populated for that repo.

  • Disk full during checkout: The checkout is cleaned up in a deferred function that runs even on error.

Next steps