# Analysis

The analysis phase runs after the facade phase and performs dependency scanning, libyear calculation, and code complexity analysis. It uses a temporary full clone created from the bare clone.

---

## On-demand full clone design

The analysis phase needs to read file contents (manifest files for dependencies, all files for code complexity). Since the facade phase uses bare clones (no working tree), a temporary full checkout is created.

### Workflow

```
Bare clone (permanent)
    |
    v
git clone /bare.git /tmp/checkout    (local, no network)
    |
    v
Run dependency scanning
Run libyear calculation
Run scc (code complexity)
    |
    v
rm -rf /tmp/checkout                 (deleted after analysis)
```

### Key points

- The full clone is created locally from the bare clone -- no network request is needed.
- The checkout is deleted immediately after analysis completes.
- If the analysis phase fails (e.g., disk full), the bare clone is unaffected.
- Disk usage temporarily doubles during analysis (bare clone + full checkout).

---

## Dependency scanning

The dependency scanner walks the full checkout looking for manifest files across 12 ecosystems.

### Supported ecosystems

| Manifest File | Ecosystem | Parser |
|---|---|---|
| `package.json` | npm (JavaScript/TypeScript) | JSON parser extracts `dependencies` + `devDependencies` |
| `requirements.txt` | Python (pip) | Line parser, handles `==`, `>=`, comments, `-r` includes |
| `go.mod` | Go | Parses `require` block |
| `Cargo.toml` | Rust (Cargo) | TOML parser extracts `[dependencies]` + `[dev-dependencies]` |
| `Gemfile` | Ruby (Bundler) | Parses `gem` declarations |
| `pom.xml` | Java (Maven) | XML parser extracts `<dependency>` elements |
| `pyproject.toml` | Python (PEP 621) | TOML parser extracts `[project.dependencies]` |
| `setup.py` | Python (setuptools) | Regex parser for `install_requires` |
| `build.gradle` | Java (Gradle) | Parses `implementation`, `compile`, `api` declarations |
| `composer.json` | PHP (Composer) | JSON parser extracts `require` + `require-dev` |
| `Package.swift` | Swift (SPM) | Parses `.package(url:)` declarations |
| `*.csproj` | .NET (NuGet) | XML parser extracts `<PackageReference>` elements |

### Output

Results are stored in `aveloxis_data.repo_dependencies`:

| Column | Description |
|---|---|
| `repo_id` | The repository |
| `dep_name` | Dependency name |
| `dep_count` | Number of times this dependency appears |
| `dep_language` | Language/ecosystem |

### Multiple manifests

If a repo contains multiple manifest files (e.g., both `package.json` and `requirements.txt`), all are scanned. Dependencies from different ecosystems are stored as separate rows.

---

## Libyear calculation

For each versioned dependency found during scanning, the libyear calculator queries its package registry to determine how out-of-date the dependency is.

### What is libyear?

Libyear measures the age of a dependency by comparing the release date of the version in use against the release date of the latest available version:

```
libyear = (latest_release_date - current_release_date) / 365
```

A libyear of 0 means the dependency is up to date. A libyear of 2.5 means the version in use was released 2.5 years before the latest version.

### Supported registries

| Registry | URL Pattern | Ecosystems |
|---|---|---|
| **npm** | `https://registry.npmjs.org/{package}` | JavaScript, TypeScript |
| **PyPI** | `https://pypi.org/pypi/{package}/json` | Python |
| **Go proxy** | `https://proxy.golang.org/{module}/@v/list` | Go |
| **crates.io** | `https://crates.io/api/v1/crates/{crate}` | Rust |
| **RubyGems** | `https://rubygems.org/api/v1/versions/{gem}.json` | Ruby |

### Version cleaning

Before querying registries, version strings are cleaned:

- Leading `v` is stripped (`v1.2.3` -> `1.2.3`)
- Constraint operators are stripped (`>=1.2.3` -> `1.2.3`, `~>1.2` -> `1.2`)
- Whitespace is trimmed

### Output

Results are stored in `aveloxis_data.repo_deps_libyear`:

| Column | Description |
|---|---|
| `repo_id` | The repository |
| `name` | Dependency name |
| `requirement` | Version requirement string from the manifest |
| `type` | Dependency type (e.g., `"runtime"`, `"development"`) |
| `package_manager` | Package manager name |
| `current_version` | Version currently in use |
| `latest_version` | Latest available version |
| `current_release_date` | Release date of current version |
| `latest_release_date` | Release date of latest version |
| `libyear` | Years between current and latest (float) |

### Rate limiting

Registry queries are not subject to GitHub/GitLab rate limits. However, some registries (notably crates.io) have their own rate limits. The libyear calculator makes requests sequentially to avoid overwhelming registries.

---

## Code complexity via scc

If [scc](https://github.com/boyter/scc) (Sloc Cloc and Code) is installed, Aveloxis runs it against the full checkout to get per-file code metrics.

### Installation

```bash
aveloxis install-tools
```

This installs `scc` via `go install github.com/boyter/scc@latest`.

### Execution

```bash
scc -f json --by-file /path/to/checkout
```

The `--by-file` flag produces per-file output (not just per-language summaries). The `-f json` flag produces machine-readable JSON output.

### Output

Results are stored in `aveloxis_data.repo_labor`:

| Column | Description |
|---|---|
| `repo_id` | The repository |
| `repo_clone_date` | When the repo was cloned |
| `rl_analysis_date` | When the analysis was run |
| `programming_language` | Language of the file |
| `file_path` | Full path within the repo |
| `file_name` | File name only |
| `total_lines` | Total lines in the file |
| `code_lines` | Lines of code (excluding comments and blanks) |
| `comment_lines` | Lines of comments |
| `blank_lines` | Blank lines |
| `code_complexity` | Cyclomatic complexity score |
| `repo_url` | Git URL of the repo |

### Snapshot rotation and history (v0.27.7)

`repo_labor` is **latest-snapshot-only**, following the house history pattern (`repo_info`, `repo_deps_scorecard`, `repo_distribution`): the current table holds exactly one per-file snapshot per repo — the most recent successful scc run — and prior snapshots live in `aveloxis_data.repo_labor_history`.

Each analysis run calls `ReplaceRepoLaborSnapshot`, which in **one transaction** rotates the repo's current rows into `repo_labor_history` and inserts the fresh per-file rows. Because rotation and insert are fused into a single store method, a caller cannot insert without rotating (the pre-v0.27.7 bug: every run stacked a full new snapshot and nothing was ever rotated or deleted — production reached 2.0M live rows / 29 GB, growing unboundedly). A failed insert rolls the rotation back too, so the previous snapshot stays current; scc failures return before the call, so a failed scan never rotates anything. A *successful* scan that finds zero source files still replaces — an empty current snapshot is the truth of the last observation.

`repo_labor_history` is declared `LIKE aveloxis_data.repo_labor INCLUDING ALL`, keeping the primary key on `repo_labor_id` (ids come from the shared parent sequence, so rotated rows never collide). Per the v0.25.1 lesson, a parent's natural-key UNIQUE constraints must not survive into a history table — `repo_labor` has none (only the BIGSERIAL PK, audited at v0.27.7), and an integration tripwire fails the build if one ever leaks in.

**One-shot migration.** The first `aveloxis migrate` after installing v0.27.7 moves every NON-latest snapshot into history, batched by keyset windows over the `repo_labor_id` primary key (the v0.26.6 bulk-backfill rule — no LIMIT-rescan loops, no per-batch global sorts). For each repo the cohort with the maximum `rl_analysis_date` stays; everything else moves. Legacy rows with `rl_analysis_date IS NULL` rotate to history as "oldest" whenever the repo has any dated snapshot; a repo whose rows are *all* NULL-dated keeps them (with no dates there is exactly one indistinguishable cohort, and it is by definition the latest). The migration is idempotent — a re-run moves nothing.

**Operator note — disk reclaim.** Moving ~90% of a 29 GB table leaves dead tuples behind in `repo_labor`. Plain `VACUUM` makes the space reusable by new rows but does **not** return it to the operating system. To actually reclaim the disk, run `pg_repack` (online, preferred) or `VACUUM FULL aveloxis_data.repo_labor` (takes an ACCESS EXCLUSIVE lock — use a maintenance window) after the migration completes.

**Downstream consumers are unchanged.** The `explorer_repo_files` / `explorer_repo_languages` materialized views and the COCOMO `labor_investment` snapshot query all filter to the latest `rl_analysis_date` per repo — exactly the cohort that stays in the current table. Analysts who want time-series questions ("how did this repo's LOC evolve?") query `repo_labor_history` directly.

### If scc is not installed

The code complexity phase is silently skipped. No error is logged. The `repo_labor` table remains empty for repos analyzed without scc.

### Materialized view

The `explorer_repo_languages` materialized view aggregates `repo_labor` data to provide per-repo language breakdowns for analytics tools.

---

## Disk usage summary

| Component | Persistence | Size |
|---|---|---|
| Bare clones | Permanent | 10 MB - 5+ GB per repo |
| Full checkouts | Temporary (deleted after analysis) | Roughly equal to bare clone |
| scc output | In-memory (written to DB) | Negligible |
| Registry responses | In-memory (written to DB) | Negligible |

For a repo with a 500 MB bare clone, the analysis phase temporarily uses an additional 500 MB for the full checkout, then frees it.

---

## Error handling

- **Missing manifest files:** Silently skipped. Not all repos have dependencies.
- **Malformed manifest files:** A warning is logged, but analysis continues with other manifests.
- **Registry errors:** If a registry query fails (timeout, 404, rate limit), the dependency's libyear is not calculated. Other dependencies are still processed.
- **scc failure:** If scc crashes or returns invalid JSON, a warning is logged and `repo_labor` is not populated for that repo.
- **Disk full during checkout:** The checkout is cleaned up in a deferred function that runs even on error.

---

## Next steps

- [Facade Commits](facade-commits.md) -- how git log data is parsed before analysis
- [Materialized Views](materialized-views.md) -- views that aggregate analysis data
- [Overview](overview.md) -- system architecture overview