From 755864960d68e5af9ddc1aa256be05bd1bc81852 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 11 Jul 2026 12:14:10 +0200 Subject: [PATCH 1/6] Add EmbeddingInversion (vec2text) example: weight-export scripts + README MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Scaffolds the standalone example for embedding inversion (decode sentence embeddings back to text) built on the SKaiNET-transformers t5 / vec2text modules: - scripts/export_vec2text_weights.py — convert the jxm/gtr__nq__32 (+correct) and gtr-t5-base HuggingFace checkpoints to flat fp16 SafeTensors SKaiNET loads. - scripts/fetch-vec2text-models.sh — idempotent fetch/convert wrapper. - scripts/dump_embedder_golden.py / dump_golden_tensors.py — golden tensors for the Kotlin parity tests. - README with how-it-works, run instructions, and the planned Compose demo. Model weights (~1.1 GB) are fetched, not checked in (.gitignore). The Compose demo app depends on the t5/vec2text modules being published. Co-Authored-By: Claude Opus 4.8 --- EmbeddingInversion/.gitignore | 5 + EmbeddingInversion/README.md | 89 ++++++++ .../scripts/dump_embedder_golden.py | 58 +++++ .../scripts/dump_golden_tensors.py | 123 ++++++++++ .../scripts/export_vec2text_weights.py | 213 ++++++++++++++++++ .../scripts/fetch-vec2text-models.sh | 28 +++ 6 files changed, 516 insertions(+) create mode 100644 EmbeddingInversion/.gitignore create mode 100644 EmbeddingInversion/README.md create mode 100644 EmbeddingInversion/scripts/dump_embedder_golden.py create mode 100644 EmbeddingInversion/scripts/dump_golden_tensors.py create mode 100644 EmbeddingInversion/scripts/export_vec2text_weights.py create mode 100755 EmbeddingInversion/scripts/fetch-vec2text-models.sh diff --git a/EmbeddingInversion/.gitignore b/EmbeddingInversion/.gitignore new file mode 100644 index 0000000..e263b42 --- /dev/null +++ b/EmbeddingInversion/.gitignore @@ -0,0 +1,5 @@ +# Model weights + goldens are fetched/generated, not checked in (~1.1 GB+). +models/ +.venv/ +scripts/*.log +__pycache__/ diff --git a/EmbeddingInversion/README.md b/EmbeddingInversion/README.md new file mode 100644 index 0000000..34f2e2f --- /dev/null +++ b/EmbeddingInversion/README.md @@ -0,0 +1,89 @@ +# EmbeddingInversion — vec2text in the SKaiNET ecosystem + +Decode sentence embeddings **back into text**, in pure Kotlin. This is a Kotlin port of +[vec2text](https://github.com/vec2text/vec2text) (Morris et al.) inference, running on +[SKaiNET](https://github.com/SKaiNET) — the KotlinDL-successor multiplatform DL engine. + +Given only the 768-d embedding vector of a sentence (from `sentence-transformers/gtr-t5-base`), +the models reconstruct an approximation of the original text: + +``` +original: jack morris is a phd student at cornell tech in new york city +step 0 (invert): cos 0.71 "jack morris is a neophyte at COL Tech ... New York City University in" +step 3 (correct): cos 0.83 "jack morris is a ph.D. tech at neorthell alumnus at Cornell University in" +``` + +It is a striking privacy demonstration: **embeddings are not anonymous** — text can be +recovered from them. + +## How it works + +Three models (all `t5-base` scale), from the `jxm/gtr__nq__32` checkpoints: + +1. **GTR embedder** — `sentence-transformers/gtr-t5-base` T5 encoder + mean pooling → target embedding. +2. **Inversion model** ("hypothesizer", `jxm/gtr__nq__32`) — projects the embedding to 16 + pseudo-tokens, feeds them to a T5 encoder-decoder, greedily decodes a first guess. +3. **Corrector** (`jxm/gtr__nq__32__correct`) — given the target, the current hypothesis and their + difference, generates a refined guess. Iterated; the hypothesis with the highest cosine + similarity to the target wins. + +The Kotlin models live upstream in **SKaiNET-transformers**: +- `llm-inference:t5` — the T5 encoder-decoder runtime + GTR embedder. +- `llm-inference:vec2text` — inversion model, corrector, and the iterative `Vec2TextInverter`. + +## Fetch the model weights + +The checkpoints (~1.1 GB) are converted from HuggingFace to flat fp16 SafeTensors that SKaiNET +loads directly. Requires Python 3.10+ with `torch safetensors huggingface_hub`: + +```bash +scripts/fetch-vec2text-models.sh # → models/{gtr_encoder,inversion,corrector}.safetensors + tokenizer +``` + +Optional golden dumps for the parity tests (needs `transformers`, and `vec2text` for the full dump): + +```bash +python scripts/dump_embedder_golden.py # embedder golden (needs transformers) +python scripts/dump_golden_tensors.py # full golden trace (needs vec2text installed) +``` + +## Run it today (via the SKaiNET-transformers tests) + +Until the `t5`/`vec2text` modules are published as artifacts, run the end-to-end round-trip from +the SKaiNET-transformers repo: + +```bash +cd /path/to/SKaiNET-transformers +VEC2TEXT_MODELS_DIR=/path/to/EmbeddingInversion/models \ + ./gradlew :llm-inference:vec2text:jvmTest --tests '*Vec2TextRoundTripTest' +# GTR embedder parity (cos > 0.999 vs reference): +VEC2TEXT_MODELS_DIR=/path/to/EmbeddingInversion/models \ + ./gradlew :llm-inference:t5:jvmTest --tests '*GtrEmbedderParityTest' +``` + +## Planned Compose Multiplatform demo (`app/`) + +A `GloVeEmbeddings`-style Compose app (desktop JVM first) is the next step. It depends on the +`t5` / `vec2text` modules being available as dependencies — either published in a +`skainet-transformers` release **> 0.34.1**, or wired via a composite build. Planned tabs: + +- **Round trip** — type text → embed → show the 768-d vector → invert → compare original vs + reconstruction, with the per-step hypothesis + cosine sparkline. +- **Vector arithmetic** — interpolate two sentence embeddings with a slider and invert the + midpoint live (why inversion matters for privacy). + +Recommended JVM args (large models): `-Xmx4g`, plus SKaiNET's SIMD flags for speed. + +## Status + +| Milestone | State | +|---|---| +| M0 weight export + golden | ✅ done (scripts here) | +| M1 T5 encoder + GTR embedder | ✅ verified (cosine 0.99999985 vs reference) | +| M2 inversion (single-shot) | ✅ working end-to-end | +| M3 corrector loop | ✅ working end-to-end | +| M4 Compose demo app | ⏳ pending module publish | +| M5 beam search + KV-cache speedup | ⏳ follow-up | + +Current decoding is greedy with a no-KV-cache O(L²) loop — correct but slow on CPU. Beam search +and a KV cache (much faster, closer reconstructions) are the main follow-ups. diff --git a/EmbeddingInversion/scripts/dump_embedder_golden.py b/EmbeddingInversion/scripts/dump_embedder_golden.py new file mode 100644 index 0000000..b1d61b1 --- /dev/null +++ b/EmbeddingInversion/scripts/dump_embedder_golden.py @@ -0,0 +1,58 @@ +#!/usr/bin/env python3 +"""Minimal golden dump for the GTR embedder parity test (needs only torch + transformers). + +For each test string, writes token ids (T5 SentencePiece, max_length=32 + EOS, no padding) +and the raw-encoder mean-pooled embedding [768] — exactly the vec2text embedder path +(AutoModel(...).encoder + mean_pool, NO Dense, NO L2). The Kotlin T5 embedder test feeds +these exact ids and compares its embedding to `embedding`. +""" +import argparse +import json +from pathlib import Path + +import torch +from transformers import AutoModel, AutoTokenizer + +REPO = "sentence-transformers/gtr-t5-base" +STRINGS = [ + "jack morris is a phd student at cornell tech in new york city", + "the quick brown fox jumps over the lazy dog", +] + + +def mean_pool(last_hidden_state, attention_mask): + mask = attention_mask.unsqueeze(-1).float() + summed = (last_hidden_state * mask).sum(dim=1) + counts = mask.sum(dim=1).clamp(min=1e-9) + return summed / counts + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--out", default="models/embedder_golden.json") + args = ap.parse_args() + + tok = AutoTokenizer.from_pretrained(REPO) + encoder = AutoModel.from_pretrained(REPO).encoder.eval() + + records = [] + for s in STRINGS: + enc = tok([s], return_tensors="pt", max_length=32, truncation=True) + with torch.no_grad(): + hidden = encoder( + input_ids=enc["input_ids"], attention_mask=enc["attention_mask"] + ).last_hidden_state + emb = mean_pool(hidden, enc["attention_mask"])[0] + records.append({ + "text": s, + "input_ids": enc["input_ids"][0].tolist(), + "embedding": emb.float().tolist(), + }) + print(f" {s[:40]!r}: ids={enc['input_ids'][0].tolist()[:8]}... dim={emb.shape[0]}") + + Path(args.out).write_text(json.dumps({"records": records}, indent=2)) + print(f"wrote {args.out}") + + +if __name__ == "__main__": + main() diff --git a/EmbeddingInversion/scripts/dump_golden_tensors.py b/EmbeddingInversion/scripts/dump_golden_tensors.py new file mode 100644 index 0000000..869d690 --- /dev/null +++ b/EmbeddingInversion/scripts/dump_golden_tensors.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Dump golden intermediate tensors from vec2text for SKaiNET parity tests. + +Requires vec2text installed (`pip install vec2text` or `pip install -e path/to/vec2text`). +Writes golden.json in --out-dir with, per test string: + - embedder token ids (max_length=32, with EOS/pad) and attention mask + - GTR mean-pooled embedding (768) + - inversion MLP output (16 x 768) = the encoder inputs_embeds + - inversion encoder memory (16 x 768) + - first-step decoder logits top-20 (id, value) + - greedy hypothesis token ids and decoded text + - corrector inputs_embeds after LayerNorm (52+hypLen x 768) for the first step + - per-step cosine similarity trace for a short recursive run + +The Kotlin BertNumericalAccuracyTest-style tests assert against these to a +relative tolerance (see EmbeddingInversion module tests). +""" + +import argparse +import json +from pathlib import Path + +import torch +import vec2text +from vec2text import analyze_utils # noqa: F401 (ensures registry import side-effects) + +TEST_STRINGS = [ + "jack morris is a phd student at cornell tech in new york city", + "the quick brown fox jumps over the lazy dog", + "embeddings are not anonymous", +] + + +def tolist(t: torch.Tensor): + return t.detach().float().cpu().reshape(-1).tolist() + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--out-dir", default="models") + ap.add_argument("--steps", type=int, default=5) + args = ap.parse_args() + out_dir = Path(args.out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + + torch.manual_seed(0) + corrector = vec2text.load_pretrained_corrector("gtr-base") + inv_trainer = corrector.inversion_trainer + model = inv_trainer.model + model.eval() + emb_tok = model.embedder_tokenizer + t5_tok = model.tokenizer + + records = [] + for s in TEST_STRINGS: + enc = emb_tok( + [s], return_tensors="pt", max_length=model.config.max_seq_length, + truncation=True, padding="max_length", + ) + with torch.no_grad(): + frozen = model.call_embedding_model( + input_ids=enc["input_ids"], attention_mask=enc["attention_mask"] + ) + inputs_embeds, attn = model.embed_and_project( + embedder_input_ids=enc["input_ids"], + embedder_attention_mask=enc["attention_mask"], + frozen_embeddings=None, + ) + enc_out = model.encoder_decoder.encoder( + inputs_embeds=inputs_embeds, attention_mask=attn + ).last_hidden_state + dec_start = torch.full((1, 1), model.encoder_decoder.config.decoder_start_token_id) + logits = model.encoder_decoder( + encoder_outputs=(enc_out,), attention_mask=attn, + decoder_input_ids=dec_start, + ).logits[0, 0] + top = torch.topk(logits, 20) + hyp = model.generate( + {"frozen_embeddings": frozen}, + {"min_length": 1, "max_length": 128, "num_beams": 1, "do_sample": False}, + ) + + # short recursive run for cosine trace + traces = [] + for n in range(1, args.steps + 1): + with torch.no_grad(): + out = vec2text.invert_embeddings( + embeddings=frozen, corrector=corrector, num_steps=n, + ) + re_emb = model.call_embedding_model( + **emb_tok(out, return_tensors="pt", + max_length=model.config.max_seq_length, + truncation=True, padding="max_length") + ) + cos = torch.nn.functional.cosine_similarity(frozen, re_emb).item() + traces.append({"step": n, "text": out[0], "cosine": cos}) + + records.append({ + "text": s, + "embedder_input_ids": enc["input_ids"][0].tolist(), + "embedder_attention_mask": enc["attention_mask"][0].tolist(), + "embedding": tolist(frozen), + "inversion_inputs_embeds": tolist(inputs_embeds), + "inversion_encoder_memory": tolist(enc_out), + "first_step_logits_topk": [ + {"id": int(i), "value": float(v)} + for i, v in zip(top.indices.tolist(), top.values.tolist()) + ], + "hypothesis_ids": hyp[0].tolist(), + "hypothesis_text": t5_tok.decode(hyp[0], skip_special_tokens=True), + "cosine_trace": traces, + }) + print(f" {s[:40]!r}: hyp={records[-1]['hypothesis_text'][:50]!r}") + + (out_dir / "golden.json").write_text(json.dumps( + {"max_seq_length": model.config.max_seq_length, + "num_repeat_tokens": model.config.num_repeat_tokens, + "records": records}, indent=2)) + print(f"wrote {out_dir/'golden.json'}") + + +if __name__ == "__main__": + main() diff --git a/EmbeddingInversion/scripts/export_vec2text_weights.py b/EmbeddingInversion/scripts/export_vec2text_weights.py new file mode 100644 index 0000000..6e7f92e --- /dev/null +++ b/EmbeddingInversion/scripts/export_vec2text_weights.py @@ -0,0 +1,213 @@ +#!/usr/bin/env python3 +"""Export vec2text gtr-base checkpoints to flat SafeTensors for SKaiNET. + +Produces in --out-dir: + gtr_encoder.safetensors sentence-transformers/gtr-t5-base T5 encoder (+ shared embedding) + inversion.safetensors jxm/gtr__nq__32 (embedding_transform.* + encoder_decoder.*) + corrector.safetensors jxm/gtr__nq__32__correct (embedding_transform_{1,2,3}.*, layernorm.*, encoder_decoder.*) + tokenizer.json, spiece.model, tokenizer_config.json (t5-base SentencePiece, shared by all models) + .config.json per-model dims/hyperparams needed by the Kotlin loaders + +Only needs: torch, safetensors, huggingface_hub. No vec2text install required — +this is pure state-dict surgery. Checkpoint key layout verified against +vec2text/models/inversion.py and corrector_encoder.py (incl. the legacy +shared-MLP remap from trainers/corrector.py::_remap_state_dict). +""" + +import argparse +import json +import shutil +from pathlib import Path + +import torch +from huggingface_hub import snapshot_download +from safetensors.torch import save_file + +GTR_REPO = "sentence-transformers/gtr-t5-base" +INVERSION_REPO = "jxm/gtr__nq__32" +CORRECTOR_REPO = "jxm/gtr__nq__32__correct" + + +def load_state_dict(repo_dir: Path) -> dict: + """Load a checkpoint state dict from a downloaded HF snapshot. + + Handles single-file and sharded layouts, safetensors and pytorch_model.bin. + """ + # Single-file variants. + for name in ("model.safetensors", "pytorch_model.bin"): + path = repo_dir / name + if path.exists(): + if name.endswith(".safetensors"): + from safetensors.torch import load_file + + return load_file(str(path)) + return torch.load(str(path), map_location="cpu", weights_only=True) + + # Sharded variants: read the index and merge all referenced shards. + for index_name, is_safe in ( + ("model.safetensors.index.json", True), + ("pytorch_model.bin.index.json", False), + ): + index_path = repo_dir / index_name + if index_path.exists(): + index = json.loads(index_path.read_text()) + shards = sorted(set(index["weight_map"].values())) + sd: dict = {} + for shard in shards: + shard_path = repo_dir / shard + if is_safe: + from safetensors.torch import load_file + + sd.update(load_file(str(shard_path))) + else: + sd.update( + torch.load(str(shard_path), map_location="cpu", weights_only=True) + ) + return sd + + raise FileNotFoundError(f"no model weights found in {repo_dir}") + + +def to_dtype(sd: dict, dtype: torch.dtype) -> dict: + # .clone() breaks any shared storage (T5 ties shared/embed_tokens/lm_head to the + # same tensor); safetensors refuses to serialize aliased storage. + return { + k: (v.to(dtype) if v.is_floating_point() else v).contiguous().clone() + for k, v in sd.items() + } + + +def drop_tied_embeddings(sd: dict) -> dict: + # T5 ties word embeddings: encoder/decoder embed_tokens and lm_head all alias + # `shared.weight` (or `encoder_decoder.shared.weight`). Keep only the `shared` + # copy; the Kotlin loader feeds it to every tied site. + tied_suffixes = ( + "encoder.embed_tokens.weight", + "decoder.embed_tokens.weight", + "lm_head.weight", + ) + return { + k: v + for k, v in sd.items() + if not any(k.endswith(s) for s in tied_suffixes) + } + + +def remap_legacy_corrector_keys(sd: dict) -> dict: + """Mirror Corrector._remap_state_dict: old checkpoints share one MLP for all three.""" + if {"embedding_transform.3.weight", "embedding_transform.3.bias"} <= sd.keys(): + for idx in ("0", "3"): + w = sd.pop(f"embedding_transform.{idx}.weight") + b = sd.pop(f"embedding_transform.{idx}.bias") + for n in ("1", "2", "3"): + sd[f"embedding_transform_{n}.{idx}.weight"] = w.clone() + sd[f"embedding_transform_{n}.{idx}.bias"] = b.clone() + return sd + + +def export_gtr_encoder(out_dir: Path, dtype: torch.dtype) -> None: + repo = Path(snapshot_download(GTR_REPO)) + sd = load_state_dict(repo) + # vec2text uses AutoModel(...).encoder: keep the shared embedding + encoder stack only. + kept = { + k: v + for k, v in sd.items() + if k == "shared.weight" or k.startswith("encoder.") + } + assert "shared.weight" in kept or "encoder.embed_tokens.weight" in kept, sorted(kept)[:5] + if "shared.weight" not in kept: + kept["shared.weight"] = sd["encoder.embed_tokens.weight"] + kept = drop_tied_embeddings(kept) + save_file(to_dtype(kept, dtype), str(out_dir / "gtr_encoder.safetensors")) + + cfg = json.loads((repo / "config.json").read_text()) + (out_dir / "gtr_encoder.config.json").write_text( + json.dumps( + { + "d_model": cfg["d_model"], + "num_layers": cfg["num_layers"], + "num_heads": cfg["num_heads"], + "d_kv": cfg["d_kv"], + "d_ff": cfg["d_ff"], + "vocab_size": cfg["vocab_size"], + "relative_attention_num_buckets": cfg["relative_attention_num_buckets"], + "relative_attention_max_distance": cfg.get( + "relative_attention_max_distance", 128 + ), + "layer_norm_epsilon": cfg["layer_norm_epsilon"], + "feed_forward_proj": cfg.get("feed_forward_proj", "relu"), + "eos_token_id": cfg.get("eos_token_id", 1), + "pad_token_id": cfg.get("pad_token_id", 0), + "max_seq_length": 32, + }, + indent=2, + ) + ) + # The t5 tokenizer files travel with the gtr snapshot. + for f in ("spiece.model", "tokenizer.json", "tokenizer_config.json"): + src = repo / f + if src.exists(): + shutil.copy(src, out_dir / f) + + +def export_vec2text_model( + repo_id: str, out_name: str, out_dir: Path, dtype: torch.dtype, corrector: bool +) -> None: + repo = Path(snapshot_download(repo_id)) + sd = load_state_dict(repo) + if corrector: + sd = remap_legacy_corrector_keys(sd) + prefixes = ("embedding_transform_1.", "embedding_transform_2.", + "embedding_transform_3.", "layernorm.", "encoder_decoder.") + else: + prefixes = ("embedding_transform.", "encoder_decoder.") + kept = {k: v for k, v in sd.items() if k.startswith(prefixes)} + missing = [p for p in prefixes if not any(k.startswith(p) for k in kept)] + assert not missing, f"{repo_id}: missing key groups {missing}; has {sorted(sd)[:10]}" + # Ensure the tied word-embedding source survives, then drop its aliases. + shared_key = "encoder_decoder.shared.weight" + if shared_key not in kept: + for cand in ("encoder_decoder.encoder.embed_tokens.weight", + "encoder_decoder.decoder.embed_tokens.weight"): + if cand in sd: + kept[shared_key] = sd[cand] + break + kept = drop_tied_embeddings(kept) + save_file(to_dtype(kept, dtype), str(out_dir / f"{out_name}.safetensors")) + + cfg = json.loads((repo / "config.json").read_text()) + (out_dir / f"{out_name}.config.json").write_text( + json.dumps( + { + "embedder_dim": cfg.get("embedder_dim", 768), + "num_repeat_tokens": cfg.get("num_repeat_tokens", 16), + "max_seq_length": cfg.get("max_seq_length", 32), + "use_ln": cfg.get("use_ln", True), + }, + indent=2, + ) + ) + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--out-dir", default="models") + ap.add_argument("--dtype", choices=("fp16", "fp32"), default="fp16") + args = ap.parse_args() + out_dir = Path(args.out_dir) + out_dir.mkdir(parents=True, exist_ok=True) + dtype = torch.float16 if args.dtype == "fp16" else torch.float32 + + print(f"exporting to {out_dir.resolve()} ({args.dtype})") + export_gtr_encoder(out_dir, dtype) + print(" gtr_encoder.safetensors done") + export_vec2text_model(INVERSION_REPO, "inversion", out_dir, dtype, corrector=False) + print(" inversion.safetensors done") + export_vec2text_model(CORRECTOR_REPO, "corrector", out_dir, dtype, corrector=True) + print(" corrector.safetensors done") + for f in sorted(out_dir.iterdir()): + print(f" {f.name:36} {f.stat().st_size / 1e6:8.1f} MB") + + +if __name__ == "__main__": + main() diff --git a/EmbeddingInversion/scripts/fetch-vec2text-models.sh b/EmbeddingInversion/scripts/fetch-vec2text-models.sh new file mode 100755 index 0000000..9b94baa --- /dev/null +++ b/EmbeddingInversion/scripts/fetch-vec2text-models.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Fetch + convert the vec2text gtr-base checkpoints into flat SafeTensors that +# SKaiNET can load. Idempotent: skips work if the outputs already exist. +# +# Usage: scripts/fetch-vec2text-models.sh [OUT_DIR] +# OUT_DIR defaults to /models +# +# Requires python3 with: torch safetensors huggingface_hub +# (the export step is pure state-dict surgery; vec2text itself is only needed +# for the optional golden-tensor dump used by the parity tests.) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +OUT_DIR="${1:-$SCRIPT_DIR/../models}" +DTYPE="${VEC2TEXT_DTYPE:-fp16}" + +mkdir -p "$OUT_DIR" + +if [[ -f "$OUT_DIR/inversion.safetensors" && -f "$OUT_DIR/corrector.safetensors" && -f "$OUT_DIR/gtr_encoder.safetensors" ]]; then + echo "vec2text models already present in $OUT_DIR — skipping. (delete them to re-fetch)" + exit 0 +fi + +echo "==> exporting vec2text gtr-base weights to $OUT_DIR ($DTYPE)" +python3 "$SCRIPT_DIR/export_vec2text_weights.py" --out-dir "$OUT_DIR" --dtype "$DTYPE" + +echo "==> done. Files in $OUT_DIR:" +ls -lh "$OUT_DIR" From 58642e1e6cc1e9a9b76998f89b02dd313035ec99 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 11 Jul 2026 12:22:15 +0200 Subject: [PATCH 2/6] Remove Python weight-conversion scripts; document conversion in README Conversion tooling doesn't belong in the Kotlin examples repo. The README now describes the one-time offline weight conversion instead (source HuggingFace checkpoints, which keys to keep per output file, tied-embedding handling, and shard merging), so weights remain reproducible without shipping Python here. Co-Authored-By: Claude Opus 4.8 --- .../scripts/dump_embedder_golden.py | 58 ----- .../scripts/dump_golden_tensors.py | 123 ---------- .../scripts/export_vec2text_weights.py | 213 ------------------ .../scripts/fetch-vec2text-models.sh | 28 --- 4 files changed, 422 deletions(-) delete mode 100644 EmbeddingInversion/scripts/dump_embedder_golden.py delete mode 100644 EmbeddingInversion/scripts/dump_golden_tensors.py delete mode 100644 EmbeddingInversion/scripts/export_vec2text_weights.py delete mode 100755 EmbeddingInversion/scripts/fetch-vec2text-models.sh diff --git a/EmbeddingInversion/scripts/dump_embedder_golden.py b/EmbeddingInversion/scripts/dump_embedder_golden.py deleted file mode 100644 index b1d61b1..0000000 --- a/EmbeddingInversion/scripts/dump_embedder_golden.py +++ /dev/null @@ -1,58 +0,0 @@ -#!/usr/bin/env python3 -"""Minimal golden dump for the GTR embedder parity test (needs only torch + transformers). - -For each test string, writes token ids (T5 SentencePiece, max_length=32 + EOS, no padding) -and the raw-encoder mean-pooled embedding [768] — exactly the vec2text embedder path -(AutoModel(...).encoder + mean_pool, NO Dense, NO L2). The Kotlin T5 embedder test feeds -these exact ids and compares its embedding to `embedding`. -""" -import argparse -import json -from pathlib import Path - -import torch -from transformers import AutoModel, AutoTokenizer - -REPO = "sentence-transformers/gtr-t5-base" -STRINGS = [ - "jack morris is a phd student at cornell tech in new york city", - "the quick brown fox jumps over the lazy dog", -] - - -def mean_pool(last_hidden_state, attention_mask): - mask = attention_mask.unsqueeze(-1).float() - summed = (last_hidden_state * mask).sum(dim=1) - counts = mask.sum(dim=1).clamp(min=1e-9) - return summed / counts - - -def main(): - ap = argparse.ArgumentParser() - ap.add_argument("--out", default="models/embedder_golden.json") - args = ap.parse_args() - - tok = AutoTokenizer.from_pretrained(REPO) - encoder = AutoModel.from_pretrained(REPO).encoder.eval() - - records = [] - for s in STRINGS: - enc = tok([s], return_tensors="pt", max_length=32, truncation=True) - with torch.no_grad(): - hidden = encoder( - input_ids=enc["input_ids"], attention_mask=enc["attention_mask"] - ).last_hidden_state - emb = mean_pool(hidden, enc["attention_mask"])[0] - records.append({ - "text": s, - "input_ids": enc["input_ids"][0].tolist(), - "embedding": emb.float().tolist(), - }) - print(f" {s[:40]!r}: ids={enc['input_ids'][0].tolist()[:8]}... dim={emb.shape[0]}") - - Path(args.out).write_text(json.dumps({"records": records}, indent=2)) - print(f"wrote {args.out}") - - -if __name__ == "__main__": - main() diff --git a/EmbeddingInversion/scripts/dump_golden_tensors.py b/EmbeddingInversion/scripts/dump_golden_tensors.py deleted file mode 100644 index 869d690..0000000 --- a/EmbeddingInversion/scripts/dump_golden_tensors.py +++ /dev/null @@ -1,123 +0,0 @@ -#!/usr/bin/env python3 -"""Dump golden intermediate tensors from vec2text for SKaiNET parity tests. - -Requires vec2text installed (`pip install vec2text` or `pip install -e path/to/vec2text`). -Writes golden.json in --out-dir with, per test string: - - embedder token ids (max_length=32, with EOS/pad) and attention mask - - GTR mean-pooled embedding (768) - - inversion MLP output (16 x 768) = the encoder inputs_embeds - - inversion encoder memory (16 x 768) - - first-step decoder logits top-20 (id, value) - - greedy hypothesis token ids and decoded text - - corrector inputs_embeds after LayerNorm (52+hypLen x 768) for the first step - - per-step cosine similarity trace for a short recursive run - -The Kotlin BertNumericalAccuracyTest-style tests assert against these to a -relative tolerance (see EmbeddingInversion module tests). -""" - -import argparse -import json -from pathlib import Path - -import torch -import vec2text -from vec2text import analyze_utils # noqa: F401 (ensures registry import side-effects) - -TEST_STRINGS = [ - "jack morris is a phd student at cornell tech in new york city", - "the quick brown fox jumps over the lazy dog", - "embeddings are not anonymous", -] - - -def tolist(t: torch.Tensor): - return t.detach().float().cpu().reshape(-1).tolist() - - -def main() -> None: - ap = argparse.ArgumentParser() - ap.add_argument("--out-dir", default="models") - ap.add_argument("--steps", type=int, default=5) - args = ap.parse_args() - out_dir = Path(args.out_dir) - out_dir.mkdir(parents=True, exist_ok=True) - - torch.manual_seed(0) - corrector = vec2text.load_pretrained_corrector("gtr-base") - inv_trainer = corrector.inversion_trainer - model = inv_trainer.model - model.eval() - emb_tok = model.embedder_tokenizer - t5_tok = model.tokenizer - - records = [] - for s in TEST_STRINGS: - enc = emb_tok( - [s], return_tensors="pt", max_length=model.config.max_seq_length, - truncation=True, padding="max_length", - ) - with torch.no_grad(): - frozen = model.call_embedding_model( - input_ids=enc["input_ids"], attention_mask=enc["attention_mask"] - ) - inputs_embeds, attn = model.embed_and_project( - embedder_input_ids=enc["input_ids"], - embedder_attention_mask=enc["attention_mask"], - frozen_embeddings=None, - ) - enc_out = model.encoder_decoder.encoder( - inputs_embeds=inputs_embeds, attention_mask=attn - ).last_hidden_state - dec_start = torch.full((1, 1), model.encoder_decoder.config.decoder_start_token_id) - logits = model.encoder_decoder( - encoder_outputs=(enc_out,), attention_mask=attn, - decoder_input_ids=dec_start, - ).logits[0, 0] - top = torch.topk(logits, 20) - hyp = model.generate( - {"frozen_embeddings": frozen}, - {"min_length": 1, "max_length": 128, "num_beams": 1, "do_sample": False}, - ) - - # short recursive run for cosine trace - traces = [] - for n in range(1, args.steps + 1): - with torch.no_grad(): - out = vec2text.invert_embeddings( - embeddings=frozen, corrector=corrector, num_steps=n, - ) - re_emb = model.call_embedding_model( - **emb_tok(out, return_tensors="pt", - max_length=model.config.max_seq_length, - truncation=True, padding="max_length") - ) - cos = torch.nn.functional.cosine_similarity(frozen, re_emb).item() - traces.append({"step": n, "text": out[0], "cosine": cos}) - - records.append({ - "text": s, - "embedder_input_ids": enc["input_ids"][0].tolist(), - "embedder_attention_mask": enc["attention_mask"][0].tolist(), - "embedding": tolist(frozen), - "inversion_inputs_embeds": tolist(inputs_embeds), - "inversion_encoder_memory": tolist(enc_out), - "first_step_logits_topk": [ - {"id": int(i), "value": float(v)} - for i, v in zip(top.indices.tolist(), top.values.tolist()) - ], - "hypothesis_ids": hyp[0].tolist(), - "hypothesis_text": t5_tok.decode(hyp[0], skip_special_tokens=True), - "cosine_trace": traces, - }) - print(f" {s[:40]!r}: hyp={records[-1]['hypothesis_text'][:50]!r}") - - (out_dir / "golden.json").write_text(json.dumps( - {"max_seq_length": model.config.max_seq_length, - "num_repeat_tokens": model.config.num_repeat_tokens, - "records": records}, indent=2)) - print(f"wrote {out_dir/'golden.json'}") - - -if __name__ == "__main__": - main() diff --git a/EmbeddingInversion/scripts/export_vec2text_weights.py b/EmbeddingInversion/scripts/export_vec2text_weights.py deleted file mode 100644 index 6e7f92e..0000000 --- a/EmbeddingInversion/scripts/export_vec2text_weights.py +++ /dev/null @@ -1,213 +0,0 @@ -#!/usr/bin/env python3 -"""Export vec2text gtr-base checkpoints to flat SafeTensors for SKaiNET. - -Produces in --out-dir: - gtr_encoder.safetensors sentence-transformers/gtr-t5-base T5 encoder (+ shared embedding) - inversion.safetensors jxm/gtr__nq__32 (embedding_transform.* + encoder_decoder.*) - corrector.safetensors jxm/gtr__nq__32__correct (embedding_transform_{1,2,3}.*, layernorm.*, encoder_decoder.*) - tokenizer.json, spiece.model, tokenizer_config.json (t5-base SentencePiece, shared by all models) - .config.json per-model dims/hyperparams needed by the Kotlin loaders - -Only needs: torch, safetensors, huggingface_hub. No vec2text install required — -this is pure state-dict surgery. Checkpoint key layout verified against -vec2text/models/inversion.py and corrector_encoder.py (incl. the legacy -shared-MLP remap from trainers/corrector.py::_remap_state_dict). -""" - -import argparse -import json -import shutil -from pathlib import Path - -import torch -from huggingface_hub import snapshot_download -from safetensors.torch import save_file - -GTR_REPO = "sentence-transformers/gtr-t5-base" -INVERSION_REPO = "jxm/gtr__nq__32" -CORRECTOR_REPO = "jxm/gtr__nq__32__correct" - - -def load_state_dict(repo_dir: Path) -> dict: - """Load a checkpoint state dict from a downloaded HF snapshot. - - Handles single-file and sharded layouts, safetensors and pytorch_model.bin. - """ - # Single-file variants. - for name in ("model.safetensors", "pytorch_model.bin"): - path = repo_dir / name - if path.exists(): - if name.endswith(".safetensors"): - from safetensors.torch import load_file - - return load_file(str(path)) - return torch.load(str(path), map_location="cpu", weights_only=True) - - # Sharded variants: read the index and merge all referenced shards. - for index_name, is_safe in ( - ("model.safetensors.index.json", True), - ("pytorch_model.bin.index.json", False), - ): - index_path = repo_dir / index_name - if index_path.exists(): - index = json.loads(index_path.read_text()) - shards = sorted(set(index["weight_map"].values())) - sd: dict = {} - for shard in shards: - shard_path = repo_dir / shard - if is_safe: - from safetensors.torch import load_file - - sd.update(load_file(str(shard_path))) - else: - sd.update( - torch.load(str(shard_path), map_location="cpu", weights_only=True) - ) - return sd - - raise FileNotFoundError(f"no model weights found in {repo_dir}") - - -def to_dtype(sd: dict, dtype: torch.dtype) -> dict: - # .clone() breaks any shared storage (T5 ties shared/embed_tokens/lm_head to the - # same tensor); safetensors refuses to serialize aliased storage. - return { - k: (v.to(dtype) if v.is_floating_point() else v).contiguous().clone() - for k, v in sd.items() - } - - -def drop_tied_embeddings(sd: dict) -> dict: - # T5 ties word embeddings: encoder/decoder embed_tokens and lm_head all alias - # `shared.weight` (or `encoder_decoder.shared.weight`). Keep only the `shared` - # copy; the Kotlin loader feeds it to every tied site. - tied_suffixes = ( - "encoder.embed_tokens.weight", - "decoder.embed_tokens.weight", - "lm_head.weight", - ) - return { - k: v - for k, v in sd.items() - if not any(k.endswith(s) for s in tied_suffixes) - } - - -def remap_legacy_corrector_keys(sd: dict) -> dict: - """Mirror Corrector._remap_state_dict: old checkpoints share one MLP for all three.""" - if {"embedding_transform.3.weight", "embedding_transform.3.bias"} <= sd.keys(): - for idx in ("0", "3"): - w = sd.pop(f"embedding_transform.{idx}.weight") - b = sd.pop(f"embedding_transform.{idx}.bias") - for n in ("1", "2", "3"): - sd[f"embedding_transform_{n}.{idx}.weight"] = w.clone() - sd[f"embedding_transform_{n}.{idx}.bias"] = b.clone() - return sd - - -def export_gtr_encoder(out_dir: Path, dtype: torch.dtype) -> None: - repo = Path(snapshot_download(GTR_REPO)) - sd = load_state_dict(repo) - # vec2text uses AutoModel(...).encoder: keep the shared embedding + encoder stack only. - kept = { - k: v - for k, v in sd.items() - if k == "shared.weight" or k.startswith("encoder.") - } - assert "shared.weight" in kept or "encoder.embed_tokens.weight" in kept, sorted(kept)[:5] - if "shared.weight" not in kept: - kept["shared.weight"] = sd["encoder.embed_tokens.weight"] - kept = drop_tied_embeddings(kept) - save_file(to_dtype(kept, dtype), str(out_dir / "gtr_encoder.safetensors")) - - cfg = json.loads((repo / "config.json").read_text()) - (out_dir / "gtr_encoder.config.json").write_text( - json.dumps( - { - "d_model": cfg["d_model"], - "num_layers": cfg["num_layers"], - "num_heads": cfg["num_heads"], - "d_kv": cfg["d_kv"], - "d_ff": cfg["d_ff"], - "vocab_size": cfg["vocab_size"], - "relative_attention_num_buckets": cfg["relative_attention_num_buckets"], - "relative_attention_max_distance": cfg.get( - "relative_attention_max_distance", 128 - ), - "layer_norm_epsilon": cfg["layer_norm_epsilon"], - "feed_forward_proj": cfg.get("feed_forward_proj", "relu"), - "eos_token_id": cfg.get("eos_token_id", 1), - "pad_token_id": cfg.get("pad_token_id", 0), - "max_seq_length": 32, - }, - indent=2, - ) - ) - # The t5 tokenizer files travel with the gtr snapshot. - for f in ("spiece.model", "tokenizer.json", "tokenizer_config.json"): - src = repo / f - if src.exists(): - shutil.copy(src, out_dir / f) - - -def export_vec2text_model( - repo_id: str, out_name: str, out_dir: Path, dtype: torch.dtype, corrector: bool -) -> None: - repo = Path(snapshot_download(repo_id)) - sd = load_state_dict(repo) - if corrector: - sd = remap_legacy_corrector_keys(sd) - prefixes = ("embedding_transform_1.", "embedding_transform_2.", - "embedding_transform_3.", "layernorm.", "encoder_decoder.") - else: - prefixes = ("embedding_transform.", "encoder_decoder.") - kept = {k: v for k, v in sd.items() if k.startswith(prefixes)} - missing = [p for p in prefixes if not any(k.startswith(p) for k in kept)] - assert not missing, f"{repo_id}: missing key groups {missing}; has {sorted(sd)[:10]}" - # Ensure the tied word-embedding source survives, then drop its aliases. - shared_key = "encoder_decoder.shared.weight" - if shared_key not in kept: - for cand in ("encoder_decoder.encoder.embed_tokens.weight", - "encoder_decoder.decoder.embed_tokens.weight"): - if cand in sd: - kept[shared_key] = sd[cand] - break - kept = drop_tied_embeddings(kept) - save_file(to_dtype(kept, dtype), str(out_dir / f"{out_name}.safetensors")) - - cfg = json.loads((repo / "config.json").read_text()) - (out_dir / f"{out_name}.config.json").write_text( - json.dumps( - { - "embedder_dim": cfg.get("embedder_dim", 768), - "num_repeat_tokens": cfg.get("num_repeat_tokens", 16), - "max_seq_length": cfg.get("max_seq_length", 32), - "use_ln": cfg.get("use_ln", True), - }, - indent=2, - ) - ) - - -def main() -> None: - ap = argparse.ArgumentParser() - ap.add_argument("--out-dir", default="models") - ap.add_argument("--dtype", choices=("fp16", "fp32"), default="fp16") - args = ap.parse_args() - out_dir = Path(args.out_dir) - out_dir.mkdir(parents=True, exist_ok=True) - dtype = torch.float16 if args.dtype == "fp16" else torch.float32 - - print(f"exporting to {out_dir.resolve()} ({args.dtype})") - export_gtr_encoder(out_dir, dtype) - print(" gtr_encoder.safetensors done") - export_vec2text_model(INVERSION_REPO, "inversion", out_dir, dtype, corrector=False) - print(" inversion.safetensors done") - export_vec2text_model(CORRECTOR_REPO, "corrector", out_dir, dtype, corrector=True) - print(" corrector.safetensors done") - for f in sorted(out_dir.iterdir()): - print(f" {f.name:36} {f.stat().st_size / 1e6:8.1f} MB") - - -if __name__ == "__main__": - main() diff --git a/EmbeddingInversion/scripts/fetch-vec2text-models.sh b/EmbeddingInversion/scripts/fetch-vec2text-models.sh deleted file mode 100755 index 9b94baa..0000000 --- a/EmbeddingInversion/scripts/fetch-vec2text-models.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env bash -# Fetch + convert the vec2text gtr-base checkpoints into flat SafeTensors that -# SKaiNET can load. Idempotent: skips work if the outputs already exist. -# -# Usage: scripts/fetch-vec2text-models.sh [OUT_DIR] -# OUT_DIR defaults to /models -# -# Requires python3 with: torch safetensors huggingface_hub -# (the export step is pure state-dict surgery; vec2text itself is only needed -# for the optional golden-tensor dump used by the parity tests.) -set -euo pipefail - -SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -OUT_DIR="${1:-$SCRIPT_DIR/../models}" -DTYPE="${VEC2TEXT_DTYPE:-fp16}" - -mkdir -p "$OUT_DIR" - -if [[ -f "$OUT_DIR/inversion.safetensors" && -f "$OUT_DIR/corrector.safetensors" && -f "$OUT_DIR/gtr_encoder.safetensors" ]]; then - echo "vec2text models already present in $OUT_DIR — skipping. (delete them to re-fetch)" - exit 0 -fi - -echo "==> exporting vec2text gtr-base weights to $OUT_DIR ($DTYPE)" -python3 "$SCRIPT_DIR/export_vec2text_weights.py" --out-dir "$OUT_DIR" --dtype "$DTYPE" - -echo "==> done. Files in $OUT_DIR:" -ls -lh "$OUT_DIR" From e012790891f480a585525045f1dd0da19b6a42a6 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 11 Jul 2026 14:55:19 +0200 Subject: [PATCH 3/6] EmbeddingInversion: composite build + runnable CLI against local modules MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wire the example as a Gradle composite build that includeBuilds the sibling ../../SKaiNET and ../../SKaiNET-transformers checkouts, so it compiles and runs against the just-merged t5 / vec2text source without waiting for a published skainet-transformers release. - settings.gradle.kts: includeBuild both repos. SKaiNET core modules auto-substitute (project name == artifactId); the two skainet-transformers-inference-* coords are mapped explicitly to :llm-inference:t5 / :vec2text (their publish artifactId differs from the Gradle project name). - :cli — a JVM application that loads the converted weights + T5 SentencePiece tokenizer and runs the full inversion round-trip (./gradlew :cli:run --args="..."). Verified running end-to-end via the composite. - Gradle 9.6.1 wrapper (matches the included builds); README updated. Co-Authored-By: Claude Opus 4.8 --- EmbeddingInversion/.gitignore | 10 +- EmbeddingInversion/README.md | 75 ++++-- EmbeddingInversion/build.gradle.kts | 1 + EmbeddingInversion/cli/build.gradle.kts | 31 +++ .../kotlin/sk/ainet/samples/vec2text/Main.kt | 78 ++++++ EmbeddingInversion/gradle.properties | 4 + .../gradle/wrapper/gradle-wrapper.jar | Bin 0 -> 48462 bytes .../gradle/wrapper/gradle-wrapper.properties | 8 + EmbeddingInversion/gradlew | 248 ++++++++++++++++++ EmbeddingInversion/gradlew.bat | 82 ++++++ EmbeddingInversion/settings.gradle.kts | 40 +++ 11 files changed, 544 insertions(+), 33 deletions(-) create mode 100644 EmbeddingInversion/build.gradle.kts create mode 100644 EmbeddingInversion/cli/build.gradle.kts create mode 100644 EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt create mode 100644 EmbeddingInversion/gradle.properties create mode 100644 EmbeddingInversion/gradle/wrapper/gradle-wrapper.jar create mode 100644 EmbeddingInversion/gradle/wrapper/gradle-wrapper.properties create mode 100755 EmbeddingInversion/gradlew create mode 100644 EmbeddingInversion/gradlew.bat create mode 100644 EmbeddingInversion/settings.gradle.kts diff --git a/EmbeddingInversion/.gitignore b/EmbeddingInversion/.gitignore index e263b42..dc06c9a 100644 --- a/EmbeddingInversion/.gitignore +++ b/EmbeddingInversion/.gitignore @@ -1,5 +1,7 @@ -# Model weights + goldens are fetched/generated, not checked in (~1.1 GB+). +# Converted model weights + goldens are generated offline, not checked in (~1.1 GB+). models/ -.venv/ -scripts/*.log -__pycache__/ + +# Gradle +.gradle/ +build/ +**/build/ diff --git a/EmbeddingInversion/README.md b/EmbeddingInversion/README.md index 34f2e2f..78c0e0b 100644 --- a/EmbeddingInversion/README.md +++ b/EmbeddingInversion/README.md @@ -31,49 +31,65 @@ The Kotlin models live upstream in **SKaiNET-transformers**: - `llm-inference:t5` — the T5 encoder-decoder runtime + GTR embedder. - `llm-inference:vec2text` — inversion model, corrector, and the iterative `Vec2TextInverter`. -## Fetch the model weights - -The checkpoints (~1.1 GB) are converted from HuggingFace to flat fp16 SafeTensors that SKaiNET -loads directly. Requires Python 3.10+ with `torch safetensors huggingface_hub`: - -```bash -scripts/fetch-vec2text-models.sh # → models/{gtr_encoder,inversion,corrector}.safetensors + tokenizer -``` - -Optional golden dumps for the parity tests (needs `transformers`, and `vec2text` for the full dump): +## Model weights + +SKaiNET loads flat **fp16 SafeTensors**. The reference vec2text checkpoints are custom +PyTorch classes on HuggingFace, so they need a one-time offline conversion (done once with any +Python + HuggingFace tooling — this Kotlin repo intentionally ships no conversion scripts). +Drop the resulting files into `models/` (git-ignored, ~1.1 GB): + +| Output file | Source checkpoint | Keep these keys | +|---|---|---| +| `gtr_encoder.safetensors` | `sentence-transformers/gtr-t5-base` | `shared.weight`, `encoder.*` | +| `inversion.safetensors` | `jxm/gtr__nq__32` | `embedding_transform.{0,3}.{weight,bias}`, `encoder_decoder.*` | +| `corrector.safetensors` | `jxm/gtr__nq__32__correct` | `embedding_transform_{1,2,3}.{0,3}.*`, `layernorm.{weight,bias}`, `encoder_decoder.*` | +| `tokenizer.json` | `t5-base` | (SentencePiece, shared by all models) | + +Conversion notes: +- Cast float tensors to fp16 and `clone()` them — T5 ties `shared.weight` / + `encoder.embed_tokens.weight` / `decoder.embed_tokens.weight` / `lm_head.weight` to one + storage, so **keep only `shared.weight`** and drop the aliases (the Kotlin loader feeds it to + every tied site; SafeTensors also refuses aliased storage). +- The `jxm/*` checkpoints are sharded `pytorch_model-0000N-of-…bin` — merge all shards via the + `.index.json` before filtering keys. +- Some older corrector checkpoints share one `embedding_transform.*` MLP; remap it into the three + `embedding_transform_{1,2,3}.*` (mirrors `Corrector._remap_state_dict`). + +The parity tests (below) compare against golden tensors dumped from the reference models: +`GtrEmbedderParityTest` needs token ids + the mean-pooled embedding from `gtr-t5-base`; +`Vec2TextRoundTripTest` needs only the converted weights above. + +## Run it (composite build — no release needed) + +This project **`includeBuild`s the sibling `../../SKaiNET` and `../../SKaiNET-transformers` +checkouts**, so it compiles against the local `t5` / `vec2text` source directly — no waiting +for a published `skainet-transformers` release. Just put the converted weights in `models/` and: ```bash -python scripts/dump_embedder_golden.py # embedder golden (needs transformers) -python scripts/dump_golden_tensors.py # full golden trace (needs vec2text installed) +./gradlew :cli:run --args="jack morris is a phd student at cornell tech in new york city" ``` -## Run it today (via the SKaiNET-transformers tests) +Environment knobs: `VEC2TEXT_MODELS_DIR` (default `./models`), `VEC2TEXT_STEPS` (default 5). -Until the `t5`/`vec2text` modules are published as artifacts, run the end-to-end round-trip from -the SKaiNET-transformers repo: +The composite requires the sibling checkouts at `../../SKaiNET` and `../../SKaiNET-transformers` +and uses Gradle 9.6.1 (matching them). SKaiNET core modules auto-substitute; the two +`skainet-transformers-inference-*` coordinates are mapped explicitly in `settings.gradle.kts` +(their publish artifactId differs from the Gradle project name). -```bash -cd /path/to/SKaiNET-transformers -VEC2TEXT_MODELS_DIR=/path/to/EmbeddingInversion/models \ - ./gradlew :llm-inference:vec2text:jvmTest --tests '*Vec2TextRoundTripTest' -# GTR embedder parity (cos > 0.999 vs reference): -VEC2TEXT_MODELS_DIR=/path/to/EmbeddingInversion/models \ - ./gradlew :llm-inference:t5:jvmTest --tests '*GtrEmbedderParityTest' -``` +> Reconstruction quality scales with correction `steps`; greedy + few steps + fp16 can produce +> rough or ``-laden output on short inputs. Beam search and a decode KV-cache (much faster, +> closer) are the M5 follow-ups. ## Planned Compose Multiplatform demo (`app/`) -A `GloVeEmbeddings`-style Compose app (desktop JVM first) is the next step. It depends on the -`t5` / `vec2text` modules being available as dependencies — either published in a -`skainet-transformers` release **> 0.34.1**, or wired via a composite build. Planned tabs: +A `GloVeEmbeddings`-style Compose app (desktop JVM first) is the next step, reusing the same +composite build. Planned tabs: - **Round trip** — type text → embed → show the 768-d vector → invert → compare original vs reconstruction, with the per-step hypothesis + cosine sparkline. - **Vector arithmetic** — interpolate two sentence embeddings with a slider and invert the midpoint live (why inversion matters for privacy). -Recommended JVM args (large models): `-Xmx4g`, plus SKaiNET's SIMD flags for speed. - ## Status | Milestone | State | @@ -82,7 +98,8 @@ Recommended JVM args (large models): `-Xmx4g`, plus SKaiNET's SIMD flags for spe | M1 T5 encoder + GTR embedder | ✅ verified (cosine 0.99999985 vs reference) | | M2 inversion (single-shot) | ✅ working end-to-end | | M3 corrector loop | ✅ working end-to-end | -| M4 Compose demo app | ⏳ pending module publish | +| M4 runnable CLI (composite build) | ✅ `./gradlew :cli:run` against local modules | +| M4 Compose demo app | ⏳ next (reuses the same composite build) | | M5 beam search + KV-cache speedup | ⏳ follow-up | Current decoding is greedy with a no-KV-cache O(L²) loop — correct but slow on CPU. Beam search diff --git a/EmbeddingInversion/build.gradle.kts b/EmbeddingInversion/build.gradle.kts new file mode 100644 index 0000000..095f889 --- /dev/null +++ b/EmbeddingInversion/build.gradle.kts @@ -0,0 +1 @@ +// Root build for the EmbeddingInversion example. All real config lives in :cli. diff --git a/EmbeddingInversion/cli/build.gradle.kts b/EmbeddingInversion/cli/build.gradle.kts new file mode 100644 index 0000000..728a59c --- /dev/null +++ b/EmbeddingInversion/cli/build.gradle.kts @@ -0,0 +1,31 @@ +plugins { + kotlin("jvm") version "2.4.0" + application +} + +repositories { + google() + mavenCentral() +} + +kotlin { + jvmToolchain(21) +} + +dependencies { + // Substituted with the local ../../SKaiNET-transformers projects via the composite build. + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.35.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.35.0") + // Substituted with the local ../../SKaiNET projects. + implementation("sk.ainet.core:skainet-lang-core:0.35.0") + implementation("sk.ainet.core:skainet-backend-cpu:0.35.0") + implementation("sk.ainet.core:skainet-io-core:0.35.0") + implementation("sk.ainet.core:skainet-io-safetensors:0.35.0") + + implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.11.0") + implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") +} + +application { + mainClass.set("sk.ainet.samples.vec2text.MainKt") +} diff --git a/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt b/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt new file mode 100644 index 0000000..53fa9d0 --- /dev/null +++ b/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt @@ -0,0 +1,78 @@ +package sk.ainet.samples.vec2text + +import kotlinx.coroutines.runBlocking +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.jsonObject +import sk.ainet.context.DirectCpuExecutionContext +import sk.ainet.io.JvmRandomAccessSource +import sk.ainet.io.safetensors.SafeTensorsParametersLoader +import sk.ainet.io.tokenizer.SentencePieceTokenizer +import sk.ainet.models.t5.GtrEmbedder +import sk.ainet.models.t5.T5Config +import sk.ainet.models.t5.T5Runtime +import sk.ainet.models.t5.loadT5Weights +import sk.ainet.models.vec2text.CorrectorModel +import sk.ainet.models.vec2text.InversionModel +import sk.ainet.models.vec2text.Vec2TextInverter +import sk.ainet.models.vec2text.Vec2TextTokenizer +import sk.ainet.models.vec2text.Vec2TextWeightLoader +import sk.ainet.lang.types.FP32 +import java.io.File + +/** + * EmbeddingInversion CLI — decode a sentence embedding back into text with vec2text on SKaiNET. + * + * Usage: + * ./gradlew :cli:run --args="" + * Model directory (default ./models) can be overridden with VEC2TEXT_MODELS_DIR. + * Needs gtr_encoder / inversion / corrector .safetensors + tokenizer.json (see README). + */ +fun main(args: Array) = runBlocking { + val modelsDir = File(System.getenv("VEC2TEXT_MODELS_DIR") ?: "models") + val text = args.joinToString(" ").ifBlank { + "jack morris is a phd student at cornell tech in new york city" + } + val steps = System.getenv("VEC2TEXT_STEPS")?.toIntOrNull() ?: 5 + + val required = listOf("tokenizer.json", "gtr_encoder.safetensors", "inversion.safetensors", "corrector.safetensors") + val missing = required.filterNot { File(modelsDir, it).exists() } + if (missing.isNotEmpty()) { + System.err.println("Missing model files in ${modelsDir.absolutePath}: $missing") + System.err.println("See README.md for how to produce them, or set VEC2TEXT_MODELS_DIR.") + return@runBlocking + } + + val ctx = DirectCpuExecutionContext() + val cfg = T5Config() + fun loader(name: String) = + SafeTensorsParametersLoader(sourceProvider = { JvmRandomAccessSource.open(File(modelsDir, name).toString()) }) + + println("Loading models from ${modelsDir.absolutePath} …") + val gtr = loadT5Weights(loader("gtr_encoder.safetensors"), ctx, FP32::class, cfg, "", withDecoder = false) + val embedder = GtrEmbedder(T5Runtime(ctx, gtr, FP32::class)) + val inversion = InversionModel(ctx, Vec2TextWeightLoader.loadInversion(loader("inversion.safetensors"), ctx, FP32::class, cfg), FP32::class) + val corrector = CorrectorModel(ctx, Vec2TextWeightLoader.loadCorrector(loader("corrector.safetensors"), ctx, FP32::class, cfg), FP32::class) + + val sp = SentencePieceTokenizer.fromTokenizerJson( + Json.parseToJsonElement(File(modelsDir, "tokenizer.json").readText()).jsonObject + ) + val codec = object : Vec2TextTokenizer { + override fun encodeForEmbedder(text: String): IntArray { + val ids = sp.encode(text).take(cfg.maxSeqLength - 1).toMutableList() + ids.add(cfg.eosTokenId) + return ids.toIntArray() + } + override fun decode(ids: IntArray): String = + sp.decode(ids.filter { it != 0 && it != cfg.eosTokenId }.toIntArray()) + } + + println("Inverting (≤$steps correction steps, greedy)…\n") + val result = Vec2TextInverter(embedder, inversion, corrector, codec) + .invert(text, numSteps = steps, maxLength = cfg.maxSeqLength) + + println("original: $text") + println("reconstructed: ${result.text}") + println("cosine: ${"%.4f".format(result.cosine)}\n") + println("trace:") + result.trace.forEach { println(" step ${it.step}: cos=${"%.4f".format(it.cosine)} \"${it.text}\"") } +} diff --git a/EmbeddingInversion/gradle.properties b/EmbeddingInversion/gradle.properties new file mode 100644 index 0000000..6b74e16 --- /dev/null +++ b/EmbeddingInversion/gradle.properties @@ -0,0 +1,4 @@ +kotlin.code.style=official +org.gradle.jvmargs=-Xmx4g -Dfile.encoding=UTF-8 +org.gradle.configuration-cache=true +kotlin.native.ignoreDisabledTargets=true diff --git a/EmbeddingInversion/gradle/wrapper/gradle-wrapper.jar b/EmbeddingInversion/gradle/wrapper/gradle-wrapper.jar new file mode 100644 index 0000000000000000000000000000000000000000..b1b8ef56b44f16b14dc800fa8103a6d89abb526f GIT binary patch literal 48462 zcma&NV{|3jwk;gnwr$(CRk3Z`Sy9Ed?Nn^ruGlsztklcC=e7I2x9>aqJFB(1eyu-q z%|3b`eLzVT6buar3JMAc2#EOW{C^)LAZQ?YaW!FjX$1*JIcZUG1yyl%HEd!6f#E+}*Jo*NafvM<-FbE0;-_L#rp}qdn%JEoAVNlEB#J^Oq`mU_#*ev4HLmc> zjXz_hFft^><#omb;Zer-%wm4hxo!wjuX3hBldg(^-RiOleKin`>KHfL3P*{k?(rji(#j2Cc0K509#>qu=-T&B!-5EBi(+ zIuTD-qfcAYgS@`Fb2^-p)4#o6A3z0&fp?~cV=CRsAeCmO4ZQ5kKgC%0el=Q&Rhd#k zaGmAbUW8uKC}-C0s~2);d{;mpsNBx9rn__66W{AhaSvJEK+c0b6ARO+l(CI7E|S5x zhaYP--@F<|99X&)9`q^2(^-Zu^Tzfm)v|gkTJHQ!G*zIg5hzoygeXZoYUEJ;iFkE# zq^r$*c|>Hmn3GapzcDYnjgSFiO^NFyTR5AH#mh%zRToMpEi(r)1$5)h455DuV}0al z!*psWuL@Ke-2gvftfMEGf9YEi^<{B@qru zINgo+YsE&LN?)1qItJoNhISp-fZ86`XR#*6xcvM~_7=JHUX;K9*=Gu5X~ zix|O2d=&C#u_w{=B$eCpJ4L*6i7={j+{Og~`Emz@&98}6s<-p^)`0fXE4cJBP{>)Ltb>JwcqI>yz z0-r-SEhC@p)XOoh|1|XgjFaREHfsu4dAGVz*k#m+V<4 zHqvlud6=;#QWHUoTR_a8Y8+heN?M%n1@0YLiaN@GuOPNd26tik7eKulTx?mM-R!1H znB6+H{^krFXg_b{y=QeCT~qR3T4}l+b!Oz9;~|3*6F<3?#|DYYW&1RtFE)ILZ!`85 zVmvrZkLTzf31unH7Cc5E0iFShqlBE9hgEnRJH1juII*vyp&xd!g`q}X_6WT6E$hhQ`Vdp9k^<)VS?lj!cTh z7FQcQAVA@jL^cXod8cnhKG2TS9+;QU6Kq>}UOY3&TL9gXbl{Fv8@WsF=z7>X0To@$ zY@Oi1uc|MdJ$>Kn{@!g_e`-I&Tpwfg9cr>(iakDX1qciCG_1y!Di#4_)lE!bWJbrp z5aUonb6m-?tiQyR_`P#~SOu+tb_ev6JO>EbEhHK@KbeT0_FDo>dl9bMg)>xmCNB*g zG5NC8ABavuTEZVGW6jP*nAqRt3W?7Iigc-EE~zpNJXRAE z>`~RO9$892j&I1kV;9U)xT8^}IeV`n{}QDtj2o-RBt`DGZUOO;O*lFCb_vpyGh*;95PfeGu!dyrmZ9VJ3Z*upg z6R-3Lr%_55$Hw1^{+KWx0#z`T7O6sXo1h;m?B_ur`X2bFz-SzDrL zpk^@B<+I6imc@7vip za%1jMB7q@1j# zz{u?YojZMW{5j$@h=v4iu2mTu7IzI|)Sxn!74=*J>1a&?Xjt z2%JhSi#4huEcD9qdR9Lj4vwmfnL{%+vQ{f-KgYeqin(OPd8+(g*Uq#TLxQjD4 zLCL%ul(V&PAPlAx8D`@K8Rc`{GPecQ<)d=KWel0ejFeeXGQ6o7601B!!I@RY&eDriADD6wP6DcFKDLZ|lO#YwnrNCZ)zRJpdxX_nPZa4j#$j6v!h|6p!dH}MY6#B`@%6=) z-HigguDACKBULnon^FKzazF|Y1{t(U5rUGnEU|}djVsWT-F>@@mNx?_$kF51QF4C5 zStKR$^3(fw85(4HGs9{mUTtn1)3PwxTN?6}j;32&vJ^BiPHfndLkdU5sOemXKGyCZ z@<7j(k>DNeo~QXyJkFWk!7(y1SB%nA3{v~P2c8ooKa4auM!el!Q_=;lJ$c5ADqE+^ zX8*|A99v;jWPrm(8=h;2ZAj|(vVbx~wQ{N%v;eYLD_BB2LAEWCs@xauyBDl(_HIBvA(XJ7B1E;O zJYCJ8xFJh7f5sr;Y#Wp_`$4Z_H4e9bGiBp?Qu&2!@%Bl2dT5evfFO*^hLDiBu2%Jl z*WAlL5PaQ7skJa(qVysky}DQquZ8U?2@UyJ8zB#=U_E>MgE%XA$CtfL31m$rATJvC zs@!crc0=128PM=Zp zW_5Czv9))n_8Ru?{pxM2F8^r%*O41}RnONbSj*piG%`nyF>6ky=|;B&k8iot(J=kyoU3p<_zaAX(1ijzf*uXA zZ_5jeC{Lks+&QeFIlmzZi3+fsF4fNW^~kvC4Q*T-vrNP!x9xnen12lZQM=1_MdW76LKX(GuW`%T~dM^YX6+ras|Xy4Qhfcq=D+z-P-ea z`T;^gj3+grr3^hwqcNTJErl$z+k>{bYFm6QV%7Opth?9+>|Dn)O@`7F@=j-XSqGPW zjUAu%b3Er@;j1%RZxVDhI3sakg-gvTLOSV7;FV6ED=(5;UG??=WADZw^=$4AyFh#}VMe3afM^pF zFa}-nM8X=K?Jy02*o02@6k{ z%O!hBhjXlXKdhy3A{xGB<##e|j3^dFv~~%v2_H{t(mN7NVeS~51?D&Ozbxa`qwZ_4 z;C#Q#fL1sua%ggucgIEHZtcY=Ag&GgE|h7Q{77D!WUq`;SSGEE0pU;aoj<7-JCAvf zduN=(tx3Mb+EUXKoax|v;8b@#HJ&Q|!g4ryrl|R>WlAv?IH`bk)I24;eE4NIq@SLK31LD4+w~#3iN{=<`<1R!t^$@K5>U6%W=%8_ANuR5 zs(IDuI18ftirTDARnGmF%;iz+4{MlMihJw_l!0Y)NttXC_t+s)V<EY>=Xin*nGX79k6vQ?beRk zy_J>@YSC_gMIG$yjO-y&o>S6xtfT27aSs>e|`x(f2R1bM}*518~%x>1Yct=18b&Z>GiS*>VB$+i2876zL)1cT zN33g=g|>xWE2)dds5m2+8Vy)m-u@NHOlGYxxjam21r1;xWtT0TgqKZrl}*LSkqFt4 zNTI1=3o%C*!-i;iWnlca$stRdwITA1?#fD~5OIqIQAM18BwO_u>hqL&OAANiF|8rG z_IZ9mp?FA-{Gq9+Ky<#NgL1gWJixfO0ziP$4T4G>vsvqC-NQh+A64F4! z-(t<=AbPSG%`mTl6BJtH~3RmvPhQlE-EUkEoBIP(_WMN zK~Fe!siee{M*ns1hkp5(2}vX#%u+T!Abh=<_gEx_QW?h4V@B>uOCEetEe01tl)^`V z(=cOLmuOB;8&&m%_6pcyrt83UXkJ`f9I&0KxY09}RTTs!l^_7~8$tPA%Hm#&$k0;# zF;O0zCGo0IN)X~SyKDoY1DW{Ulce|V9w=ld;U`z$t$>8U!Gu8V?_LAJAudt3eI#*! z2i9~F=kP5m>!bmb%1e~b1!1gz01Py(Yw5gOsFN#o1a&d|=PpgN(#UVreY9^99I0iG zaYE@>(C^V7pnoB~#w$2C1_TIb1N5Je&iao?S2A*TF>@vpHg`31{uk<9{zf_}s&z%dL-Fo)C$yl$%pAdqU!HJgp zh_{m1imk{&{ScyeuziqZHu5cto0{S}^BlXu% z0~;>_yHGd#?Kt8ErxK)z6ojj5SacQobw)-8`c!$HOI*V6eyqou{1Upm%_p!BY^t(D zDtn(oQ!jff`ddGSD;P8Hes!v)OKW-*>mS&#i0ow87;h>(=Cu0>b4)|=EegbN5=Xkh z9Ge13=3z#sk+fT<)PuUUf_%Nx@l!P?t*mni^94p^Ax6b2SVL5U>9dHH!H4DL4}@?@ z?Gpq$C**OmWliYA{5s<|EZ@QI2{-K#brFxfA~AIqq&-WSALHWQ8}%mvaNFasrtnE{ zg=sB4-RF!?)nf{>Wo~kNFgYefoFHBcSr*;iF9B!R=5Np|jv>Uf+mcarG-XGy*kP{z zISVyoPcl_9cOg-@613Qx16OGF#sH&2NTHDa_}vyidmxS~pMfY#AeQvu?AXpWNzi7A z*6&7a7!C9HRU+N{>WYTh0GXoBnXw{lQby^XShgDOw@e8TP}9Y*oFV4MVF#@Ds2A+A zXBEt3a@-IIl)TOcXx;0P;|ihR%Tq@DXeG5p-O{!T7Sg$s1 z8OA4iOx-!>6eK^x{jU-0SvByimK|nZik5zKIvvWVGE)4=x^&5Nx%Qgje!k3VoizaB zip#?$u(R8u{wUFC>tVR8oA%7fs?xEu(gYn>y6BB%vwPR9&RoZE%%RK! zl#Qnkl^+Y*Y4L{Xk(YX&aGj|zSpqO_;C3CTepA!L#4EXO|(eA`Fi+2EQ3!C zo^SpVP?{chQ3uaxu7y>w213e22cdA#l-M2kStPE%sq6vE4M*?3At!S7tIp(tQg(Ml zECjeJw8)*#LYYk_+Txv3rxsH9jJZBRrHp29yJ(^;_PEdn%#U1q`r89}38;XeF{ee& zsZEsUbJ{LtwOjU{vjL(Wvs2!Bx;#^Mzld&TjS@oo3kk=0P36MC-Ie6eHNN&{8b^s z0@jcbdejrrj!>r#Wu=3H1dgjeOI}NkhmE}K+UK&M>%7b!n&{0Zixk%^)6#@=V~IZN zxG>9kl&STQth}qScidfg58d2dF|v_U<@+V^eE@$4x;7oS3)MvWusA?9+%rN>aY#eA_6 zic@S(@e9$9tQM-&-7>X8~#n{5G}nuOu=dSyN+b~jA;_SExZ1H9Q1A}}Rz;XtXUIOP0~ zZzS|~T+%de-nGI$s?wxaJoe+99vmo%xm8o8SNEsAqAE)4LNvHc-1AX24C4k4u3vZmov^_VcxgGxapV(8)_K(^8= z2d{xCrmk(x&514Ly?e{Mf6}h3=oeP7+ZE{%B^c-kK8g0W{tYw3q%zty_Rd@1nbnyHMwabNp-sSyzpV4v>QsnKcQjF67%g~n&3t^1MesVxCzfJ5b=SOI#YfPP^^JGQw=9L1RCMFbrU{8O0LWOUdBK#j&{`tzXX zpe2_{+-8$a+o#%8MUlL4$yK`*--z&3{@Y?jP!m{g5nM+Ht=bD3o}Ok~sBQ_!^!->! z?NDVtyLXzmGYCEmjSCDK*q?Aq1;8fz9l9|z@~l{)R6GfKELc^(nV+TjjI^n0M+S0i z@YOu*Tk>|M6a0_n$(E;#^1Zgif<-CpYiMvyT+Y*9Z?&~IKSwsLa5Q#p_?FqK3lKIw zlp6Hk%lio6)yq>m-`QT2Nj-q!aX7~Hlm^Xh6FNbw z$#ri(Kk*GUHXORu@`aYQU@ zB~S-oIO^~abRPocemkm!W73dbb!j^_xgo_@#W#6p12>w^{){VfeX?U71Xyn9&E zHa1#*!4c;?r}jv7dMN`g#&R_S215)dccDOJr=uz%LIz@zia+LIFjRakROr?P zQ|Xw0Pa8o7&W=fw17`+SqepsQ-Os5v3ncD5|N?N(AHH&`>hLY+CLOluJ z_ErpaT49zK(UcdNmQ%iA-`jS`A_1c|$W86{d_T_T2V-HH3xUqpX0QJSH%i>1i>#vK z&y{;5)^pMB=u;&_DEWakQU>j&+opIrBf~2GUh{`kG{|Z&2Z}5dwG}>Y{W_uQHaR$_ zYH%}$c`CGC-FGCetRdQ@RZ2-%ucC_|R?mHzYEnqC%u9zRBH8wx7po`=EVPMpq+hL2 zTdjVhQn$)++17^cn;<3=bxJy0Z$U;i3AqJMPJO&SuieU&0eVX?eLEEI7Av@#PV_ZQ zsa>I>B5HE996O$z6HyJfhEt^aC><@AnzeN`xs@lv>^pPFtcodrcGyqPSB?#C`Piu0 zh5=hAW|OtT9hs*G?7}@*mG_f7ae@-Nz4{qvne66kco^uD$(JbCo2ttqUm-SMy@kx% z!eDt?5>w5)M!E#C!b#Iu9GqyhUs|QoYWHtR{4espRS-LUt=viY2iygF=-j3kcU#uF z{ka2=zsOuLR}s;&PbbrB`zty&NfZpV*Y;~i*W$EH0JOGS&FMS%VK@)f*%OOrcU3P9 zq4zjhMpx}oc`PWtP!o5Bdlp=(A***TZwVwuZbuB1Pibv5uiHvW{PsE-k5IfCgUz~l z0nMeZU0R>(ajoQ0G%Il)z0BgRR*bsdz5NcqJ<)niF6|PUO0i}<4)q>6wx4K(5>Y_I z4$WMkbCOQFs(krBnl zx85i0*7%Zm(&nKNP?AQ}d~6@?D9dO%@}ouN2paSR;zyUqJuw)1SRy=g%o;g(BD|Bh ztnKV(4fcBgDJ~M@%}n-6ow3xOhnC>C^d?PbS(9=TnO)k5p+W;pu2F4eiG7ts zJVL4M(NiZPQDy*9`H>-P0GWY#=UTnh8feiNF}hCs`8^ZDKy;XIL^9K4Ps&y^#DQSE z-?J z@YOQ9NQi>ZP>^ix5K`R07kWj?`R(B?E*OyR1$Vd;8p%2Y2zEYt4CJM~gVX%MO(E1B zzXhsHn~R1ifq9~dtzuH!*3&W;r`D(Sjrc)m#EI%`Car;CMWcU0c+0r?O!)HpjEvyP zb^;pO-Bn6e-+>dS^o{q&8yEH9v}vuXX`W;NPRlwJdX|59`z?~z{pFE!^u{3k{KkJ55^ zD;F0ldy9W*`d5YP|0(E6|K%}9|D^SIq>wO)4^cJ+yCa&xl*3}hpvcQ1eP_k;@>tz= zOZnw)#fxHc81jPcTM#)jgy|0?n0(jd3IPu-lJ&Tm`#F1)o$GTwYp@dlqy-qiHFCHS zKgikMUx|%x=_%B)>n_y^+HvD2=nP`}-G_0A7)I$yc4`tXS-On8qOkNp>Q^$|Ew%Jm zYx34*(*Z3SF}xw$CA?nG9O3ZH7l)@Dp4EyH>8eXDb}AFz)k*T53iA~gRu&e15u@|% z9Rw?69nQOeJhv^^unjd-VGFwbDzf9K{i(U{xxHyM@-aI+0qP{TU0G~w+Fs>taL#Ik z4+92(Z7n%+okd478;__0GkE`&(C`k8h@?UNnM=F%A~2|TKo)q9F<5`s)KwxJRw~k; z4giS~|8AIVG;rde6I^W6m9fliR^7YT*>&x7wv^?xu(5p45n{|2F>x%?9Jq+~Tqo9# zChbeGm@9!(s;uIKae_4h@`~yIj`Tqct+-M>d>~2PCiQ?UmFUioyy&~h_DTBQ--W|q zqA^UaJMTz4tEggQ*_cQ_LA7j7bLyz8#cpGggy;YBVk!%oSdufoh5-FYAQ)v=d$Bi`G$^~ zm!O;En#M9uCykPzLZ5SHa%?hDHP5P;T4HN0L6J*r9DAvC1WWPOrd{*obfr3yJ?Kl3 z^_6dnXRoi4<$Tr!=4mhHg6ig~BatHR zv%ZMJr-`8w_JyFEzUSQdp0HT>|9QQG?IXj$7Rbx4E)%HauDyY!tedHP ztIbq;D)ckd-eirAHOG7icBH23*ApHA@nG*Jdh}~G?L5C^Xw^+nLWG+>hRi&(fnpY5 z?^hj4si6I{m1u^%i_yk$tco}28X8|}g5*tAEZYF37$f(+xT%XvO^`i^Ig}%cydrwF zlpL!xdO->&@q|8MiJrAxt;z2CP*a+EvV`_2& z<1=p{zjhmmYVkpx#RV=#zuy&7^2Trn=H$nT{OBVF*0z|QH!NxBF%gbqT!BEx zKB!SsSUwSo1Zr?kMM%N)@hG=&m`vRQ6QK6=oIvnUI+|C)dGKM@jNwqG2Xi8;YCUHYRh? zbl@DN-za)+0F9kw>Yv=ioL)01uFp7@AVEB0AH-nmB%j$RC_totFy4BKd;OPCMUMBb zu3oUUK`|{AvkM+@KPZD4Tn$(VlQi&aWV*Uf@DO|FQjLOoVw&C@z~Um*h%Ka-C=n4H z@(Lf&MDJXNS{3Hs@J)11(zo9tGp>wS^b9{Q1WN=Ktn>ZieRZS?k`gb7P4n?cl^7^* zG5-oARAG#i<*z`J0ski%;QCLD-T$AbOHq<{KxIb4=QJRn@MGj=ns0WhZX+uX z=oTjz`o-VviMt1mB0W1vA*7oq1ENz{<*-EU)U;r*ODfV!G-?hdnzhM@rRZ=|qaFTN zX*t~$gc-)M7GS{#34R-n`B)eAPfebN46~61R?j^(Pg3TXR1PyQrO7Mf@xf<3VL0`4 zh(i?-SktJu8Oj?KIy4p@%5ZH;P&p5LB8 z^}7P)9h}vUP+1Hd3nNzNcbR`%1>dSZbWhiXe-CcB+s9e)_w<{bypZ(@cQT`P@ch=d zSOPhExgI31MVFPsClEXe>$~qYQ+d}7(!BE*9y%AjQ47BMDt=#>`1ie)|ES{pFFdHa zI)CK`f3x>)DtZnm!f5=e@g;3iK^jf!RU6hpjYu^V#q0uWLuJ-6={Ua3gDi9#*P7;- z`rm*5)n{2QE{UZ01PVy@_9(amogzzOwYcVgp2>LsJ(}hKbX_!ayZ7=U{!p{BHussVj(W z2z3$zu7h$KK<%}P0YBJ+)0unV*xD&6GusXqs=M=Cl&fP@Ttzfq?>H9TW#qDId+C7? zhD;;HOxDJR4dc_xI7-b6N6nZ@bUWueDk<_9Rju2I*o(i)M0&~%C^ zc)a<25M<^NrsjAccydV2HJu_-1W>b;xrB~Mi@c7FrW-94$-GnKXvF7( zA68!d!gkIo8(URS{(u{zRtrF}B$9@*)KH9POqOW-B$za4Sg-A&PM*on$>$o#L7pH~ z&YW8oJX3T!!@2r4Rr6ac0ZDbtB1b5yc$5}7oZSDvGF0FWTpZ#r7@GfM^MmC-p{9Qj z_JmmlTxO(^(NHqBc$ECU$jQp^;)%xnyr$qvNTd`R@j$8JppDCGQAHQ7?fja9McCUZ^;``VW$1+G#=<;K{_OfH- z_$fp~S3K`;jPNNZnkB@=DFQy3{6+Bq9nOf3~dr4q8zD_t{P4-^%<4kj!U z0aj`=#@G*w?!4fpM? z8Pwb15(Ka*TtDN-2aWK>*hh{R_C}*e*vSTkHdM(ETM!JrJ=1h?(_WL}2p#QXjrKZ_ z0k_yu^;~)#*r>sQP7d_4VBRvWJCzw#TxA{*hktwQI3ST{8{>3$KHJIgMGK6I!d}Q zinmfq&RLRxX8P)_@@vVr0gPu7*)uU<%xS{|Eg;*w1}2=C&?7B zSX?OLt-gZO+<4@tLeF+K0~*|xwMD__KxWgGfsUpj)KyeCM3J-f*uxe|xk;Dlqq%1< zL(PaY@U(>Z#k!C!B45JlmE^~wHSH;r1c^kWTG9_VT~1LN6$a6Yg@kNF?&b0hs+5Dw=0j zR(wcEYmdfgojx+Hzu89*C}4$I7^?^vYKhF(`>=MC)VeeFR}}?j#XeLnp8OhW9%9ND zt6utD8DHnQj5@YJv+$USdN{8apQir2)Z{8_s!BABmG2O#pz5lSh|gf#CI8X4I|U4g zhQwk=VEV+j+-KNxuIk96Bi%^(Sf9}A7o$zHJ5mV~)qP))QQY&^>9}z9z9)PWpw>8T z7#NWNEtnUoUl{DP5(lmy<3;tpLJ3hG|;CGB`3**uH0tf9>;7w;Aq9SRVg1FDpI5y~rY#B|eCNpAXD z9692@_%$t2^nu&4lU~(~_iVf|Cs|mXs-xKlY$-~FZB$!oDK#)JgHZCG)ySDURM=@(i zCpd{Er89|l&)(&5>L6LuWY3yC6)`jPz(Po8pY=AYIBnx3y2Qx6*sT42mpR$zwx!!< zHHCc~tbF^-bje?bo#~Q59Dmw_-VcliCn^FfI*EV)U1NkNA`6Cm=^%j`%M?1Zxa=1U zn#DPNc32&XHHfUfmPx*J+3_GA&g-_pd#wO=Q^5bdhzmm)>s@yO0q|>ROV(hkhJWf@ zqWjI#+9Wx%C+!kp&kxX|XPS5m9CBC&3r>}SwdFd#YF_W78A*CN6mFC)qzOjM);Z&v z#MjdXXMw63v*tbvY+$tDmuHNFunOlRM#qe|eV&|$98!xy{n)-=N?lrkr0_}U^sz|x zs0y);(2Dooa;(9zHzRi=I{GSVcv!6jl%ck@)>JODfR? z%aI)0HvbhzY9K7eYsntq#JvWzj$WCuoyGoPY7;LSPfZlFiWU)X?(-p}s4FXQcpIp00;%Jv;k0t@2vBu4i;rh-?{z}cHTLL9Rz zT8r(1Ws*H~EyH+adP$cGv|7HkeS9p6eOEI*`idH3twkEJ*72|ey4JgISglGV0Vo@qe#)f-=|g%l$S&Onwl@mmdn|sjXXYaQ4MlfzjiK1* zY&hWQyc9?G2}2s1fYnQ}LXpq{!&Kr97d?=a?_xXAU0SXrZE?T+=9os2*v9%Csph*M zW{}m4+PIRmHEI;<=c5$PMrfg#MTs);4Tb_0**o}*cimSWRcxo(;G&&NV+-?W7v*%4ACG#t5J zQP=$g-(mN*;B6s)d9JNkF0#Zz_WA>J;{=2a!IJsiqCV!YLjJ(wUJ`3b$>qcZ!HjDT z2xm;fMSbtJ|3o~tc!jJ+U8a)vX@NcxU8y#u!Puq%R~{sps0msRFO2!GM4}786S7* zxgNmf{q@|Sdnf6_he>gEGX7Hn)uih5nL&&t4`O{?V;;bdl1U~9RAnjNmt~1UPC3mh zrR8ZtHzz1(yOYSK$OjKf;InJ+7mH$WfqI^OG3dhA+S!YmIgRv>2H78?<6A=~%E{ug^P+^b*+f=j32&Nv&Ypq?DcH&Busg^AUDE|p; z8(tQxZs1+0gUX<5~Ah zT0cGckI5%nM~d`uaMJ$o%2bt^##I0UdaQ2>-bpsP4P1Vk8r7EOSr+a!D*Z4shiKFL z35Lvs^i;#;G{%ksUUo8(Nj2DY?u5->J8kqS_#{B`HqS(UkzR|K5&6XI_#FH4?$ znMXeTb$nmr1`|{n*#5H1T%vtU4-H)vrtAchme!ZG#@c+Hrf4uxx$;VU(Dr~N-ich4 zMKpdwot^bPY#kBILFgi?i3W_kV%vn2J+%R5x}TL8I?B~o#VXlmr?i=y`yJi-><;X* zPCDrsU51x;mkr+t18lPs=6)r^gEh2$saaA!qv_< zKQP13J}ptHaUjT_(*x+P}wfV-}57aU3rp#3AB&~e3%y}0ju#22u5@mUIT!GA{* zd%-e2DTmr#$(P6^$&N0oCgR)F9IPR~!Q!x6YI*7dx6LR6n8tj(#1~!0rofeMtT#g* zW%-p@V09>&o>iz0j66K^soJWg(o9#T(8Xx-P3?;J|t~nIDSGPq(?-B zOoNnc5HZhsW(m6!J+yj~kjmjV6GKvhO>%^v5`O2I@4B$Z!~DgelYWdC4P>YfmI$TR zq`atDEhIt5ua)PS;Yz1`FX@3Na6j^uBx_rNKTmgboWGwE6O5;iQiN6Q8>ZX%ApVJS zTEf6oj=@?7klS(JaijG|(gO@dTgxB3#H)4&?+@VWkTc)dl;qK|uv;WRI*cG2`6PiF z4+svy+Bfn&Fs57Jz6i!C(w$w@VWPAbRGak~oN>3vUg|Mmk0NpfURt0*DSJ_e*Gi8I zqshW4F}L&aS8x~4*#{4vOc`gKW99cx*L^69fgPj#?++q9LidItd}<@&#E{ZGz7g|c zFX$uKJ;Qv^NpN*e&EL;l@1br8j8oxO3e`g<911L_jr~Xb0)t$x$A~dFay9(}gt4&L zyb=1<`|)_7(!^xJ14xLBGKXO3`R^_;F01 zG70TiF<5(=pRsJYj!^XjLl_vFJOQPhN#Pkr#G0-m#xG>q)GAHjE4WFhe7Zi83;gte zdDv6+)qrgh3F0}$gPmtb9-Ff1m|xDD$6jX)Dcd5Ms-(@nKM_3)2+hfh6@Cs@-=%Z_ zIinf|ck6rN{EOadGmJ-rzvxZnAL)(mf108HL2v&m)%=a*?3CnX2ZfOQY?ha_11m@UzRqlkhrVbQ@0M(tSSTerx}IH@Dn2={w$iGqU#`v}PuV7I&A9JYNP%sqMn z1bTq*Ok{V>SlVH8H*4X-lO?VzaDQzAaLvc1tTL+To)YOuj^V8mQ?)K-FT(s_!ds-O zeb$rKRR-~g^+_aiGtH6kbJ)!K^ie;ipJ8e;>iy2}73i(1RY-~!(tk2zPj;pwB4k1a zVa~7lF^EE`UH=#eb**88zBH%!WkO0S?_Zu0KpRtXN+XMsAwfT56IZI}&cs+R5N~p3 zlQH7o$(zsQQBPIRmD)i>TfdcgCSKbVVD;VCmO3l1VNbV&rWc9o>Pk>ex!)Nap%NtP z&kKIFMm@k9-HeXj2$((SmG+a-dXvl7q(7n=8)cELHf!@Le+X)=++(}pKC*dcns?>G zVa*fV{2FDIJNaK_jq)WE9MvxiTm6sI%YUn|S=oP0Z`vE#GMZa`4V5byxmv0@8@Zb~ zyBOJuTAG>Im^uIL@!ZrWJy6xL{%n;pEwY87Y^xYSfmmgRcgcEDfz4TJ#{;n|g>8(> zv$(RLnp4oD1Mj>H@ar|0RCy}E{GwvuKOf1FS}O&z-Q)MmCVEK{p~b2xFj@lTn}#s4xg7h+r;n$TZDlT2AXAv z7R^$J?R|*xL^>7HI}e>7{HszA#Y_e8=~8*3zy_J$ejuhByeI0I!w-&%MW7Q-FGMKU z8qPm&IdU3w#^#`d%Vcn&q^w;EEr|w2F@ax^`R;a@p>l`U-T%~f&^`#zG}qdSV)A<0 z^*U=#=#o&gd{o+*s#j$xf+2y^t1Wj9_h}(DNi^aK#jI}z)v1rk-H)gocbgc`wB*?$ zfg~22r!^VEN+n>U8|3{Ebe#!9k|dF8lV*9c&9H~&g|$Ymc-2O^j9w$Q^I)ldd}5zv zQkBFDS2TxDn`p}-{-`br?tUCgyfr0Wbf3QeATbp=9sN|e90U^eVOu0~VT$1A5))@C zPcwzUn7bP^Gd~hLA@8EwiklMmlc^(;uPE%tLecC-iZ$_~jNJnZYn1A%r}=VE(-LG; znh6Q+b;zKz_N7)0SH7t~u#)e>Pr194w7xp;V&CpmJw5j6zBO%yB zjVf*iveYaWlrE~+p8YYym=-QmTd_F!`)ATishn6(oD}hTE2AqnVPF_os`ca^ET@@Z zoo~4YJASOBn<;8#(#3G>n1E)&@JA^3LV7mK^kaJ$((~ASWup3G(%#8O%xFX8XSiN~ zUF0&gDyT`FzIjtA`<-+9RXEKbwu%RtcrG!#-aoN0aj)i z(G|=#b_!z{o1}cIyw#n=j~Ac|NnR@<-CW$c%JFBFTi5JW0BX#4k2o2w{L0EglSN7E zFUcmFVF&U6NBA7!t`Lut>faDk>pW>Lz9BSzsqWvnI<+L#wg=zw+aeL6=70S773#Rq zG@fVM9=1ZibB`>L>hKz>rHG}`pX;dZD>I!_x~u>jsx3;0d$`Q%t7d<8^lkl8w0WZ3 z(HGiok6h^#G2EzIH}G*;!U8FW>@|C+wE+z{@e{wwWEkzUEiT0aDJo2JwZR{zcX$Bz ze2pzE&vKCc6@vE*GIv1LZ=qSg~HR)Jf|ljt#^m2hZF4z|32*7{hd|u`C7{C zjG>}`{SC3Dnc~5%D4yBa!V@}xSBtQ$ZWY^qs3)9jTuIXYMgPF5E0*&A0B(=JEntcVgC%ZO4UKHyuzuSblKNHWJ}OzVpeS z?8|{P8FtkJ=~%YMf1h*@o-YsZkLVQU!43cY~nWEmBt#&Ar%7WClZK8 zSe-!M)B8((tj^wSIm3?e5oe&mQs6BAE#Y7K*^boU^Z#aITL%-H zul5Gx*FKM}n~RnE*Ko3}nXrk8nTw0Ok-d?{|KMda<$n9cFHzkfb4wa&Dp0x>XjayP zg-KZ^Ayey*gb`NecHls@$a-2|Z!Xe^@P`uYYo`Q*jKzDQGPFf^GDQ5rd(-X3n)&f|bD>?`-DktKL<0hWK!cPS>L^@|VH6## zG*0#NtGfzpZpt+e{yL@K$|Lg*JfO%I+hp&kR;NxOJ+y2H49xZA7=^RKObPZi6 zL&R70!l_{PTFcxI#h+WsO^Y<`hE*z1vg9n7nG-6n0xBU8F8yDd}=?${Kl$qim3(S98@^W*vvSs{l zU}!oUIXap-i#nT`er(?avm4Q4-snuM&-cwu#-M{K8n;l1gP$ z3sw?`ls1z%eb%&mNBvLuEci8}-Q`|kUw6;F0-pHb?+A)+BLSn7_@my}6u%J=Ub~(* zU1n~wcfO|73IBZF;|Bhy$0FeO^>lmmZz?ZuZC8$p6<>B{Lsp-*mS05IVU00ergKWv z(LIsLS=?(>QLLQQ?bdTpyO?iiEL`;>(XJw^lA*7FCd|$g@c3VRy#tUf-Lfs*_HNs@ zZQC|>+qT`k+qP}nwz1o`ZNC1_y*J{2=fCentcZ%LwW?M`<;L&dcdwa@4GT@LCkltq=Xfy+OasOLT!lXrqy` zEW9YuDcfQtJ$oJ|Ln|b|q*_a|YPgCbBBfQ|5;-1(P3R`sK~3T`TtVV6yrtDbioJKI zPDV1BAaj#O~V^ll>$# zNC?nv_r5RiH^A2t<)qzcvns9Qd$_UU$`jN;KUSNqMCQiCFCi3A$*D#(v=FXCqz$SB zyC8vjHyJhMy$5kCi}FBy0NdSCJa6{q(|*9I^zwX1NHX*dHOIDB8bsI3_{(*-kkQV@ng|lWd*nWx!(xQ1stGMcRDjH=YUQvY2^uCZuO%-0Jw5az*F1nW_|h zR~z5DT4j&Z7527|#z9b}pmRW}p^|OrU(TWox^&Kn>YUn%%JlZJ^16vzy|O|GnZsf3 zSXEMjOhuYZlh*ikE0&zHt5va@6&GI{1&D+NPop@Tss&f!V4;}nqX@iOvdonoDa}J_ zE-u%qrrUpYVYSGU5NeXJr?#B#3dkObD8uk*U|u*zS;T2YgAk;_kdF0s4A6A*YGO4)#dKwYLQi+*i=C3N85d93 zAe#Lng7EX?@}-FPvIdp0y!`J@^1tg|IHwZ=C-i6LW7u!d>#==7<(?=6?caFCo;)AM zwwV6XHIU7}%D3 z75#&7SiVq=f6k4N*gy{?o~K9`+fsId8Co*62ksPHLm=SB>G)@44I(Fbs1stfE==|e z5WM)k7Hs~OwT#*$%<~0|BEb_6HV0F0=kYy;P zdAZbN(@{*9FL}4bSi-&#J^2;N`G{J?KFD@i^8BEXQq3$Q#~shvw_cx5r%ZlgHz2&Y z*cU<9UD1(G6qg=Yx{LRix``xh^Yi7@j|r7hm00t{(0ei78ZQbt`JV={$XlXvX91YH zxbI<;-YQG@9xrY>Ar~yWklR>hQ-X6TUxD-S!;~b9lu;Tu@f59S=euifnkTO2C*G;S z@TJZ5{$VG<^ThBbq_74=9q9r7DxC6VBngr@olJ}~W87-NEagn(;M*)7Oj2!(TG+}U zsLu!TV4B7DH{}gtanAHawLkpH5_$jk$0~;0`rM1Hjkl;4D-KsjXTl<*z|E`_8Nlb6 zroi&vNu(socja8wZ}9J>;D}esqgs4BR?_u7ZyELz2k%GQjtG%Vx+yeS&QI*AK1Q~e z;1-8)WjT?WqB>et(n%42u5UPI+!F^B7Hx#oW{i;??}{9#vpvk}lwvHPB$=-+pnIAL zGBd3sTO%TRGFw?`Nh>DzU#VeO7C?`w!-QT4ZgBE!WsS1clJ&i=m$ zHn^;?BNx^_wESMCsSKfxi542WFvUJUh%GpT-JP-b+D|wh`H$h4?*AT6uKyK)=>%&^oOXr5Al10+ld z9x<66pEk?hlV|$s!otJ~_Kz3DcB~XFzWq<@HMwvNFc2}VQuS$6g{U$+nN4G0`E zua0)-H1D8k;mm6E{(!pNomCz*qxv$pI3NvG>(+Q4AcJvK#K8 zb9SOKS@GC!pN|JW#<}*37GFj>D1wi~_)k#-N5izNy0%(q7hMm?oL_Ju8jMFGA9bKb zv$!gbC9lC0>Unx?+*3GF(6ZZH<(4j|5-Om02Y2z2IG_&xn+2Z`6;N1An(~^lQwwUQ zOiKj)?fuj7EGlb8nv@wDs4us&o=Bt%l*TAhB{h=R+Pddpm83-ms{V0T&ofYt=D7dS=Kr=V{~wzR|1=j_+3Fh+3mcp0J6k#Z&$+yVt*OJ$s$BYK zRx!5u|IH#%N;9@dV#r@$o(;Dy3GBon{2-)SK+R!>`0yL(nq~lFeelQy_)_BZt2i}m z8rSXb0|MpaMQpG<_IaUCD@=+=`KtLmC}H1)-vV;8Y!fw&`K2B6oou$QOj%XL`Ye$dX*5~GV? zjoCc8{4m*B_lFn=K@#mp@(*Vga>;sjA3Ds|(a_aGGbuFi)9-z>)&hY^h=PM>jvvAt z$Q7Zfbr%lPeu2OFHW3uNyavs`ezAXnB`OuCGx+U1e%!gwF?S3T3XLaG+BzOfiLB-f zLsTI!R2nT{#3)Z+EHpqiKXE$CK-~2S!*Tvgi)l{*o7SZiuHQf&N=jK$gt6|+nF)`Gm z!Txq?dNfctW^}=z-436nDud8w974=Iuf~cqED93ykXqf1w8FZK9fiO>iyHhGH6`Xa zy99CYP)x3@)FSqPdVt-Br1$H%x6;EwpuBzZ?#_D^RUI0KPMzf^_Q2rPhK)0jFB8Xm zlV*;2seylEHqM|s4!E5>k-zx$17R0R2*LcwM(ea^%K>Rf92id$mc6SChy+Lhh?+zh zvO6({dx7GOFjsuW1#TIks9C3Y1NS^K;IL#Bmt5WRAnNcc>QhlO{Vj2vmon)s*asQd z33&IEDekAAXHibwHHW4Kjin6FB;UgbL))#+*%fRgjq!Uy)J$xt^A4P* z=wpGU$DPMXW)DL%DW!nu39E+G5tKB@YM$r#?rOf~PwEaIWOZ?-rZteokPGZsqWYS4;B z|0LjjIbp)2Q9#;HApIi0rAAv&MKYgXU3KhsoOYe|YT)zr{({<}EXL67@nFgE$g8n) zlwsHK7H3m?1l)9j7MVEeKIFU&$Urel=||l_I+%2%vpEWGJ4%Ae=4~9emV-GN((dey zu%{X&7)-JZ@$2L0Yqtni7;-H%fWs%8= z=kT2S6oOA<-_q!hTShh=6tYB`my{cf^+Lx>yzS~3hAy^=8Fn4^M9*a;F$7-pPb`5WTTi>BH<(hQt<2d>L}bEO@qeR~R5CV6M#}U~hOs$t?sI z7o&N-naKA!$TJ z>&^XTo(>zGjv|b*XTI$ut5?7&&KtRH*Xif1`>gBEp7*Joo(B{{&6%EYr?;2euFLC6 zyxINGDCvA&Z9Ke6+p?I9Q!BMcUI`b0h}(?yqWH@VsM zQOR!?^5j*fLK3_B=$34i3+r{u7IgD)M~W2q7y3L-307k;BupXtBuqlRxD3=-rhwa9 z?bS^@iS*Hnd^;p2cOp}nC~VDSN?;3$3z!yI^$)`1W?UAhtCjjqn>M&ph0;8EaiL{z zu|C4KQm1Ko&6~iXk*x&^ph_a+*qDsevtmcT;T0k>1Tvc@2_|YU#phijBjGm~(FAS> zlUlF>J!lV+cX^mbgNt|q+%c)}o#I2L8tL)BII4PpHABevx1oqq4Fk=enLf)lPJppehzt;iO9UQ2qK{ycJZ}25$Em8#QCj@IGeY)Ih;t1C_j5#Indn9> z?q%Mr*&t<`FGYDnXUw!Q9F(&(vc=j2NyA|}`{O%(aBk4&ic|F*CyG^zcJTh7Jbkku znj-MdZ0aPz3?=kXncCW=-<;dP;J9T1y-C;{aJj^)J(P2N6H-0wO?ZvS=U!GHKVCK< z=aWv?u%5>H&8MwXa49`eLmGW<%;nt}*#2=)K*`axE(dLvH|fGa6F34#8tRY?cr_y0 ze3Ys0rp;JgADiP65s|!r+v;Bhhv}`Vm{n>M24Hc%zOJ&UhG2A;(vSJbsM4>fU{u2_ z-6VIhEcV`qxROML_k8tmxBr)-{ z0Nki4Ka!>@`U^UZ)eJ*+dVEKh%hU52puWKbEG44AD>zWsBPQobQCa)OTlz41wS`U5 zA(_e!#MIkQ_D?<^L@2G~TpSiQGc{2i*D?M}9=ed6<%52)rPN_&_Zz}kJyQ*xrss+n z+*}R)Uzw_8MN}8>Nin$jkrHrz;R3n*HT*JD&M9fIRS?wRHq#A#i(f4q5+z;_5Ij)k z55fi>(u^$A=GCiS!o_k6hWVWf;@9>(C^LB-^lw%JYn+7v`}UC04jw=#dbI?>PxGb< z^hYM;a|^$Xv8HwRyEFBlC0EGDeVFD zsI=F15ChE=aHP6tL~Ao9#WHh`H@ZcicgWiJi5Wg12JkaFg6%fLuw^#2^+FGSBYJC) zcLQaBfXhJJeIf<*h>U>kVP9*cRCfKc<$@qO~wd*)<>-)SK6P zJ@I^4#us1Hf$yt#&=?VaIkhDY^^W;!&OFd#L5S3wEK(42b#OVRSI3Yn=DLC>djb3m zOx*FMX7ymI4;B56>=L7Cv?Opmx_j#kUAIX{b-S2c8Z$v=gOMvo?-ij^Qg7+-IsiMdRFM)v7G{O9O zb{zD!lmDA*H)}70ZFQ4xTkLM$F*jknM@CK!9fA;1rEyA1T;kT|rRhl7MQ@3Z8K3<$ zthbXo^c6w1sy3usEhrD|+wtJ{DqW>!SzzMAYG&n5P_48!FI7^!mt^UsJ=Ii%VFz|f zC`{_0n8zVxPB%8P&U9wpG3=awF3lq(pY)ZY+X0iPX>u?nXvOVKqHlZ!kPr!p?==9sB_~DS`Wz) z-C{l?ZU7>v`xhem*b=STWhZXwe7a@WUN>CeYu(sj2^yMe+X__p(O0XKfx z%AXEQxVFsfTzy)ozm#eCQhr*;4iF$jVCn@40VgXeH%1E z29UQ3y$aVZ3TOp-E~*g`Gz^slv`Lf|RO$MFBa@P)tKRuI=cc?XxIqzmXgmw~OWv_3 z79M~sk*g{jtNxD4ShkFGO@d3`N{)-(L`+B$P3o{T)|L%BE`c71nj=koezdtBY4~a%t^5r3-m!3Kj%V`9dB?v%w?BxOI$&~!jUNWa z@o8Q~I6n%f3*aDLLYK<|4FU2X@*``7jnlDRq5+VebLwb4vJVL_1XDYFTUc;$dW3relP0}p?81NZ&{!uRJU{&9)O%uEL4Mkts~ z&T=;)Kjl_c^Tc3YX*8y9Lb`*cpyU^wFHkn{Z--k1SA~|n0bO2_YwyEVv91paW(>>D z5A?fn$`0!!94mEWTUFmE5+yocu&wZDj;aE3+jOFJ95*T%`pKWaqKNiaixt!T^#`@p zHlA$6Fj^5&7!Hb19 zHyE9zQWe<12XmH)8IDIOtwPeM zHRd&LKn-qMRQRtyy5LYzR9#*8JDBD2K-E^^INa=#S{XA+rW5XKtg>7Nn^Of&Vhir! z+P>KycTUF|e~Hw_vAX%ap<+u9o9)jcAVaw~|4zkmS zZa8>nl~i|D8zjQ^%<{;ZR6cbVD>%?nlBzUD&(9h}VOpBkVW!AuVW!MGuz;OfTWE_| z{yi!0mE#74$DH%4$iv357s-5PS(g3aXJUS?=I-+Jz4Y{Czu2{VMepL1!wV0l8b0k) zSH~&|HJ~YYm{WKY&gKO*WNzB=l|JE3C?T`VIh$Fi$wHFx68QWYRy%ziF%z4Zc<{>B zjkGSyv*i{+F*O@tKQ!EDM%7xw!z{Yx)~Woo$kr{Z7+t7ve;X$MoE{R-LVe22TZY;% zOIFYRqSw}4;Mcno^z?O*G8Q`&wbgNV%>E*DX{fnqK*lP#K0dvcU3endLW%GugLOH< z>Y{oG#ECe$UPvO#$t@?@GA5JFE*6oY@?+$jRxnx(BiZ8q{AuRkwymR+;{*D6-bh*) z-5@PC8lo`?K**Ec9*n$U>OJRjK0H$J@vnMoQZa4ti zMegzJ2oft=1Y+aEG$4JE9{t_I{tH*SwKVixk$IyL|hvQq*qu&_4C6X zp>36)v+qAXl|OfXL8koN-RrhNjjA36)N;pjmTkOO>jg}c>35j<2gH)fb7QYv#8VV2-AXJ1-O{Vpi$uIz3lMp3dl`?Wwpp>|6_$}|ROmbQ- z+O3VID2pdMNR%dc(_#%+-P-%bNIb5Irk&d>rOY(_mq8%P;dkWuH0mR4vhl=r?rV5g z%=n2Yz2%@f5#I6!(KxF>D%1-3IyJU|VW-!(l$}cWBQtobb>#9D+>HlD>@kp+qgiCj zU_Y+2nP+9m^gw~vIRygs?R~aXBZ*Vk8cFZj_&b8(pTaY{Y}cTT z*fRuKeL3=89rk16#2TNQ%KL}Ryx)%5M0MHy=A(uL9M*f_;^wBL-FO~J+@|(7I)GQF zGxu8y$fzRDE)xoI0MCR3S^FKd3Mzir$&35HZu)9V$~5*Kk^r{%vt!7ISD#%fswRS1 z7x8ugQ&u(usOPXbN5Z5URhEFc|NLc;g}f4JzVjlUxu&$T#yH-Omy4s=$~b=B<)v}= z;R7RHY}oe#TExRVjM2_)jF*Q3%G{)3ZZqgSTa^}wnjk_InITrx)tW> zN_A5pLZ9CogVv`5^1_9Jm_n4I&Od-1kC6YSPp-Oxyt0!D zIplg&zC_?4NKvoQui_?BUY3EYOP5n0W0#hYf21a%4Fg1xeEs;w-CE2d_X6pd9A`2e zuiIRY)}Lqe0J(eXdpq{`UG}5w@h=I2qwDlnybY&n3-F)3(mWK*z~Y1=sqQ352UCF4 zQlI=T^y5Lp>gG~>1T94`()}Z4=w<|*zIWTL=+#(!PT$k6nPOoI-RVk#s?iWB=$tTc z;v`#9_oLoCy7W1j8Mn^hfr?}kDKcERb3jxH4>hafqve(?N%m6{o48;*Aj`VQb5)Ul zHK-31_Fm*+OH8EXSzh8{$7fljqN=ahTv<75(Rp-SR$Zz#EMGFOcXfT5%J^HHx8x@r zP2)nIWHes~>%OVy%4>O3(0{X?N*ukyQv5>kKb>M|32-D&p%1(V8j7s?3w|Lp63nOV z937ts^a~AioVI92W$?353}~XMK~{A}5JkKH5b=n9Ciq@IDBAB;Z!IUAV+ciiDvH*j zMD^3Dk+a${QM5$azio{#f^OHOx>LnJ+5kbRm4^N`5ii4(4>XD|b?3s1jrWv1Z}MFy zT9v+!?Ds9SiLUpcRnr?JG+C=^SKkC=BwXt~F8Tyir)=)czcAl$Z)2R5pR!H;e=OVl z8*}D=$~ONscK(|=^G~^sSitaqkw<2U?vov$hY7)fa=I8~62|7IuK10w(qZq9BnSjK zt$S9yI^QU{77(-&cteiu27n8-8*tNC&-dMPS#upD2hi$Q=J$O0#Os?xwTN{WtSzZC zp0+5nsTrDO-C3RykP7Y)6z8U{uiQ@973Pg|STBrbPO4R4VU>jA3ZJD%OK)mD`u%Bq zjUA|-$B9L(11X}nY*naJ%@8ESe`WsFWU8vR= z2;2}9@)$?_zbc_riw26%Kg!e8Kd<=z-OEDxpIr0*^LqcyFQ+uzy_6rD_)MF*+Au)L zK+sV!gc8RX!}1A93BeHY86igj>{s@tCS@2Inb@Wg|3Ir$G(TxPHZ`*>y-_zsskEEv zlcqu`YL%;Yn6XuOyEIg6vQ;HLymz>grb&Gw(*q#A5?6USh=@|D2=%(`I*cmsk7f^9^}}P? z?OW5EW$5ivagZURMyiQ!)dSTd0?Cq6Pu{r&OKRfiuu+&nj(M|bhppFk4ze_}sSz1;);PvKNiaE=q^G|5w^Vy2SN zBs0Xts91C^d0dq<=JmXesd8D;1K5UvF9?WTYl6d%lJqXxN`Pj}5LxPgSRE$%)Se9Nn;^;MLmXCiH$)23AiNRlj3 zB5S`@U11=y{xj(rqgS3zSUD^dhUILAwb|IZt>UN#gv=Rm63ig{MK*6HQPQQC{?1ODO*flB7}Q(AO3hFI}(g&O+0tS_v* zssss=fjAF6c7M%h{bJFcbm>-<=R>Xa4X{qGb3|a97zk+R8pO+p(k2^QM<;%(sz0y~ zRB?%#!Lct8vXEtAzqvF2#xo$NsieLB9TCSs^E_?X{@2BD7<@uv#vvJzQhJD^v3!dT zl|$vIA|g+p5nMz|Au5{UAyp|$2kfI)S~hhN0%yOnr(#(o-&bKg$Y+VeF{*sx3Du~N znZWwrE{QHx{GA?2J*uLTQ+AKA)Nbt+N2AXvftlF`pev3SOJ$4`MSDf=HiGkA5i0UO zd~$T7PLbVXMt2^U57wmD5}@X1U>&QO#B&jZ0J18_+exP+Z@5Me9xd0Jbq&L^e7(>X zNNZ(5fx4(0i?cEE=!j+2!b@EfJXIo&j};GwfS*019h#N=Yt|*|0J4`!D5 zN_q7;3^d-)FNmK&7&H^rwGK+yh}q{Hpt?|PFC?Fm#mlG5xknmlrQ>IgB05c3KF~=a zh6K*nAvP~CiOXlXY$wlxYQ8_)WN;>NeiQS5Mb-&Nuox?GER-8$-`li(QhmzUy}Keq zW@+_RPM`C|bx|r{2{VLpv4kQKehI>QOprT%3zknCxVb_F`5u!3W#trOn>06Z6D*XH z=M)M2!jWK4RGLfuttE%E2P@F6hVZljI&jmjn43^ zPJ~{D)br75_H1XB8(ej-Emk3-$#Qk8x9>hEB<9vjxJQ=EG&)&*v=3TD&pvVnxeR-) z?Lb+YlOky39f%jYERz8;%h7@zQH?O%8>!r^nUZ(>IPqq+lbCHA8Ax24#IZ@dwzGe_ zNr{+ocSoD-L2*Xdg%@t^OiJbgq#@1W&4(>T_SLJKpM5HrJSQaRRfbG&uyI9+T~>My zyWR{C12~~%bhg$$vJk%xRx<*^v~v)B^3%hV33i~-tUvA5Sfb|5i=rmc9n>)2!GqKa z^P&<_F>DtK$|77CJ5xuKX-Q%!OtxP3n%EsDQrn82M%6F*?l55XtzSVcMPQG0ZuQjl zmq*Ic&aackwk$S6PqbQ!TT;VJDSX~x&h0RoXfrD8&a{@qUZfVn6$ilU9V(GVzCpk^ zP$Zf;Ui%dnVGK2;ueF6kZ zFhW{mY7j^Tftei%owFtP`AO&4M?tOT( z;Htw$hS6rDA9#f<0l{2DA~U)NOfScqg!^m^q#5Caibizsnh)JfGIIAiSiC=S%J|_X-AWeS|ich7A5v3!>zaS0qG@+}6 zF+61ADkXR}zFbZ1mX?PdOp=@C9DI^|;2Tz^0qedK3>_4z?WYMY85qL(rt=Zq14q`G zmX)L~hGa0K_F1zeK5O`YjYkt&x-#C=rX%}-v%xC}Z95zssU#Mk{YR8Je z@U4Wha=tl!xo6aPg=VsfWT-Uw*s!bATd!Jrcam6JES#?b>09?3j3HtW9zjdZo{@vm z;Qsw!K~TU*LK!uvRJbS;OkNH2Wt%Y^x3I4&v!zodO!!r6#`%hm7yl~tBXG|sE%(t= zztYj^vC$ivB^+7S$l7s@do8-L_omu&g;hi4Q7^#p%DB);DAqKLC_yf{M--fbVCW4Q zpLSAJpyR=Jw|FpZ7!OY9&`o&H;FE5C-006%H7z?V^+c?EUl19l4m+%pxM%W-d$e~- zt(|&Ex@CFK^ihfbnmM|@OUuO+x=YOaa6Up`MZSv=z+ zj&v;Xfs>|(JoZyyf*n#2H&qEvkEBqz1th01TIY?cy1siJEZd%upf04|88q_e^UcqIJI$qO^tX{0Q=;ytn*d0;d>W zpbMg2hvsXQ_P18QOkwPq?4dM+V|(uRBPZ<<$bpw08v0vS$9$VUpbm=Fv(IMqMe~ij zM>0rOq>iZMoC}d%y?jB;97(AMLyv&6Zzi(5LIvB?<#Ywf0)mZ_~Rdangdl z&@8jcCHuwoEo63_;{rqY2HFx=n@YZylX9a} zl&P9Yv{)Lgc|b3Q1o2l|SANshLidoYfmF5?I`bsF`E$9kGP};}K?$qva#L^~CH` z!TFGfb4WF(Bq_ENC#V_OREgx>tR!Qa(Jg2?b%7g;M5AE-&>&(JHfZkcmN2s4eJeN!nCrcl9Way`gTk=o|nGo|BD1pGHLvB0ih$H-WM^@K##RBrgEQ`4$CSNzg z8QjInTy|bpvXE2PqeM9*$mGvZ!Ps7Fn?$@*V_0OIlsGq$7xq#m0A&oC)8WX5OB{I{& z&m4D92ULj=J&5P>4A>lRn(KPS@|aiq-&TfHnOC`uYpkgbZ!za!sgrKX&HmC&DR$Qw znLUwmqe#(ab!;OBsne)NG--Cm>qV#<+25uf(vCyt?AGIMoJse#4t}n3bFn42(girok)X zsLlF0m3f3uPV@^VjN3J zs7vW$dREOUH=t;vnxK-_6qp*ejG&zM*m*>v9wu&xniWe@+eJ-67VZtoVET-b0X5{6 zr(c*Y=7z@KB`=B#zMR8)M_(&sn@t?LtNkyD`lrk0nJapT+`Ued`PVEyOY{v7f2Alh zxP{mY>C3kmqt~@Sx9=weAH3PUD&9e;-4Z?DM%u2JrA~7?nOo3Fg!@?ilHRb~Q9Vh0 zS~k)vttP$Xy9A>{?$-j{oKIM^!~^qOk9nFfO9U;uX<{Z}MGPU&T0}pPw4d7EHF*^c z(1Qo888T#p5hW(|Q-(yg#r6vVzhg0gpd>56bb9oH0wu}%3M)p2fxFLEy>QG4R_-h8 zU+Al?!eBv?3%sHzLA?4>j0E@%7$S|RYf_S$ylY+ z4n%*ot_mG#p83HvVERPUjJRH!Ay-9T%yQe2biJr+b%|?XeE(`??bZyWEqp{h5`F<$ z|26&q>X&o$0crC>TI-zNN~}*w7-kFnefLs z2fQs{{%-wM-9ryBgJ*Iuv&{5yuKy+Eoc^si>??Jju|gyAn_Uf`ajXB1%g`EBtwiQ1 zx^awk%lc*V?-yf2mx&<2oHk?3d{TaxpMu&Sc>d+t2h>+*DNg;iw%P+Pbq56MHt1{8 zuC!j;1YlpBL2hXi-rks7|L=db0Mz7?nWiEF08stMZRP$Sn6!kAqm#as74d%`|J5u1 zZ`hY{-1iNNl z1=2bj@r1^~3~TeQTAAId%fY2ha|!FRU6VMpiAkkk@VViqVwhBxz8SBI0v70InyyD6 z3Bn|Jj3nVomoatTh{xa7jx;yvi_UnW_#l*M<|9E)rOc4j#iVycL>cKHTtp3#k-nKL z+7?|mS#aSINetxl?nE8)%Zyk>!C1k`<{`huyPwZD2`YbK4!99|Okznl56^r1}88nU&cpyn*~f zRP2FGaX0@#FpvKuii!WfqnQ6~#DBA2l_uoxjK6W&?wmdns)%IKg2?m;9KE4d3H+J4 z{P-@21_oU4WQ76zv4`7rf2c8VBqkLlTWX8sn;VP7*r9$|Zvr<123Vyh&st-dNnOt) zxtL4AjW-w3bde9fPrdt&)f0toUJ2&UdD?Duy5Ap7dEF=0V82i93p+Kxkri{*^!QAa z`)V#?MO?Egc}EaN?0rV`N9>*U}noU~6E-WouZiR;Mgh z;i}OVBurvrDpRj7!i%ICbMj)VT&(w5JB7dEWs8$MSfbZaa1D^jw$rlh41JSI!*+g5 zc`HjldKt~dEdKiq-t`OW#SHiFi#h4kU3|pR`S;CF5SvpIp|Cl8#>|qEO zL6o_yj`uN0$wSqXQfj)_qWIKrnS3$j-u8y`GrF8k5xy*m3E_xC>4xG+3@28lsi2dl zG->G?bNPxG)$u+RlKOK*4722EnDvKFTfCP}MVn#i1AP7T_HVVXeMTs4JO zpT_!OPG@)cEQ+es9a7Q~8ZJxuwg`RN6PqI_ZGrR{=g#vc28nWQy+I8dcb5dFR^-u; z&&P%sTVJJ;F`R;9s*$hDbF31St>mkHWdp=P*}5fF!x?lQhPw$TMi}e=#xDm^PWJok zBklIX+F!cN8)z!@No~Er@9ywmEwj?-&7I}xh?Aw0SPtK(3EQ+5LHqwwu+}k1p;#vH zrvh`dw3QgL-4@kIQ!Av--?{@#~s8|+dQ;(;Mo#ndpY6spn{3TJBv8{Ee0%vgX2)N zCCV1=Y(p9TH+hpYR^mG9QF6nF>tHb9wDPpXRlL7F+QvVV*IK(W=+D|wiR-*I;elS7 zY`O=x^{a5b-2CDtug6c%+y!Jb>;Y$1|5k+KbP-$ndnLz+PK~0IJ6_kenCmP!NG!nT z0oX@l4sD#DBU$@kjnc{sh4baeOf!mqY{x0?+@X-P%tFTkGt+fK8Xnl}SW!g#bX7&^ z+2;eo?q}&im*rirs}E*eubvzp8ZZ##(eDL0O^$sfaX!0;rmj^d#vG<0v5$vbadqkM z;c@S>jXq)Rz%lvuo_XtEk0U!0-X%0LG%_Oo&y;sC!y!Vzbv!1e%gjo7+E(!P5CXQg zglw~&%zv|GAITU4^EUXYL*ba5L|+fG{n2f#<$P`;XXQzw!rFG>1xIQtjYXPCx$0Tg z_y1H9*k8*NMu;cG(T9I5k|_z+!6-KvLctWLG?awCF`Wto6>5{_B*kX_J!#TlRfW|Q zTxT2;H#0}=YR;55U1N;$dTp5H%;k}GCmbbyfA00QK5!SnK;wWT_=y7G3YX(F_2ej zekKG-;-FFYlnsInfBS-ue-l(=JyzlnCV;dv+bFa!pd>$1xZyr37BgGGzr|0+^O~0j z15^}t&e-E6dU|#)QNVmuka5beLq1^$=n5hx6Mg@fLV!rjf(f07zjUyE!{MRr^$O81 z9c&-SdtEZ{pn(T}h6ZnUS7wPMBn?d!5HMe!BHRBbb05=@24O?2h_`+1 zSkky=Y6p<;hK&MFs_UV3Pi4-ZFlQ5qOdAaJ4>=1O04Q<~*!bCF?FPS~o{er4?b z@BAktYAQF=_~SF#TF%vAsN~HdgBetV+7Sn}tl<@KS7SOg0f&fC(;da%oL1YWSL+*m zGM#5P_te#*^#`lcd2E#Bzrd<*Ozyihcs6GM{UIN@;iOnS-MRs~qr?3IfIIow<-ibm z1axfeXk3WdOtrvL9~RrkL@RPE27Wm{vO5xg=Y{Si6xRMyB}nHWVL(7VUs(tiyCf+=eFX z^v*e{k1Tj6MkZdZ0LiaYY^zFpCUo+Dxx=bBlNeU*IS#VeeOAzI)Vt^$zh$j^EZMHM z**h+Kz~xZ6N@mz-#ETTbxO`K|Nr-N;@=2jQ#7ZgkFx(W;GWygjB|Jx@jU+qS`t!IrL_@Mh#X_TZx%@ z^4p_*L+-*ol_Bw(5gpCY^}j0qLkVl4eKqJivQEuSwK~_wQU=a?(Pr}B&EB% zySux)K|s1&x?55}O1is2>5>k~O$h(?yyyFj*W>Z~9|mI&_Fz2Mnsd!nbFSyU4NmP* zk_r34gxePNOJ$h6cykvyCw$qW0>}3|r&9U*AFcQWu@^Z90;YM#zVCO^+rx zNH@pXoqevqr|SqP@$wvXr8J@&d_JP>=uXmMSW8G@sN0shx}NXhJ^U;k3^P3*Y9*{X zT_){Q>`WUL%w79gi?=u4Dq=QB^rnC>Qexc!1mCKET58qi_4>ylhJterN@VVP&{9R} zf`VGjgzL=<92XlYXsi4V{!C1%tpasaKFas6LJV)K-=vfm;P_v(pq!FX4Y?&YsVKhO zR%%faHzRDbQ!M3E;64T2WnRzcuczPxKYjJ4E?oK+r6|}!&xa}zY4)CB2A?|sZ9Z0a z|7}5bo3I!eu5axh5J}j*49lzaa_Zc8rw3g>pdb(cSDK@($H8DyJ~4-_*`cwZ$s? ze5h6-?o%Yb`5-tXa|0?FF6Y2tk6?PhbB~VSfa6cTW01)6;9^4dE+jka44m<(+qOx| zS7+%A4{cV1vYAlL_6DE@7TAVxXLfPEJy)0APHnPc=nL6sYxCkc(#=FY#J=VU)@bgA z0_~_L;7&Dz1PtGWxfn&<4}Ma94p>_udw=f*7k4kv58VQ0lC!J^kehlmGtWV4Mi6UiYHz1L*lE`k@;g5_yK$-= zZtu<-NFGqxlm4JpB#T7g%Ex-iNmQO!&y7g$cHfwbO|=&7md}4l4Mn9|n24rEQ^>Ux zYO+gTedMAD(2~_1Q6k*FOpy38A*yn7gLcbXj?+s+U;2tl$BG4xn$@hHmfNzSfuA*V zDR8OI{FbT?yi6r34Q}@hSTAGKo2ggB19-#DmV2x|Zadz2|rHCQV8f=qYq3S-XQKr)V!L{fbjC(JB{i1oZ ziF#JsGKmxT>@0|5a3}*}b2#dWUIr!i`8n>4;r7E*)&qvB!SvEbZkC%_T$i>HF_iTK znSw(apn9nYdcK)KaXd!E__$?es}T}>(H*ztldjGo3~FxJOQHIwDEbA;V7L2u0y+iR zI z`Ta|+1SVzj1fro-ACvhOxw!`lkeVnt+5zUv+2Q>l6W3DEHS!?GkLeUc=jF=*DYi;4 zgAmXvqwtL98S&@oBP*(OL2;6Q!{jJ!x!SIzc(UKP=n25KVnzea3MJKb=3u8Cm>iLlc zo>?@$-95+WQf~)EAZt_5R=Kx&-+eesXf5(h%iWVsgV-k<5sR4Bt?SzA!_Si!Vs17{ z{6tvfF)5Sptk|88Zta~Yi^wNgFB3D>72<4rA$j}O^elvaJgTjo4ShF~YmiNpHeGbr zyKXGp)-!&Ibd!z^zbI+4QbF?)fGbwcwDyLFza9Z}=ghoEC1>_-5DRf*_-4`0`D_3% z-j$9^NUELnMfu|?&hgFGHu3n@;Oi!chfyGFC1tj zysM2L<;pVB&eZILeivP-DG6^E!_0P@Pv$*0)yMcNP8S ztipdgy#t~iDVyOeruzZb?;xzt0NZ53utk9^3ZvN}(iFQco`XI5+!2~Bt*g7s$UI9V zqTk}E=N|5KTZK~u!6+3ngR++0rc2UcL~b2^1ySOpH^5EkBa;19dk^IoLT_D(^eYV? zh)u!~KjQmm97L8GO!T6q$6zM-+4)P@I(QCal||#8B$YWzh+EnD6~{;lGD;KM(2Z~x zbfm^>#(c>3<`9QS(Mb$0_NoT37Om8`p*ft5u4+)-eY&scXqIdG8ph(=r%k3w~PVLOXd zvY%SJgzTUS)}20bSmIE#Ku2ArE#^+hFkz~5s)Jq}y~;DcyBxahE*PlD`+}A(u^rn<&8zczVDn%^A5dk-Vy_mr0qL*uM z+kH(G>dhnCDc>o`r?(AIs+^*rfe)ECTkV3CYD3Q#19fXQhe<>BD4P`WFJ{4fglrGp zMC#o(hLNzR_6BG%EOWFS0kBYlhLR^aX`ly0}L;y&ATq9Kgir+g(JSTR7eC^Kd70rtk@Qwh@u3M8?jc zvgkQ+ER2q@6iY?Es?2yUOPXy52HHmmw09OlCy8i1JSX$cFQ?Kz?WxLaD*;xXXdOZ= zBkjariS2=U=4{ztOD4WdLby%7@-N=%81G7r_onmAC}*~wh&dH`ElcXAaT1YCg!*3c zydPyIQxoLY1}B)t!AYV-sVm|=v@yqXQI~?W4Le?d1`+uZEGOQ|ee*VGf zrT|&74wW?}lFB{`V02N9RseY6=RHwR+vczuOFPU6KW$IutXl`cwNkIGa12qG zrJ%bP3TNk7J?}yS3x6XEWxoN1EKl;n-Jr)OR82@8A-lLcqJ0m!DhivFnJu)P!CIZozRj3Dupfu>UuxP6njtRWN0x(t)#GPjJ(W*QX;@KZebajIc;dm zCW~hL0jRsrD=aVq-P|3Oy{?-lW2lzd!ihrjVFr)oLbOS5oQOiE*S-!;?Lbx&bB@wB zIBCNkoH#5Y8I#5PlHx>EpLUEIfBnTV;pU3R%nfkZ z!YFhE-!>M@7lKEDX})s?nHWmd;*DDNM6GEm7PaY{ePtQ7vU*E6^Yo7t_xmKXg?pIw zLetbL($kGYR?TwDFJ{6?y@??DP->A;k*WI-u5h`r_Fj=a1?c8CaYv_fx+w3Y&sz)# z5l!Eerg8T>?FtY$ym)%@xf}a@V)bx@rCghzp-=;#(K|s@NOO*IZA)NzB23n8Oyp`N z6Y_)!pjq5GpOl;|9mspLVAjuk4Swf>dB>Z+oWGfksTiJHt6LL8{)`TN&}5mlo&S@f zn?k$j;4E88b8ms}U06xznINvR%znonws$*X0nXu~KR;D&0=; zq1MxLBj~1VFmZ3_rpJ&0B|edG0LL4z$TA%JtOE-~IHfCXompV+wy z8-&6rt-RaR;6BG2HZ5IoYkQ!W1K80!*5H1C5|T&@US7!VmLWU9nG%2IR0sf%g(q;p zir%R2#OCiM-FRbfu?u|_l)-Q7I{}F_K#B)nXF9wXSLm-9xO`&}clEL58GaMK6`1Uo zQKob~3zs=o{h-kD;27bhfCkdw{8=X?mD$rB(iIfJLV2z}Inma$btemM>{3VY_dH`c zRmH*W_;0{4Bi*0y!=kq3gCg}!KzsqQv(?<&2%Y|52_E_JZZE7axCF6;pWKz-h9;(1 zFEg|lBDp{TkLtU9pc8X{8!)$h;lT}wYiX`cFvH{sCC$IJ1nrkGsX1R-c54t zLc9jBHVaK(PZqQAK)*w|rQxaCi@4yDsR;BKp_0+QMY4^V@oQdty=y?g5jigp7$EqZ zjDUR~x@7qfAlguTFi<0JZx{E(?05$3ZrE!(`+7JwC(6-O)0zPfL-;9#k~GMZLtGy?nM#)>2+T`kNj ze-Cd%!Vd{3rx0cOIo+1L-plN7F!@)*0?vWum?{xsvwILKF<=UycOWzqNrt^1DAHo{ z&>l4+Ab^}}aY{#leq4;cq6#<-V$Ho7UKVZ81@Wh+CFOY)SxBEZUOMd5^n&4mJBI5y zhiL&%RP$EK=dU%dsx>v_%dKWSAnH{~OU>To6_twC8@+RTFwOV zjN#5sZh{G`WWFrn$+vV8xa_EdxGegTh$iG5fdf8|IkR2eF_u{^F!2%tv7EYty{ytY zfTzxF4)ngPoP_WTG|Fer08u&Q$%>o}_7yWw_VUke{^I-nDIPLL`#{~ep5)0hW*8ez z$=vvIc7ys0bTt^Z4cC$pSAr8jP+)*}S0n5;J4~41b{%cIM*fv_$1_a{7~CzEGF*%a zmo!~DyV(mH=a!>N6aTXY|l>8fd_G+w#(nF|q5jcLBA z13?#dl>PPCA}RNzqD6oVO(@OKym{I-Pa5JmLRwqW$FBiUBnL+P2)@~J(ec|s_sm!R2@$OKicGYN*2GqU(J&T z{Lqn)*=vxuAX1Gv0Dk!C`pCTtlDrGq_gKcHI?^jian>rS^UL?G0{-ilaNK#DTyw56 z{Mo5FbQ?Hew~5Kllovle5o!-n7?EA%~9 z%jQnBip8H@%a9KGo;gZW59-6s%P>_Y62@fk&z9tt_3vec<8wZNl}y-DPVJOG|Iin_ z626Fx(_8z21@R?Y6h3=m$wyZ(m0~u^gGm$C_>_E9bIWd}w}}Fi6`vO0&SEgSdVWB! z70oGSTwI5)%Dq)n3w0Upp_=|g;_;3OZw=}>WJUsdX*M=A4EsAwYD>0ZPrKc^Y`%(P zR4QJgyJNu4aNup&3279U6_ zdbsfLmw#jb+-(ai0SJf=$M4ESh--^XS307Zgwt`pJ8{}aNm%u@LRcdGx zw~H)F7#NIpX{7#kW5V(1H5 zz5AdL#5;!Xs~elu2h{fX{pR6_V=3+&^ruJ{iTx$`s^O_)RYD@?{ol+}(o43PDCFcy z>6@z&ig(9lnQ&Je#^YG*qG0nV5izc-nDi1Oya!vptC5L&xq!LbWas62!Jk9@Hgg$u zcf|NzytpAfC_?Eo)ZG&ywyD+)KyrtAk@F|5=o#Mda4t2W8yW1la)U@5zE9jn2t8L( zX81%5B2%>F4iIQQ*!=|^;t?PSN?@8gFwrSJ@S3$#y8xt&xUbuD-u=7}9#eLWR72-qTT@xu+BTcA6}iClYMq3D|3PS&w~_olnHK zbbUG}X3XIIUV2VpcbYSqR^lWK`E;G4pb|N_JYdhO-P9g;3Pq zx#XGZHE!5Xc?m~}&3$AbIXJZLI=xQV><&VT5CXbQ&*Kz10ue(bo$2A61QOcN*>`p;EOKRNXLPtn*{8w3F-Cleb(>;Dq;Q;C(4 zd?J7xq=(1C&}V+H(IjuWE!QWIPhSF^7YZk!fUfOIo+QzqwU^5k7P>3Y8U%-;?GA!O zHYcntF5ohIP^By2K2uO|W-gA~czK@O*61M(U{K*rXX`j+=FR!L5*bC z8%ZNoC}V;XL!Kpb>sP)JkSj_sf;rwMx2$<+g%bK77T7~8tSw-VD@GV=JA)2g5Hs@& zN(X^2sMAj;J;5fpbBvQ$s%Wr@mKo`t|+60qbQv%_fRc(1N8*2fDS zc~Y)?i3pyo`Y`?2GK=TmHMB1Sk?@)-KhzR}Oj=qWo(Ut-uUx}_lC%xNatZzBfmEBJ zSB2ILfPtS-VxP5RivoeD?|F1}MKFC}S2DXwe+>&i*)@^(pNc<0Ylm@t;ENoizkQkG z#jnpbKyf#qNVcsT*VPwT{GWW9AfDFmg(z^eN2;&JR3~wRYIg?8~`b z6w+Q}ETeZ#j>1Z?z5425VK$AnXI=J;)o?YW1AC@*n=7rc0xy8rmLo~Jcb!bgn3ceG zv1@S2g~rpP*}ia;hD~CRV%Kn2XA_Ux$o_4-22CZ*sM5r!eGy6Peeyw==5WHgAUBr! zfvRYibkq^Pj~pB0`BIi)Xx#xu3H)+%OM`sS+HY@3+2tFUh{#~*CgyA#2A6>lqfn z6S5O{6{Wk3D3`MS+HG^VfwulGBaN;h`#huNIg<4%zjQE;0edb^GBt_26eM9Eg~2<= z%x&8wNd;sz2J(b`T`Vn+b%GZu!pg_&@u44I_b|jc_M^Ast*GX% z~cER`C{E`DzN*%y4r>@ti4A$Le2~6EEK|BE&%nFopIQQ zN!-D9pX<=ija}?3M}Wur)SnR4!Q^=N{TZI>K-5OX+PuZ@ecEdP)O|3 z;Z49IgbEtgSJg(*(Aa^$Aoi=5ZV6^_E4HzP)mn?bbRzqSk-Q@}P! zU^@l7uS{R0FQ1#*uh%#!jP+VDBI7|deK+xz-o;cMwsFQa_N6oU`m|HL^uTLD=QXI? zqFiDND9*>fT!W9Zuh{5;R})jH-(6Au;dQ~kD`bIM)20??E{+DjC_(m7K9a=~L+3%m zmtNX7LSUw(wb78YdD4gQYKDwb0w6BK=Xyc%RRPAvWSvJs>w0h2R385!%w)PxhWr&M01bMie zx>a1ez2u_4;Q$qR#^a%(z`bD;W}PcbW;gZp$;XJ(jj16;20aY3xp5(V_)^EWM`}Gr zK#ADYB0DVWY&9JP_oH)FDL~K(Y0HNT%jo5+7MAC6`q*B*BqP)IfOA zSs1}p4ht#5?g87B?XYTl`HxLvWh($kg4e|Fz2Zvohr;hXR?n)(=s&V%ugp%$J_YTVFooJk<#&j9b704}aM+b!QM* zY2B{6NUDF@2GpzM?B-{6Ghg#rk|qw*Qr=FO%CA^HN`cxwni?*?^I8;o%^2I|#b!@H z!~kFZVrVLm*xR}zG$0!nJB)j{!+gufR3EieNl0$mvb9e%%PXc-huMH^XTw*p?1 zYyBDhW(uaF%N2hMyCTWakzvUi@hY_+R8p{u`b*vcrP^U z_*g|+yWK|d2olI`sQ^ThBwo*25*7;P@yH3tB(f9HU$-isz0RnuWHIEzUyNIb?n@Re zv$Du(b|ul3b3Fq0U>?6%DxBrqHZ@M!(Q9Sr<$XXSD&RZR=lmi8#WaVOpR03FJ!gJX7}xq)vi!L65L~h`COI7w7PQN!xMG^TmKZsOTAK%u z#7EYSymBa>Y&`4@Ffm&lxog|JGhG>BPx$u;Ig zhanra)@5TBV{@8(le)od=MZScTHK2=8cikHIuNW>^0PQLiQ-@U95r?P0sc?spnX8XB-Fwp8ZN9nk*gQNY==j2)0kCP> zDS3wH9LV%ani_3bU2|xy#zAU$rwL<`uAe~6y>{(&G8kQVUiZh>m`rur~bZ0XVL~QQ(q<_ClM)5o8+`+95hA?X0lOj&2f6?i%}xEm~y3R zZA1w3h^*;MJ*GFdRrP9o(a}EeSy$0MRB1H>ND#EI?o(ILX|D1yXsML7Jz;PiQelZ+ zp!i9t0BZQ}Y0c!zH|4A21GdDR7i)Cpg{XY}^=@lm1vWb9>y^p4F^Fj{5|XH~U(`1y zf0U&kUb4c0uQ(#`!MNRwE;%*DP}`saRhM}Q@8)WSInEKkDq_N)ih@A^4cDIuzpTR1 zg1^TRqQx;vVRq~}7XnA(a3&`_p-X}Rp+M!R82&a9yRuU2)qbcH!*(OuBG-ZxL$7^3 zk&b$I^~5I@OdQRRR`nvwa|Z8Ax*#R#RSH|9#$u7?>1oDhG*RHFDlwSr4bi&61QLwz zDLzl|vh{cbR+{+2Riced&uLkYy9`dK_ScE8u`N&ueqg2cUruA%=)P)#35CF58vwV> zIFPBlmMmvWShXzwjAC;X9Q9dnE`&F@@U8Utn=nx1ySEfLX(0((;LiiMhO*{o z332vyIVs;A+_1A?y(oW|?Fl2oUa(^_iON_+oYqiYgd}-iq2eyFl8e*2C7b|Q$7#)w zm1s2=sH^Fdv2u>d+BWU{?4KqFr-5CP>KbEH1xpYDVVij6M-c8AG=ym^@?d!I(P`9u z(W@77VDq{wy0<#R`)C@Tr;x*YPD61$^u=U&KnFrtLk+}c7XYQ}!}&%5t49-o8#I6j z8$BWc@|_PmISg)MZFq}`=(Tu&Y0*gn=!zUT%R6}HnzGC1I3zr#o#GHqMQG@>OzQj7okNAF z(psjhjkl6sE-6TI^GhnVg0K&Qnd~;28l$D{!$=pSZL9m)_hz5f__8{k;McQxsl7yL zoV4+ZL@DetHhsB+u&|Sr*#=j%+t!eitu!F$RMK>tLL_&GeKR_!oe^eQ=FnS3U9fs4 zI?FrCXlH>RT``+eW}G!(+Yec7JR&Y?WJi( zmoa%r*|6?kWI2MyMWFR&UR94W?=gsTJxJ}_*g_YkdUWL!owBrj-lX=Hx;)8+BIbFr zftcCqOWQ7{96mH7cGBrD==xgg7+$j^gyKT_a)O9QZ?{T>TX!jrkd>J#Cm|;2;tO2| z=43{SY5NJhTQKQ*&oeNy$u#WO!de&b$r+usOzH|f+vA&o_9PCcYXVad((7s>b=O!Z zxvTY)LL%1i&SDV@+C7(o`!I)3_ln}{m?q?=Y~@fKh>zj!lY5>N_O3$Ml2U5KPx+(7 zN0LYrf4JaN?NRvbXSVht{+PCc8`(XyfG??_f2D8e;jKH>`WI|T!;WbjqP9zrm*ZR7KW`bM%aMZ4>;lijsSslVlc+pT}&WfxFuQSMv0}uM1%mqJA$7GWa z6pIIode$f6LrBHlm1tMmunGE`=P4W`HIGYvT#t8kYINF0AA{{c=jGrCMA7YO`<&7m znPRW=3T+R(iyAEZD5LAgt+0a^)JQ95Y} zArV<65fxQBr;(Bl?f2HlYs0 ziGdJ%;O|#epl^W;RG_kRG@~>7OHhi=$l8MLJ1b@ZM>7{2pdviba?Qm47dPlXx4gn5 zAS((u#k2^#&-gl#^es}6f5-WyC+g41pS(8g(F7)M06uYiwe0*BfoQ)={+9!*<1+zM zpe4zFKtG#={Y zWh|VWfPQ@cp#n$BpCHi$Fq3A1NJ*f0`j5@bc=iX#zgcbujwXNJ%$8|Sv|Ql8_W^R* zf9Tq6-~sy2ga7Yw^MCDC&}KYbVj#*CIDmc}rk9j|j8g*IG1;2^%l?~tkPAUa! zoPSK-#rj{#|LUpVSk(V~Fn@15{M8crTjX;6d-DGbxPRIH@BK7?9A#=eKOijruWrUa zH|Ben#;-;|-(p?xH>CfwTj$T*@7>LQyk=br|G@pFquD<@LjKJ8-uCLNSK7B=k^Fbg zA3CS~4E^4B>8qpGw|FJ}1N48^U;fBn>u1XM)-XTrI(OM$QvTNt=KtpC^fUK+i;S=aQLge^)V~~G-zzMBoxuDS=O(|*`v;1gKX3c@GJ`*k za60qfF#ev4`Df+EpE=)Gb$=Bt{1(v`f5!Qj&icO6_{Yu)@%|;?4@$*8G>EADkeqE{m7WL`BO#91q`=2-V`_;N1uP(+}zs&l(<<*~)e?RN~b;0jj z5a;|l`5!F*{S5hjw(!SY+EDOI$ls&#chmVlGroU@`a19UEsRQj$M}a?NO>s;-~$;5 R2np~f1o-$>Q}y+){|A@R9n$~+ literal 0 HcmV?d00001 diff --git a/EmbeddingInversion/gradle/wrapper/gradle-wrapper.properties b/EmbeddingInversion/gradle/wrapper/gradle-wrapper.properties new file mode 100644 index 0000000..db6d194 --- /dev/null +++ b/EmbeddingInversion/gradle/wrapper/gradle-wrapper.properties @@ -0,0 +1,8 @@ +distributionBase=GRADLE_USER_HOME +distributionPath=wrapper/dists +distributionUrl=https\://services.gradle.org/distributions/gradle-9.6.1-bin.zip +networkTimeout=10000 +retries=0 +retryBackOffMs=500 +zipStoreBase=GRADLE_USER_HOME +zipStorePath=wrapper/dists diff --git a/EmbeddingInversion/gradlew b/EmbeddingInversion/gradlew new file mode 100755 index 0000000..249efbb --- /dev/null +++ b/EmbeddingInversion/gradlew @@ -0,0 +1,248 @@ +#!/bin/sh + +# +# Copyright © 2015 the original authors. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# SPDX-License-Identifier: Apache-2.0 +# + +############################################################################## +# +# gradlew start up script for POSIX generated by Gradle. +# +# Important for running: +# +# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is +# noncompliant, but you have some other compliant shell such as ksh or +# bash, then to run this script, type that shell name before the whole +# command line, like: +# +# ksh gradlew +# +# Busybox and similar reduced shells will NOT work, because this script +# requires all of these POSIX shell features: +# * functions; +# * expansions «$var», «${var}», «${var:-default}», «${var+SET}», +# «${var#prefix}», «${var%suffix}», and «$( cmd )»; +# * compound commands having a testable exit status, especially «case»; +# * various built-in commands including «command», «set», and «ulimit». +# +# Important for patching: +# +# (2) This script targets any POSIX shell, so it avoids extensions provided +# by Bash, Ksh, etc; in particular arrays are avoided. +# +# The "traditional" practice of packing multiple parameters into a +# space-separated string is a well documented source of bugs and security +# problems, so this is (mostly) avoided, by progressively accumulating +# options in "$@", and eventually passing that to Java. +# +# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS, +# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly; +# see the in-line comments for details. +# +# There are tweaks for specific operating systems such as AIX, CygWin, +# Darwin, MinGW, and NonStop. +# +# (3) This script is generated from the Groovy template +# https://github.com/gradle/gradle/blob/3d91ce3b8caaf77ad09f381f43615b715b53f72c/platforms/jvm/plugins-application/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt +# within the Gradle project. +# +# You can find Gradle at https://github.com/gradle/gradle/. +# +############################################################################## + +# Attempt to set APP_HOME + +# Resolve links: $0 may be a link +app_path=$0 + +# Need this for daisy-chained symlinks. +while + APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path + [ -h "$app_path" ] +do + ls=$( ls -ld "$app_path" ) + link=${ls#*' -> '} + case $link in #( + /*) app_path=$link ;; #( + *) app_path=$APP_HOME$link ;; + esac +done + +# This is normally unused +# shellcheck disable=SC2034 +APP_BASE_NAME=${0##*/} +# Discard cd standard output in case $CDPATH is set (https://github.com/gradle/gradle/issues/25036) +APP_HOME=$( cd -P "${APP_HOME:-./}" > /dev/null && printf '%s\n' "$PWD" ) || exit + +# Use the maximum available, or set MAX_FD != -1 to use that value. +MAX_FD=maximum + +warn () { + echo "$*" +} >&2 + +die () { + echo + echo "$*" + echo + exit 1 +} >&2 + +# OS specific support (must be 'true' or 'false'). +cygwin=false +msys=false +darwin=false +nonstop=false +case "$( uname )" in #( + CYGWIN* ) cygwin=true ;; #( + Darwin* ) darwin=true ;; #( + MSYS* | MINGW* ) msys=true ;; #( + NONSTOP* ) nonstop=true ;; +esac + + + +# Determine the Java command to use to start the JVM. +if [ -n "$JAVA_HOME" ] ; then + if [ -x "$JAVA_HOME/jre/sh/java" ] ; then + # IBM's JDK on AIX uses strange locations for the executables + JAVACMD=$JAVA_HOME/jre/sh/java + else + JAVACMD=$JAVA_HOME/bin/java + fi + if [ ! -x "$JAVACMD" ] ; then + die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +else + JAVACMD=java + if ! command -v java >/dev/null 2>&1 + then + die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. + +Please set the JAVA_HOME variable in your environment to match the +location of your Java installation." + fi +fi + +# Increase the maximum file descriptors if we can. +if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then + case $MAX_FD in #( + max*) + # In POSIX sh, ulimit -H is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + MAX_FD=$( ulimit -H -n ) || + warn "Could not query maximum file descriptor limit" + esac + case $MAX_FD in #( + '' | soft) :;; #( + *) + # In POSIX sh, ulimit -n is undefined. That's why the result is checked to see if it worked. + # shellcheck disable=SC2039,SC3045 + ulimit -n "$MAX_FD" || + warn "Could not set maximum file descriptor limit to $MAX_FD" + esac +fi + +# Collect all arguments for the java command, stacking in reverse order: +# * args from the command line +# * the main class name +# * -classpath +# * -D...appname settings +# * --module-path (only if needed) +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables. + +# For Cygwin or MSYS, switch paths to Windows format before running java +if "$cygwin" || "$msys" ; then + APP_HOME=$( cygpath --path --mixed "$APP_HOME" ) + + JAVACMD=$( cygpath --unix "$JAVACMD" ) + + # Now convert the arguments - kludge to limit ourselves to /bin/sh + for arg do + if + case $arg in #( + -*) false ;; # don't mess with options #( + /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath + [ -e "$t" ] ;; #( + *) false ;; + esac + then + arg=$( cygpath --path --ignore --mixed "$arg" ) + fi + # Roll the args list around exactly as many times as the number of + # args, so each arg winds up back in the position where it started, but + # possibly modified. + # + # NB: a `for` loop captures its iteration list before it begins, so + # changing the positional parameters here affects neither the number of + # iterations, nor the values presented in `arg`. + shift # remove old arg + set -- "$@" "$arg" # push replacement arg + done +fi + + +# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' + +# Collect all arguments for the java command: +# * DEFAULT_JVM_OPTS, JAVA_OPTS, and optsEnvironmentVar are not allowed to contain shell fragments, +# and any embedded shellness will be escaped. +# * For example: A user cannot expect ${Hostname} to be expanded, as it is an environment variable and will be +# treated as '${Hostname}' itself on the command line. + +set -- \ + "-Dorg.gradle.appname=$APP_BASE_NAME" \ + -jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \ + "$@" + +# Stop when "xargs" is not available. +if ! command -v xargs >/dev/null 2>&1 +then + die "xargs is not available" +fi + +# Use "xargs" to parse quoted args. +# +# With -n1 it outputs one arg per line, with the quotes and backslashes removed. +# +# In Bash we could simply go: +# +# readarray ARGS < <( xargs -n1 <<<"$var" ) && +# set -- "${ARGS[@]}" "$@" +# +# but POSIX shell has neither arrays nor command substitution, so instead we +# post-process each arg (as a line of input to sed) to backslash-escape any +# character that might be a shell metacharacter, then use eval to reverse +# that process (while maintaining the separation between arguments), and wrap +# the whole thing up as a single "set" statement. +# +# This will of course break if any of these variables contains a newline or +# an unmatched quote. +# + +eval "set -- $( + printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" | + xargs -n1 | + sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' | + tr '\n' ' ' + )" '"$@"' + +exec "$JAVACMD" "$@" diff --git a/EmbeddingInversion/gradlew.bat b/EmbeddingInversion/gradlew.bat new file mode 100644 index 0000000..8508ef6 --- /dev/null +++ b/EmbeddingInversion/gradlew.bat @@ -0,0 +1,82 @@ +@rem +@rem Copyright 2015 the original author or authors. +@rem +@rem Licensed under the Apache License, Version 2.0 (the "License"); +@rem you may not use this file except in compliance with the License. +@rem You may obtain a copy of the License at +@rem +@rem https://www.apache.org/licenses/LICENSE-2.0 +@rem +@rem Unless required by applicable law or agreed to in writing, software +@rem distributed under the License is distributed on an "AS IS" BASIS, +@rem WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +@rem See the License for the specific language governing permissions and +@rem limitations under the License. +@rem +@rem SPDX-License-Identifier: Apache-2.0 +@rem + +@if "%DEBUG%"=="" @echo off +@rem ########################################################################## +@rem +@rem gradlew startup script for Windows +@rem +@rem ########################################################################## + +@rem Set local scope for the variables, and ensure extensions are enabled +setlocal EnableExtensions + +set DIRNAME=%~dp0 +if "%DIRNAME%"=="" set DIRNAME=. +@rem This is normally unused +set APP_BASE_NAME=%~n0 +set APP_HOME=%DIRNAME% + +@rem Resolve any "." and ".." in APP_HOME to make it shorter. +for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi + +@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script. +set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m" + +@rem Find java.exe +if defined JAVA_HOME goto findJavaFromJavaHome + +set JAVA_EXE=java.exe +%JAVA_EXE% -version >NUL 2>&1 +if %ERRORLEVEL% equ 0 goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH. 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +"%COMSPEC%" /c exit 1 + +:findJavaFromJavaHome +set JAVA_HOME=%JAVA_HOME:"=% +set JAVA_EXE=%JAVA_HOME%/bin/java.exe + +if exist "%JAVA_EXE%" goto execute + +echo. 1>&2 +echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME% 1>&2 +echo. 1>&2 +echo Please set the JAVA_HOME variable in your environment to match the 1>&2 +echo location of your Java installation. 1>&2 + +"%COMSPEC%" /c exit 1 + +:execute +@rem Setup the command line + + + +@rem Execute gradlew +@rem endlocal doesn't take effect until after the line is parsed and variables are expanded +@rem which allows us to clear the local environment before executing the java command +endlocal & "%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %* & call :exitWithErrorLevel + +:exitWithErrorLevel +@rem Use "%COMSPEC%" /c exit to allow operators to work properly in scripts +"%COMSPEC%" /c exit %ERRORLEVEL% diff --git a/EmbeddingInversion/settings.gradle.kts b/EmbeddingInversion/settings.gradle.kts new file mode 100644 index 0000000..2ebe288 --- /dev/null +++ b/EmbeddingInversion/settings.gradle.kts @@ -0,0 +1,40 @@ +pluginManagement { + repositories { + google() + mavenCentral() + gradlePluginPortal() + } +} + +dependencyResolutionManagement { + repositories { + google() + mavenCentral() + } +} + +rootProject.name = "embedding-inversion" + +// Composite build: consume the local SKaiNET + SKaiNET-transformers checkouts directly +// (source), so this example builds against the just-merged t5 / vec2text modules without +// waiting for a published release. Gradle substitutes every `sk.ainet.core:*` and +// `sk.ainet.transformers:*` dependency with the matching local project by coordinate. +// +// Both are included here at the root so transformers resolves `sk.ainet.core:*` from this +// same composite — do NOT also set `useLocalSkainet` in the transformers build, or SKaiNET +// would be included twice. +// SKaiNET core modules auto-substitute (project name == published artifactId). +includeBuild("../../SKaiNET") +// The t5 / vec2text modules publish as `skainet-transformers-inference-*`, which differs +// from their Gradle project names (`t5`, `vec2text`), so auto-substitution can't match — +// map the coordinates to the local projects explicitly. +includeBuild("../../SKaiNET-transformers") { + dependencySubstitution { + substitute(module("sk.ainet.transformers:skainet-transformers-inference-t5")) + .using(project(":llm-inference:t5")) + substitute(module("sk.ainet.transformers:skainet-transformers-inference-vec2text")) + .using(project(":llm-inference:vec2text")) + } +} + +include(":cli") From f61b811cb0db22ddf4ba6169d740f97ac3bb52d3 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sat, 11 Jul 2026 20:30:19 +0200 Subject: [PATCH 4/6] EmbeddingInversion: Compose desktop UI + consume released SKaiNET core 0.36.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add :app — a Compose for Desktop demo with two tabs: * Round trip: type text -> embed -> 768-d embedding strip -> invert, showing the per-step hypotheses with a live cosine bar as each correction completes. * Vector arithmetic: interpolate two sentence embeddings with a slider and invert the blend (text you never wrote, decoded from a vector). The UI drives the correction loop step by step via the public InversionModel / CorrectorModel APIs so each hypothesis renders as it's produced (CPU decode is slow). Runs the load + inference off the UI thread. - Now that SKaiNET core 0.36.0 is on Maven Central, drop the ../../SKaiNET composite: core resolves from Central (pinned by sk.ainet:skainet-bom:0.36.0). Only SKaiNET-transformers stays a composite (t5/vec2text not yet published). - Declare plugin versions once at the root (apply false) to silence the "Kotlin plugin loaded multiple times" warning across :app/:cli. Both :app and :cli compile; the CLI round-trip is verified end-to-end. Co-Authored-By: Claude Opus 4.8 --- EmbeddingInversion/README.md | 33 ++- EmbeddingInversion/app/build.gradle.kts | 47 ++++ .../sk/ainet/samples/vec2text/ui/Engine.kt | 117 +++++++++ .../sk/ainet/samples/vec2text/ui/Main.kt | 242 ++++++++++++++++++ EmbeddingInversion/build.gradle.kts | 8 +- EmbeddingInversion/cli/build.gradle.kts | 17 +- EmbeddingInversion/settings.gradle.kts | 15 +- 7 files changed, 450 insertions(+), 29 deletions(-) create mode 100644 EmbeddingInversion/app/build.gradle.kts create mode 100644 EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt create mode 100644 EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt diff --git a/EmbeddingInversion/README.md b/EmbeddingInversion/README.md index 78c0e0b..f77731f 100644 --- a/EmbeddingInversion/README.md +++ b/EmbeddingInversion/README.md @@ -59,22 +59,33 @@ The parity tests (below) compare against golden tensors dumped from the referenc `GtrEmbedderParityTest` needs token ids + the mean-pooled embedding from `gtr-t5-base`; `Vec2TextRoundTripTest` needs only the converted weights above. -## Run it (composite build — no release needed) +## Run it -This project **`includeBuild`s the sibling `../../SKaiNET` and `../../SKaiNET-transformers` -checkouts**, so it compiles against the local `t5` / `vec2text` source directly — no waiting -for a published `skainet-transformers` release. Just put the converted weights in `models/` and: +Put the converted weights in `models/`, then either: ```bash +# Desktop GUI (Round trip + Vector arithmetic tabs): +./gradlew :app:run + +# Or the CLI: ./gradlew :cli:run --args="jack morris is a phd student at cornell tech in new york city" ``` -Environment knobs: `VEC2TEXT_MODELS_DIR` (default `./models`), `VEC2TEXT_STEPS` (default 5). +Environment knobs: `VEC2TEXT_MODELS_DIR` (default `../models` for the app, `./models` for the +CLI), `VEC2TEXT_STEPS` (CLI, default 5). + +### Build setup + +- **SKaiNET core** is consumed from Maven Central (**0.36.0**, pinned by the `sk.ainet:skainet-bom` + platform) — no local `SKaiNET` checkout needed. +- **`t5` / `vec2text`** are not yet published, so **`SKaiNET-transformers` is a composite build**: + `settings.gradle.kts` `includeBuild`s `../../SKaiNET-transformers` and maps the two + `skainet-transformers-inference-*` coordinates to the local `:llm-inference:t5` / `:vec2text` + projects (their publish artifactId differs from the Gradle project name, so auto-substitution + can't match them). Uses Gradle 9.6.1 to match that build. -The composite requires the sibling checkouts at `../../SKaiNET` and `../../SKaiNET-transformers` -and uses Gradle 9.6.1 (matching them). SKaiNET core modules auto-substitute; the two -`skainet-transformers-inference-*` coordinates are mapped explicitly in `settings.gradle.kts` -(their publish artifactId differs from the Gradle project name). +Once `skainet-transformers` is released, drop the composite and depend on the published +coordinates directly. > Reconstruction quality scales with correction `steps`; greedy + few steps + fp16 can produce > rough or ``-laden output on short inputs. Beam search and a decode KV-cache (much faster, @@ -98,8 +109,8 @@ composite build. Planned tabs: | M1 T5 encoder + GTR embedder | ✅ verified (cosine 0.99999985 vs reference) | | M2 inversion (single-shot) | ✅ working end-to-end | | M3 corrector loop | ✅ working end-to-end | -| M4 runnable CLI (composite build) | ✅ `./gradlew :cli:run` against local modules | -| M4 Compose demo app | ⏳ next (reuses the same composite build) | +| M4 runnable CLI (composite build) | ✅ `./gradlew :cli:run` | +| M4 Compose desktop app | ✅ `./gradlew :app:run` — Round trip + Vector arithmetic tabs | | M5 beam search + KV-cache speedup | ⏳ follow-up | Current decoding is greedy with a no-KV-cache O(L²) loop — correct but slow on CPU. Beam search diff --git a/EmbeddingInversion/app/build.gradle.kts b/EmbeddingInversion/app/build.gradle.kts new file mode 100644 index 0000000..32dd9e6 --- /dev/null +++ b/EmbeddingInversion/app/build.gradle.kts @@ -0,0 +1,47 @@ +import org.jetbrains.compose.desktop.application.dsl.TargetFormat + +plugins { + kotlin("jvm") + id("org.jetbrains.compose") + id("org.jetbrains.kotlin.plugin.compose") +} + +repositories { + google() + mavenCentral() +} + +kotlin { + jvmToolchain(21) +} + +dependencies { + // SKaiNET core from Maven Central, version aligned by the published BOM (0.36.0). + implementation(platform("sk.ainet:skainet-bom:0.36.0")) + implementation("sk.ainet.core:skainet-lang-core") + implementation("sk.ainet.core:skainet-backend-cpu") + implementation("sk.ainet.core:skainet-io-core") + implementation("sk.ainet.core:skainet-io-safetensors") + + // t5 / vec2text — substituted with the local SKaiNET-transformers projects (composite build). + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.35.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.35.0") + + implementation("org.jetbrains.kotlinx:kotlinx-coroutines-swing:1.11.0") + implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") + + implementation(compose.desktop.currentOs) + implementation(compose.material3) +} + +compose.desktop { + application { + mainClass = "sk.ainet.samples.vec2text.ui.MainKt" + jvmArgs += listOf("-Xmx4g") + nativeDistributions { + targetFormats(TargetFormat.Dmg, TargetFormat.Msi, TargetFormat.Deb) + packageName = "EmbeddingInversion" + packageVersion = "1.0.0" + } + } +} diff --git a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt new file mode 100644 index 0000000..1d00de1 --- /dev/null +++ b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt @@ -0,0 +1,117 @@ +package sk.ainet.samples.vec2text.ui + +import kotlinx.serialization.json.Json +import kotlinx.serialization.json.jsonObject +import sk.ainet.context.DirectCpuExecutionContext +import sk.ainet.io.JvmRandomAccessSource +import sk.ainet.io.safetensors.SafeTensorsParametersLoader +import sk.ainet.io.tokenizer.SentencePieceTokenizer +import sk.ainet.lang.tensor.Tensor +import sk.ainet.lang.tensor.plus +import sk.ainet.lang.tensor.times +import sk.ainet.lang.types.FP32 +import sk.ainet.models.t5.GtrEmbedder +import sk.ainet.models.t5.T5Config +import sk.ainet.models.t5.T5Runtime +import sk.ainet.models.t5.loadT5Weights +import sk.ainet.models.vec2text.CorrectorModel +import sk.ainet.models.vec2text.InversionModel +import sk.ainet.models.vec2text.Vec2TextInverter +import sk.ainet.models.vec2text.Vec2TextWeightLoader +import java.io.File + +/** One correction step surfaced to the UI. */ +data class Step(val step: Int, val text: String, val cosine: Float) + +/** + * Loads the gtr-base checkpoints once and drives the vec2text inversion loop **step by step** + * (using the public InversionModel / CorrectorModel APIs) so the UI can render each hypothesis + * as it's produced — the full loop takes minutes on CPU. + */ +class Vec2TextEngine private constructor( + private val embedder: GtrEmbedder, + private val inversion: InversionModel, + private val corrector: CorrectorModel, + private val sp: SentencePieceTokenizer, + private val cfg: T5Config, +) { + private fun encodeForEmbedder(text: String): IntArray { + val ids = sp.encode(text).take(cfg.maxSeqLength - 1).toMutableList() + ids.add(cfg.eosTokenId) + return ids.toIntArray() + } + + private fun decode(ids: IntArray): String = + sp.decode(ids.filter { it != 0 && it != cfg.eosTokenId }.toIntArray()) + + /** Embed [text] into its `[768]` GTR sentence embedding. */ + fun embed(text: String): Tensor = embedder.embed(encodeForEmbedder(text)) + + /** Raw float view of an embedding, for visualization. */ + fun toFloats(v: Tensor): FloatArray = v.data.copyToFloatArray() + + /** Linear interpolation `(1-alpha)*a + alpha*b` of two embeddings. */ + fun interpolate(a: Tensor, b: Tensor, alpha: Float): Tensor = + (a * (1f - alpha)) + (b * alpha) + + /** + * Invert [target] into text over [steps] correction rounds, calling [onStep] after the + * initial hypothesis (step 0) and each correction. Returns the best-cosine hypothesis. + */ + fun invert(target: Tensor, steps: Int, onStep: (Step) -> Unit): Step { + var hypIds = inversion.invert(target, maxLength = cfg.maxSeqLength) + var hypText = decode(hypIds) + var cos = cosineOf(target, hypText) + var best = Step(0, hypText, cos) + onStep(best) + + for (s in 1..steps) { + val hypEmb = embed(hypText) + hypIds = corrector.correct(target, hypEmb, hypIds, maxLength = cfg.maxSeqLength) + hypText = decode(hypIds) + cos = cosineOf(target, hypText) + val step = Step(s, hypText, cos) + onStep(step) + if (cos > best.cosine) best = step + } + return best + } + + private fun cosineOf(target: Tensor, text: String): Float = + Vec2TextInverter.cosine(target, embed(text)) + + companion object { + val REQUIRED = listOf( + "tokenizer.json", "gtr_encoder.safetensors", "inversion.safetensors", "corrector.safetensors", + ) + + /** Resolve the models dir: `$VEC2TEXT_MODELS_DIR`, else `../models`, else `models`. */ + fun modelsDir(): File { + System.getenv("VEC2TEXT_MODELS_DIR")?.let { return File(it) } + for (c in listOf("../models", "models")) { + val f = File(c) + if (File(f, "tokenizer.json").exists()) return f + } + return File("../models") + } + + fun missing(dir: File): List = REQUIRED.filterNot { File(dir, it).exists() } + + /** Blocking load of all three models (call off the UI thread). */ + suspend fun load(dir: File): Vec2TextEngine { + val ctx = DirectCpuExecutionContext() + val cfg = T5Config() + fun loader(name: String) = + SafeTensorsParametersLoader(sourceProvider = { JvmRandomAccessSource.open(File(dir, name).toString()) }) + + val gtr = loadT5Weights(loader("gtr_encoder.safetensors"), ctx, FP32::class, cfg, "", withDecoder = false) + val embedder = GtrEmbedder(T5Runtime(ctx, gtr, FP32::class)) + val inversion = InversionModel(ctx, Vec2TextWeightLoader.loadInversion(loader("inversion.safetensors"), ctx, FP32::class, cfg), FP32::class) + val corrector = CorrectorModel(ctx, Vec2TextWeightLoader.loadCorrector(loader("corrector.safetensors"), ctx, FP32::class, cfg), FP32::class) + val sp = SentencePieceTokenizer.fromTokenizerJson( + Json.parseToJsonElement(File(dir, "tokenizer.json").readText()).jsonObject + ) + return Vec2TextEngine(embedder, inversion, corrector, sp, cfg) + } + } +} diff --git a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt new file mode 100644 index 0000000..8814138 --- /dev/null +++ b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt @@ -0,0 +1,242 @@ +package sk.ainet.samples.vec2text.ui + +import androidx.compose.foundation.Canvas +import androidx.compose.foundation.layout.Arrangement +import androidx.compose.foundation.layout.Column +import androidx.compose.foundation.layout.Row +import androidx.compose.foundation.layout.Spacer +import androidx.compose.foundation.layout.fillMaxSize +import androidx.compose.foundation.layout.fillMaxWidth +import androidx.compose.foundation.layout.height +import androidx.compose.foundation.layout.padding +import androidx.compose.foundation.rememberScrollState +import androidx.compose.foundation.verticalScroll +import androidx.compose.material3.Button +import androidx.compose.material3.CircularProgressIndicator +import androidx.compose.material3.MaterialTheme +import androidx.compose.material3.OutlinedTextField +import androidx.compose.material3.Slider +import androidx.compose.material3.Tab +import androidx.compose.material3.TabRow +import androidx.compose.material3.Text +import androidx.compose.runtime.Composable +import androidx.compose.runtime.LaunchedEffect +import androidx.compose.runtime.getValue +import androidx.compose.runtime.mutableStateListOf +import androidx.compose.runtime.mutableStateOf +import androidx.compose.runtime.remember +import androidx.compose.runtime.rememberCoroutineScope +import androidx.compose.runtime.setValue +import androidx.compose.ui.Alignment +import androidx.compose.ui.Modifier +import androidx.compose.ui.geometry.Offset +import androidx.compose.ui.graphics.Color +import androidx.compose.ui.unit.dp +import androidx.compose.ui.window.Window +import androidx.compose.ui.window.application +import kotlinx.coroutines.Dispatchers +import kotlinx.coroutines.launch +import kotlinx.coroutines.withContext +import sk.ainet.lang.types.FP32 +import kotlin.math.abs +import kotlin.math.min + +private sealed interface Load { + data object Loading : Load + data class Missing(val dir: String, val files: List) : Load + data class Ready(val engine: Vec2TextEngine) : Load + data class Failed(val message: String) : Load +} + +fun main() = application { + Window(onCloseRequest = ::exitApplication, title = "Embedding Inversion — vec2text on SKaiNET") { + MaterialTheme { App() } + } +} + +@Composable +private fun App() { + var load by remember { mutableStateOf(Load.Loading) } + LaunchedEffect(Unit) { + val dir = Vec2TextEngine.modelsDir() + val missing = Vec2TextEngine.missing(dir) + load = if (missing.isNotEmpty()) { + Load.Missing(dir.absolutePath, missing) + } else try { + Load.Ready(withContext(Dispatchers.Default) { Vec2TextEngine.load(dir) }) + } catch (e: Throwable) { + Load.Failed(e.message ?: e.toString()) + } + } + + Column(Modifier.fillMaxSize().padding(20.dp)) { + Text("Embedding Inversion", style = MaterialTheme.typography.headlineSmall) + Text( + "Decode a sentence embedding back into text (vec2text on SKaiNET). Embeddings are not anonymous.", + style = MaterialTheme.typography.bodyMedium, + ) + Spacer(Modifier.height(16.dp)) + when (val l = load) { + is Load.Loading -> Row(verticalAlignment = Alignment.CenterVertically) { + CircularProgressIndicator(Modifier.height(20.dp)); Spacer(Modifier.height(8.dp)) + Text(" Loading models (~1.1 GB)…") + } + is Load.Missing -> Text( + "Models not found in ${l.dir}\nMissing: ${l.files.joinToString()}\n" + + "Put the converted weights there or set VEC2TEXT_MODELS_DIR (see README).", + color = MaterialTheme.colorScheme.error, + ) + is Load.Failed -> Text("Failed to load models: ${l.message}", color = MaterialTheme.colorScheme.error) + is Load.Ready -> Tabs(l.engine) + } + } +} + +@Composable +private fun Tabs(engine: Vec2TextEngine) { + var tab by remember { mutableStateOf(0) } + val titles = listOf("Round trip", "Vector arithmetic") + TabRow(selectedTabIndex = tab) { + titles.forEachIndexed { i, t -> Tab(selected = tab == i, onClick = { tab = i }, text = { Text(t) }) } + } + Spacer(Modifier.height(16.dp)) + when (tab) { + 0 -> RoundTripTab(engine) + else -> VectorArithmeticTab(engine) + } +} + +@Composable +private fun RoundTripTab(engine: Vec2TextEngine) { + val scope = rememberCoroutineScope() + var text by remember { mutableStateOf("jack morris is a phd student at cornell tech in new york city") } + var steps by remember { mutableStateOf(5f) } + var running by remember { mutableStateOf(false) } + var vector by remember { mutableStateOf(null) } + val trace = remember { mutableStateListOf() } + + Column(Modifier.verticalScroll(rememberScrollState())) { + OutlinedTextField(text, { text = it }, Modifier.fillMaxWidth(), label = { Text("Text to embed & invert") }) + StepsSlider(steps) { steps = it } + RunButton("Embed → invert", running) { + running = true; trace.clear(); vector = null + scope.launch { + withContext(Dispatchers.Default) { + val target = engine.embed(text) + vector = engine.toFloats(target) + engine.invert(target, steps.toInt()) { trace.add(it) } + } + running = false + } + } + vector?.let { EmbeddingStrip(it) } + Reconstruction(original = text, trace = trace) + } +} + +@Composable +private fun VectorArithmeticTab(engine: Vec2TextEngine) { + val scope = rememberCoroutineScope() + var a by remember { mutableStateOf("the weather in paris is cold and rainy") } + var b by remember { mutableStateOf("the food in tokyo is fresh and delicious") } + var alpha by remember { mutableStateOf(0.5f) } + var running by remember { mutableStateOf(false) } + var vector by remember { mutableStateOf(null) } + val trace = remember { mutableStateListOf() } + + Column(Modifier.verticalScroll(rememberScrollState())) { + OutlinedTextField(a, { a = it }, Modifier.fillMaxWidth(), label = { Text("Sentence A") }) + OutlinedTextField(b, { b = it }, Modifier.fillMaxWidth(), label = { Text("Sentence B") }) + Text("Interpolation A ${"%.2f".format(1 - alpha)} ↔ B ${"%.2f".format(alpha)}") + Slider(alpha, { alpha = it }, valueRange = 0f..1f) + RunButton("Invert interpolated embedding", running) { + running = true; trace.clear(); vector = null + scope.launch { + withContext(Dispatchers.Default) { + val target = engine.interpolate(engine.embed(a), engine.embed(b), alpha) + vector = engine.toFloats(target) + engine.invert(target, 5) { trace.add(it) } + } + running = false + } + } + Text( + "Inverting a blend of two sentence embeddings — text you never wrote, decoded from a vector.", + style = MaterialTheme.typography.bodySmall, + ) + vector?.let { EmbeddingStrip(it) } + Reconstruction(original = "(interpolated embedding)", trace = trace) + } +} + +@Composable +private fun StepsSlider(steps: Float, onChange: (Float) -> Unit) { + Text("Correction steps: ${steps.toInt()}") + Slider(steps, onChange, valueRange = 0f..20f, steps = 19) +} + +@Composable +private fun RunButton(label: String, running: Boolean, onClick: () -> Unit) { + Row(verticalAlignment = Alignment.CenterVertically) { + Button(onClick = onClick, enabled = !running) { Text(label) } + if (running) { + Spacer(Modifier.height(8.dp)) + CircularProgressIndicator(Modifier.height(18.dp).padding(start = 12.dp)) + Text(" working… (CPU decode is slow)") + } + } + Spacer(Modifier.height(12.dp)) +} + +/** A horizontal strip visualizing the 768-d embedding: one column per (downsampled) dim, red<0= 0) Color(0f, 0.35f, 1f, value.coerceIn(0f, 1f)) + else Color(1f, 0.2f, 0.2f, (-value).coerceIn(0f, 1f)) + drawRect(c, topLeft = Offset(i * cw, 0f), size = androidx.compose.ui.geometry.Size(cw + 1, size.height)) + } + } + Spacer(Modifier.height(12.dp)) +} + +@Composable +private fun Reconstruction(original: String, trace: List) { + if (trace.isEmpty()) return + Spacer(Modifier.height(12.dp)) + Text("original: $original", style = MaterialTheme.typography.bodyMedium) + val best = trace.maxByOrNull { it.cosine } + if (best != null) { + Text("reconstructed: ${best.text}", style = MaterialTheme.typography.titleMedium) + Text("best cosine: ${"%.4f".format(best.cosine)}") + } + Spacer(Modifier.height(8.dp)) + Text("steps", style = MaterialTheme.typography.labelMedium) + Column { + for (s in trace) { + Row(Modifier.fillMaxWidth().padding(vertical = 2.dp), verticalAlignment = Alignment.CenterVertically) { + Text("${s.step}", Modifier.height(20.dp)) + CosineBar(s.cosine) + Text(" ${"%.3f".format(s.cosine)} ${s.text}", style = MaterialTheme.typography.bodySmall) + } + } + } +} + +@Composable +private fun CosineBar(cosine: Float) { + Canvas(Modifier.height(12.dp).fillMaxWidth(0.18f).padding(horizontal = 8.dp)) { + drawRect(Color(0.85f, 0.85f, 0.85f), size = size) + drawRect( + Color(0.2f, 0.6f, 0.3f), + size = androidx.compose.ui.geometry.Size(size.width * cosine.coerceIn(0f, 1f), size.height), + ) + } +} diff --git a/EmbeddingInversion/build.gradle.kts b/EmbeddingInversion/build.gradle.kts index 095f889..4e6287c 100644 --- a/EmbeddingInversion/build.gradle.kts +++ b/EmbeddingInversion/build.gradle.kts @@ -1 +1,7 @@ -// Root build for the EmbeddingInversion example. All real config lives in :cli. +// Root build for the EmbeddingInversion example. Plugin versions are declared once here +// (apply false) so the :cli and :app subprojects can apply them without repeating versions. +plugins { + kotlin("jvm") version "2.4.0" apply false + id("org.jetbrains.compose") version "1.10.1" apply false + id("org.jetbrains.kotlin.plugin.compose") version "2.4.0" apply false +} diff --git a/EmbeddingInversion/cli/build.gradle.kts b/EmbeddingInversion/cli/build.gradle.kts index 728a59c..7fa44c0 100644 --- a/EmbeddingInversion/cli/build.gradle.kts +++ b/EmbeddingInversion/cli/build.gradle.kts @@ -1,5 +1,5 @@ plugins { - kotlin("jvm") version "2.4.0" + kotlin("jvm") application } @@ -13,14 +13,17 @@ kotlin { } dependencies { - // Substituted with the local ../../SKaiNET-transformers projects via the composite build. + // SKaiNET core: version aligned by the published BOM (0.36.0), resolved from Maven Central. + implementation(platform("sk.ainet:skainet-bom:0.36.0")) + implementation("sk.ainet.core:skainet-lang-core") + implementation("sk.ainet.core:skainet-backend-cpu") + implementation("sk.ainet.core:skainet-io-core") + implementation("sk.ainet.core:skainet-io-safetensors") + + // t5 / vec2text: substituted with the local ../../SKaiNET-transformers projects (composite); + // the version here is ignored by the substitution. implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.35.0") implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.35.0") - // Substituted with the local ../../SKaiNET projects. - implementation("sk.ainet.core:skainet-lang-core:0.35.0") - implementation("sk.ainet.core:skainet-backend-cpu:0.35.0") - implementation("sk.ainet.core:skainet-io-core:0.35.0") - implementation("sk.ainet.core:skainet-io-safetensors:0.35.0") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.11.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") diff --git a/EmbeddingInversion/settings.gradle.kts b/EmbeddingInversion/settings.gradle.kts index 2ebe288..88d9f37 100644 --- a/EmbeddingInversion/settings.gradle.kts +++ b/EmbeddingInversion/settings.gradle.kts @@ -15,17 +15,11 @@ dependencyResolutionManagement { rootProject.name = "embedding-inversion" -// Composite build: consume the local SKaiNET + SKaiNET-transformers checkouts directly -// (source), so this example builds against the just-merged t5 / vec2text modules without -// waiting for a published release. Gradle substitutes every `sk.ainet.core:*` and -// `sk.ainet.transformers:*` dependency with the matching local project by coordinate. +// SKaiNET core is consumed from Maven Central (0.36.0, pinned by the skainet BOM in +// cli/build.gradle.kts) — no local ../../SKaiNET checkout needed. // -// Both are included here at the root so transformers resolves `sk.ainet.core:*` from this -// same composite — do NOT also set `useLocalSkainet` in the transformers build, or SKaiNET -// would be included twice. -// SKaiNET core modules auto-substitute (project name == published artifactId). -includeBuild("../../SKaiNET") -// The t5 / vec2text modules publish as `skainet-transformers-inference-*`, which differs +// The t5 / vec2text modules are not yet published, so SKaiNET-transformers stays a composite +// build. Its modules publish as `skainet-transformers-inference-*`, which differs // from their Gradle project names (`t5`, `vec2text`), so auto-substitution can't match — // map the coordinates to the local projects explicitly. includeBuild("../../SKaiNET-transformers") { @@ -38,3 +32,4 @@ includeBuild("../../SKaiNET-transformers") { } include(":cli") +include(":app") From 533646dedde5ecdcdd1e8ff635bc67389a6d90a0 Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Sun, 12 Jul 2026 15:17:06 +0200 Subject: [PATCH 5/6] EmbeddingInversion: consume everything from Maven Central (0.36.0), drop composite SKaiNET-transformers 0.36.0 (incl. the t5 / vec2text modules) is now on Maven Central, so the example no longer needs any composite build or local checkout: - settings.gradle.kts: remove the SKaiNET-transformers includeBuild + dependency substitutions. Plain project now. - :cli and :app: depend on skainet-transformers-inference-{t5,vec2text}:0.36.0 directly; SKaiNET core stays aligned via sk.ainet:skainet-bom:0.36.0. - README: drop composite instructions; document pure Maven Central consumption. Verified: :cli and :app compile from Central with --refresh-dependencies, and the CLI round-trip runs end-to-end against the published artifacts. Co-Authored-By: Claude Opus 4.8 --- EmbeddingInversion/README.md | 26 +++++++++---------------- EmbeddingInversion/app/build.gradle.kts | 6 +++--- EmbeddingInversion/cli/build.gradle.kts | 7 +++---- EmbeddingInversion/settings.gradle.kts | 17 ++-------------- 4 files changed, 17 insertions(+), 39 deletions(-) diff --git a/EmbeddingInversion/README.md b/EmbeddingInversion/README.md index f77731f..9b37180 100644 --- a/EmbeddingInversion/README.md +++ b/EmbeddingInversion/README.md @@ -76,30 +76,22 @@ CLI), `VEC2TEXT_STEPS` (CLI, default 5). ### Build setup -- **SKaiNET core** is consumed from Maven Central (**0.36.0**, pinned by the `sk.ainet:skainet-bom` - platform) — no local `SKaiNET` checkout needed. -- **`t5` / `vec2text`** are not yet published, so **`SKaiNET-transformers` is a composite build**: - `settings.gradle.kts` `includeBuild`s `../../SKaiNET-transformers` and maps the two - `skainet-transformers-inference-*` coordinates to the local `:llm-inference:t5` / `:vec2text` - projects (their publish artifactId differs from the Gradle project name, so auto-substitution - can't match them). Uses Gradle 9.6.1 to match that build. - -Once `skainet-transformers` is released, drop the composite and depend on the published -coordinates directly. +Everything is consumed from **Maven Central at 0.36.0** — no composite build, no local +checkouts. SKaiNET core is version-aligned by the `sk.ainet:skainet-bom` platform; the +inversion models come from `sk.ainet.transformers:skainet-transformers-inference-{t5,vec2text}`. > Reconstruction quality scales with correction `steps`; greedy + few steps + fp16 can produce > rough or ``-laden output on short inputs. Beam search and a decode KV-cache (much faster, > closer) are the M5 follow-ups. -## Planned Compose Multiplatform demo (`app/`) +## Compose desktop app (`app/`) -A `GloVeEmbeddings`-style Compose app (desktop JVM first) is the next step, reusing the same -composite build. Planned tabs: +`./gradlew :app:run` opens a Compose for Desktop window with two tabs: -- **Round trip** — type text → embed → show the 768-d vector → invert → compare original vs - reconstruction, with the per-step hypothesis + cosine sparkline. +- **Round trip** — type text → embed → 768-d embedding strip → invert; the per-step hypotheses + stream in with a live cosine bar as each correction completes. - **Vector arithmetic** — interpolate two sentence embeddings with a slider and invert the - midpoint live (why inversion matters for privacy). + blend (text you never wrote, decoded from a vector — why inversion matters for privacy). ## Status @@ -109,7 +101,7 @@ composite build. Planned tabs: | M1 T5 encoder + GTR embedder | ✅ verified (cosine 0.99999985 vs reference) | | M2 inversion (single-shot) | ✅ working end-to-end | | M3 corrector loop | ✅ working end-to-end | -| M4 runnable CLI (composite build) | ✅ `./gradlew :cli:run` | +| M4 runnable CLI | ✅ `./gradlew :cli:run` (Maven Central 0.36.0) | | M4 Compose desktop app | ✅ `./gradlew :app:run` — Round trip + Vector arithmetic tabs | | M5 beam search + KV-cache speedup | ⏳ follow-up | diff --git a/EmbeddingInversion/app/build.gradle.kts b/EmbeddingInversion/app/build.gradle.kts index 32dd9e6..03c74d9 100644 --- a/EmbeddingInversion/app/build.gradle.kts +++ b/EmbeddingInversion/app/build.gradle.kts @@ -23,9 +23,9 @@ dependencies { implementation("sk.ainet.core:skainet-io-core") implementation("sk.ainet.core:skainet-io-safetensors") - // t5 / vec2text — substituted with the local SKaiNET-transformers projects (composite build). - implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.35.0") - implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.35.0") + // t5 / vec2text — published on Maven Central. + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.36.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.36.0") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-swing:1.11.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") diff --git a/EmbeddingInversion/cli/build.gradle.kts b/EmbeddingInversion/cli/build.gradle.kts index 7fa44c0..17823a4 100644 --- a/EmbeddingInversion/cli/build.gradle.kts +++ b/EmbeddingInversion/cli/build.gradle.kts @@ -20,10 +20,9 @@ dependencies { implementation("sk.ainet.core:skainet-io-core") implementation("sk.ainet.core:skainet-io-safetensors") - // t5 / vec2text: substituted with the local ../../SKaiNET-transformers projects (composite); - // the version here is ignored by the substitution. - implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.35.0") - implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.35.0") + // t5 / vec2text — published on Maven Central. + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.36.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.36.0") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.11.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") diff --git a/EmbeddingInversion/settings.gradle.kts b/EmbeddingInversion/settings.gradle.kts index 88d9f37..1342fd2 100644 --- a/EmbeddingInversion/settings.gradle.kts +++ b/EmbeddingInversion/settings.gradle.kts @@ -15,21 +15,8 @@ dependencyResolutionManagement { rootProject.name = "embedding-inversion" -// SKaiNET core is consumed from Maven Central (0.36.0, pinned by the skainet BOM in -// cli/build.gradle.kts) — no local ../../SKaiNET checkout needed. -// -// The t5 / vec2text modules are not yet published, so SKaiNET-transformers stays a composite -// build. Its modules publish as `skainet-transformers-inference-*`, which differs -// from their Gradle project names (`t5`, `vec2text`), so auto-substitution can't match — -// map the coordinates to the local projects explicitly. -includeBuild("../../SKaiNET-transformers") { - dependencySubstitution { - substitute(module("sk.ainet.transformers:skainet-transformers-inference-t5")) - .using(project(":llm-inference:t5")) - substitute(module("sk.ainet.transformers:skainet-transformers-inference-vec2text")) - .using(project(":llm-inference:vec2text")) - } -} +// Everything is consumed from Maven Central now: SKaiNET core + SKaiNET-transformers +// (incl. the t5 / vec2text modules) are all published at 0.36.0. No composite build. include(":cli") include(":app") From 09c6fc659b72b84c1d1881ea82ea5294ab70a01c Mon Sep 17 00:00:00 2001 From: Michal Harakal Date: Tue, 14 Jul 2026 13:35:23 +0200 Subject: [PATCH 6/6] EmbeddingInversion: enable beam search via transformers 0.37.0 (local publish) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Consume skainet-transformers-inference-{t5,vec2text} 0.37.0 (adds beam search) from a scoped mavenLocal until it lands on Central; SKaiNET core stays on Central 0.36.0. - CLI: VEC2TEXT_BEAM (sequence beam width) + VEC2TEXT_TOKEN_BEAMS env knobs. - App: a "Beam width" slider; the engine runs streaming sequence-level beam (inversion.invertBeam + corrector.correctBeam ranked by cosine each round), emitting the best-of-beam hypothesis per step so the UI still updates live. - README: beam usage + the publishToMavenLocal command to reproduce 0.37.0. Verified end-to-end against the local 0.37.0 artifacts: beam ×3 reconstructs "...ph.d. technologist ... Cornell University" at cosine 0.82 (vs ~0.77 greedy) on the README example. Co-Authored-By: Claude Opus 4.8 --- EmbeddingInversion/README.md | 37 ++++++++++--- EmbeddingInversion/app/build.gradle.kts | 9 ++- .../sk/ainet/samples/vec2text/ui/Engine.kt | 55 ++++++++++++++++--- .../sk/ainet/samples/vec2text/ui/Main.kt | 11 +++- EmbeddingInversion/cli/build.gradle.kts | 9 ++- .../kotlin/sk/ainet/samples/vec2text/Main.kt | 7 ++- EmbeddingInversion/settings.gradle.kts | 1 + 7 files changed, 103 insertions(+), 26 deletions(-) diff --git a/EmbeddingInversion/README.md b/EmbeddingInversion/README.md index 9b37180..b174d4f 100644 --- a/EmbeddingInversion/README.md +++ b/EmbeddingInversion/README.md @@ -71,18 +71,36 @@ Put the converted weights in `models/`, then either: ./gradlew :cli:run --args="jack morris is a phd student at cornell tech in new york city" ``` -Environment knobs: `VEC2TEXT_MODELS_DIR` (default `../models` for the app, `./models` for the -CLI), `VEC2TEXT_STEPS` (CLI, default 5). +CLI environment knobs: `VEC2TEXT_MODELS_DIR` (default `../models` for the app, `./models` for +the CLI), `VEC2TEXT_STEPS` (default 5), `VEC2TEXT_BEAM` (sequence beam width, default 1 = greedy), +`VEC2TEXT_TOKEN_BEAMS` (T5 token-level beam, default = `VEC2TEXT_BEAM`). The GUI has a **Beam +width** slider. Example: + +```bash +VEC2TEXT_BEAM=3 ./gradlew :cli:run --args="jack morris is a phd student at cornell tech in new york city" +# beam ×3 lifts cosine ~0.77 → ~0.82 vs greedy at one step (and is proportionally slower). +``` ### Build setup -Everything is consumed from **Maven Central at 0.36.0** — no composite build, no local -checkouts. SKaiNET core is version-aligned by the `sk.ainet:skainet-bom` platform; the -inversion models come from `sk.ainet.transformers:skainet-transformers-inference-{t5,vec2text}`. +SKaiNET core comes from **Maven Central 0.36.0** (aligned by the `sk.ainet:skainet-bom` platform). +The inversion models `sk.ainet.transformers:skainet-transformers-inference-{t5,vec2text}` are used +at **0.37.0** (adds beam search); until that lands on Central it's resolved from the **local Maven +cache** (a scoped `mavenLocal`). To (re)publish it, from a `SKaiNET-transformers` checkout with +`VERSION_NAME=0.37.0`: + +```bash +./gradlew :llm-bom:publishToMavenLocal :transformer-core:publishToMavenLocal \ + :llm-core:publishToMavenLocal :llm-inference:t5:publishToMavenLocal \ + :llm-inference:vec2text:publishToMavenLocal -PsignAllPublications=false +``` + +No composite build, no source checkouts. Once transformers 0.37.0 is on Central, drop the +`mavenLocal` repository from `cli/` and `app/` `build.gradle.kts`. -> Reconstruction quality scales with correction `steps`; greedy + few steps + fp16 can produce -> rough or ``-laden output on short inputs. Beam search and a decode KV-cache (much faster, -> closer) are the M5 follow-ups. +> Reconstruction quality scales with correction `steps` and beam width; greedy + few steps + fp16 +> can produce rough or ``-laden output on short inputs. A decode KV-cache (much faster, +> compounding with beam) is the remaining M5 follow-up. ## Compose desktop app (`app/`) @@ -103,7 +121,8 @@ inversion models come from `sk.ainet.transformers:skainet-transformers-inference | M3 corrector loop | ✅ working end-to-end | | M4 runnable CLI | ✅ `./gradlew :cli:run` (Maven Central 0.36.0) | | M4 Compose desktop app | ✅ `./gradlew :app:run` — Round trip + Vector arithmetic tabs | -| M5 beam search + KV-cache speedup | ⏳ follow-up | +| M5 beam search | ✅ `VEC2TEXT_BEAM` / GUI slider (transformers 0.37.0) | +| M5 decode KV-cache speedup | ⏳ follow-up | Current decoding is greedy with a no-KV-cache O(L²) loop — correct but slow on CPU. Beam search and a KV cache (much faster, closer reconstructions) are the main follow-ups. diff --git a/EmbeddingInversion/app/build.gradle.kts b/EmbeddingInversion/app/build.gradle.kts index 03c74d9..140d0a9 100644 --- a/EmbeddingInversion/app/build.gradle.kts +++ b/EmbeddingInversion/app/build.gradle.kts @@ -9,6 +9,8 @@ plugins { repositories { google() mavenCentral() + // SKaiNET-transformers 0.37.0 (beam search) from the local Maven cache until it lands on Central. + mavenLocal { mavenContent { includeGroupAndSubgroups("sk.ainet") } } } kotlin { @@ -23,9 +25,10 @@ dependencies { implementation("sk.ainet.core:skainet-io-core") implementation("sk.ainet.core:skainet-io-safetensors") - // t5 / vec2text — published on Maven Central. - implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.36.0") - implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.36.0") + // t5 / vec2text 0.37.0 (beam search) — from the local Maven cache until it lands on Central + // (publish with: SKaiNET-transformers `./gradlew publishToMavenLocal -PsignAllPublications=false`). + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.37.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.37.0") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-swing:1.11.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") diff --git a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt index 1d00de1..f05b984 100644 --- a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt +++ b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Engine.kt @@ -57,26 +57,65 @@ class Vec2TextEngine private constructor( /** * Invert [target] into text over [steps] correction rounds, calling [onStep] after the * initial hypothesis (step 0) and each correction. Returns the best-cosine hypothesis. + * + * [beamWidth] > 1 or [tokenBeams] > 1 enables beam search (better reconstructions, slower): + * [tokenBeams] is the T5 token-level beam per generation; [beamWidth] keeps that many + * hypotheses across correction rounds, ranked by cosine to [target]. */ - fun invert(target: Tensor, steps: Int, onStep: (Step) -> Unit): Step { + fun invert( + target: Tensor, + steps: Int, + beamWidth: Int = 1, + tokenBeams: Int = 1, + onStep: (Step) -> Unit, + ): Step = + if (beamWidth <= 1 && tokenBeams <= 1) invertGreedy(target, steps, onStep) + else invertBeam(target, steps, beamWidth.coerceAtLeast(1), tokenBeams.coerceAtLeast(1), onStep) + + private fun invertGreedy(target: Tensor, steps: Int, onStep: (Step) -> Unit): Step { var hypIds = inversion.invert(target, maxLength = cfg.maxSeqLength) var hypText = decode(hypIds) - var cos = cosineOf(target, hypText) - var best = Step(0, hypText, cos) + var best = Step(0, hypText, cosineOf(target, hypText)) onStep(best) for (s in 1..steps) { - val hypEmb = embed(hypText) - hypIds = corrector.correct(target, hypEmb, hypIds, maxLength = cfg.maxSeqLength) + hypIds = corrector.correct(target, embed(hypText), hypIds, maxLength = cfg.maxSeqLength) hypText = decode(hypIds) - cos = cosineOf(target, hypText) - val step = Step(s, hypText, cos) + val step = Step(s, hypText, cosineOf(target, hypText)) onStep(step) - if (cos > best.cosine) best = step + if (step.cosine > best.cosine) best = step } return best } + /** Streaming sequence-level beam: emit the best-of-beam hypothesis after each round. */ + private fun invertBeam(target: Tensor, steps: Int, beamWidth: Int, tokenBeams: Int, onStep: (Step) -> Unit): Step { + var beams = rank(target, inversion.invertBeam(target, maxOf(beamWidth, tokenBeams), cfg.maxSeqLength)).take(beamWidth) + var best = Step(0, beams.first().text, beams.first().cos) + onStep(best) + + for (s in 1..steps) { + val pool = ArrayList() + for (b in beams) pool += corrector.correctBeam(target, embed(b.text), b.ids, tokenBeams, cfg.maxSeqLength) + beams = rank(target, pool).take(beamWidth) + val sb = beams.first() + val step = Step(s, sb.text, sb.cos) + onStep(step) + if (step.cosine > best.cosine) best = step + } + return best + } + + private class Cand(val ids: IntArray, val text: String, val cos: Float) + + private fun rank(target: Tensor, idsList: List): List = + idsList.asSequence() + .map { ids -> decode(ids) to ids } + .distinctBy { it.first } + .map { (text, ids) -> Cand(ids, text, cosineOf(target, text)) } + .sortedByDescending { it.cos } + .toList() + private fun cosineOf(target: Tensor, text: String): Float = Vec2TextInverter.cosine(target, embed(text)) diff --git a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt index 8814138..6eef2ba 100644 --- a/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt +++ b/EmbeddingInversion/app/src/main/kotlin/sk/ainet/samples/vec2text/ui/Main.kt @@ -111,6 +111,7 @@ private fun RoundTripTab(engine: Vec2TextEngine) { val scope = rememberCoroutineScope() var text by remember { mutableStateOf("jack morris is a phd student at cornell tech in new york city") } var steps by remember { mutableStateOf(5f) } + var beam by remember { mutableStateOf(1f) } var running by remember { mutableStateOf(false) } var vector by remember { mutableStateOf(null) } val trace = remember { mutableStateListOf() } @@ -118,13 +119,14 @@ private fun RoundTripTab(engine: Vec2TextEngine) { Column(Modifier.verticalScroll(rememberScrollState())) { OutlinedTextField(text, { text = it }, Modifier.fillMaxWidth(), label = { Text("Text to embed & invert") }) StepsSlider(steps) { steps = it } + BeamSlider(beam) { beam = it } RunButton("Embed → invert", running) { running = true; trace.clear(); vector = null scope.launch { withContext(Dispatchers.Default) { val target = engine.embed(text) vector = engine.toFloats(target) - engine.invert(target, steps.toInt()) { trace.add(it) } + engine.invert(target, steps.toInt(), beam.toInt(), beam.toInt()) { trace.add(it) } } running = false } @@ -175,6 +177,13 @@ private fun StepsSlider(steps: Float, onChange: (Float) -> Unit) { Slider(steps, onChange, valueRange = 0f..20f, steps = 19) } +@Composable +private fun BeamSlider(beam: Float, onChange: (Float) -> Unit) { + val w = beam.toInt() + Text("Beam width: ${if (w <= 1) "1 (greedy)" else "$w (slower, better)"}") + Slider(beam, onChange, valueRange = 1f..4f, steps = 2) +} + @Composable private fun RunButton(label: String, running: Boolean, onClick: () -> Unit) { Row(verticalAlignment = Alignment.CenterVertically) { diff --git a/EmbeddingInversion/cli/build.gradle.kts b/EmbeddingInversion/cli/build.gradle.kts index 17823a4..d140211 100644 --- a/EmbeddingInversion/cli/build.gradle.kts +++ b/EmbeddingInversion/cli/build.gradle.kts @@ -6,6 +6,8 @@ plugins { repositories { google() mavenCentral() + // SKaiNET-transformers 0.37.0 (beam search) from the local Maven cache until it lands on Central. + mavenLocal { mavenContent { includeGroupAndSubgroups("sk.ainet") } } } kotlin { @@ -20,9 +22,10 @@ dependencies { implementation("sk.ainet.core:skainet-io-core") implementation("sk.ainet.core:skainet-io-safetensors") - // t5 / vec2text — published on Maven Central. - implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.36.0") - implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.36.0") + // t5 / vec2text 0.37.0 (beam search) — from the local Maven cache until it lands on Central + // (publish with: SKaiNET-transformers `./gradlew publishToMavenLocal -PsignAllPublications=false`). + implementation("sk.ainet.transformers:skainet-transformers-inference-t5:0.37.0") + implementation("sk.ainet.transformers:skainet-transformers-inference-vec2text:0.37.0") implementation("org.jetbrains.kotlinx:kotlinx-coroutines-core:1.11.0") implementation("org.jetbrains.kotlinx:kotlinx-serialization-json:1.11.0") diff --git a/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt b/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt index 53fa9d0..3c0c726 100644 --- a/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt +++ b/EmbeddingInversion/cli/src/main/kotlin/sk/ainet/samples/vec2text/Main.kt @@ -33,6 +33,8 @@ fun main(args: Array) = runBlocking { "jack morris is a phd student at cornell tech in new york city" } val steps = System.getenv("VEC2TEXT_STEPS")?.toIntOrNull() ?: 5 + val beamWidth = System.getenv("VEC2TEXT_BEAM")?.toIntOrNull() ?: 1 + val tokenBeams = System.getenv("VEC2TEXT_TOKEN_BEAMS")?.toIntOrNull() ?: beamWidth val required = listOf("tokenizer.json", "gtr_encoder.safetensors", "inversion.safetensors", "corrector.safetensors") val missing = required.filterNot { File(modelsDir, it).exists() } @@ -66,9 +68,10 @@ fun main(args: Array) = runBlocking { sp.decode(ids.filter { it != 0 && it != cfg.eosTokenId }.toIntArray()) } - println("Inverting (≤$steps correction steps, greedy)…\n") + val mode = if (beamWidth > 1 || tokenBeams > 1) "beam ×$beamWidth, token-beams ×$tokenBeams" else "greedy" + println("Inverting (≤$steps correction steps, $mode)…\n") val result = Vec2TextInverter(embedder, inversion, corrector, codec) - .invert(text, numSteps = steps, maxLength = cfg.maxSeqLength) + .invert(text, numSteps = steps, maxLength = cfg.maxSeqLength, sequenceBeamWidth = beamWidth, tokenBeams = tokenBeams) println("original: $text") println("reconstructed: ${result.text}") diff --git a/EmbeddingInversion/settings.gradle.kts b/EmbeddingInversion/settings.gradle.kts index 1342fd2..bad86d7 100644 --- a/EmbeddingInversion/settings.gradle.kts +++ b/EmbeddingInversion/settings.gradle.kts @@ -7,6 +7,7 @@ pluginManagement { } dependencyResolutionManagement { + // Repositories are declared per-module (:cli, :app) so their own blocks apply; see there. repositories { google() mavenCentral()