Repository navigation
feat(fetchers): add public social media metadata support - #188
Merged
Merged
Conversation
4 tasks done
chaliy
added a commit
that referenced
this pull request
Nov 1, 2026
<!-- claude-projects-attribution: {"github_login":"chaliy"} -->
_Requested by **Михайло** · [project
thread](https://claude.ai/code/project/chan_019YEkBJuZMHJdXrjom86cGX?thread=cmsg_019YEkBJuZMHJdXrjom86cGXFSAmywce8BSJgNQ4YNsrks)_
## What changed
Release v0.7.0: workspace version 0.7.0, CHANGELOG section with
Highlights / Breaking Changes / What's Changed, knowledge log entry.
Two release-blocking fixes for embedding, found while wiring bashkit:
- **Markkit 0.3** (just published) so bashkit, which also embeds markkit
0.3, links one copy.
- **`fetchkit-cli` depends on `fetchkit` with `default-features =
false`** and gains matching `reqwest-tls` / `docs-pdf` features (default
on). Before, any `fetchkit-cli` dependency forced reqwest TLS
(aws-lc-sys) back on.
### Highlights
- Embed the exact `fetchkit` CLI with `fetchkit_cli::run(args, &dyn
CliHost)` (#189)
- TikTok, Vimeo, SoundCloud oEmbed metadata; YouTube Shorts/live/embed
URLs (#188)
- Build without a TLS stack when supplying your own `HttpTransport`
## Before / After
```
cargo tree -p fetchkit-cli --no-default-features -e normal -i aws-lc-sys
error: package ID specification `aws-lc-sys` did not match any packages
```
Default builds unchanged. `cargo test --workspace`, clippy (`-D
warnings`, also `fetchkit-cli --no-default-features`), fmt and OKF check
pass locally.
## Risk
- Low. Merging runs `release.yml` (tag, GitHub Release) and
`publish.yml` (`fetchkit`, then `fetchkit-cli`).
- Breaking (minor bump): `reqwest-tls` split, documented in CHANGELOG.
### Checklist
- [x] Unit tests are passed
- [x] Smoke tests are passed
- [x] Documentation is updated
- [x] Knowledge is up to date and not in conflict
---
_Generated by [Claude
Code](https://claude.ai/code/session_01Wy7MByPXsW4jHxfLeTfGEc)_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Fetch public TikTok and Vimeo videos and SoundCloud tracks/playlists as bounded Markdown metadata through their official oEmbed endpoints. The shared registry enables this automatically in Rust, CLI, MCP and Python. YouTube now recognizes Shorts, live, embed and privacy-enhanced embed URLs alongside watch/short links.
Titles, authors, thumbnails and descriptions appear when supplied; descriptions become Markdown. Embed HTML is ignored. Output explicitly states that media/transcripts are not retrieved by the new fetcher. TikTok currently supplies only minimal embed metadata, so it can return the platform and canonical URL without a title. Existing YouTube captions remain best-effort.
Why
Social pages commonly need JavaScript or show login/navigation shells. Public oEmbed metadata provides useful structured output without credentials or a browser. Supported URL matching stays narrow; unsupported pages retain the default HTML fetcher.
Before / After
Before: TikTok, Vimeo and SoundCloud had no specialized fetcher; YouTube Shorts/live/embed paths fell through to HTML fetching. YouTube matched watch and youtu.be URLs only.
After: real release-CLI runs (
fetchkit fetch URL --output json) returned:https://www.tiktok.com/@scout2015/video/6718335390845095173tiktok_videohttps://www.youtube.com/shorts/jNQXAC9IVRwyoutube_videohttps://vimeo.com/863362136vimeo_videohttps://soundcloud.com/forss/flickermoodsoundcloud_trackSoundCloud description output now includes
[https://www.dswshoes.com/tv_commercial.jsp?m=october2007](https://www.dswshoes.com/tv_commercial.jsp?m=october2007)rather than raw anchor HTML. The old Vimeo sample76979871returned HTTP 404 andVimeo oEmbed unavailable: HTTP 404, confirming failures are preserved.Validation: 597 workspace tests passed (2 existing ignored); all 18 live fetcher tests passed with no skips; formatting, Clippy
-D warnings, warning-free docs, release build excluding Python, and both OKF checks passed. Offline regression coverage includes registration, canonicalization, metadata rendering, malformed/incomplete JSON, output caps, HEAD, provider errors, policy enforcement and YouTube variants/query injection.Two stale live fixtures now use Rust’s current
mainbranch and httpbin’s dedicated HTML endpoint, retaining HTTP/content assertions.Risk
Checklist