Add TweetFeed Lookup expansion/hover module - #898
Merged
Conversation
Queries https://api.tweetfeed.live/v1/ioc (no API key) for an ip-src/ip-dst/ domain/hostname/url/md5/sha256 attribute and returns misp_standard results: microblog objects for the source tweets (reporter, hashtags, first/last seen), co-reported IOCs as attributes referenced with 'mentions', a domain-ip object for registration/hosting, and text/link attributes for AI context, IP network metadata, cross-feed corroboration and campaign membership. Nothing is tagged and no confidence score is asserted: the data is community reported and unverified. Tests mock requests.get with verbatim API captures (13 cases). Docs blocks added by hand following the generator layout. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Member
|
Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
New
expansion+hovermodule TweetFeed Lookup (misp_modules/modules/expansion/tweetfeed.py).TweetFeed is a free, CC0 feed of URLs, domains, IPs and MD5/SHA-256 hashes that security researchers share on X/Twitter, extracted every 15 minutes since 2021. MISP already consumes it as a bulk feed; this module adds the per-attribute half: given an
ip-src,ip-dst,domain,hostname,url,md5orsha256attribute it callsGET https://api.tweetfeed.live/v1/ioc?value=<v>(no API key, 365-day window plus an older archive) and returnsmisp_standardresults.Mapping
microblogobject (typeTwitter,url,usernameparsed from the tweet URL, onehashtagper tag,creation-datewhen unambiguous), referenced to the queried attribute withmentions;to_idsfalse) referenced from the microblog object withmentions;reg) -> adomain-ipobject (apex, IPs,registration-date, registrar / NS / ASN / age intext);text/linkattributes withdisable_correlation;microblogobjects built from the archived tweet id.Deliberately no tag and no confidence score are set from this data: it is community reported and not independently verified. The AI summary is labelled as LLM-generated in the attribute comment.
Tests and docs
tests/test_tweetfeed.py: 13 pytest cases that mockrequests.getwith verbatim captures of the live API (domain, url, ip, sha256, md5, corroborated, archive-only, miss, error paths, hover call without uuid, introspection/version). Same style astests/test_xposedornot.py.flake8clean at the project's 120-column limit.README.mdbullet and thedocumentation/README.md/documentation/mkdocs/expansion.mdblocks were added by hand following the generator layout; happy to regenerate if you prefer.Note for reviewers
While writing the tests I found that adding an object with
misp_event.add_object(**obj)drops the references added earlier withobj.add_reference(...): a standaloneMISPObjectdoes not serialiseObjectReference, so the unpacked dict never carries them. Passing the instance (misp_event.add_object(obj)) keeps them. This module uses the instance form; other modules that use the**objform afteradd_referencemay be affected.Author: Daniel López (TweetFeed maintainer).