Build a Twitter dataset for machine learning

#Where Twitter datasets come from

Every Twitter (X) dataset starts in one of two places: a collection someone already published, or tweets you collect yourself through an API. A published set is free and loads in minutes. A collection of your own matches your topic, language, dates and labels, because you choose them; our guide on how to scrape Twitter compares every way to pull tweets in 2026.

The best-known public sets were gathered between 2009 and 2022, while Twitter still gave researchers free access to its API, and several no longer hand out tweet text at all.

• Sentiment140

  Size

  1.6M tweets, 498 for testing

  Collected

  2009

  What you get today

  Text with positive or negative labels taken from emoticons

• TweetEval (Cardiff NLP)

  Size

  7 classification tasks

  Collected

  Benchmark from 2020

  What you get today

  Text and labels for sentiment, emotion, irony, hate, offensive language, emoji and stance

• COVID-19-TweetIDs

  Size

  Over 1B tweets

  Collected

  From January 2020

  What you get today

  Tweet IDs only

• ArchiveTeam Twitter archive

  Size

  About 16B records

  Collected

  2011 to 2022

  What you get today

  Raw JSON, no labels

• SNAP Twitter7 (Stanford)

  Size

  476M tweets

  Collected

  2009

  What you get today

  Nothing: withdrawn at Twitter's request

For a generic sentiment or emotion baseline, one of the labeled sets is the fastest start. The community-kept awesome-twitter-data list (opens in a new tab) tracks hundreds more and marks which ones still share text. For anything specific, such as reactions to your product, an event from this year, a language the archives skip or a label scheme of your own, you build the dataset.

#The catch with public Twitter datasets

Public Twitter datasets share four problems: they are old, most of them hold only IDs, those IDs decay every year, and their labels were made for someone else's task.

• Old text. Every tweet in Sentiment140 is 140 characters or shorter, because Twitter only raised the limit to 280 in November 2017. A model trained on 2009 language meets 2026 slang, topics and post lengths it has never seen.
• IDs instead of text. A dataset shared with others usually carries only post IDs and user IDs, so many research sets, the big COVID-19 collections among them, are lists of numbers. You get the text back by rehydrating the IDs: looking each one up again through an API.
• Decay. A tweet that was deleted, or whose author went private or got suspended, no longer comes back. The rows you lose are not random, so the rebuilt set leans toward accounts that are still active.
• Borrowed labels. Sentiment140's labels came from emoticons, not from people reading the tweets, and benchmark labels answer the benchmark's question, not yours.

For a model that has to understand this year's posts, that leaves one route: a fresh set, cleaned and labeled for one task.

#How to build a fresh Twitter dataset with Sorsa API

For a fresh dataset, use Sorsa API, a Twitter (X) API and scraper: one API key, X's search operators back to 2006, about 20 tweets per request from $0.07 per 1,000, and 20 requests per second on every plan, so a 1M-tweet topic set takes about 42 minutes to collect. Every tweet arrives with its author's full profile at no extra cost, and nothing touches an X account of yours.

Pick the endpoint by what defines your dataset:

• A topic, keywords or a date range: POST /search-tweets (opens in a new tab).
• One account's posts: POST /user-tweets (opens in a new tab), covered in our guide to download all tweets from a user.
• A hand-picked group of accounts, read as one X List: GET /list-tweets (opens in a new tab).
• Reply threads for conversation data: POST /comments (opens in a new tab).

1. Get a key. Sign up and copy your API key from the dashboard. The first 100 requests are free, with no card.

2. Write the query. Search operators shape the dataset before you pay for it: lang:en keeps one language, -filter:retweets drops copies, since: and until: fix the window. Test a query by hand in the free Twitter Advanced Search tool first.

3. Send a test request. One request returns about 20 tweets and a cursor to the next page.

   curl -X POST "https://api.sorsa.io/v3/search-tweets" \
     -H "ApiKey: YOUR_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"order": "latest",
          "query": "crashing lang:en -filter:retweets since:2026-10-01"}'
   

   {
     "tweets": [
       {
         "id": "2115807747265175863",
         "full_text": "The new beta keeps crashing on launch for me",
         "created_at": "2026-10-29T14:07:31Z",
         "lang": "en",
         "likes_count": 12,
         "reply_count": 3,
         "retweet_count": 0,
         "conversation_id_str": "2115807747265175863",
         "in_reply_to_tweet_id": null,
         "is_reply": false,
         "made_with_ai": false,
         "user": {
           "id": "1710221540156513285",
           "username": "maya_trains_models",
           "followers_count": 412,
           "created_at": "2023-10-06T09:12:44Z",
           "verified": false
         }
       }
     ],
     "next_cursor": "DAADDAABCgABHV5mIq0VsAoKAAIdXmUz8Kqg"
   }
   

   Show full response

4. Page through and save raw JSONL. Send next_cursor back until it comes back empty, and write every tweet untouched, one JSON object per line. Cleaning runs later on a copy, so you never pay to collect the same tweets twice.

   import json
   import time
   
   import requests
   
   URL = "https://api.sorsa.io/v3/search-tweets"
   HEADERS = {"ApiKey": "YOUR_API_KEY"}
   QUERY = "crashing lang:en -filter:retweets since:2026-10-01"
   
   
   def collect(query, path, limit=50_000):
       body = {"query": query, "order": "latest"}
       saved = 0
       with open(path, "a", encoding="utf-8") as out:
           while saved < limit:
               r = requests.post(URL, headers=HEADERS, json=body, timeout=30)
               if r.status_code == 429:  # over 20 requests per second
                   time.sleep(1)
                   continue
               r.raise_for_status()  # 403: no key, or no requests left
               page = r.json()
               for tweet in page.get("tweets", []):
                   tweet["_query"] = query
                   out.write(json.dumps(tweet, ensure_ascii=False) + "\n")
                   saved += 1
               if not page.get("next_cursor"):
                   break
               body["next_cursor"] = page["next_cursor"]
       return saved
   
   
   print(collect(QUERY, "raw_tweets.jsonl"), "tweets saved")
   

Run one query per class or topic, each into its own file, and the per-file counts tell you at once whether a class is too thin. To go further back than this year, the same endpoint reaches every public tweet since 2006; our guide to historical Twitter data covers date slicing for large backfills.

#Rehydrate a tweet ID dataset

To rebuild a published ID dataset, send its tweet IDs to POST /tweet-info-bulk (opens in a new tab) on Sorsa API, up to 100 per request, and keep whatever comes back. A list of 1M IDs costs from $14 and takes about 8 minutes at 20 requests per second; the same 1M posts cost $5,000 in post reads on the official X API.

import json

import requests

URL = "https://api.sorsa.io/v3/tweet-info-bulk"
HEADERS = {"ApiKey": "YOUR_API_KEY"}

with open("tweet_ids.txt", encoding="utf-8") as f:
    ids = list(dict.fromkeys(x.strip() for x in f if x.strip()))

found = set()
with open("rehydrated.jsonl", "w", encoding="utf-8") as out:
    for i in range(0, len(ids), 100):
        body = {"tweet_links": ids[i:i + 100]}
        r = requests.post(URL, headers=HEADERS, json=body, timeout=30)
        r.raise_for_status()
        for tweet in r.json().get("tweets", []):
            found.add(tweet["id"])
            out.write(json.dumps(tweet, ensure_ascii=False) + "\n")

missing = [x for x in ids if x not in found]
print(f"recovered {len(found)} of {len(ids)}; {len(missing)} gone")

Report the recovery rate with the dataset: how many IDs came back, out of how many, on which date. Two datasets rebuilt from the same IDs a year apart are different datasets, and a reader of your results needs to know which one you trained on.

Good to know

Keep tweet IDs as text from start to finish. An ID has 19 digits, while a spreadsheet keeps 15 significant digits and a JavaScript number about 16, so an ID that passes through Excel or JSON.parse comes back rounded and matches no tweet. Read ID files as strings, and save them in quotes.

#Clean and deduplicate the tweets

Clean a tweet dataset in four passes: drop retweets and exact duplicates by ID, collapse near-duplicate text, filter by language and author, and swap handles and links for placeholders. Each pass uses fields that come back in every tweet.

• id

  What it does in a dataset

  The primary key; store it as a string

• full_text

  What it does in a dataset

  The text you clean, label and train on

• lang

  What it does in a dataset

  Keeps each split in one language

• conversation_id_str

  What it does in a dataset

  Groups a thread, so the split can keep it together

• in_reply_to_tweet_id

  What it does in a dataset

  Links a reply to its parent for conversation pairs

• made_with_ai

  What it does in a dataset

  Marks posts X labels as made with AI; leave them out of human-text sets

• user.followers_count, user.created_at

  What it does in a dataset

  Author signals for spam and bot filters

• likes_count, reply_count

  What it does in a dataset

  A quality floor: posts nobody engaged with are often spam

Near-duplicates are the pass most pipelines skip. Retweets with a comment, copied posts, bot campaigns and the same complaint pasted by ten accounts all survive an ID check, and they inflate whatever they say. Normalize the text first (lowercase, no links, no handles, no punctuation), then compare word shingles: two posts that share most of their three-word runs say the same thing. The checker below runs that test on whatever you paste.

For text the model will read, replace each handle with @user and each link with http, the convention Cardiff NLP's tweet models use. The model then learns how people write, not which accounts and links appear most often. Keep the original full_text beside the cleaned one, so you can change the cleaning later without collecting again.

import hashlib
import json
import re


def norm(text):
    text = re.sub(r"https?://\S+", " ", text.lower())
    text = re.sub(r"^rt @\w+:?|@\w+", " ", text)
    text = re.sub(r"[^\w\s#]", " ", text)
    return " ".join(text.split())


seen, rows = set(), []
for line in open("raw_tweets.jsonl", encoding="utf-8"):
    t = json.loads(line)
    if t.get("retweeted_status") or t.get("made_with_ai"):
        continue
    clean = norm(t["full_text"])
    key = hashlib.sha1(clean.encode()).hexdigest()
    if key in seen or len(clean.split()) < 4:
        continue
    seen.add(key)
    rows.append(t)

The hash catches exact copies after normalization; for fuzzy matches at the scale of millions, MinHash with locality-sensitive hashing does the shingle comparison without comparing every pair.

#Label the dataset

Label a tweet dataset in two layers: a gold set of a few hundred tweets that people label by hand, and the bulk pre-labeled by a model and checked by a person where the model is unsure. The gold set never goes into training; it is the test that tells you whether the bulk labels are good enough.

Write the label rules down before anyone starts, with an example of each class and of the hard cases: sarcasm, questions, quotes of someone else's opinion. Two people labeling the same 100 tweets and comparing results show where the rules are vague. Store with each label who or what produced it (human, model, query), so you can drop or reweigh the weak ones later.

A search query can act as a weak label: tweets collected for "keeps crashing" lean negative. Use that only to bootstrap, because a query label teaches the model the query's words, not the sentiment.

#Split train and test without leaking

Split a tweet dataset by conversation and by time, never row by row: hold out the newest weeks as the test set, and keep every thread on one side of the line. A random split puts a post and its replies, or two copies of the same text, into both train and test, and the test score then measures memory instead of skill.

import hashlib

rows.sort(key=lambda t: t["created_at"])
cut = rows[int(len(rows) * 0.9)]["created_at"]  # newest 10% = test
test = [t for t in rows if t["created_at"] >= cut]

threads = {t.get("conversation_id_str") for t in test}
rest = [t for t in rows if t["created_at"] < cut
        and t.get("conversation_id_str") not in threads]


def bucket(t):
    key = t.get("conversation_id_str") or t["id"]
    return int(hashlib.md5(key.encode()).hexdigest(), 16) % 10


train = [t for t in rest if bucket(t) != 0]
valid = [t for t in rest if bucket(t) == 0]

For tasks about accounts rather than posts, such as Twitter bot detection, group by user.id instead, so no account appears on both sides.

#Save it in the format your model reads

Save a tweet dataset as JSONL, one record per line, with the tweet ID in every record. JSONL streams, appends and loads straight into pandas, the Hugging Face datasets library and most training scripts. Which fields go in each line depends on the job.

• Classifier or topic model

  What each JSONL line holds

  id, cleaned text, label, label_source, lang, created_at

• Evaluation set for a model or a prompt

  What each JSONL line holds

  id, text, the expected answer, and the rule it tests

• RAG index for retrieval

  What each JSONL line holds

  id, text, the post URL, created_at and the author ID, so answers can cite the post

{
  "id": "2115807747265175863",
  "text": "the new beta keeps crashing on launch for me",
  "label": "negative",
  "label_source": "human",
  "lang": "en",
  "created_at": "2026-10-29T14:07:31Z",
  "split": "test"
}

Ship a dataset card with the files: the queries and dates, the day you collected or rehydrated, the recovery rate, how labels were made, what was filtered out and what the set should not be used for. Hugging Face's dataset cards follow the same outline, and the next person to train on your data, often you in six months, needs every line of it.

#What a Twitter dataset costs

A 1M-tweet dataset costs $5,000 in post reads on the official X API, against from $70 on Sorsa API by search or from $14 when you rehydrate known IDs. Switch the size to see the totals for a fresh collection by search.

Two limits on the official X API change the math for datasets. Its pay-per-use plan stops at 3M post reads a month, so a 10M-tweet set needs an Enterprise contract, which X's price list (opens in a new tab) points to above that cap. And from November 22, 2026, X also bills the author profile that comes back with each post as a user read, $0.010 more per post (X's notice (opens in a new tab)). On Sorsa API the author's full profile is part of every tweet, and failed requests are never charged.

Time is the other half of the budget. At 20 requests per second, Sorsa API collects about 400 tweets a second by search and up to 2,000 a second by ID, so 1M fresh tweets take about 42 minutes and a 10M-tweet set about 7 hours on one key.

#How to share and train on a tweet dataset

Public research datasets ship as tweet and user IDs, and anyone with the IDs gets the full text back through an API: Sorsa API rehydrates 1M IDs in about 8 minutes.

• Sharing. Publish the tweet IDs and user IDs with a rehydration script, the way public research datasets do; POST /tweet-info-bulk turns 100 IDs back into full posts per request.
• Training. Keep the full text you collect for your own models: classifiers, topic models, evaluation sets, retrieval indexes and language-model fine-tunes are the common jobs for tweet datasets.
• People. Replace handles before training, keep no profiles of individuals, and remove tweets their authors deleted when you refresh the set.

Sorsa API returns public data only and never posts, likes or follows. Universities with a research project can also ask about academic access to Sorsa API.

#Start your dataset with a test pull

Public Twitter datasets give you a baseline; a fresh one gives you your own topic, your own dates and labels that fit your task. Build it with Sorsa API: from $0.07 per 1,000 tweets by search and from $0.014 per 1,000 by ID, at 20 requests per second, with the author's profile in every tweet. Your 100 free requests collect the first 2,000 tweets, enough to test the query, the cleaning and the split before you pay for anything.

#FAQ