How to Scrape Twitter (X) in 2026: Methods, Tools, and Costs

By Sorsa Editorial

Reviewed by Keksich, founder of Sorsa, marketer and X API researcher.

Updated July 2026: verified the open-source library status against GitHub and PyPI (twscrape, Scweet, and Tweety work; snscrape, Twint, and ntscraper are dead; upstream Twikit is currently broken and running on a community fork). X's pay-per-use API pricing and its January 2026 Terms of Service were re-checked the same month.

Most people who search "how to scrape Twitter" do not actually want a scraper. They want the data. Those are different problems: the data is the goal, and a scraper is a standing commitment to residential proxies, burner accounts that get banned, and re-reverse-engineering X every couple of weeks.

We build and run Sorsa API, a read-only Twitter/X data API, so treat this section as interested. The tradeoff is still worth stating plainly, because the same profiles, tweets, search results, and followers that a scraper fights to extract come back as clean JSON from one REST call, with no proxies, logins, or maintenance on your side. Pricing starts from $0.02 per 1,000 tweets and $0.01 per 1,000 profiles on batch endpoints, up to 50x cheaper than the official X API, on a flat 20 requests per second across every plan. There is no application or approval queue, the first call takes about three minutes, and the first 100 requests are free with no card.

If you still want to build the scraper, for control, for learning, or because your workflow genuinely needs it, this guide is the honest, current version: what data is actually reachable, what each approach involves, which tools live and which are dead, what it really costs, and where the legal lines fall.

Table of Contents

• Can you still scrape Twitter in 2026?
• What data can you scrape (and what is off-limits)
• Is it legal to scrape Twitter?
• How X.com works under the hood
• Method 1: Headless browser scraping
• Method 2: Open-source Python libraries
• Method 3: AI-assisted scraping
• Method 4: Managed scraping services
• How to find tweet and profile URLs to scrape
• What scraping actually costs
• Method 5: A read-only data API (no scraping)
• Which method should you use?
• FAQ
• How we verified this guide

Can you still scrape Twitter in 2026?

Yes. Public Twitter data is still scrapeable in 2026 without the official API, but every easy path has closed. Three shifts explain the difficulty, and the rest of this guide is really about working around them.

First, X locked down anonymous guest access, which is the single change that killed the old free stack. Second, it moved the official API to pay-per-use with no free read tier, so "just use the API" now costs money from the first request. Third, it tightened bot detection on every public page, so even a working scraper runs into IP blocking and rate limits quickly.

The payoff is still real. X remains one of the richest sources of live public opinion anywhere, which is why teams keep scraping it for brand monitoring, sentiment analysis, competitor research, lead generation, trend detection, and machine-learning datasets. The difficulty is the whole reason a guide like this exists: the work is no longer in parsing HTML, it is in keeping up with access controls that change every few weeks.

What data can you scrape (and what is off-limits)

Before writing any code, know where the wall is. Without a logged-in session, you can reach public profiles, individual tweets and their visible replies, and embedded media metadata. Anything behind authentication, protected accounts, direct messages, full follower lists, and complete search results, stays out of reach unless you scrape with a real account, which carries suspension risk.

Anonymous access has softer limits too. Reply depth is shallow, and rate-limited guest views cut off long timelines before you reach the end. Full historical archives require heavy scroll automation, and even then you hit the guest timeline cap. If you need history without scraping, a data API can return tweets back to 2006 through search, and full follower and following lists.

The rest of this guide covers only the public, unauthenticated surface. That is both the safest line legally and the one that does not get your account banned.

Is it legal to scrape Twitter?

Scraping public Twitter data sits in a genuine split: U.S. courts have largely protected it, while X's own Terms of Service flatly prohibit it. Both are true at once, and which one matters depends on whether your risk is criminal-law exposure or contract exposure.

On the case-law side, the balance leans toward public data. In 2022 the Ninth Circuit affirmed that scraping publicly available information does not violate the Computer Fraud and Abuse Act, the hiQ v. LinkedIn line being the most cited. X has tested this directly and lost: a federal court dismissed X Corp. v. Bright Data in May 2024, finding X's claims against the scraper largely preempted, and X's suit against the Center for Countering Digital Hate was dismissed on First Amendment grounds. Accessing genuinely public data is not a CFAA crime.

On the contract side, X's terms are the harder constraint. The current terms, effective January 15, 2026, bar crawling or scraping "in any form, for any purpose" without prior written consent. They add a liquidated-damages clause: anyone who requests, views, or accesses more than 1,000,000 posts in any 24-hour period in violation of the terms agrees to pay $15,000 (or the same figure in euros in the EU, EFTA, and UK) per million posts. The 2026 update also redefines "Content" to cover AI prompts and outputs, adds a misuse clause aimed at jailbreaking and prompt injection, and sets a Texas venue with a class-action waiver. Breaching the terms is a contract matter, not a crime, but X can suspend accounts, block IPs, and point to that clause at scale.

Three rules keep public scraping on defensible ground: pull only public data, never protected accounts, DMs, or anything behind a login; do not hoard, keep scraped data only as long as your use case needs; and respect rate limits, since overloading servers is where legal risk turns concrete. EU readers carry one more layer, because public posts can still contain personal data, so GDPR applies whenever you store or process it. This is informational, not legal advice; if your use case involves sensitive data or high volume, talk to a lawyer.

How X.com works under the hood

Understanding X's architecture explains why every scraper eventually breaks. If you have scraped other sites, X's defenses are in a different league, and the reasons are structural, not incidental.

X is a React single-page application. Load a profile or tweet URL and the server returns a near-empty HTML shell. JavaScript then requests a guest token from the backend and fires GraphQL queries to fetch the actual data, which the browser renders. There is almost nothing useful in the initial HTML, so simple fetch-and-parse scraping returns nothing. That design gives X three chokepoints.

Guest tokens are temporary credentials required for every GraphQL call. They are tied to your IP, expire within a few hours, and the way they are issued changes every few weeks. When X shifts token issuance, every scraper relying on the old method stops instantly. This is the exact mechanism that killed the free libraries: they all assumed an anonymous access path that no longer exists.

GraphQL operation IDs (doc_ids) are identifiers embedded in X's JavaScript bundle that tell the backend which operation to run. Fetching a profile, searching tweets, and loading a timeline each need a different one. X rotates them every two to four weeks, a scraper tracks eight to twelve at once, and there is no documentation, so scraper maintainers reverse-engineer them from minified JavaScript and then do it again a fortnight later.

Rate limits and detection are the third layer. X enforces roughly 300 requests per hour per IP for guest sessions. Datacenter IPs get flagged within a request or two. TLS fingerprinting catches headless browsers whose network stack does not perfectly mimic a real one, and cookie validation flags suspicious session patterns. If an account gets flagged, our free shadowban checker confirms it in a few seconds.

These defenses have not been static. The table below is a factual record of what X changed and when, which is why a scraping guide even twelve months old can point you at dead tools.

The takeaway is that X is not a stable target. It ships defensive changes roughly every two to four weeks, and any scraper is a subscription to keeping up.

Method 1: Headless browser scraping

The most common do-it-yourself approach automates a real browser, loads X pages, and reads the data the page itself requests. It works because it runs the same JavaScript X expects, so the page fetches its own current doc_ids and the content renders the way it would for a human.

In practice the scraper launches a headless Chromium, navigates to a tweet or profile, waits for it to render, and reads the tweet object out of the page's own background responses: text, timestamps, engagement counts, media URLs, and the author profile, nested under a legacy key that production code then flattens into the handful of fields it needs. Profiles, timelines, and search follow the same pattern, though search and long timelines are gated behind a login.

What you can get: public profiles, individual tweets, visible replies, quote tweets, and embedded media. Essentially anything on the public interface.

What you cannot get without a login: protected accounts, DMs, full follower lists, and complete search or long timelines. Authenticated scraping reaches those but puts the account at risk.

What keeping it running costs. Datacenter IPs are blocked almost instantly, so a browser scraper only survives on residential proxy traffic, typically $50 to $200 a month depending on volume, and it needs constant retry logic for tokens that expire mid-session and for endpoints that return empty results when doc_ids rotate. This works today and will break within two to four weeks of X's next update, so plan on ten to fifteen hours a month keeping it alive.

Method 2: Open-source Python libraries

Instead of building from scratch, you can use a library that wraps X's internal API. Some are actively maintained and currently working; several widely-recommended ones are dead; and one former default is now broken upstream. Here is the honest picture, re-verified against GitHub and PyPI in July 2026.

The status column is the whole point of this table, and it moves. Twikit was long the default recommendation because it is async, well documented, and had the largest community, but its published PyPI release broke against X's 2026 webpack and transaction-ID changes, and users have moved to a maintained fork that keeps the same twikit import. If a tutorial still tells you to pip install twikit and expect it to just work, check the date and the open issues first.

twscrape is the most-cited of the maintained libraries. It is async-only, read-only by design, and has no built-in resume, so a large multi-day pull needs your own checkpoint logic, and like every library in the table it depends on an X account that X can suspend.

The graveyard (do not waste your time)

If a tutorial recommends any of these as a current solution, its date is the tell. The X scraping landscape turns over fast, which is exactly why we date-stamp the working table: a library that worked last quarter can be dead this one.

The catch with every working library

Each library in the working table needs a logged-in X account, and that carries consequences. X suspends accounts showing automated behavior, so the account behind the scraper is always at risk, and even an actively maintained library breaks when X pushes an update, at which point you wait on the maintainer's response time or on someone forking it.

Method 3: AI-assisted scraping

AI-assisted scraping replaces brittle CSS selectors with a language model that reads the page and extracts what you describe in plain English. The best-known open-source option is ScrapeGraphAI, a Python library where you write a prompt like "get the text, author, and like count of each tweet" and it works out the structure, then re-adapts when X changes its layout instead of breaking on a renamed element.

This is a real answer to the maintenance problem, with real tradeoffs. The model still has to reach the page, so you carry the same proxy, account, and anti-bot requirements as any other method, none of which the LLM solves. Every extraction now also spends tokens, which adds cost and latency and makes output less deterministic than a fixed parser. It fits prototyping and layouts that change often, and it fits poorly for cheap, high-volume, repeatable collection where you want predictable cost per record. If predictable cost and zero maintenance are the goal, a structured data API is the cleaner version of what AI scraping is reaching for.

Method 4: Managed scraping services

Managed services run the scraping infrastructure for you. You send a query or URL, they return structured data, and proxy rotation, token management, and anti-bot measures are their problem. The names you will compare are Bright Data, Apify, and Scrapfly, and Scweet also offers a hosted version on Apify with a small free tier. The upside is no code to maintain and no proxies to manage. The downsides are cost at scale and vendor dependency: if their scraper breaks, you wait for their fix, and pricing models vary from per tweet to per compute unit to per gigabyte of proxy traffic, which makes true per-record cost hard to compare on the sticker.

This is a big enough decision to deserve its own breakdown. For real per-1,000 costs, reliability, and hidden fees across each managed option, see our Twitter scrapers comparison.

How to find tweet and profile URLs to scrape

Every scraping method above needs targets: the specific tweet and profile URLs you want to pull. X's own search is login-walled, so the practical workaround is Google, which indexes public tweets. A site: query is often the fastest way to build a target list without an account.

Three patterns cover most needs. site:x.com inurl:status <keyword> surfaces individual tweets about a topic. site:x.com <name> finds profile pages for a person or brand. Adding Google's date-range tools filters for recent posts. The one honest caveat is that Google's index lags real-time X by hours to days, so for breaking content it will not be current, but for assembling a list of profiles and posts to scrape it works well. If you need real-time discovery instead of a static list, that is precisely where search-by-query beats crawling, and a search API returns matching tweets directly.

What scraping actually costs

Most tutorials quote the tool's sticker price and stop. The real cost of scraping X is the tool plus residential proxies plus developer time plus the recurring cost of things breaking. Here is the full picture across the routes.

The row that decides it is maintenance. Developer time is the most expensive line on the list, and it does not show up until the first time X rotates its doc_ids and your pipeline goes quiet on a Friday. Teams we have worked with routinely spend ten to fifteen hours a month keeping a self-built X scraper alive, plus the proxy bill on top. As a rough sense of scale, pulling 10,000 tweets by browser automation runs roughly 50 to 100 requests and a few dollars of residential proxy traffic, before you count the hours. The free tools are not free once you price your own time.

Method 5: A read-only data API (no scraping)

If you have read this far, the honest takeaway is that scraping X is possible but costly in time, money, and ongoing effort, and for many use cases you do not need to scrape at all. A read-only X data API returns the same information through plain REST endpoints: profiles, tweets, search, followers, engagement, community data, all as clean JSON, one API key in the header, and no proxies, guest tokens, or doc_ids for you to manage.

Here is a profile lookup with Sorsa's API:

curl -H "ApiKey: YOUR_KEY" \
  "https://api.sorsa.io/v3/info?username=elonmusk"

That returns the full profile object in one line: ID, username, display name, bio, follower and following counts, tweet count, verification status, images, and creation date. Compare it to a browser-automation script plus the proxy setup, token management, and retry logic around it.

Sorsa covers 40 endpoints across users, tweets, search, followers, verification, communities, lists, and trends, on a flat 20 requests per second with no per-endpoint windows. Pricing starts from $0.02 per 1,000 tweets and $0.01 per 1,000 profiles on batch endpoints, where calls like /info-batch (up to 100 profiles) and /tweet-info-bulk (up to 100 tweets) each count as a single request. The first 100 requests are free with no card, you can test any endpoint without code in the playground, and the three-minute quickstart gets a first call working with no approval. Coming from the official X API, the migration guide maps the switch endpoint by endpoint.

The tradeoff is real and worth naming: an API is read-only, so it cannot post, like, or follow, and you depend on the provider rather than your own code. For read-heavy work that needs to keep running, that is usually the point.

In practice: a monitoring pipeline that stopped breaking

A mid-sized fintech analytics team came to us running a self-built Playwright scraper for real-time monitoring across a few hundred finance and competitor accounts. It worked until it didn't: every time X rotated its doc_ids the pipeline went silent, and an engineer lost a day re-reverse-engineering the new identifiers and rotating proxies. Across developer hours and residential proxies, the real monthly cost had crept past $2,000 for data they treated as routine.

They moved the read side to a data API and kept building only the parts that were actually theirs. The recurring engineering drain went to zero, because the same REST calls returned the same JSON every day no matter what X changed on its end, and the spend dropped to a fraction of the proxy-plus-hours total. The cost reduction is a real property of switching from per-account scraping with proxies to flat per-request billing; the maintenance reduction is simply what happens when you stop running a scraper. We kept the example anonymized because naming a client and its monitoring targets can expose both.

Which method should you use?

There is no single winner. The right choice depends on your volume, your budget, and how much downtime you can tolerate.

Building your own scraper, with a headless browser or an open-source library, gives you full control of the extraction in exchange for account-ban risk, proxy spend, and maintenance every time X changes something. A managed scraper moves that infrastructure to a vendor, at a higher price and with a dependency on their fixes. Use a read-only data API when you need reliable read access, you cannot afford weekly breakage, or you are shipping a product on top of X data. And stay on the official X API for anything that writes, posting, DMs, follows, since no scraper or read API does write actions safely, plus Ads and compliance data.

For a wider look at providers, see our guide to Twitter/X API alternatives; for what the official API now costs, our X API pricing breakdown; and if you are wondering whether the free tier survived, is the Twitter API free in 2026.

FAQ

Can you still scrape Twitter in 2026?

Yes, public Twitter data is still scrapeable in 2026 without the official API, but the easy paths are gone. The free libraries that relied on anonymous access are dead, guest browsing is rate-limited, and X rotates its internal guest tokens and GraphQL identifiers every two to four weeks. Working methods today are headless-browser automation, a maintained open-source library on a logged-in account, a managed scraper, or a read-only data API.

Can you scrape Twitter without logging in?

You can, but with heavy limits. Without authentication, headless browser scraping reaches public profiles and individual tweets, while full search, complete timelines, deep threads, and follower lists are restricted or blocked. Every open-source library that provides full data access requires either account credentials or session tokens, which is what unlocks the complete data.

What is the best tool to scrape Twitter in 2026?

It depends on your job. Among open-source libraries, twscrape, Scweet, and Tweety were still maintained as of July 2026, while Twikit's upstream release is currently broken by X's 2026 changes and runs on a community fork, so verify its status before relying on it. All of them need a logged-in X account and regular fixes. For zero-maintenance access with no proxies or account risk on your side, a read-only data API like Sorsa returns the same data through REST endpoints, from $0.02 per 1,000 tweets with the first 100 requests free.

Can you scrape Twitter with Python?

Yes, Python is the most common language for it, either through headless-browser automation with Playwright or through one of the maintained libraries (twscrape, Scweet, Tweety), each of which needs a logged-in account and breaks when X changes. Tweepy exists too but wraps the official paid API rather than scraping. If an API fits better than a scraper, our Twitter API Python guide walks through the REST route.

Is it legal to scrape Twitter?

Scraping genuinely public data is generally legal in the U.S., because courts have found that accessing public web data does not violate the Computer Fraud and Abuse Act. It can still breach X's Terms of Service, which is a contract issue rather than a crime and can get an account or IP banned. Accessing non-public data crosses the CFAA line. In the EU, GDPR applies when public posts contain personal data you store or process. This is informational, not legal advice.

Does the official X API still have a free tier?

No. X discontinued the free tier and moved to pay-per-use pricing for new developers. You buy credits upfront and pay per resource: about $0.005 per post read, $0.010 per user profile, and $0.015 per post created, with a cap of 2 million post reads per month on standard accounts. Our companion guide on whether the X API is free covers the current options.

How often does X break scrapers?

Every two to four weeks on average. The usual triggers are changes to how guest tokens are issued, rotations of the GraphQL operation IDs, and new anti-bot detection layers. Any scraper, do-it-yourself or library-based, needs regular updates to keep working, and this recurring maintenance is the single biggest hidden cost of the scraping approach.

How we verified this guide

This guide is built on our own work running an X data API, plus a fresh review of each tool's current state. In this pass we re-checked the open-source libraries directly on GitHub and PyPI to confirm what is maintained (twscrape, Scweet, and Tweety are active in 2026; snscrape, Twint, and ntscraper are not; upstream Twikit's release is broken and running on a community fork), read X's Terms of Service effective January 15, 2026 for the scraping and liquidated-damages language, and used the EFF summary of the Ninth Circuit CFAA ruling for the case-law section. Pricing for our own API comes from the Sorsa docs. X's Terms of Service and the official X API pricing were re-verified in July 2026; the open-source library breakdown was re-checked the same month. Libraries and platform terms change quickly, so re-check anything time-sensitive against the source before you rely on it.