twitter-scraper

twitter-scraper

A port of the now-archived n0madic/twitter-scraper to Node.js.

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse-engineered. No API rate limits. No tokens needed. No restrictions. Extremely fast.

You can use this library to get the text of any user's Tweets trivially.

Many things have changed since X (the company formerly known as Twitter) was acquired in 2022:

• Several operations require logging in with a real user account via scraper.login(). While we are not aware of confirmed cases caused by this library, any account you log into with this library is subject to being banned at any time. You have been warned.
• Twitter's frontend API does in fact have rate limits (#11). The rate limits are dynamic and sometimes change, so we don't know exactly what they are at all times. Refer to rate limiting for more information.
• Twitter's authentication requirements and frontend API endpoints change frequently, breaking this library. Fixes for these issues typically take at least a few days to go out.

Installation

This package requires Node.js v16.0.0 or greater.

NPM:

npm install @the-convocation/twitter-scraper

Yarn:

yarn add @the-convocation/twitter-scraper

TypeScript types have been bundled with the distribution.

Usage

Most use cases are exactly the same as in n0madic/twitter-scraper. Channel iterators have been translated into AsyncGenerator instances, and can be consumed with the corresponding for await (const x of y) { ... } syntax.

Browser usage

This package directly invokes the Twitter API, which does not have permissive CORS headers. With the default settings, requests will fail unless you disable CORS checks, which is not advised. Instead, applications must provide a CORS proxy and configure it in the Scraper options.

Proxies (and other request mutations) can be configured with the request interceptor transform:

const scraper = new Scraper({
  transform: {
    request(input: RequestInfo | URL, init?: RequestInit) {
      // The arguments here are the same as the parameters to fetch(), and
      // are kept as-is for flexibility of both the library and applications.
      if (input instanceof URL) {
        const proxy =
          'https://corsproxy.io/?' + encodeURIComponent(input.toString());
        return [proxy, init];
      } else if (typeof input === 'string') {
        const proxy = 'https://corsproxy.io/?' + encodeURIComponent(input);
        return [proxy, init];
      } else {
        // Omitting handling for example
        throw new Error('Unexpected request input type');
      }
    },
  },
});

corsproxy.io is a public CORS proxy that works correctly with this package.

The public CORS proxy corsproxy.org does not work at the time of writing (at least not using their recommended integration on the front page).

Next.js 13.x example:

'use client';

import { Scraper, Tweet } from '@the-convocation/twitter-scraper';
import { useEffect, useMemo, useState } from 'react';

export default function Home() {
  const scraper = useMemo(
    () =>
      new Scraper({
        transform: {
          request(input: RequestInfo | URL, init?: RequestInit) {
            if (input instanceof URL) {
              const proxy =
                'https://corsproxy.io/?' + encodeURIComponent(input.toString());
              return [proxy, init];
            } else if (typeof input === 'string') {
              const proxy =
                'https://corsproxy.io/?' + encodeURIComponent(input);
              return [proxy, init];
            } else {
              throw new Error('Unexpected request input type');
            }
          },
        },
      }),
    [],
  );
  const [tweet, setTweet] = useState<Tweet | null>(null);

  useEffect(() => {
    async function getTweet() {
      const latestTweet = await scraper.getLatestTweet('twitter');
      if (latestTweet) {
        setTweet(latestTweet);
      }
    }

    getTweet();
  }, [scraper]);

  return (
    <main className="flex min-h-screen flex-col items-center justify-between p-24">
      {tweet?.text}
    </main>
  );
}

Edge runtimes

This package currently uses cross-fetch as a portable fetch. Edge runtimes such as CloudFlare Workers sometimes have fetch functions that behave differently from the web standard, so you may need to override the fetch function the scraper uses. If so, a custom fetch can be provided in the options:

const scraper = new Scraper({
  fetch: fetch,
});

Note that this does not change the arguments passed to the function, or the expected return type. If the custom fetch function produces runtime errors related to incorrect types, be sure to wrap it in a shim (not currently supported directly by interceptors):

const scraper = new Scraper({
  fetch: (input, init) => {
    // Transform input and init into your function's expected types...
    return fetch(input, init).then((res) => {
      // Transform res into a web-compliant response...
      return res;
    });
  },
});

Bypassing Cloudflare bot detection

In some cases, Twitter's authentication endpoints may be protected by Cloudflare's advanced bot detection, resulting in 403 Forbidden errors during login. This typically happens because standard Node.js TLS fingerprints are detected as non-browser clients.

To bypass this protection, you can use the optional CycleTLS fetch integration to mimic Chrome browser TLS fingerprints:

Installation:

npm install cycletls
# or
yarn add cycletls

Usage:

import { Scraper } from '@the-convocation/twitter-scraper';
import {
  cycleTLSFetch,
  cycleTLSExit,
} from '@the-convocation/twitter-scraper/cycletls';

const scraper = new Scraper({
  fetch: cycleTLSFetch,
});

// Use the scraper normally
await scraper.login(username, password, email);

// Important: cleanup CycleTLS resources when done
cycleTLSExit();

Note: The /cycletls entrypoint is Node.js only and will not work in browser environments. It's provided as a separate optional entrypoint to avoid bundling binaries in environments where they cannot run.

See the cycletls example for a complete working example.

Cookie-based authentication

If you're encountering error 399 ("Incorrect. Please try again") or Twitter's suspicious activity detection during login, you can use cookies exported from an already-authenticated browser session instead. This approach:

• Avoids Twitter's anti-bot protection that blocks automated logins
• No need to store or handle passwords in code
• Uses your established browser session
• Bypasses rate limiting on authentication endpoints

Step 1: Export cookies from your browser

Using Chrome/Edge:

1. Log in to X.com in your browser
2. Open DevTools (F12) → Application tab → Cookies
3. Click the URL bar that says "Filter cookies" and press Ctrl+A to select all cookies
4. Copy all cookies (they'll be in format: name1=value1; name2=value2; ...)

Using Firefox:

1. Log in to X.com in your browser
2. Open DevTools (F12) → Storage tab → Cookies → https://x.com
3. Find the ct0 cookie and copy its value
4. Find the auth_token cookie and copy its value
5. Construct the cookie string: ct0=<value>; auth_token=<value>

Tip: You can use the Cookie-Editor extension to export cookies in a convenient format.

Step 2: Use cookies in your code

import { Cookie } from 'tough-cookie';
import { Scraper } from '@the-convocation/twitter-scraper';

// Your cookie string from browser (name=value; name2=value2; ...)
const cookieString = 'ct0=abc123; auth_token=xyz789; lang=en; ...';

// Parse the cookie string
const cookies = cookieString
  .split(';')
  .map((c) => Cookie.parse(c))
  .filter(Boolean);

// Create scraper and set cookies
const scraper = new Scraper();
await scraper.setCookies(cookies);

// Verify authentication works
const isLoggedIn = await scraper.isLoggedIn();
if (isLoggedIn) {
  console.log('✓ Successfully authenticated with cookies!');
  // Now you can use authenticated features
  const profile = await scraper.getProfile('username');
}

Cookies expire over time. If authentication fails, you may need to export fresh cookies from your browser.

Rate limiting

The Twitter API heavily rate-limits clients, requiring that the scraper has its own rate-limit handling to behave predictably when rate-limiting occurs. By default, the scraper uses a rate-limiting strategy that waits for the current rate-limiting period to expire before resuming requests.

This has been known to take a very long time, in some cases (up to 13 minutes).

You may want to change how rate-limiting events are handled, potentially by pooling scrapers logged-in to different accounts (refer to #116 for how to do this yourself). The rate-limit handling strategy can be configured by passing a custom implementation to the rateLimitStrategy option in the scraper constructor:

import { Scraper, RateLimitStrategy } from '@the-convocation/twitter-scraper';

class CustomRateLimitStrategy implements RateLimitStrategy {
  async onRateLimit(event: RateLimitEvent): Promise<void> {
    // your own logic...
  }
}

const scraper = new Scraper({
  rateLimitStrategy: new CustomRateLimitStrategy(),
});

More information on this interface can be found on the RateLimitStrategy page in the documentation. The library provides two pre-written implementations to choose from:

• WaitingRateLimitStrategy: The default, which waits for the limit to expire.
• ErrorRateLimitStrategy: A strategy that throws if any rate-limit event occurs.

Contributing

Setup

This project currently requires Node 18.x for development and uses Yarn for package management. Corepack is configured for this project, so you don't need to install a particular package manager version manually.

The project supports Node 16.x at runtime, but requires Node 18.x to run its build tools.

Just run corepack enable to turn on the shims, then run yarn to install the dependencies.

Basic scripts

• yarn build: Builds the project into the dist folder
• yarn test: Runs the package tests (see Testing first)

Run yarn help for general yarn usage information.

Testing

This package includes unit tests for all major functionality. Given the speed at which Twitter's private API changes, failing tests are to be expected.

yarn test

Before running tests, you should configure environment variables for authentication.

TWITTER_USERNAME=    # Account username
TWITTER_PASSWORD=    # Account password
TWITTER_EMAIL=       # Account email
TWITTER_COOKIES=     # JSON-serialized array of cookies of an authenticated session
PROXY_URL=           # HTTP(s) proxy for requests (optional)

Commit message format

We use Conventional Commits, and enforce this with precommit checks. Please refer to the Git history for real examples of the commit message format.