#TwitterAPIScraping

[realdataapi1]

Twitter Scraper - Twitter Profile Extractor

RealdataAPI / twitter-data-scraper

Scrape Twitter Data about users, including user profiles, follower count, followings, hashtags, tweets, retweets, threads, images, statistics, videos, history, replies, and other data fields using Twitter Data Scraper. Our scraper to extract Twitter data is accessible in multiple countries, including Canada, France, Australia, Germany, the USA, the UK, Spain, etc.

 Customize me!  Report an issue Social Media

Readme

API

Input

Related actors

Which Twitter Data Can This Twitter Scraper Extract?

Twitter Scraper loads mentioned Twitter URLs and profiles to scrape the below data.

User data like Twitter username, follower count, following count, location, images, username, banner, etc.

Retweets, list of tweets, profile replies.

Latest, video tweets, people, search hashtags, pictures, or get top.

Insights for every tweet, including replies, favorites, retweets, etc.

Twitter Scraper on our platform allows you to scrape Twitter data at scale. It also allows scraping data more than the official Twitter API because you don't need a registered application, Twitter account, or API key, and it has no restrictions.

You can load the source platform for the Twitter handles list or use Twitter links like trending topics, searches, or hashtags.

Why Use Real Data API Twitter Scraper?

Crawling the Twitter platform will give you access to over five hundred million tweets daily. You can collect any required data in multiple ways.

Monitor discussions about your city, country, products, or brand.

Observe attitudes, new trends, and fashions as they enter the market.

Track your competitors to check their popularity and how to beat them.

Monitor market and investor sentiments to ensure the safety of your Investments.

Use Twitter information to train your artificial intelligence and machine learning prototypes for academic research.

Study customer habits, target underdeveloped fields, or develop new products depending on customer pain points.

Spot fake news by learning patterns of how people spread fake information.

Discover discussions about services and travel destinations, and use local knowledge best.

How to Use Twitter Scraper?

To learn more about using this Twitter Scraper, check out our stepwise tutorial or watch the video.

Can I Scrape Twitter Data Legally?

Yes, you can extract publicly available data from Twitter. But you must note that you may get private data in your output. GDPR and other regulations worldwide protect personal data, respectively. They don't allow you to extract personal information without genuine reason or prior permission. You can consult your lawyers if you are confused or unsure whether your reason is genuine.

Do You Want More Options to Scrape Twitter Data?

If you wish to extract specific Twitter data quickly, try the targeted Twitter data scraper options below.

Twitter URL Scraper

Twitter Image Scraper

Twitter History Scraper

Easy Twitter Search Scraper

Twitter Info Scraper

Twitter Profile Scraper

Twitter History Hashtag Scraper

Twitter Explore Scraper

Twitter Latest Scraper

Twitter Explore Scraper

Twitter Video Scraper

Tips & Tricks

item-6

The scraper has the default option to extract using search queries, but you can also try Twitter URLs or Twitter handles. If you plan to use the URL option, check the below allowable URL types.

Searches:https://twitter.com/search?q=tesla&src=typed_query

Profiles: https://twitter.com/elonmusk

Topics: https://twitter.com/i/topics/933033311844286464

Retweets with quotes: https://twitter.com/elonmusk/status/1356524205374918659/retweets/with_comments

Explore: https://twitter.com/explore

Trending topics: https://twitter.com/search?q=%23FESTABBB21&src=trend_click&vertical=trends

Hashtag: https://twitter.com/hashtag/WandaVision

Statuses: https://twitter.com/elonmusk/status/1356381230925635591

Events: https://twitter.com/i/events/1354736314923372544

List: https://twitter.com/i/lists/1611381299687694336

Logging In Using Cookies

The option to log in using cookies allows you to use the already initialized cookies of the existing user. If you try this option, the scraper will try to avoid the block from the source platform. For example, the scraper will reduce the running speed and introduce a random delay between two actions.

We highly recommend you don't use a private account to run the scraper unless there is no other option. Instead, you can create a new Twitter account so that Twitter won't ban your personal account.

Use Chrome browser extensions like EditThisCookie to log in using existing cookies. Once you install it, open the source platform in your browser, login into Twitter using credentials, and export cookies using a browser extension. It will give you a cookie array to use as an input value login cookie while logging in.

If you try to log out from the Twitter account with the submitted cookies, the scraper will invalidate them, and the scraper will stop its execution.

Check out the below video tutorial to sort it out.

Input Parameters

Twitter Data Output

You can export the scraped dataset in multiple digestible formats like CSV, JSON, Excel, or HTML. Every item in the scraped data set contains a different tweet in the following format. [{ "user": { "protected": false, "created_at": "2009-06-02T20:12:29.000Z", "default_profile_image": false, "description": "", "fast_followers_count": 0, "favourites_count": 19158, "followers_count": 130769125, "friends_count": 183, "has_custom_timelines": true, "is_translator": false, "listed_count": 117751, "location": "", "media_count": 1435, "name": "Elon Musk", "normal_followers_count": 130769125, "possibly_sensitive": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1576183471", "profile_image_url_https": "https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg", "screen_name": "elonmusk", "statuses_count": 23422, "translator_type": "none", "verified": true, "withheld_in_countries": [], "id_str": "44196397" }, "id": "1633026246937546752", "conversation_id": "1632363525405392896", "full_text": "@MarkChangizi Sweden’s steadfastness was incredible!", "reply_count": 243, "retweet_count": 170, "favorite_count": 1828, "hashtags": [], "symbols": [], "user_mentions": [ { "id_str": "49445813", "name": "Mark Changizi", "screen_name": "MarkChangizi" } ], "urls": [], "media": [], "url": "https://twitter.com/elonmusk/status/1633026246937546752", "created_at": "2023-03-07T08:46:12.000Z", "is_quote_tweet": false, "replying_to_tweet": "https://twitter.com/MarkChangizi/status/1632363525405392896", "startUrl": "https://twitter.com/elonmusk/with_replies" }, { "user": { "protected": false, "created_at": "2009-06-02T20:12:29.000Z", "default_profile_image": false, "description": "", "fast_followers_count": 0, "favourites_count": 19158, "followers_count": 130769125, "friends_count": 183, "has_custom_timelines": true, "is_translator": false, "listed_count": 117751, "location": "", "media_count": 1435, "name": "Elon Musk", "normal_followers_count": 130769125, "possibly_sensitive": false, "profile_banner_url": "https://pbs.twimg.com/profile_banners/44196397/1576183471", "profile_image_url_https": "https://pbs.twimg.com/profile_images/1590968738358079488/IY9Gx6Ok_normal.jpg", "screen_name": "elonmusk", "statuses_count": 23422, "translator_type": "none", "verified": true, "withheld_in_countries": [], "id_str": "44196397" }, "id": "1633021151197954048", "conversation_id": "1632930485281120256", "full_text": "@greg_price11 @Liz_Cheney @AdamKinzinger @RepAdamSchiff Besides misleading the public, they withheld evidence for partisan political reasons that sent people to prison for far more serious crimes than they committed./n/nThat is deeply wrong, legally and morally.", "reply_count": 727, "retweet_count": 2458, "favorite_count": 10780, "hashtags": [], "symbols": [], "user_mentions": [ { "id_str": "896466491587080194", "name": "Greg Price", "screen_name": "greg_price11" }, { "id_str": "98471035", "name": "Liz Cheney", "screen_name": "Liz_Cheney" }, { "id_str": "18004222", "name": "Adam Kinzinger #fella", "screen_name": "AdamKinzinger" }, { "id_str": "29501253", "name": "Adam Schiff", "screen_name": "RepAdamSchiff" } ], "urls": [], "media": [], "url": "https://twitter.com/elonmusk/status/1633021151197954048", "created_at": "2023-03-07T08:25:57.000Z", "is_quote_tweet": false, "replying_to_tweet": "https://twitter.com/greg_price11/status/1632930485281120256", "startUrl": "https://twitter.com/elonmusk/with_replies" }] ...

Search Using Advanced Feature

Use this type of pre-designed search with Advanced Search as a starting link, like https://twitter.com/search?q=cool%20until%3A2021-01-01&src=typed_query

Workaround to Get Maximum Tweets Limit

Twitter returns only 3200 tweet posts per search or profile by default. If you require more tweets than the maximum limit, you can split your starting links using time slices as the below URL samples.

https://twitter.com/search?q=(from%3Aelonmusk)%20since%3A2020-03-01%20until%3A2020-04-01&src=typed_query&f=live

https://twitter.com/search?q=(from%3Aelonmusk)%20since%3A2020-02-01%20until%3A2020-03-01&src=typed_query&f=live

https://twitter.com/search?q=(from%3Aelonmusk)%20since%3A2020-01-01%20until%3A2020-02-01&src=typed_query&f=live

Each link is from the same account - Elon Musk, but we separated them by a 30-day monthly timeframe, like January > February > March 2020. You can create it using the advanced search option on Twitter. https://twitter.com/search If you want, you can use larger time intervals for a few accounts that don't post regularly.

Other restrictions contain-

You can cap live tweets by max one day in the past.

Flying can cap most search results at about hundred and fifty tweets like Top, Pictures, and Videos.

Extend Output Function

This output parameter function allows you to change your dataset output shape, split data arrays into different items, or categorize the output. async ({ data, item, request }) => { item.user = undefined; // removes this field from the output delete item.user; // this works as well const raw = data.tweets[item['#sort_index']]; // allows you to access the raw data item.source = raw.source; // adds "Twitter for ..." to the output if (request.userData.search) { item.search = request.userData.search; // add the search term to the output item.searchUrl = request.loadedUrl; // add the raw search URL to the output } return item; }

Item filtering: async ({ item }) => { if (!item.full_text.includes('lovely')) { return null; // omit the output if the tweet body doesn't contain the text } return item; }

Separating into multiple data items and changing the entire result: async ({ item }) => { // dataset will be full of items like { hashtag: '#somehashtag' } // returning an array here will split in multiple dataset items return item.hashtags.map((hashtag) => { return { hashtag: `#${hashtag}` }; }); }

Extend Scraper Function

This factor permits you to extend scraper working and can simplify extending the default scraper function without owning a custom version. For instance, you can contain a trending topic search for every page visit. async ({ page, request, addSearch, addProfile, addThread, customData }) => { await page.waitForSelector('[aria-label="Timeline: Trending now"] [data-testid="trend"]'); const trending = await page.evaluate(() => { const trendingEls = $('[aria-label="Timeline: Trending now"] [data-testid="trend"]'); return trendingEls.map((_, el) => { return { term: $(el).find('> div > div:nth-child(2)').text().trim(), profiles: $(el).find('> div > div:nth-child(3) [role="link"]').map((_, el) => $(el).text()).get() } }).get(); }); for (const { search, profiles } of trending) { await addSearch(search); // add a search using text for (const profile of profiles) { await addProfile(profile); // adds a profile using link } } // adds a thread and get replies. can accept an id, like from conversation_id or an URL // you can call this multiple times but will be added only once await addThread("1351044768030142464"); }

extendScraperFunction contains additional data variables. async ({ label, response, url }) => { if (label === 'response' && response) { // inside the page.on('response') callback if (url.includes('live_pipeline')) { // deal with plain text content const blob = await (await response.blob()).text(); } } else if (label === 'before') { // executes before the page.on('response'), can be used for intercept request/response } else if (label === 'after') { // executes after the scraping process has finished, even on crash } }

Twitter Scraper with Real Data API Integrations

Lastly, using Real Data API Integrations, you can connect Twitter Scraper with almost any web application or cloud service. You can connect with Google Drive, Google Sheets, Airbyte, Make, Slack, GitHub, Zapier, etc. Further, you can use Webhooks to carry out an activity once an event occurs, like an alert when Twitter Scraper completes its execution.

Using Twitter Scraper with Real Data API Platform

The Real Data API platform gives you programmatic permission to use scrapers. We have organized the Twitter Scraper API around RESTful HTTP endpoints to allow you to schedule, manage, and run Real Data API Scrapers. The actor also lets you track actor performance, create and update versions, access datasets, retrieve results, and more.

To use the scraper using Python, try our client PyPl package, and to use it using Node.js, try our client NPM package.

Check out the API tab for code examples or explore Real Data API reference documents for details.

[realdataapi1]

Twitter Scraper - Twitter Profile Extractor

Extract Twitter profiles easily with our Twitter Scraper. Analyze tweets, followers, and insights using our powerful Twitter Profile Extractor for detailed data analysis.