Dataset: kuben-developer/tiktok-videos-4b on Hugging Face, 4.5 billion TikTok video records, 289 GB, Parquet/zstd.
Most TikTok scrapers drive a headless browser or parse a JSON blob from the web page. Both approaches are the wrong layer. They are slow, fragile, and they miss most of the interesting fields. The TikTok Android app does not use either. It talks to a private HTTP+JSON API, the same one the app hits when you scroll, and that API is fast, stable, and returns far more structured data than anything you can pull from the web.
Someone just used that API to collect 5.94 billion videos, 3.23 billion creator profiles, and 2.8 billion comments in roughly three weeks. They then uploaded 4.5 billion of those video records as a single open dataset to Hugging Face: 4,501,811,789 rows, 27 Parquet files compressed with zstd, totaling about 289 GB. It is the largest public TikTok dataset available, released as-is for research, with no login, no account, and no session cookie anywhere in the pipeline.
The dataset is live, free, and queryable with DuckDB without a full download.
What exactly was scraped and how?
The collection method is documented in full at tiktok-api.seeksocial.io. The core idea is that TikTok's Android app, com.zhiliaoapp.musically, communicates with a set of private HTTP endpoints that return JSON. These endpoints are faster than the public web API and return considerably richer objects. The scraper reproduces the exact request signing the app uses, from anonymous device registrations.
Getting in is hard in a specific way. Four unrelated things must be correct simultaneously: a device credential TikTok issued, a valid request signature, the correct regional host, and a TLS handshake that looks like a phone. Miss any one and the request fails. There is no login anywhere in the system, which means account-gated content like DMs, private videos, or who liked what is out of reach and stays out of reach.
The system exposes 24 endpoints, each with a measured success rate. They cover creator profiles, every video a creator has posted, follower and following lists, TikTok's own similar-creator graph, full video detail with statistics, comments and replies, sounds with their videos, hashtags, keyword search, trending shelves, and camera effects.
The dataset uploaded to Hugging Face contains one row per video with these fields: content_id, create_time, desc (the caption), mentions, duration, is_video, music_id, music_title, views, likes, comments, shares, saves, country, language, and is_ad. Every content_id appears exactly once.
The chart above shows the gap. At 4.5 billion rows and 289 GB, this dataset dwarfs typical academic TikTok studies, which tend to operate on samples of a few million rows, by three orders of magnitude.
The system also collected 3.23 billion creator profiles and 2.8 billion comments, though only the video dataset was uploaded. The creator identity is deliberately excluded from the published data. There is no author ID, username, or profile data in the Parquet files. You can group videos by sound, hashtag, or caption, but not by who posted them.
Why does the mobile API matter if web scraping exists?
The web endpoints most scrapers target parse a hydration blob from the HTML. TikTok renders its public pages from a JSON payload inside a script tag, and you walk to webapp.user-detail or webapp.video-detail. That path changes every few weeks when TikTok rotates the key structure. The SocialCrawl guide walks through this approach, and it works, but it is inherently brittle.
The mobile API is a different layer entirely. It does not depend on HTML structure or hydration blobs. It returns structured JSON directly, with fields the web layer simply does not expose. The saves field, for instance, gives you bookmark counts. That is often the earliest signal that something is gaining traction, and it is not available from the web endpoints.
The chart above compares the two. The mobile API returns roughly 16 structured fields per video, while the web endpoints return about 7. That 2.3x gap matters when you are building engagement models or trend detection. More fields means more signal, and more signal means your downstream models have more to work with.
There is also a speed argument. Headless browsers are slow because they render a full page, execute JavaScript, and wait for hydration. The mobile API returns JSON in a single round trip. At the scale of billions of requests, that difference is the difference between finishing in three weeks and finishing in three months.
What does this mean for builders working with social data?
If you build anything that touches social platform data, recommendation systems, trend detection, or content analysis, this dataset and the method behind it change your calculus in several ways.
- Training data at scale. 4.5 billion rows with engagement metrics is enough to train serious content understanding models. Captions, view counts, like ratios, comment counts, and save rates are all present. If you have been working with scraped samples of 50,000 videos, you now have access to four orders of magnitude more data, for free.
- Sound-level analysis. The
music_idandmusic_titlefields let you join videos by sound, which is how trends actually propagate on TikTok. You can build a trending-sound tracker from this dataset without touching the live API. - Engagement normalization. The
savesfield is quietly the most valuable column. Bookmarks correlate with intent in a way that views and likes do not. If you have been proxying engagement with likes alone, you have been leaving signal on the table. - Query without downloading. The dataset is Parquet over zstd, which means DuckDB can query it remotely without a full 289 GB pull. The Hugging Face listing includes a working DuckDB example that aggregates plays by sound from 2025 onward. Start with a single 10 GB file before pulling all 27.
- API architecture lesson. If you are building a platform with a public web layer and a private mobile API, this is your threat model. The mobile API is discoverable, reproducible, and faster than your web endpoints. Someone will reverse-engineer it. Plan for that.
For teams building recommendation engines or content classifiers, the dataset offers something that API-gated access does not: scale without rate limits. TikTok's official Research API, where available, caps requests and requires approval. This dataset sidesteps that entirely, at the cost of being a snapshot rather than a live feed.
What are the legal and ethical landmines?
This is where the story gets complicated. The dataset uploader is explicit about the risks.
Collection was contrary to TikTok's terms of service. The dataset is not affiliated with, endorsed by, or connected to TikTok or ByteDance. If your company builds on this data, you are building on data that was collected in violation of a platform's ToS, and that carries enforcement risk. TikTok could issue takedowns, and ByteDance has shown willingness to pursue scrapers legally.
More seriously, captions are written by real people. The dataset contains personal data under the GDPR, the UK GDPR, and the CCPA, regardless of the fact that the content was publicly posted. The Hugging Face listing states this plainly: if you are in a jurisdiction those regulations cover, that obligation is yours the moment you download it. You cannot use this data to identify, profile, target, or contact individuals.
There is also a mirror of the dataset under a different account, which raises questions about provenance and persistence. If you build a pipeline on top of this data, you should expect it could disappear.
The deliberate exclusion of creator identity is a mitigation, not a solution. Captions can contain identifying information. Mentions contain account IDs. The country and language fields are TikTok's own inferences, not verified, and the uploader warns they are wrong often enough that you should not treat them as ground truth.
What should you do with this dataset right now?
If you are a researcher or a builder evaluating social data at scale, here is the pragmatic read.
Start by querying one Parquet file locally. The Hugging Face listing includes a DuckDB snippet that aggregates plays by sound from 2025 onward. One file is about 10 GB and holds roughly 167 million videos. Pull that, run a few queries, and see whether the fields and coverage match your use case before committing to the full 289 GB download.
If you are training models on this data, shuffle your batches. Rows are grouped by creator, not randomized. Reading sequentially gives you highly correlated batches that will wreck your gradient estimates. The uploader flags this explicitly.
Do not treat engagement counts as comparable across videos collected at different times. Every number is a snapshot taken at collection time, somewhere in a three-week window. A video collected on day one and a video collected on day twenty have had different amounts of time to accumulate views. Normalize for age before comparing raw counts.
Do not treat this as a census. It is 27 of 32 storage partitions, split on a hash of the creator ID. That makes it an unbiased random subset of what was collected, but what was collected is itself not all of TikTok. Coverage skews toward creators the system could reach through the crawling strategy, and that strategy is not documented in full.
For a sense of how LLMs interact with scraped and potentially biased data, consider the lessons from our earlier reporting on how labs filter nearly half of AI conversations. The filtering problem cuts both ways: what gets scraped shapes what models learn, and what gets left out is invisible.
If you are building a product on TikTok data, weigh the ToS and privacy risk carefully. The dataset is useful for research and prototyping. Shipping it in a commercial product is a different decision. If you are in the EU or UK, the GDPR exposure alone should make you pause before integrating this into anything user-facing.
The real story is the API layer
The dataset is impressive. 4.5 billion rows is a serious corpus. But the lasting impact is the documentation of TikTok's mobile API. Twenty-four endpoints with measured success rates, full request signing details, and a working implementation that collected nearly 6 billion records in three weeks. That is a blueprint, and it is now public.
For platform teams, the lesson is clear. Your private mobile API is not private. It is reverse-engineerable, it returns more data than your web layer, and someone will build a system on it that scales to billions of requests. If your threat model only covers web scraping, you are missing the larger exposure surface.
For builders, the question is whether the three-week collection window and the snapshot nature of the data fit your use case. If you need historical trends, you have them. If you need real-time monitoring, you do not. If you need a training corpus for content understanding, this is the best one available right now. If you need something you can ship in a product, the legal and ethical questions may outweigh the data quality.
