# Data freshness and scraping

> When trackagoat updates profiles, posts and stats, how to trigger an update yourself, and what to do when data looks stale.

## Overview

trackagoat fetches creator data through a background scraper service, per platform account. Data is **not** fetched in real time: it's collected on a schedule and stored so you can chart trends over time.

> **Note:**
>
> trackagoat is not affiliated with TikTok or Instagram. Data is collected from public profiles and posts. Stats reflect what was observed at update time, so they can be hours behind what you see on TikTok or Instagram directly.

## Three scrape types

| Type | What it fetches | Runs for |
|------|----------------|----------|
| **Creator Profiles** | Follower count, total likes, video count, bio, avatar | Each tracked account |
| **Video Stats** | View, like, comment, share, and save counts | Each tracked video |
| **Discover Videos** | New videos posted by a tracked account since last check | Each tracked account |

Each type runs on its own schedule and can be triggered independently.

### Discovery completeness

Each discovery pass fetches a creator's most recent posts. To make sure nothing is permanently missed, trackagoat also runs a cost-bounded **reconcile**: every tracked creator periodically (roughly once a month) gets a deep re-scan that walks further back through their history and fills in any posts an earlier pass skipped.

Accounts with a detected gap (where the number of posts trackagoat has captured is well below the post count shown on their platform profile) are **prioritized** for reconcile, so real gaps are typically closed within a cycle or two rather than waiting for the full monthly rotation. You can see which creators currently have a gap in the **Tracking Gaps** widget on the [Program Health](https://www.trackagoat.com/docs/guides/program-health.md) page.

> **Note:**
>
> Photo carousels (TikTok `/photo/` posts) are fully supported: they're discovered, shown alongside videos with a link to the carousel, and their stats are updated on the same age-based schedule as videos while they're new (below).

## Automatic schedule

Updates run automatically in the background. You don't need to do anything once an account is added.

| What | How often |
|------|-----------|
| **Profiles** (followers, likes, post count) | Daily |
| **New posts** (discovery) | About twice a day (once a day on Free) |
| **Full-history re-scan** of each account | About every 30 days, sooner for accounts with a detected gap |
| **Accounts whose last check failed** | Retried within about 2 hours |
| **Video stats** | By the post's age (below) |

Video stats are re-checked most often while a post is new, because that's when the numbers move:

| Post age | TikTok video stats re-checked | TikTok photo carousel stats re-checked |
|----------|-------------------------------|----------------------------------------|
| Under 3 days | Every 12 hours | Every 12 hours |
| 3 to 7 days | Every 24 hours | Every 24 hours |
| 7 to 30 days | Every 48 hours | Every 48 hours |
| 30 to 90 days | Weekly | Weekly |
| Over 90 days | Every 2 weeks | Monthly |

Instagram follows a similar curve and slows down sooner as posts age.

When a new-posts check sees a TikTok post, the views, likes, comments, shares and saves it reads are saved as a stats update too, so recent posts are often fresher than this schedule.

Posts waiting in the [Tracking Inbox](https://www.trackagoat.com/docs/guides/inbox.md) aren't on this schedule, but posts under 30 days old get their stats updated once a day, so you decide on current numbers.

### Plan floors

Your plan sets the **minimum time between runs** of each update type for your org. The scheduler enforces these floors: if your org's saved interval is shorter than your plan allows, it runs at the plan floor instead. They are floors, not the refresh rate of any single video: even on Ultra, a video is re-checked on the age-based schedule above (12 hours at most often).

| Plan | Profiles | New posts | Video stats |
|------|----------|-----------|-------------|
| **Free** | 24 hours | 24 hours | 24 hours |
| **Starter** | 12 hours | 12 hours | 4 hours |
| **Ultra** | 6 hours | 4 hours | 1 hour |

Org admins can set longer intervals, never shorter than the plan floor.

On **Free**, every update type runs at most once a day, so new posts and video stats are checked daily, including posts under 3 days old.

## Manual updates ("Update Now")

Org admins can trigger an update immediately without waiting for the next scheduled run. This is useful when you just added a creator and want fresh data right away, or when you know a big video just dropped.

**Where to find manual triggers:**

- **Creator detail page**: "Update Now" button in the header. Refreshes that one creator's profile. The "Find New Videos" option in the ⋯ menu runs a discovery scan for that creator specifically.
- **Video detail page → Details tab**: "Update Now" fetches fresh stats for that single video.
- **Org Settings → Scraping tab**: triggers an org-wide update for all creators/videos at once. Three separate buttons, one per job type.

When you click "Update Now", a dialog opens that:
1. Explains what will be fetched and for whom.
2. Shows your remaining daily quota (see below).
3. Asks you to confirm before anything runs.

After confirming, a toast confirms the scrape started. The page data updates automatically when the job completes (usually under a minute).

## Daily update quota

Manual updates count against a daily quota that resets at **midnight in your org's time zone**. Scheduled updates don't count. The Free plan has no manual updates; see the **Manual updates/day** column in [Plans and limits](https://www.trackagoat.com/docs/guides/limits.md) for each plan's current allowance.

**What counts as one update:**
- Clicking "Update Now" on a creator page = 1
- Clicking "Update Now" on a video page = 1
- Running an org-wide update type from the Scraping tab = 1 (however many creators or videos the org has)

Adding a video does **not** count: its first stats update starts right away and is free. If you add many videos within a minute, some of them get their first stats from the next scheduled update instead.

When you reach the limit, the "Update Now" dialog shows "Daily limit reached" and the button stays disabled until midnight in your org's time zone. Org admins can see today's usage under **Org Settings → Usage**.

## Viewing update status

**Creator list**: a data-freshness badge on each creator shows when their profile was last updated.

**Video list**: each video shows a tier badge indicating how often it's scheduled for updates. Hover the badge to see the next scheduled update time.

**Video detail → Details tab**: shows the tier and next scheduled update.

**Org Settings → Scraping tab** (admin only): shows the last run time, status (OK / Partial / Failed), and processed count for each job type across the whole org.

## Data freshness

Stats are **snapshots**, not live counters. A view count of 12,400 is what we saw at the last update, not necessarily the current count on the platform. Every number shows when it was fetched, and creators and videos carry a freshness badge (Fresh, Turning stale, Stale).

Charts are built from the stored snapshot history. A newly added creator or video shows no chart data until its first update completes.

## What if an update fails?

If the scraper encounters an error (a platform rate limit, a data-provider outage, a handle that no longer exists), the job is retried automatically with exponential backoff.

A creator that fails **5 consecutive** profile updates is automatically **auto-disabled** to stop wasting quota on a handle that may be gone. Auto-disabled creators stop being updated until an org admin re-enables them (subject to a 24-hour cooldown). See [Creators and accounts](https://www.trackagoat.com/docs/guides/accounts.md#auto-disabled-creators) for the re-enable flow.

If you notice a creator hasn't updated in a long time, check:

**Step 1: Check the handle**

**Is the account's handle still valid?** The account may have been deleted, renamed, or made private.

**Step 2: Check the Scraping tab**

**Is the org-wide Scraping tab showing errors?** If all job types are failing, the scraper service itself may be down: contact support.

**Step 3: Try a manual update**

**Try a manual "Update Now"**: if it succeeds, the prior failure was transient.

## Scraping and plan limits

The `min_*_hours` scrape frequency settings are **floors on scheduled scrapes**: they don't limit how often you can do manual triggers (that's the `max_manual_scrape_triggers_per_day` quota above).

See [Plans and limits](https://www.trackagoat.com/docs/guides/limits.md) for all limit keys and how enforcement works.

- [Limits](https://www.trackagoat.com/docs/guides/limits.md): Plan tiers and scrape frequency floors.
- [Program Health](https://www.trackagoat.com/docs/guides/program-health.md): Monitor data freshness and pipeline status.

---
Source: https://www.trackagoat.com/docs/guides/scraping