Use Case | How an Edtech Platform replaced manual fraud review with a daily risk-scoring system

When a trust and safety team is reviewing school accounts one at a time across disconnected tools, fraud screening does not scale. It accumulates. For an Edtech platform, Stackdrop replaced that manual process with a daily scoring system that surfaces risk across 10,000 schools, ranking and explaining it before an analyst opens a single account.

What was the trust and safety team doing before this existed?

The team was verifying school accounts manually, one at a time, using a collection of simple Retool apps, each built around two or three queries, and Jira tickets that held school information gathered during previous reviews. When an analyst needed to assess a school, they moved between systems: checking identity data in one place, payment history in another, cross-referencing a ticket that may or may not have been updated recently. Nothing was consolidated. There was no score, no ranked list, and no way to see at a glance which accounts warranted immediate attention versus which could wait.

The team knew what fraud looked like. The bottleneck was the time it took to assemble the picture. Each assessment required manual coordination across tools that were never designed to talk to each other, and the process didn't scale with the size of the school population.

How does the scoring system work?

The system runs daily across the full school population, approximately 10,000 schools. It Ingests Data from multiple sources simultaneously: the platform's own Data Warehouses, Stripe identity verification records, ChargeBee subscription and billing events, SIFT risk signals, fraud detection services, and the trust and safety team's Jira ticket history, where previous reviews and decisions are logged.

The scoring system delivers a highly detailed and scalable assessment of school risk by evaluating multiple dimensions of account activity simultaneously. It provides an extensive view of operational, financial, behavioural, and content-related risk signals, enabling analysts to identify problematic patterns with a level of consistency and coverage that would be difficult to achieve manually at scale.

The result is a consolidated daily risk profile for each school that not only quantifies overall risk exposure but also provides clear visibility into the specific factors contributing to elevated concern, allowing investigations and enforcement actions to be prioritized efficiently and accurately.

How do analysts interact with the results?

The analyst-facing dashboard presents a ranked list of schools ordered by current risk score. From the list, an analyst can open a detailed view on any school and see its identity verification status, active subscription plan, flagged course topics, recent signal flags, and a breakdown of how risk is distributed across the five indicator categories. Flags appear in plain language: SIFT score above threshold, down to student billing issues.

The school's score is tracked over time, creating a historical risk profile. This changes how triage decisions get made: a school with a high score that appeared overnight reads differently from one whose indicators have been accumulating.

Analysts can submit a correction directly from the dashboard if a data point looks wrong, creating a feedback loop without a separate ticket or an email to another team. The review lands in a dedicated queue where it can be actioned and tracked.

How do admins tune the system without developer involvement?

The admin dashboard gives fraud operations managers direct control over how the algorithm weights each indicator category. Weights are adjustable through the interface, and a simulation mode lets admins see how a proposed change would shift score distributions across the school population before it is committed. Blacklisted schools are managed in the same place.

This matters because fraud patterns shift. A weighting scheme calibrated to one threat profile may need adjustment as new patterns emerge. Giving the operations team control over the model's sensitivity, without creating a developer dependency for every change, is what makes the system durable beyond its initial configuration.

If you are evaluating a similar build or want to understand how the workflow architecture was structured, get in touch with the Stackdrop team.

FAQ

How do you process 10,000 schools daily without overloading the system? The driver workflow batches schools into groups of 1,000 and processes them sequentially. This batching architecture avoids saturation and query timeouts that arise when running large, multi-source queries in a single pass. Each batch runs as an independent workflow execution, so failures are isolated and retryable without restarting the full daily run.

What happens when a data source is temporarily unavailable? The algorithm is built to handle partial data without failing the run. The entire system serves as an ETL that saves high-level information for front-end use. Analysts see which signals are absent in the detailed school view, so gaps in the data are visible rather than silently absorbed into the score.

Can new data sources or indicators be added later? Yes. The component structure is designed for extension. Each module handles its own retrieval and scoring logic independently, so a new data source can be integrated into the relevant component without restructuring the rest of the algorithm. The admin weight-tuning interface means new indicators can be calibrated operationally once they are live, severity can be tuned, and blacklisted values can be edited without a change to the code.

Why was Retool used for the analyst and admin dashboards? The trust and safety team was already working inside Retool apps as part of their daily workflow. Building the new dashboards in Retool meant analysts could start using the scored output without a context switch to a new tool, and admins got direct access to weight configuration without waiting on engineering to expose a settings panel. The existing Retool environment also gave the team a familiar starting point for submitting corrections and managing the review queue.