In this tutorial, we demonstrate how to build a complete, production-style pipeline for detecting and redacting personally identifiable information (PII) using the OpenAI Privacy Filter model. The process begins by setting up the environment and loading a token classification model that can identify multiple categories of sensitive data, including names, emails, phone numbers, addresses, and secrets. We then design helper functions to normalize labels, extract structured spans, and transform raw model outputs into usable formats. Next, we implement a configurable redaction system that replaces sensitive entities with meaningful placeholders, preserving privacy while maintaining contextual clarity. Throughout the tutorial, we test the pipeline on curated examples, convert outputs into structured dataframes, and prepare the system for batch processing and real-world usage.
To get started, we install all required libraries and set up the runtime environment, configuring device selection and initializing output paths. After loading the tokenizer and model, we create a token classification pipeline with aggregation. The model is trained to recognize labels such as account_number, private_address, private_email, private_person, private_phone, private_url, and private_date. We define a mapping for each label to a corresponding placeholder (e.g., [PRIVATE_EMAIL], [PRIVATE_PHONE]), and write functions to extract entity spans from the classifier output. The redaction function iterates over detected entities in reverse order to avoid offset issues, replacing each with the appropriate placeholder. Finally, we test the pipeline on sample texts, display results in a dataframe, and discuss how to extend the approach for batch processing.