文字列とテキスト処理

Why Tokenisation Matters

• A typical language model has a vocabulary of 50,000–100,000 tokens.
• Each token maps to a number (its ID), which the model actually processes.
• The way text is tokenised affects cost - more tokens means more computation and higher API fees.

Pattern Matching - Finding Needles in Haystacks

A core string operation is searching for a pattern within a larger text. Does this email contain the word "urgent"? Does this code contain a security vulnerability?

The Simple Approach

Slide the pattern along the text, checking character by character:

text:    "the cat sat on the mat"
pattern: "cat"

Position 0: "the" → no match
Position 1: "he " → no match
Position 4: "cat" → match found at index 4!

This is O(n × m) in the worst case, where n is the text length and m is the pattern length. For short patterns, it's fine. For scanning millions of documents, we need smarter approaches.

Classic String Challenges

String Reversal

Reversing a string is simple but reveals how you think about data:

reverse("hello") → "olleh"

AI uses reversal in sequence-to-sequence models - for instance, some early translation models reversed the input sentence to improve accuracy.

Palindrome Detection

A palindrome reads the same forwards and backwards: "racecar", "madam", "level".

is_palindrome(text):
    return text == reverse(text)

Anagram Detection

Two words are anagrams if they contain the same characters in a different order: "listen" and "silent".

The elegant solution? Count character frequencies using a hash map:

are_anagrams(word1, word2):
    return character_counts(word1) == character_counts(word2)

This connects directly to the frequency counting pattern from the previous lesson - hash maps make it O(n).

Regular Expressions - Pattern Matching on Steroids

Regular expressions (regex) let you describe patterns rather than exact text:

| Pattern | Matches | Use Case | |---------|---------|----------| | \d+ | One or more digits | Extracting numbers from text | | [A-Z][a-z]+ | A capitalised word | Finding proper nouns | | \b\w+@\w+\.\w+\b | Email addresses | Data extraction | | (cat\|dog\|bird) | Any of three words | Classification keywords |

AI data pipelines use regex extensively for data cleaning - removing HTML tags, extracting dates, standardising phone numbers, and filtering out unwanted characters before training.

Real-World AI Text Processing Pipeline

Here's a simplified view of how an AI processes text:

1. Raw text → "The café's Wi-Fi isn't working!!!"
2. Lowercasing → "the café's wi-fi isn't working!!!"
3. Removing punctuation → "the cafés wifi isnt working"
4. Tokenisation → ["the", "café", "s", "wi", "fi", "isn", "t", "working"]
5. Token IDs → [1, 8432, 82, 5901, 3344, 2817, 83, 1562]
6. Into the model → Numbers the AI can actually process

Every step involves string operations - slicing, searching, replacing, and splitting.

Key Takeaways

• Strings are sequences of characters - the raw material for all text-based AI.
• Tokenisation bridges human language and machine learning by splitting text into processable pieces.
• Pattern matching and regex are essential tools for cleaning and extracting data.
• Classic string problems (reversal, palindromes, anagrams) build the thinking skills needed for AI text processing.
• Every message you send to ChatGPT passes through a pipeline of string operations before the model sees it.