---
title: Multiple Tokenizers Per Field
description: Apply different token configurations to the same field
canonical: https://docs.paradedb.com/documentation/tokenizers/multiple-per-field
---

In many cases, a text field needs to be tokenized multiple ways. For instance, using the [unicode](/documentation/tokenizers/available-tokenizers/unicode)
tokenizer for search, and the [literal](/documentation/tokenizers/available-tokenizers/literal) tokenizer for [Top K ordering](/documentation/sorting/topk).

To tokenize a field in more than one way, append an `alias=<alias_name>` argument to the additional tokenizer configurations.
The alias name can be any string you like. For instance, the following statement tokenizes `description` using both the simple and literal tokenizers.

```sql
CREATE INDEX search_idx ON mock_items
USING bm25 (
  id,
  (description::pdb.literal),
  (description::pdb.simple('alias=description_simple'))
) WITH (key_field='id');
```

Under the hood, two distinct fields are created in the index: a field called `description`, which uses the literal tokenizer,
and an aliased field called `description_simple`, which uses the simple tokenizer.

To query against the aliased field, cast it to `pdb.alias('alias_name')`:

<CodeGroup>
```sql SQL
-- Query against `description_simple`
SELECT description, rating, category
FROM mock_items
WHERE description::pdb.alias('description_simple') ||| 'Sleek running shoes';

-- Query against `description`
SELECT description, rating, category
FROM mock_items
WHERE description ||| 'Sleek running shoes';
```

```python Django
from paradedb import Match, ParadeDB

# Query against `description_simple`
MockItem.objects.extra(
    where=["(description::pdb.alias('description_simple')) ||| 'Sleek running shoes'"]
).values('description', 'rating', 'category')

# Query against `description`
MockItem.objects.filter(
    description=ParadeDB(Match('Sleek running shoes', operator='OR'))
).values('description', 'rating', 'category')
```

```ruby Rails
# Query against `description_simple`
MockItem.search(:description_simple)
        .matching_any("Sleek running shoes")
        .select(:description, :rating, :category)

# Query against `description`
MockItem.search(:description)
        .matching_any("Sleek running shoes")
        .select(:description, :rating, :category)
```

</CodeGroup>

<Note>
If a text field uses multiple tokenizers and one of them is [literal](/documentation/tokenizers/available-tokenizers/literal), we recommend aliasing
the other tokenizers and leaving the literal tokenizer un-aliased. This is so queries that `GROUP BY`, `ORDER BY`, or aggregate the
text field can reference the field directly:

<CodeGroup>
```sql SQL
CREATE INDEX search_idx ON mock_items
USING bm25 (
  id,
  (description::pdb.literal),
  (description::pdb.simple('alias=description_simple'))
) WITH (key_field='id');

SELECT description, rating, category
FROM mock_items
WHERE description @@@ 'shoes'
ORDER BY description
LIMIT 5;
```

```python Django
from paradedb import ParadeDB, ParseWithField

MockItem.objects.filter(
    description=ParadeDB(ParseWithField(query='shoes'))
).values('description', 'rating', 'category').order_by('description')[:5]
```

```ruby Rails
MockItem.search(:description)
        .parse("shoes")
        .select(:description, :rating, :category)
        .order(:description)
        .limit(5)
```

</CodeGroup>

</Note>