​

tokenize_dataset

This is a method of the TrainModel class in the train module.

​

Signature

def tokenize_dataset(dataset: Any) -> Any

​

Parameters

​

Returns

​

Uses

• tokenize_function
• tokenized_dataset.remove_columns

​

Used By

• TrainModel.train_model

​

Source