Pool简介
Pool 是在CatBoost中用作训练模型的数据结构。
Pool的案例应用
train_dataPool = Pool(data=[[12, 14, 16, 18], [23, 25, 27, 29], [32, 34, 36, 38]],
label=[10, 20, 30],
weight=[0.1, 0.2, 0.3]) # weight各自权重默认设置为1
Pool源代码解读
class Pool Found at: catboost.core
class Pool(_PoolBase):
"""
Pool used in CatBoost as a data structure to train model from.
"""
def __init__(
self,
data,
label=None,
cat_features=None,
text_features=None,
embedding_features=None,
column_description=None,
pairs=None,
delimiter='\t',
has_header=False,
ignore_csv_quoting=False,
weight=None,
group_id=None,
group_weight=None,
subgroup_id=None,
pairs_weight=None,
baseline=None,
feature_names=None,
thread_count=-1):
"""
Pool is an internal data structure that is used by CatBoost.
You can construct Pool from list, numpy.ndarray, pandas.
DataFrame, pandas.Series.
Parameters
----------
Pool 是在CatBoost中用作训练模型的数据结构。
data : list or numpy.ndarray or pandas.DataFrame or pandas.Series or FeaturesData or string. Data source of Pool. If list or numpy.ndarrays or pandas.DataFrame or pandas. Series, giving 2 dimensional array like data. If FeaturesData - see FeaturesData description for details, 'cat_features' and 'feature_names' parameters must be equal to None in this case. If string, giving the path to the file with data in catboost format. If path starts with "quantized://", the file has to contain quantized dataset saved with Pool.save().
label : list or numpy.ndarrays or pandas.DataFrame or pandas.Series, optional (default=None). Label of the training data. If not None, giving 1 or 2 dimensional array like data with floats. If data is a file, then label must be in the file, that is label must be equals to None cat_features : list or numpy.ndarray, optional (default=None). If not None, giving the list of Categ features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type
text_features : list or numpy.ndarray, optional . (default=None), If not None, giving the list of Text features indices or names. If it contains feature names, Pool's feature names must
be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type
embedding_features : list or numpy.ndarray, optional . (default=None). If not None, giving the list of Embedding features indices or names. If it contains feature names, Pool's feature names must be defined: either by passing 'feature_names' parameter or if data is pandas.DataFrame (feature names are initialized from it's column names). Must be None if 'data' parameter has FeaturesData type
column_description : string, optional (default=None). ColumnsDescription parameter. There are several columns description types: Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp. All columns are Num as default, it's not necessary to specify this type of columns. Default Label column index is 0 (zero). If None, Label column is 0 (zero) as default, all data columns are Num as default. If string, giving the path to the file with ColumnsDescription in column_description format.
pairs : list or numpy.ndarray or pandas.DataFrame or string.The pairs description. If list or numpy.ndarrays or pandas.DataFrame, giving 2 dimensional. The shape should be Nx2, where N is the pairs' count. The first element of the pair is the index of winner object in the training set. The second element of the pair is the index of loser object in the training set. If string, giving the path to the file with pairs description.
delimiter : string, optional (default='\t'). Delimiter to use for separate features in file. Should be only one symbol, otherwise would be taken only the first character of the string.
has_header : bool optional (default=False). If True, read column names from first line.
ignore_csv_quoting : bool optional (default=False). If True ignore quoting '"'.
weight : list or numpy.ndarray, optional (default=None). Weight for each instance. If not None, giving 1 dimensional array like data.
group_id : list or numpy.ndarray, optional (default=None). group id for each instance. If not None, giving 1 dimensional array like data.
group_weight : list or numpy.ndarray, optional .(default=None).Group weight for each instance. If not None, giving 1 dimensional array like data.
subgroup_id : list or numpy.ndarray, optional . (default=None) .subgroup id for each instance. If not None, giving 1 dimensional array like data.
pairs_weight : list or numpy.ndarray, optional .(default=None). Weight for each pair. If not None, giving 1 dimensional array like pairs.
baseline : list or numpy.ndarray, optional (default=None).Baseline for each instance. If not None, giving 2 dimensional array like data.
feature_names : list or string, optional (default=None). If list - list of names for each given data_feature. If string - path with scheme for feature names data to load. If this parameter is None and 'data' is pandas.DataFrame feature names will be initialized from DataFrame's column names. Must be None if 'data' parameter has FeaturesData type
thread_count : int, optional (default=-1).Thread count for data processing. If -1, then the number of threads is set to the number of CPU cores.
data : 列表或numpy。ndarray或pandas.DataFrame或pandas.Series或特性数据或字符串。池的数据源。如果列表或numpy。ndarrays或pandas.DataFrame或pandas.Series，给出类似数据的二维数组。如果FeaturesData -参见FeaturesData描述的详细信息，在这种情况下'cat_features'和'feature_names'参数必须等于None。如果字符串，以catboost格式提供文件的路径。如果path以"quantized://"开头，则文件必须包含保存在Pool.save()中的量化数据集。
label : 列表或numpy。ndarrays或pandas.DataFrame或pandas.Series,可选(默认= None)。训练数据的标签。如果不是None，则给出1维或2维的数组，如带有浮点数的data。如果data是一个文件，那么label必须在文件中，即label必须等于None cat_features: list或numpy。ndarray,可选(默认= None)。如果不是None，则给出类别特征索引或名称的列表。如果包含特性名称，则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None
text_features : 列表或numpy。ndarray,可选的。(default=None)，如果不是None，则给出文本特性的索引或名称。如果它包含特性名称，则池的特性名称必须
be defined: 通过传递'feature_names'参数或如果数据是pandas.DataFrame特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None
embedding_features : list或numpy。ndarray,可选的。(默认= None)。如果不是None，则给出嵌入特性的索引或名称。如果包含特性名称，则必须定义池的特性名称:通过传递'feature_names'参数或如果data是pandas。DataFrame(特性名称从它的列名初始化)。如果“data”参数具有FeaturesData类型，必须为None
column_description : 字符串，可选(默认为None)。ColumnsDescription参数。有几个列描述类型:Label, Categ, Num, Auxiliary, DocId, Weight, Baseline, GroupId, Timestamp。默认情况下，所有列都是Num，没有必要指定这种类型的列。默认标签列索引为0(零)。如果为None，则Label column默认为0(0)，所有数据列默认为Num。如果是字符串，则以column_description格式给出该文件的路径。
pairs : 列表或numpy。ndarray或pandas.DataFrame或字符串。对描述。如果列表或numpy。ndarrays或pandas.DataFrame，给出二维。形状应该是Nx2，其中N是对的计数。pair的第一个元素是训练集中赢家对象的索引。对的第二个元素是输家对象在训练集中的索引。如果是字符串，则给出文件的路径和对的描述。
delimiter : 字符串，可选(默认='\t')。用于文件中单独特性的分隔符。应该只有一个符号，否则将只取字符串的第一个字符。
has_header : bool可选(默认为False)。如果为True，则从第一行读取列名。
ignore_csv_quoting :bool可选(默认为False)。如果为真，请忽略“”。
weight :列表或numpy。ndarray,可选(默认= None)。每个实例的权重。如果不是None，则给出类似data的一维数组。
group_id : list或numpy。ndarray,可选(默认= None)。每个实例的组id。如果不是None，则给出类似data的一维数组。
group_weight : list或numpy。ndarray,可选的。(默认= None)。每个实例的组权重。如果不是None，则给出类似data的一维数组。
subgroup_id : list或numpy。ndarray,可选的。(default=None) .subgroup每个实例的id。如果不是None，则给出类似data的一维数组。
pairs_weight :列表或numpy。ndarray,可选的。(默认= None)。每双的重量。如果不是None，则给出一个一维数组。
baseline：列表或numpy。ndarray,可选(默认= None)。每个实例的基线。如果不是None，则给出像data这样的二维数组。
feature_names : 列表或字符串，可选(默认为None)。If list -每个给定data_feature的名称列表。If string - path with scheme for feature names要加载的数据。如果此参数为None且'data'为pandas.DataFrame特性名称将从DataFrame的列名初始化。如果“data”参数具有FeaturesData类型，必须为None
thread_count : nt，可选(默认=-1)。用于数据处理的线程计数。如果-1，则线程数设置为CPU核数。
if data is not None:
self._check_data_type(data)
self._check_data_empty(data)
if pairs is not None and isinstance(data, STRING_TYPES) !
= isinstance(pairs, STRING_TYPES):
raise CatBoostError("data and pairs parameters should
be the same types.")
if column_description is not None and not isinstance
(data, STRING_TYPES):
raise CatBoostError("data should be the string type if
column_description parameter is specified.")
if isinstance(data, STRING_TYPES):
if any(v is not None for v in [cat_features,
text_features, embedding_features, weight, group_id,
group_weight,
subgroup_id, pairs_weight, baseline, label]):
raise CatBoostError(
"cat_features, text_features, embedding_features,
weight, group_id, group_weight, subgroup_id, pairs_weight, "
"baseline, label should have the None type when
the pool is read from the file.")
if (feature_names is not None) and (not isinstance
(feature_names, STRING_TYPES)):
raise CatBoostError(
"feature_names should have None or string type
when the pool is read from the file.")
self._read(data, column_description, pairs,
feature_names, delimiter, has_header, ignore_csv_quoting,
thread_count)
else:
if isinstance(data, FeaturesData):
if any(v is not None for v in [cat_features,
text_features, embedding_features, feature_names]):
raise CatBoostError(
"cat_features, text_features,
embedding_features, feature_names should have the None
type"
" when 'data' parameter has FeaturesData type")
elif isinstance(data, np.ndarray):
if (data.dtype.kind == 'f') and (cat_features is not
None) and (len(cat_features) > 0):
raise CatBoostError("'data' is numpy array of
floating point numerical type, it means no categorical features,"
" but 'cat_features' parameter specifies nonzero
number of categorical features")
if (data.dtype.kind == 'f') and (text_features is not
None) and (len(text_features) > 0):
raise CatBoostError(
"'data' is numpy array of floating point
numerical type, it means no text features,"
" but 'text_features' parameter specifies
nonzero number of text features")
if (data.dtype.kind != 'O') and (embedding_features
is not None) and (len(embedding_features) > 0):
raise CatBoostError(
"'data' is numpy array of non-object type, it
means no embedding features,"
" but 'embedding_features' parameter specifies
nonzero number of embedding features")
elif isinstance(data, scipy.sparse.spmatrix):
if (data.dtype.kind == 'f') and (cat_features is not
None) and (len(cat_features) > 0):
raise CatBoostError("'data' is scipy.sparse.
spmatrix of floating point numerical type, it means no
categorical features,"
" but 'cat_features' parameter specifies nonzero
number of categorical features")
if (text_features is not None) and (len(text_features)
> 0):
raise CatBoostError(
"'data' is scipy.sparse.spmatrix, it means no text
features,"
" but 'text_features' parameter specifies
nonzero number of text features")
if (embedding_features is not None) and (len
(embedding_features) > 0):
raise CatBoostError(
"'data' is scipy.sparse.spmatrix, it means no
embedding features,"
" but 'embedding_features' parameter specifies
nonzero number of embedding features")
if isinstance(feature_names, STRING_TYPES):
raise CatBoostError(
"feature_names must be None or have non-string
type when the pool is created from "
"python objects.")
self._init(data, label, cat_features, text_features,
embedding_features, pairs, weight, group_id, group_weight,
subgroup_id, pairs_weight, baseline, feature_names,
thread_count)
super(Pool, self).__init__()
def _check_files(self, data, column_description, pairs):
"""
Check files existence.
"""
if data.find('://') == -1 and not os.path.isfile(data):
raise CatBoostError("Invalid data path='{}': file does not
exist.".format(data))
if column_description is not None and column_description.
find('://') == -1 and not os.path.isfile(column_description):
raise CatBoostError("Invalid column_description
path='{}': file does not exist.".format(column_description))
if pairs is not None and pairs.find('://') == -1 and not os.
path.isfile(pairs):
raise CatBoostError("Invalid pairs path='{}': file does not
exist.".format(pairs))
def _check_delimiter(self, delimiter):
if not isinstance(delimiter, STRING_TYPES):
raise CatBoostError("Invalid delimiter type={} : must be
str().".format(type(delimiter)))
if len(delimiter) < 1:
raise CatBoostError("Invalid delimiter length={} : must
be > 0.".format(len(delimiter)))
def _check_column_description_type(self,
column_description):
"""
Check type of column_description parameter.
"""
if not isinstance(column_description, STRING_TYPES):
raise CatBoostError("Invalid column_description type={}:
must be str().".format(type(column_description)))
def _check_string_feature_type(self, features, features_name):
"""
Check type of cat_feature parameter.
"""
if not isinstance(features, (list, np.ndarray)):
raise CatBoostError("Invalid {} type={}: must be list() or
np.ndarray().".format(features_name, type(features)))
def _check_string_feature_value(self, features,
features_count, features_name):
"""
Check values in cat_feature parameter. Must be int indices.
"""
for indx, feature in enumerate(features):
if not isinstance(feature, INTEGER_TYPES):
raise CatBoostError("Invalid {}[{}] = {} value type={}:
must be int().".format(features_name, indx, feature, type
(feature)))
if feature >= features_count:
raise CatBoostError("Invalid {}[{}] = {} value: index must
be < {}.".format(features_name, indx, feature, features_count))
def _check_pairs_type(self, pairs):
"""
Check type of pairs parameter.
"""
if not isinstance(pairs, (list, np.ndarray, DataFrame)):
raise CatBoostError("Invalid pairs type={}: must be list(),
np.ndarray() or pd.DataFrame.".format(type(pairs)))
def _check_pairs_value(self, pairs):
"""
Check values in pairs parameter. Must be int indices.
"""
for pair_id, pair in enumerate(pairs):
if (len(pair) != 2):
raise CatBoostError("Length of pairs[{}] isn't equal to
2.".format(pair_id))
for i, index in enumerate(pair):
if not isinstance(index, INTEGER_TYPES):
raise CatBoostError("Invalid pairs[{}][{}] = {} value
type={}: must be int().".format(pair_id, i, index, type(index)))
def _check_data_type(self, data):
"""
Check type of data.
"""
if not isinstance(data, (STRING_TYPES, ARRAY_TYPES,
SPARSE_MATRIX_TYPES, FeaturesData)):
raise CatBoostError(
"Invalid data type={}: data must be list(), np.ndarray(),
DataFrame(), Series(), FeaturesData " + " scipy.sparse matrix or
filename str().".format(type(data)))
def _check_data_empty(self, data):
"""
Check that data is not empty (0 objects is ok).
note: already checked if data is FeatureType, so no need
to check again
"""
if isinstance(data, STRING_TYPES):
if not data:
raise CatBoostError("Features filename is empty.")
elif isinstance(data, (ARRAY_TYPES,
SPARSE_MATRIX_TYPES)):
data_shape = np.shape(data)
if len(data_shape) == 1 and data_shape[0] > 0:
if isinstance(data[0], Iterable):
data_shape = tuple(data_shape + tuple([len(data
[0])]))
else:
data_shape = tuple(data_shape + tuple([1]))
if not len(data_shape) == 2:
raise CatBoostError("Input data has invalid shape: {}.
Must be 2 dimensional".format(data_shape))
if data_shape[1] == 0:
raise CatBoostError("Input data must have at least one
feature")
def _check_label_type(self, label):
"""
Check type of label.
"""
if not isinstance(label, ARRAY_TYPES):
raise CatBoostError("Invalid label type={}: must be array
like.".format(type(label)))
def _check_label_empty(self, label):
"""
Check label is not empty.
"""
if len(label) == 0:
raise CatBoostError("Labels variable is empty.")
def _check_label_shape(self, label, samples_count):
"""
Check label length and dimension.
"""
if len(label) != samples_count:
raise CatBoostError("Length of label={} and length of
data={} is different.".format(len(label), samples_count))
def _check_baseline_type(self, baseline):
"""
Check type of baseline parameter.
"""
if not isinstance(baseline, ARRAY_TYPES):
raise CatBoostError("Invalid baseline type={}: must be
array like.".format(type(baseline)))
def _check_baseline_shape(self, baseline, samples_count):
"""
Check baseline length and dimension.
"""
if len(baseline) != samples_count:
raise CatBoostError("Length of baseline={} and length of
data={} are different.".format(len(baseline), samples_count))
if not isinstance(baseline[0], Iterable) or isinstance(baseline
[0], STRING_TYPES):
raise CatBoostError("Baseline must be 2 dimensional
data, 1 column for each class.")
try:
if np.array(baseline).dtype not in (np.dtype('float'), np.
dtype('float32'), np.dtype('int')):
raise CatBoostError()
except CatBoostError:
raise CatBoostError("Invalid baseline value type={}: must
be float or int.".format(np.array(baseline).dtype))
def _check_weight_type(self, weight):
"""
Check type of weight parameter.
"""
if not isinstance(weight, ARRAY_TYPES):
raise CatBoostError("Invalid weight type={}: must be
array like.".format(type(weight)))
def _check_weight_shape(self, weight, samples_count):
"""
Check weight length.
"""
if len(weight) != samples_count:
raise CatBoostError("Length of weight={} and length of
data={} are different.".format(len(weight), samples_count))
if not isinstance(weight[0], (INTEGER_TYPES,
FLOAT_TYPES)):
raise CatBoostError("Invalid weight value type={}: must
be 1 dimensional data with int, float or long types.".format(type
(weight[0])))
def _check_group_id_type(self, group_id):
"""
Check type of group_id parameter.
"""
if not isinstance(group_id, ARRAY_TYPES):
raise CatBoostError("Invalid group_id type={}: must be
array like.".format(type(group_id)))
def _check_group_id_shape(self, group_id, samples_count):
"""
Check group_id length.
"""
if len(group_id) != samples_count:
raise CatBoostError("Length of group_id={} and length
of data={} are different.".format(len(group_id), samples_count))
def _check_group_weight_type(self, group_weight):
"""
Check type of group_weight parameter.
"""
if not isinstance(group_weight, ARRAY_TYPES):
raise CatBoostError("Invalid group_weight type={}: must
be array like.".format(type(group_weight)))
def _check_group_weight_shape(self, group_weight,
samples_count):
"""
Check group_weight length.
"""
if len(group_weight) != samples_count:
raise CatBoostError("Length of group_weight={} and
length of data={} are different.".format(len(group_weight),
samples_count))
if not isinstance(group_weight[0], (FLOAT_TYPES)):
raise CatBoostError("Invalid group_weight value type={}:
must be 1 dimensional data with float types.".format(type
(group_weight[0])))
def _check_subgroup_id_type(self, subgroup_id):
"""
Check type of subgroup_id parameter.
"""
if not isinstance(subgroup_id, ARRAY_TYPES):
raise CatBoostError("Invalid subgroup_id type={}: must
be array like.".format(type(subgroup_id)))
def _check_subgroup_id_shape(self, subgroup_id,
samples_count):
"""
Check subgroup_id length.
"""
if len(subgroup_id) != samples_count:
raise CatBoostError("Length of subgroup_id={} and
length of data={} are different.".format(len(subgroup_id),
samples_count))
def _check_feature_names(self, feature_names,
num_col=None):
if num_col is None:
num_col = self.num_col()
if not isinstance(feature_names, Sequence):
raise CatBoostError("Invalid feature_names type={} :
must be list".format(type(feature_names)))
if len(feature_names) != num_col:
raise CatBoostError("Invalid length of feature_names={} :
must be equal to the number of columns in data={}".format(len
(feature_names), num_col))
def _check_thread_count(self, thread_count):
if not isinstance(thread_count, INTEGER_TYPES):
raise CatBoostError("Invalid thread_count type={} : must
be int".format(type(thread_count)))