文章目录
1 前言
大抵来说，从事机器算法框架工作的人员基本有两个比较大的流派。一种是算法派，一直以来都是从事算法方面的工作，算法的理论基础与算法调优经验非常充足，对于算法有较深的理解。另外一种是架构工程派，这类同学以前从事搜、广、推以及大数据等方面的工作，工程架构经验非常丰富，有自己的工程设计理念，对于计算性能有着极致的追求与探索。两种流派并无孰优孰略的问题，最终都会殊路同归，不过由于其研究经历，在学习方式、学习理念上有一定的差异，对于框架的描述也有所不同。
笔者属于先从事工程架构，然后转到算法建模领域，接着又杀回到算法框架领域，所以本章描述会综合算法与工程的思维方式进行阐述。
对于一个大的软件项目来说，有几个比较重要点：
- 优秀的设计理念如：软件分层、分阶段、良好的灵活性等；
- 高效的性能：较高的并发支持度QPS、较低的延迟Latency等；
- 优秀的易用性：可以不通过修改框架代码支持业务目标，较强的包容性；
那么，对于TensorFlow来说，支持声明式的编程，灵活高效，支持预编译等机制，那么他是如何做到的呢。其实大家在上大学的时候，我们学计算机的时候，老师都和我们讲过，程序 = 数据结构 + 算法。那么我们就来看看TensorFlow的数据结构与算法到底有何独到之处。
- 首先，TensorFlow支持任意的深度模型，要做到这点，需要将原子封装化，包括数据框架化和计算框架化，用白话来说，就是函数的接口要一致，数据的封装接口也要一致，这样才能支持任意的组合。那么对于Tensorflow的计算图来说，Tensor就是这个封装，也可以理解为接口数据的载体。
- 其次，Tensor作为统一的数据对外封装接口，大多是接口性质的，大多是临时的，所以对于需要常驻存储的数据节点来说我们采用其他的形式，比如Variable，也可以理解为模型数据的载体。
- 再次，上述已经统一了数据接口，然后对于算子来说，我们也需要统一算子（也就是函数）的接口，并且要保留灵活性，同时要支持用户自己定义的OP算子，也可以理解为模型计算的载体。
- 再次，通过算法模型同学定义模型结构，组装成计算图，定义出工程架构的计算的DAG。
- 再次，通过计算图DAG，用会话机制驱动图的运行。
以上完成一个具备“屠龙之技”的深度学习框架。
2 框架数据载体：张量
张量是TensorFlow框架对于整体数据流管理的重要的数据结构，通过提供统一的框架数据载体，从而更加方面的定义数学表达式，更加准确的描述数学模型。在实际的模型运转过程中，模型运转表达式中的数据就有张量来进行承载。TensorFlow提供Tensor和SparseTensor两种张量抽象，分别表示Dense数据与Sparse数据，后者旨在通过设计紧凑的数据节奏解决与优化高维稀疏的内存占比。
3 Dense数据载体:张量-Tensor
从TensorFlow的名字上，其实大家可以看出Tensor是一个非常重要的内容。在TensorFLow中，所有的数据都通过张量的形式对外表达。
2.1 定义
- 展示形式 ：从数据维度来进行考量，可以理解Tensor为一个N维数组。
- 零阶张量表达标量-Scalar
- 一阶张量表达向量-Vector
- N阶张量表达N维数组
- 具体实现
从TensorFlow代码的实现来看，Tensor并不直接存储实际的数据，大家可以理解Tensor为一个引用或者是指针，通过对实际数据进行封装，添加一层，对外展示出统一的样式，也就是说Tensor就是一个壳，也可以理解就是一个句柄，他存储的是张量的元信息以及指向实际数据的内存缓冲区的指针。那么为什么采用这样的设计呢。主要是为了实现内存复用，拒绝内存拷贝带来的时空开销。当某个前置操作的输出值被多个后置操作使用的时候，可以通过指针就行多重引用，无需进行重复存储，并且通过引用计数的方式进行数据的生命周期的管控，合理的进行内存调度，类似于Java虚拟机的垃圾回收机制。Tensor的定义代码如下。
- Python实现，代码路径 tensorflow/pyhton/framework/ops.py
@tf_export("Tensor")
class Tensor(_TensorLike):
"""Represents one of the outputs of an `Operation`.
A `Tensor` is a symbolic handle to one of the outputs of an
`Operation`. It does not hold the values of that operation's output,
but instead provides a means of computing those values in a
TensorFlow `tf.compat.v1.Session`.
This class has two primary purposes:
1. A `Tensor` can be passed as an input to another `Operation`.
This builds a dataflow connection between operations, which
enables TensorFlow to execute an entire `Graph` that represents a
large, multi-step computation.
2. After the graph has been launched in a session, the value of the
`Tensor` can be computed by passing it to
`tf.Session.run`.
`t.eval()` is a shortcut for calling
`tf.compat.v1.get_default_session().run(t)`.
In the following example, `c`, `d`, and `e` are symbolic `Tensor`
objects, whereas `result` is a numpy array that stores a concrete
value:
```python
# Build a dataflow graph.
c = tf.constant([[1.0, 2.0], [3.0, 4.0]])
d = tf.constant([[1.0, 1.0], [0.0, 1.0]])
e = tf.matmul(c, d)
# Construct a `Session` to execute the graph.
sess = tf.compat.v1.Session()
# Execute the graph and store the value that `e` represents in `result`.
result = sess.run(e)
"""
# List of Python operators that we allow to override.
OVERLOADABLE_OPERATORS = {
# Binary.
"__add__",
"__radd__",
"__sub__",
"__rsub__",
"__mul__",
"__rmul__",
"__div__",
"__rdiv__",
"__truediv__",
"__rtruediv__",
"__floordiv__",
"__rfloordiv__",
"__mod__",
"__rmod__",
"__lt__",
"__le__",
"__gt__",
"__ge__",
"__ne__",
"__eq__",
"__and__",
"__rand__",
"__or__",
"__ror__",
"__xor__",
"__rxor__",
"__getitem__",
"__pow__",
"__rpow__",
# Unary.
"__invert__",
"__neg__",
"__abs__",
"__matmul__",
"__rmatmul__"
}
# Whether to allow hashing or numpy-style equality
_USE_EQUALITY = tf2.enabled()
def __init__(self, op, value_index, dtype):
"""Creates a new `Tensor`.
Args:
op: An `Operation`. `Operation` that computes this tensor.
value_index: An `int`. Index of the operation's endpoint that produces
this tensor.
dtype: A `DType`. Type of elements stored in this tensor.
Raises:
TypeError: If the op is not an `Operation`.
"""
if not isinstance(op, Operation):
raise TypeError("op needs to be an Operation: %s" % op)
self._op = op
self._value_index = value_index
self._dtype = dtypes.as_dtype(dtype)
# This will be set by self._as_tf_output().
self._tf_output = None
# This will be set by self.shape().
self._shape_val = None
# List of operations that use this Tensor as input. We maintain this list
# to easily navigate a computation graph.
self._consumers = []
self._id = uid()
self._name = None
@property
def op(self):
"""The `Operation` that produces this tensor as an output."""
return self._op
@property
def dtype(self):
"""The `DType` of elements in this tensor."""
return self._dtype
@property
def graph(self):
"""The `Graph` that contains this tensor."""
return self._op.graph
@property
def name(self):
"""The string name of this tensor."""
if self._name is None:
if not self._op.name:
raise ValueError("Operation was not named: %s" % self._op)
self._name = "%s:%d" % (self._op.name, self._value_index)
return self._name
@property
def device(self):
"""The name of the device on which this tensor will be produced, or None."""
return self._op.device
@property
def shape(self):
"""Returns the `TensorShape` that represents the shape of this tensor.
The shape is computed using shape inference functions that are
registered in the Op for each `Operation`. See
`tf.TensorShape`
for more details of what a shape represents.
The inferred shape of a tensor is used to provide shape
information without having to launch the graph in a session. This
can be used for debugging, and providing early error messages. For
example:
```python
c = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
print(c.shape)
==> TensorShape([Dimension(2), Dimension(3)])
d = tf.constant([[1.0, 0.0], [0.0, 1.0], [1.0, 0.0], [0.0, 1.0]])
print(d.shape)
==> TensorShape([Dimension(4), Dimension(2)])
# Raises a ValueError, because `c` and `d` do not have compatible
# inner dimensions.
e = tf.matmul(c, d)
f = tf.matmul(c, d, transpose_a=True, transpose_b=True)
print(f.shape)
==> TensorShape([Dimension(3), Dimension(4)])
```
In some cases, the inferred shape may have unknown dimensions. If
the caller has additional information about the values of these
dimensions, `Tensor.set_shape()` can be used to augment the
inferred shape.
Returns:
A `TensorShape` representing the shape of this tensor.
"""
if self._shape_val is None:
self._shape_val = self._c_api_shape()
return self._shape_val
.......
- C++实现，代码路径 tensorflow/core/framework/tensor.h
class Tensor {
public:
/// \brief Creates a 1-dimensional, 0-element float tensor.
///
/// The returned Tensor is not a scalar (shape {}), but is instead
/// an empty one-dimensional Tensor (shape {0}, NumElements() ==
/// 0). Since it has no elements, it does not need to be assigned a
/// value and is initialized by default (IsInitialized() is
/// true). If this is undesirable, consider creating a one-element
/// scalar which does require initialization:
///
/// ```c++
///
/// Tensor(DT_FLOAT, TensorShape({}))
///
/// ```
Tensor();
/// \brief Creates a Tensor of the given `type` and `shape`. If
/// LogMemory::IsEnabled() the allocation is logged as coming from
/// an unknown kernel and step. Calling the Tensor constructor
/// directly from within an Op is deprecated: use the
/// OpKernelConstruction/OpKernelContext allocate_* methods to
/// allocate a new tensor, which record the kernel and step.
///
/// The underlying buffer is allocated using a `CPUAllocator`.
Tensor(DataType type, const TensorShape& shape);
/// \brief Creates a tensor with the input `type` and `shape`, using
/// the allocator `a` to allocate the underlying buffer. If
/// LogMemory::IsEnabled() the allocation is logged as coming from
/// an unknown kernel and step. Calling the Tensor constructor
/// directly from within an Op is deprecated: use the
/// OpKernelConstruction/OpKernelContext allocate_* methods to
/// allocate a new tensor, which record the kernel and step.
///
/// `a` must outlive the lifetime of this Tensor.
Tensor(Allocator* a, DataType type, const TensorShape& shape);
/// \brief Creates a tensor with the input `type` and `shape`, using
/// the allocator `a` and the specified "allocation_attr" to
/// allocate the underlying buffer. If the kernel and step are known
/// allocation_attr.allocation_will_be_logged should be set to true
/// and LogMemory::RecordTensorAllocation should be called after the
/// tensor is constructed. Calling the Tensor constructor directly
/// from within an Op is deprecated: use the
/// OpKernelConstruction/OpKernelContext allocate_* methods to
/// allocate a new tensor, which record the kernel and step.
///
/// `a` must outlive the lifetime of this Tensor.
Tensor(Allocator* a, DataType type, const TensorShape& shape,
const AllocationAttributes& allocation_attr);
/// \brief Creates a tensor with the input datatype, shape and buf.
///
/// Acquires a ref on buf that belongs to this Tensor.
Tensor(DataType type, const TensorShape& shape, TensorBuffer* buf);
/// \brief Creates an empty Tensor of the given data type.
///
/// Like Tensor(), returns a 1-dimensional, 0-element Tensor with
/// IsInitialized() returning True. See the Tensor() documentation
/// for details.
explicit Tensor(DataType type);
private:
// A tag type for selecting the `Tensor` constructor overload that creates a
// scalar tensor in host memory.
struct host_scalar_tag {};
class HostScalarTensorBufferBase;
template <typename T>
struct ValueAndTensorBuffer;
// Creates a tensor with the given scalar `value` in CPU memory.
template <typename T>
Tensor(T value, host_scalar_tag tag);
public:
// A series of specialized constructors for scalar tensors in host memory.
//
// NOTE: The `Variant` host-scalar constructor is not defined, because Variant
// is implicitly constructible from many different types, and this causes
// ambiguities with some compilers.
explicit Tensor(float scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(double scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(int32 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(uint32 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(uint16 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(uint8 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(int16 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(int8 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(tstring scalar_value)
: Tensor(std::move(scalar_value), host_scalar_tag{}) {}
explicit Tensor(complex64 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(complex128 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(int64 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(uint64 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(bool scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(qint8 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(quint8 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(qint16 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(quint16 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(qint32 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(bfloat16 scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(Eigen::half scalar_value)
: Tensor(scalar_value, host_scalar_tag{}) {}
explicit Tensor(ResourceHandle scalar_value)
: Tensor(std::move(scalar_value), host_scalar_tag{}) {}
// NOTE: The `const char*` host-scalar constructor is provided as a
// convenience because otherwise passing a string literal would surprisingly
// construct a DT_BOOL tensor.
explicit Tensor(const char* scalar_value)
: Tensor(tstring(scalar_value), host_scalar_tag{}) {}
/// Copy constructor.
Tensor(const Tensor& other);
/// \brief Move constructor. After this call, <other> is safely destructible
/// and can be assigned to, but other calls on it (e.g. shape manipulation)
/// are not valid.
Tensor(Tensor&& other);
~Tensor();
/// Returns the data type.
DataType dtype() const { return shape_.data_type(); }
/// Returns the shape of the tensor.
const TensorShape& shape() const { return shape_; }
/// \brief Convenience accessor for the tensor shape.
///
/// For all shape accessors, see comments for relevant methods of
/// `TensorShape` in `tensor_shape.h`.
int dims() const { return shape().dims(); }
/// Convenience accessor for the tensor shape.
int64 dim_size(int d) const { return shape().dim_size(d); }
/// Convenience accessor for the tensor shape.
int64 NumElements() const { return shape().num_elements(); }
bool IsSameSize(const Tensor& b) const {
return shape().IsSameSize(b.shape());
}
- 支持的数据类型
2.2典型案例
import tensorflow as tf
t_a = tf.constant([1.0, 2.0], name="t_a") # tf.constant是一个算子，这个计算的结果被张量t_a引用，并且对其生命周期负责
t_b = tf.constant([3.0, 4.0], name="t_b")
t_c = tf.add（t_a, t_b, name="add")
print t_c
输出
Tensor（"add:0", shape(2,), dtype=float32)
从这个示例可以看出，首先TensorFlow的张量与Python中的Numpy的数组是不一样的，他不仅仅有数据，还有一些元信息，其中主要的元信息包括以下三个：
- 名字 - name：名字是Tensor的唯一标识，同时也表明了这个张量是通过什么方式计算出来的，前面讲解计算图的时候讲过，计算图通过算子的组合与数据的流通进行实现，Tensor就是这个数据的流动，通过Tensor的名字进行标识。Tensor从计算图来说就是算子计算结果，同时算子计算结果可能有多个，所以张量的命名方式可以表示为“node：src_out_index”。其中node为计算这个Tensor节点的名字，src_out_index表示当前的张量是来自计算节点的第几个输出。比如上面的“add:0”就说嘛是计算节点add的第一个输出（与数组下标一致，都是从0开始计数）
- 维度 - shape：维度是Tensor的形状信息（维度信息），如上述代码所示，比如最后的Tensor t_c的输出是shape=（2，），说明这个张量t_c就是一维数组，长度是2。维度是张量是十分重要的信息，后续涉及到的一些多维数组（矩阵）的操作，都需要维度争取。。
- 类型 - dtype：类型是Tensor的数据类型信息，每个张量都需要有一个唯一的类型信息，如果不填写，框架会根据数据进行自行推理填充。同时TensorFlow对所有参与运算的张量进行类型检查，如果发现类型不匹配就会进行报错，无法进行。
4 总结
通过上面的描述，可以简单总结下，Tensor基本具备几个作用。
- 作为整个框架的统一的数据接口
- 中间结果：实现对中间结果的引用，算子如果看做一个一个节点的话，Tensor及时充当静态数据图在运行是的数据流动管道
- 最终结果：当计算图构造运行完成后，通过张量可以获取最终的计算结果，也就是最终
下一章节继续分析稀疏张量：SparseTensor，敬请期待。
5 感悟
很多时候，很多情况下，有些事情是非常微妙的，对于一个领域的认知，从不熟悉到得心应手，其实是要花费大量的时间的，但是如果我们很多事情连卖出第一步的勇气都没有，那永远都只能原地踏步。有时候第一步走出去，就成功 一半。
世界上唯一可以不劳而获的就是无知，唯一可以无中生有的就是梦想，虽然世间残酷，但只要你愿意走，总会有路。
想干什么和能干什么是两回事，但是如果连想都不敢想，就不要不说，不要去做了。