【Tools】Anaconda Operaction

专为数据科学和机器学习工作流程而设计,是一个开源包管理器,环境管理器,以及Python和R编程语言的分发。它通常用于大规模数据处理,科学计算和预测分析。pip install xxx ,在特定环境里使用pip,下载的包会存在特定环境的目录里面,例如:D:\Anaconda3\envs\nlp\Lib\site-packages\fasttext; conda install xxx ,不管在什么环境下载的包,都统一放在一个目录里面:D:\Anaconda3\pkgs\fasttext, 如果另一个环境也需要这个包,不需要重新下载;

Anaconda:

conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
conda config --add channels http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/Paddle/
conda config --set show_channel_urls yes
#-------
conda config --add channels conda-forge   #添加conda 源
#查看所有的 conda .condarc 配置文件内容
conda config --get
  • 要充分利用conda 包的好处,遇到需要安装什么包的优选conda:https://anaconda.org/wdroz/openpose-gpu
  • 通过源码编译的是在其他已经都编译的没有的状况下,或者需要修改源码
  • 设置不进入conda环境,新版的anaconda(python3.7)安装后自动进入虚拟环境base,执行命令:

? conda config --set auto_activate_base false

conda package

  • A conda package is a compressed tarball file (.tar.bz2) or .conda file that contains:

    • system-level libraries.
    • Python or other modules.
    • executable programs and other components.
    • metadata under the info/ directory.
    • a collection of files that are installed directly into an install prefix.
  • package structure:

    ├── bin            #relevant binaries for package
    │   └── pyflakes
    ├── info     # package metadata
    │   ├── LICENSE.txt
    │   ├── files
    │   ├── index.json
    │   ├── paths.json
    │   └── recipe
    └── lib       # relevant library files
        └── python3.5
    
    • metadata contain dependencies to system core, low-level libraries and can contain links to software files that are automatically downloaded when executed.

    • index.json

      • metadata about the package including platform, version, dependencies, and build info
    {
      "arch": "x86_64",
      "build": "py37hfa4b5c9_1",
      "build_number": 1,
      "depends": [
        "depend > 1.1.1"
      ],
      "license": "BSD 3-Clause",
      "name": "fun-packge",
      "platform": "linux",
      "subdir": "linux-64",
      "timestamp": 1535416612069,
      "version": "0.0.0"
    }
    
    • paths.json
      • a list of files in the package, along with their associated SHA-256, size in bytes, and the type of path (eg. hardlink vs. softlink)
    {
      "paths": [
        {
          "_path": "lib/python3.7/site-packages/fun-packge/__init__.py",
          "path_type": "hardlink",
          "sha256": "76f3b6e34feeb651aff33ca59e0279c4eadce5a50c6ad93b961c846f7ba717e9",
          "size_in_bytes": 2067
        },
        {
          "_path": "lib/python3.7/site-packages/fun-packge/__config__.py",
          "path_type": "hardlink",
          "sha256": "348e3602616c1fe4c84502b1d8cf97c740d886002c78edab176759610d287f06",
          "size_in_bytes": 87519
        },
        ...
    }
    

Conda Dir

  • root_dir:
  • pkgs: contains decompressed packages, ready to be linked in conda environments. Each package resides in a subdirectory corresponding to its canonical name.
  • envs: conda environments
    • bin
    • include
    • share
.condarc
  • Where conda looks for packages.
  • how conda uses a proxy server.
  • Where conda lists known environments.
  • Whether to update the Bash prompt with the currently activated environment name.
  • Whether user-built packages should be uploaded to Anaconda.org.
  • What default packages or features to include in new environments.
Conda 下载流程
【Tools】Anaconda Operaction

Conda follows these steps when installing a package:

  1. Downloading and processing index metadata.
  2. Reducing the index.
  3. Expressing the package data and constraints as a SAT problem.
  4. Running the solver.
  5. Downloading and extracting packages.
  6. Verifying package contents.
  7. Linking packages from package cache into environments.

包首先现在到这个目录下

【Tools】Anaconda Operaction

环境打包

#  1.创建环境的快照或备份
conda create --name snapshot --clone myenv

#  2. 复制环境  相同操作系统
# 生成 spec list 文件
conda list --explicit > spec-list.txt
pip freeze > requirements.txt
# 创建环境
conda create --name myname --file spec-list.txt

#  3. 生成environment.yml 跨平台操作系统复现环境
# 导出environment.yml 文件, 包括 pip 安装软件
conda env export > environment.yml
#创建环境
conda env create -f environment.yml
pip install -r requirements.txt

#  4.Pack 打包,包括该环境中安装的软件包的所有二进制文件, conda-pack 指定平台和操作系统,目标计算机必须集有与源计算机相同的平台和操作系统
# 安装
conda install -c conda-forge conda-pack
# 打包一个环境
# pack environment my_env into my_env.tar.gz
conda pack -n my_env
# pack environment my_env into out_name.tar.gz
conda pack -n my_env -o out_name.tar.gz
# pack environment located at an explicit path into my_env.tar.gz
conda pack -p /explicit/path/to/my_env
# 创建环境
mkdir -p my_env
tar -xzvf my_env.tar.gz -C my_env
# Use Python without activating or fixing the prefixes. Most Python
# libraries will work fine, but things that require prefix cleanups
# will fail.
./my_env/bin/python

# Activate the environment. This adds `my_env/bin` to your path
source my_env/bin/activate

# Run Python from in the environment
(my_env) $ python

# Cleanup prefixes from in the active environment.
# Note that this command can also be run without activating the environment
# as long as some version of Python is already installed on the machine.
(my_env) $ conda-unpack

包管理

#方式一
#	文件夹那放到对应python 环境目录下lib->sitepackages然后import 就行。例如 winguiauto.py 文件放在Anaconda3\envs\pytorch-openpose\Lib\site-packages, 直接import winguiauto ,可能需要重启下
#Tips  python 目录(pytorch-openpose) >where python
	#C:\Users\dell\Anaconda3\envs\pytorch-openpose\python.exe
	#C:\Users\dell\Anaconda3\python.exe	#C:\Users\dell\AppData\Local\Microsoft\WindowsApps\python.exe    
#方式二: 批量包下载 
# 新建 .txt 文件 内容(h5requipy,numpy,tensorflow==2.0.0-alpha0)单独成行,
pip install -r requirement.txt #进行下载  -i https://pypi.tuna.tsinghua.edu.cn/simple/

#方式三: 直接 conda install / pip install 进行下载

#方式四: 离线下载
#	直接下载的whl文件,通过pip install (路径+whl文件名)
#	可以下载到本地 anaconda\install\Lib\site-packages路径下,或者在线下载安装
#查看版本
conda --version
conda update conda
#环境管理
conda create --name pysot python=3.7
pip install -r requirements.txt
conda info --envs        #conda env list
conda activate pysot
conda deactivate pysot
#根据environment.yml 创建环境
conda env create -f environment.yml  #yml 第一行是环境名称
conda remove --name ENVNAME --all
#环境迁移
#包管理
conda search package
conda install package
conda uninstall package --name env
conda install scipy=0.15.0  #下载指定版本
conda list  # list package
conda list -n env  #查看一个环境中所有的包
#删除包
conda remove -n myenv scipy
#install with channel
conda install scipy --channel conda-forge
#本地下载   本地下载不会解决包依赖问题
conda install /packages-path/packages-filename.tar
conda config --set env_prompt ‘({name})‘

查看包依赖

  • The tarball is unarchived in the pkgs directory
  • Files are hard-linked to the install path
  • Shebang lines and other instance of a place-holder prefix are replaced with the install prefix
  • The metadata is updated, so that conda knows that it is installed
  • the process of command conda create -n py3k python=3.7
  • Environments are simple: just link the package to a different directory
  • Hand-links are very cheap, and very fast
  • Conda environments are completely independent installations of everything
    • No fiddling with pythonPath or symlinking site-package
  • Activating an environment just means changing your path so that its bin/ or scripts/ comes first

【Tools】Anaconda Operaction

To find what packages are depending on a specific package in your environment, there is not one specific conda command. It requires a series of steps:

  1. List the dependencies that a specific package requires to run: conda info package_name
  2. Find your installation’s package cache directory: conda info
  3. Find package dependencies. By default, Anaconda/Miniconda stores packages in ~/anaconda/pkgs/ (or ~/opt/pkgs/ on macOS Catalina). Each package has an index.json file which lists the package’s dependencies. This file resides in ~anaconda/pkgs/package_name/info/index.json.
  4. Now you can find what packages depend on a specific package. Use grep to search all index.json files as follows: grep package_name ~/anaconda/pkgs/*/info/index.json

包导入原理:

  • 怎么加载?——加载方式和用法

    1. import package 读这个包的__init__.py,也就是说导入包的本质是执行包下面的__init__.py文件,执行结束后会包名的目录下生成一个"pycache / init.cpython-36.pyc" 文件
    2. import module 读整个模块的内容
    3. import package1.package2.package3.module4 package读取__init__.py,module读取整个模块内容,按顺序读注意:因为package是读取__init__.py,所以调用的时候必须在__init__.py有引用的东西才能调用,否则会报错。
    4. 从指定路径导入:
    import sys
    sys.path.append(‘/path/to/folder/containing/my_package‘)  #注意是要导入包的上级目录
    import my_package	
    
  • 案例

├── sub_ui.py├── main.py├── test/│   └── test1.py└── model/    └── model1.py#-----  同级目录导入 main.py导入sub_ui.pyimport sub_ui  或from sub_ui import *#-----  子目录,main.py导入model/model1.py需要在model/下建立__init__.py空文件,让编译器认为这是一个模块。    import model.model1或    from model.model1 import *#----   跨目录导入 跨目录,model1.py导入test/test1.py需要在test1/下建立__init__.py空文件import sys sys.path.append("..") import test.test1 或者 import sys sys.path.append("..") import test.test1 import *
  •  从哪里导入:python中,每个.py文件被称之为模块,每个具有__init__.py文件的目录被称为包。只要模块或者包所在的目录在sys.path中,就可以使用import 模块或import 包来使用.对于python来说,所有被加载到内存的模块都是放在sys.modules里面,所以执行import时会首先去该列表中查询是否已添加。如果已经在sys.modules中,那就简单了,只需要将该module的name添加到我们正在调用该module的本地空间中。如果还没有放在sys.modules中,就需要在sys.path所有路径的目录中去按顺序查找该模块的文件,这些文件一般后缀为".py"、".pyo"、".pyc"、".pyd"、".dll",找到这些模块后就可以将这些模块添加到sys.modules中,再将module name导入到本地。
>>> import sys>>> sys.modules{‘copy_reg‘: <module ‘copy_reg‘ from ‘/usr/lib/python2.7/copy_reg.pyc‘>, ‘sre_compile‘: <module ‘sre_compile‘ from ‘/usr/lib/python2.7/sre_compile.pyc‘>, ‘_sre‘: <module ‘_sre‘ (built-in)>, ‘encodings‘: <module ‘encodings‘ from ‘/usr/lib/python2.7/encodings/__init__.pyc‘>, ‘site‘: <module ‘site‘ from ‘/usr/lib/python2.7/site.pyc‘>, ‘__builtin__‘: <module ‘__builtin__‘ (built-in)>, ‘sysconfig‘: <module ‘sysconfig‘ from ‘/usr/lib/python2.7/sysconfig.pyc‘>, ‘__main__‘: <module ‘__main__‘ (built-in)>, ‘encodings.encodings‘: None, ‘abc‘: <module ‘abc‘ from ‘/usr/lib/python2.7/abc.pyc‘>, ‘posixpath‘: <module ‘posixpath‘ from ‘/usr/lib/python2.7/posixpath.pyc‘>, ‘_weakrefset‘: <module ‘_weakrefset‘ from ‘/usr/lib/python2.7/_weakrefset.pyc‘>, ‘errno‘: <module ‘errno‘ (built-in)>, ‘encodings.codecs‘: None, ‘sre_constants‘: <module ‘sre_constants‘ from ‘/usr/lib/python2.7/sre_constants.pyc‘>, ‘re‘: <module ‘re‘ from ‘/usr/lib/python2.7/re.pyc‘>, ‘_abcoll‘: <module ‘_abcoll‘ from ‘/usr/lib/python2.7/_abcoll.pyc‘>, ‘types‘: <module ‘types‘ from ‘/usr/lib/python2.7/types.pyc‘>, ‘_codecs‘: <module ‘_codecs‘ (built-in)>, ‘encodings.__builtin__‘: None, ‘_warnings‘: <module ‘_warnings‘ (built-in)>, ‘genericpath‘: <module ‘genericpath‘ from ‘/usr/lib/python2.7/genericpath.pyc‘>, ‘stat‘: <module ‘stat‘ from ‘/usr/lib/python2.7/stat.pyc‘>, ‘zipimport‘: <module ‘zipimport‘ (built-in)>, ‘_sysconfigdata‘: <module ‘_sysconfigdata‘ from ‘/usr/lib/python2.7/_sysconfigdata.pyc‘>, ‘mpl_toolkits‘: <module ‘mpl_toolkits‘ (built-in)>, ‘warnings‘: <module ‘warnings‘ from ‘/usr/lib/python2.7/warnings.pyc‘>, ‘UserDict‘: <module ‘UserDict‘ from ‘/usr/lib/python2.7/UserDict.pyc‘>, ‘encodings.utf_8‘: <module ‘encodings.utf_8‘ from ‘/usr/lib/python2.7/encodings/utf_8.pyc‘>, ‘sys‘: <module ‘sys‘ (built-in)>, ‘codecs‘: <module ‘codecs‘ from ‘/usr/lib/python2.7/codecs.pyc‘>, ‘readline‘: <module ‘readline‘ from ‘/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so‘>, ‘_sysconfigdata_nd‘: <module ‘_sysconfigdata_nd‘ from ‘/usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc‘>, ‘os.path‘: <module ‘posixpath‘ from ‘/usr/lib/python2.7/posixpath.pyc‘>, ‘_locale‘: <module ‘_locale‘ (built-in)>, ‘sitecustomize‘: <module ‘sitecustomize‘ from ‘/usr/lib/python2.7/sitecustomize.pyc‘>, ‘signal‘: <module ‘signal‘ (built-in)>, ‘traceback‘: <module ‘traceback‘ from ‘/usr/lib/python2.7/traceback.pyc‘>, ‘linecache‘: <module ‘linecache‘ from ‘/usr/lib/python2.7/linecache.pyc‘>, ‘posix‘: <module ‘posix‘ (built-in)>, ‘encodings.aliases‘: <module ‘encodings.aliases‘ from ‘/usr/lib/python2.7/encodings/aliases.pyc‘>, ‘exceptions‘: <module ‘exceptions‘ (built-in)>, ‘sre_parse‘: <module ‘sre_parse‘ from ‘/usr/lib/python2.7/sre_parse.pyc‘>, ‘os‘: <module ‘os‘ from ‘/usr/lib/python2.7/os.pyc‘>, ‘_weakref‘: <module ‘_weakref‘ (built-in)>}>>> sys.path[‘‘, ‘/usr/lib/python2.7‘, ‘/usr/lib/python2.7/plat-x86_64-linux-gnu‘, ‘/usr/lib/python2.7/lib-tk‘, ‘/usr/lib/python2.7/lib-old‘, ‘/usr/lib/python2.7/lib-dynload‘, ‘/home/ldd/.local/lib/python2.7/site-packages‘, ‘/usr/local/lib/python2.7/dist-packages‘, ‘/usr/lib/python2.7/dist-packages‘, ‘/usr/lib/python2.7/dist-packages/gtk-2.0‘]

Error record:

  • Missing dependencies for SOCKS support. 解决方法: unset all_proxy

linux 安装Anaconda


#本地安装pip install ~/Dowloads/torch-0.2.0.post3-cp35-cp35m-manylinux1_x86_64.whl#conda 安装本地包conda install --use-local pytorch-0.4.0-py35_cuda8.0.61_cudnn7.1.2_1.tar.bz2#change the channelconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --set show_channel_urls yes#下载安装curl -O https://repo.anaconda.com/archive/Anaconda3-5.1.0-Linux-x86_64.sh#通过SHA-256校验和通过加密哈希验证来验证安装程序的数据完整性sha256sum Anaconda3-5.1.0-Linux-x86_64.sh#安装bash Anaconda3-5.1.0-Linux-x86_64.sh#激活环境,否则无法找到conda命令source ~/.bashrcconda info#用来查看conda 内部安装环境版本等信息。#验证安装是否正确 罗列conda 安装所有软件conda list#创建 一个环境my_env  Python版本为3conda create --name my_env python=3   #使用 sudo 命令创建,否则无法在环境中注册#在环境my_env35中安装numpy包 conda install --name my_env35 numpy#激活环境 ,这样会有个前缀#更新 conda使用程序conda update conda#更新 或卸载 anaconda 安装包conda remove -n envname --all   #delete all the package and environmentconda update anaconda conda install anaconda-cleanrm -rf anacondasudo gedit ~/.bashrc #删除环境变量source ~/.bashrc #使环境变量生效

Pycharm :

#PyCharm:https://www.jetbrains.com/pycharm/download/#section=linuxcd downloads/pycharm-2018.1/binbash pycharm.sh#进入file setting 设置选择编译器

【Tools】Anaconda Operaction

Win环境配置

F:\Anaconda3F:\Anaconda3\ScriptsF:\Anaconda3\Library\bin#shell 中加入这三个目录,才可以在cmd 中使用conda

Jupyter使用

若想要关闭已经打开的终端和“ipynb”格式的笔记本,仅仅关闭其页面是无法彻底退出程序的,需要在Running页面点击其对应的“Shutdown”。

  • 安装: conda install jupyter notebook

  • 修改默认的主目录:jupyter notebook --generate-config

  • 启动目录: jupyter notebook, http://localhost:8888; 可以对端口进行映射处理

    • 指定端口号来启动: jupyter notebook --port <port_number>
    • 启动服务但不打开网页: jupyter notebook --no-browser
    • 在conda 虚拟环境中运行启动 jupyter 命令,kernel 会自动关联所有的conda环境;
  • 帮助命令: jupyter notebook --help

  • 将你conda创建的环境与Jupyter Notebook相关联: conda install nb_conda

  • 加载本地python文件:%load Python文件的绝对路径

  • 直接运行本地python文件:%run Python文件的绝对路径; !python3 Python文件的绝对路径; !shell命令;

  • 学习链接: https://www.jianshu.com/p/91365f343585#conda

  • ModuleNotFoundError: No module named ‘jupyter_nbextensions_configurator‘

    • pip install jupyter_nbextensions_configurator
      
  • Error executing Jupyter command ‘notebook‘: [Errno 2] No such file or directory

    • jupyter notebook --port=11220 --allow-root #不适用sudo 权限就可以执行了
  • Jupyter 常用快捷键

    ctrl + / #进行注释

    cell操作
    Ctrl+enter 执行本cell
    shift+enter 执行本cell且 向下建立一个新cell

    A 向上建立一个cell
    B 向下建立一个cell
    d d 删除cell
    O 收起output 或者打开output

    在cell里面的快捷键
    Esc+m m 把cell切换至markdown模式
    Esc+y y 把cell切换至code模式
    Esc+l l 显示行数
    Esc + F 在代码中查找、替换,忽略输出。
    Esc + O 在 cell 和输出结果间切换。

    复合快捷键
    shift 选中多个cell
    然后执行 shift+M 合并选中cell

Error Record

.1. 代理问题

#Anaconda 使用conda指令出现ValueError: Unable to determine SOCKS version from socks://127.0.0.1:1080/ 解决办法printenv | grep all_proxy  #查看代理unset all_proxy && unset ALL_PROXY   #export all_proxy=""  export all_proxy="socks5://127.0.0.1:1080"export http_proxyexport https_proxyunset http_proxyunset https_proxyconda install pysocks     #下次创建环境是同时下载 pip和pysocks 包source ~/.bashrc   

每次在虚拟环境内部使用pip的时候都需要获取虚拟环境内部的 pip命令路径:/Users/stefanannihilater/anaconda3/envs/setests3/bin/pip,显然这样非常的繁琐。

如果想在虚拟环境内部使用简单的pip install package调用虚拟环境内部的pip命令的话,只需要我们在创建虚拟环境的时候指定pip只对虚拟环境生效,而不影响全局库:

conda create -n 虚拟环境名 pip pysocks
  • pip:是为了指定pip命令只对当前虚拟环境生效
  • pysocks:是pip命令的依赖,如果不写,在虚拟环境内使用 pip命令的时候会出现Missing dependencies for SOCKS support.的报错。

下次可以把包下载到本地,然后局域网上传服务器,减少服务器流量开销;

  • 错误记录SSL

    • 方式一:环境变量是否配好:
    D:\Anaconda3; D:\Anaconda3\Scripts; D:\Anaconda3\Library\bin 
    

【Tools】Anaconda Operaction

.2. cudatoolkit=11.1. 下载失败

conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/conda config --set show_channel_urls yesconda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/conda install pytorch torchvision torchaudio cudatoolkit=11.1

.3. torchvision 库无法使用

Download error with CIFAR10/Mnist based on torchvision.datasets

  • uninstall and reinstall torchvision
conda unstall torchvisionconda install pytorch torchvision torchaudio cudatoolkit=11.1

.4. pip 代理

【Tools】Anaconda Operaction

.5. GPU资源问题

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

  • GPU被沾满了,没有分配资源;

【Tools】Anaconda Operaction

上一篇:《道路工程》——(六)横断面组成


下一篇:互联网的发展 服务器的发展