docker和Singularity的简单使用命令

caigui

2023-05-23

生信工具

107

docker和Singularity的简单使用命令

docker 菜鸟教程: https://www.runoob.com/docker/docker-tutorial.html

做个小笔记，方便用

1. docker

1. docker安装

# 安装docker: https://docs.docker.com/engine/install/debian/

# 卸载旧版本
for pkg in docker.io docker-doc docker-compose podman-docker containerd runc; do sudo apt-get remove $pkg; done
# https://docs.docker.com/engine/install/debian/#uninstall-docker-engine
# sudo apt-get purge docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin docker-ce-rootless-extras
# sudo rm -rf /var/lib/docker
# sudo rm -rf /var/lib/containerd
# sudo rm /etc/apt/sources.list.d/docker.list
# sudo rm /etc/apt/keyrings/docker.asc

# Add Docker's official GPG key:
sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update

# To install the latest version, run:
# 这一步需要开启clash
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# Verify that the installation is successful by running the hello-world image:
sudo docker run hello-world

2. docker设置镜像

https://geekery.cn/free-service/docker-hub-mirror.html#windows
https://xuanyuan.me/blog/archives/1154

curl -sSL https://www.geekery.cn/sh/docker/set_docker_mirror.sh | sudo bash
# [成功] Docker 镜像站配置完成。已添加（如未存在）: https://hub.geekery.cn https://ghcr.geekery.cn https://quay.geekery.cn https://gcr.geekery.cn

# 此时，以下命令失败
docker pull hello-world
# 此时，以下命令成功
docker pull hub.geekery.cn/library/hello-world

3. docker运行

# 使用docker
# pull
docker pull nservant/hicpro:latest
# 删除镜像
docker rmi nservant/hicpro:latest
# 查看已经安装的镜像
docker images
# 查看运行的容器
docker ps
# 查看所有容器运行状态
docker ps -a
# 查看卷
docker volume ls
# 删除卷
docker volume rm cg_laravel_sk_sail-meilisearch

# 查看容器内标准输出
docker logs 8e4a57f49da2 #容器ID
# 停止容器
docker stop 8e4a57f49da2
# 重启容器
docker restart 8e4a57f49da2


# run一个容器（等效于create+start） 不指定tag时，默认以latest镜像运行
docker run -it nservant/hicpro /bin/bash
# 退出容器
exit

# 如果在run的时候加入-d参数，容器会后台运行
# docker attach + exit 可以进入容器、退出容器，并会导致容器停止
# docker exec + exit 可以进入容器、退出容器，容器不会停止

# exec进入容器举例
docker exec -it f5b4e7215245 /bin/bash

# 清理掉所有处于终止状态的容器
docker container prune

#进入容器 并挂载一个工作目录，-v参数的冒号（:）左边是本地目录，右边是docker容器中的目录
docker run -v /mnt/caigui/41_sk_genome_129M/37_hicpro:/mnt/caigui/41_sk_genome_129M/37_hicpro -itd nservant/hicpro:latest /bin/bash

# docker运行hicpro的实例
docker run -v /mnt/caigui/41_sk_genome_129M/37_hicpro:/mnt/caigui/41_sk_genome_129M/37_hicpro -itd nservant/hicpro:latest /bin/bas
docker ps -a
docker exec -it f5b4e7215245 /bin/bash
cd /mnt/caigui/41_sk_genome_129M/37_hicpro
/HiC-Pro_3.0.0/bin/HiC-Pro -i /mnt/caigui/41_sk_genome_129M/37_hicpro/fastq -o /mnt/caigui/41_sk_genome_129M/37_hicpro/Resultes -c /mnt/caigui/41_sk_genome_129M/37_hicpro/config_hicpro.txt

# 额外的一个小例子
# 运行一个web程序
docker pull training/webapp
docker run -d -P training/webapp python app.py
docker ps #查看开放的端口
# 浏览器访问http://10.10.81.248:32768/
docker top #来查看容器内部运行的进程

2. Singularity

# 安装Singularity
# 在http://neuro.debian.net/选择系统版本和下载源
wget -O- http://neuro.debian.net/lists/buster.cn-hf.full | sudo tee /etc/apt/sources.list.d/neurodebian.sources.list
sudo apt-key adv --recv-keys --keyserver hkps://keyserver.ubuntu.com 0xA5D32F012649A5A9
sudo apt-get update
sudo apt-get install -y singularity-container

# pull
singularity pull shub://DoaneAS/HiC-Pro:latestles

# 直接用里面的命令，举例
singularity exec sing/DoaneAS-HiC-Pro-master-latest.simg HiC-Pro -i /mnt/caigui/41_sk_genome_129M/37_hicpro/RAWDATA -o /mnt/caigui/41_sk_genome_129M/37_hicpro/Resultes -c /mnt/caigui/41_sk_genome_129M/37_hicpro/config_hicpro.txt
singularity exec sing/DoaneAS-HiC-Pro-master-latest.simg /home/caigui/software/HiC-Pro-3.1.0/bin/utils/digest_genome.py -r DpnII -o /mnt/caigui/41_sk_genome_129M/37_hicpro/reference/genome_DpnII.bed /mnt/caigui/41_sk_genome_129M/37_hicpro/reference/genome.fa

3. 关于docker和singularity的使用

mac电脑，可以连外网，可以docker pull，不能用singularity lab服务器，不能连外网，不可以docker pull，能用singularity。

经过在lab上使用clash-for-linux配置后，已经可以用singularity拉取docker的镜像了，但是docker pull还是用不了。且需要注意节点之前差异很大，有的很快（5MB/s）很顺畅就拉下来了，有的很慢，而且拉到一半会断掉。
在mac上拉取镜像后，打包成tar文件上传，然后使用docker load，singularity bulid也可

docker load -i <镜像名称>.tar
singularity build <image_name>.sif <image_name>.tar

4. singularity升级

# 最近在使用nfcore-rnaseq时，发现v2版本的singularity不能使用v3版本构建的img，因此需要进行升级
singularity --version
# 2.6.1-dist

# 卸载old版本  
sudo rm /etc/apt/sources.list.d/neurodebian.sources.list*
sudo apt-get remove singularity-container
sudo apt-get purge singularity-container
sudo rm -rf \
    /usr/local/libexec/singularity \
    /usr/local/var/singularity \
    /usr/local/etc/singularity \
    /usr/local/bin/singularity \
    /usr/local/bin/run-singularity \
    /usr/local/etc/bash_completion.d/singularity

# 安装new version
# > https://docs.sylabs.io/guides/4.2/admin-guide/installation.html

# 安装一些依赖

# go: https://go.dev/doc/install
wget https://go.dev/dl/go1.23.2.linux-amd64.tar.gz
sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.23.2.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin #写到~/.zshrc
go version

# Install Dependencies
# Ensure repositories are up-to-date
sudo apt-get update
# Install debian packages for dependencies
sudo apt-get install -y \
   autoconf \
   automake \
   cryptsetup \
   fuse \
   fuse2fs \
   git \
   libfuse-dev \
   libglib2.0-dev \
   libseccomp-dev \
   libtool \
   pkg-config \
   runc \
   squashfs-tools \
   squashfs-tools-ng \
   uidmap \
   wget \
   zlib1g-dev

# Download SingularityCE from a release
cd ~/sf/ 
# wget有问题时，也可以去github自己下载
export VERSION=4.2.0 && # adjust this as necessary \
    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
    tar -xzf singularity-ce-${VERSION}.tar.gz && \
    cd singularity-ce-${VERSION}

# Compile Singularity
cd ~/sf/singularity-ce-4.2.0
./mconfig && \
    make -C ./builddir && \
    sudo make -C ./builddir install

# 测试
source ~/.zshrc
singularity version
# 4.2.0 #真新啊