Ansible + Docker 部署 Apache Nifi 1.28 单用户集群
1. 准备工作
1.1 主机列表
IP | 主机名 | 内存(GB) | CPU核数 | 磁盘 | 操作系统 | CPU 架构 |
---|---|---|---|---|---|---|
10.0.0.13 | arc-pro-dc01 my.registry.com |
16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
10.0.0.14 | arc-pro-dc02 | 16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
10.0.0.15 | arc-pro-dc03 | 16 | 1 | 500GB | CentOS 7.9.2009 | x86_64 |
1.2 已安装服务
版本 | arc-pro-dc01 | arc-pro-dc02 | arc-pro-dc03 | |
---|---|---|---|---|
Ansible | 2.9.27 | ✅ | ||
Harbor | v2.13.2 | ✅ | ||
Docker | 28.1.1 | ✅ | ✅ | ✅ |
Docker Compose | v2.39.2 | ✅ | ✅ | ✅ |
Zookeeper | 3.6.4 | ✅ | ✅ | ✅ |
说明:
- 每个服务器的 IP 均是静态的
- 每个服务器的防火墙都已关闭
- 每个服务器的 SELINUX 已经禁用
- 每个服务器均存在一个管理员用户 admin,该用户可以免密码执行 sudo 命令;
- 在 arc-pro-dc01 机器上,可以使用 admin 用户免密码 ssh 到其他机器;
- 服务器之间的时间同步;
- 所有操作均使用 admin 用户完成;
- 私有镜像仓库地址:https://my.registry.com:10443。
为使集群满足以上要求,参考下列文章进行配置:
- 使用 VMware Workstation 安装 CentOS-7 虚拟机
- 用 Ansible 批量完成 CentOS 7 操作系统基础配置
- 使用 Ansible 批量安装 Docker
- Docker 私有镜像仓库 Harbor 安装部署带签名认证
- Ansible + Docker 部署 Zookeeper 3.6 集群
1.3 集群规划
版本 | arc-pro-dc01 | arc-pro-dc02 | arc-pro-dc03 | |
---|---|---|---|---|
Nifi | 1.28.1 | ✅ | ✅ | ✅ |
1.4 镜像准备
找一个可以连接互联网的、已经安装了 docker 的服务器,下载镜像
docker pull apache/nifi:1.28.1
docker pull apache/nifi-toolkit:1.28.1
docker image save apache/nifi:1.28.1 -o nifi.1.28.1.tar.gz
docker image save apache/nifi-toolkit:1.28.1 -o nifi-toolkit:1.28.1.tar.gz
将 nifi.1.28.1.tar.gz 上传到本集群任意一台服务器,导入镜像:
docker load -i nifi.1.28.1.tar.gz
docker load -i nifi-toolkit:1.28.1.tar.gz
docker tag apache/nifi:1.28.1 my.registry.com:10443/library/apache/nifi:1.28.1
docker tag apache/nifi-toolkit:1.28.1 my.registry.com:10443/library/apache/nifi-toolkit:1.28.1
# 上传到私服
docker push my.registry.com:10443/library/apache/nifi:1.28.1
docker push my.registry.com:10443/library/apache/nifi-toolkit:1.28.1
2. 配置最佳实践
2.1 增加最大打开文件描述符和用户最大创建进程数的值
/etc/security/limits.conf
* hard nofile 50000
* soft nofile 50000
* hard nproc 10000
* soft nproc 10000
2.2 增加可用的 TCP socket 端口数量
扩大 **临时端口范围(ephemeral ports)**,这样系统在创建大量 TCP 连接时不会过早耗尽端口。/etc/sysctl.conf
net.ipv4.ip_local_port_range="10000 65000"
2.3 设置 socket 在关闭前保持在 TIMED_WAIT 状态的时间
# 查看内核版本
$ uname -r
3.10.0-1160.119.1.el7.x86_64
# 其中:3.10.0 就是内核版本# for kernel 2.6
sudo sysctl -w net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait="1"# for kernel 3.0
sudo sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait="1"
2.4 关闭交换分区
/etc/sysctl.conf
vm.swappiness = 0
2.5 Nifi 仓库分区关闭 atime(访问时间戳)
存放 NiFi 各类仓库的分区:- Content Repository(内容库)
- FlowFile Repository(流文件库)
- Provenance Repository(溯源库)
这些目录会产生大量磁盘读写操作。
atime
是 Linux 文件系统的 访问时间戳,每次文件被读取时都会更新一次 atime
。
默认启用时,哪怕只是读文件,也会导致一次额外的写操作(更新元数据)。
在 NiFi 这种高 IO 场景下,这种额外写操作会拖慢整体性能。
所以推荐 关闭 atime(即 noatime
选项)。
关闭 atime
后,磁盘少了很多无用的写操作,因此 吞吐量(throughput)提升明显,NiFi 处理流数据的速度会更快。
实现方法:
在 /etc/fstab
中,把存放 NiFi 仓库的分区挂载参数加上 noatime
。例如:
/dev/sdb1 /nifi-repository ext4 defaults 0 2
修改为:
/dev/sdb1 /nifi-repository ext4 defaults,noatime 0 2
3. Ansible 文件
3.1 Ansible 目录结构
说明:在 arc-pro-dc01 机器上,执行 ansible 命令的基础目录为 /home/admin/ansible
$ tree /home/admin/ansible/
/home/admin/ansible/
├── ansible.cfg
├── hosts
└── nifi├── disable_swap.yml├── docker-compose.yml.j2├── start-nifi-container.yml└── sysctl.yml
3.2 ansible.cfg
[defaults]
inventory=./hosts
host_key_checking=False
3.3 hosts
[nifi]
arc-pro-dc01
arc-pro-dc02
arc-pro-dc03
3.4 disable_swap.yml
---
- name: Disable swap on all cluster nodeshosts: nifigather_facts: falsebecome: yestasks:- name: Turn off all swap immediatelycommand: swapoff -aregister: swapoff_resultchanged_when: swapoff_result.rc == 0- name: Backup fstab before modifyingcopy:src: /etc/fstabdest: "/etc/fstab.backup"owner: rootgroup: rootmode: 0644- name: Comment out swap entries in fstab to disable on bootlineinfile:path: /etc/fstabregexp: '^([^#].*\s+swap\s+.*)$'line: '# \1'backrefs: yes- name: Check runtime swap statuscommand: swapon --summaryregister: swap_statuschanged_when: falsefailed_when: false- name: Show runtime swap statusdebug:msg: >Runtime swap is {{'disabled' if swap_status.stdout == '' else 'ENABLED'}}- name: Check if /etc/fstab contains swap entrycommand: awk '!/^#/ && $3=="swap" {print}' /etc/fstabregister: fstab_swapchanged_when: falsefailed_when: false- name: Show fstab swap statusdebug:msg: >fstab swap entry is {{'(NOT disabled permanently)' if fstab_swap.stdout != '' else '(disabled permanently)'}}
3.5 sysctl.yml
---
- hosts: nifigather_facts: falsebecome: yesvars:sysctl_config_file: /etc/sysctl.confsysctl_params:net.ipv4.ip_local_port_range: "10000 65000"net.netfilter.nf_conntrack_tcp_timeout_time_wait: 1tasks:- name: Ensure sysctl.conf parameterslineinfile:path: '{{ sysctl_config_file }}'regexp: '^{{ item.key }}\s*='line: '{{ item.key }}={{ item.value }}'state: presentloop: "{{ sysctl_params | dict2items }}"- name: Apply sysctl paramscommand: sysctl -p- name: Show modified sysctl.conf linesshell: "grep -E '^({{ sysctl_params.keys() | join('|') }})' {{ sysctl_config_file }}"register: sysctl_conf_check- name: Print modified sysctl.conf linesdebug:msg: "{{ sysctl_conf_check.stdout_lines }}"
3.6 docker-compose.yml.j2
services:nifi:image: my.registry.com:10443/library/apache/nifi:1.28.1restart: unless-stoppedcontainer_name: nifihostname: {{ inventory_hostname }}network_mode: hostenvironment:NIFI_WEB_HTTPS_PORT: 8443SINGLE_USER_CREDENTIALS_USERNAME: adminSINGLE_USER_CREDENTIALS_PASSWORD: {{ SINGLE_USER_CREDENTIALS_PASSWORD }}NIFI_CLUSTER_IS_NODE: trueNIFI_CLUSTER_ADDRESS: {{ inventory_hostname }}NIFI_CLUSTER_NODE_PROTOCOL_PORT: 11443NIFI_ZK_CONNECT_STRING: {{ NIFI_ZK_CONNECT_STRING }}ulimits:nofile:soft: 50000hard: 50000nproc:soft: 10000hard: 10000volumes:- {{ conf_dir }}/conf:/opt/nifi/nifi-current/conf- {{ data_dir }}/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository- {{ data_dir }}/content_repository:/opt/nifi/nifi-current/content_repository- {{ data_dir }}/provenance_repository:/opt/nifi/nifi-current/provenance_repository- {{ data_dir }}/status_repository:/opt/nifi/nifi-current/status_repository- {{ log_dir }}:/opt/nifi/nifi-current/logs
3.7 start-nifi-container.yml
---
- name: Start Nifi Containerhosts: nifibecome: truegather_facts: falsevars:nifi_image: my.registry.com:10443/library/apache/nifi:1.28.1nifi_toolkit_image: my.registry.com:10443/library/apache/nifi-toolkit:1.28.1 nifi_owner: adminnifi_group: adminhttps_port: 8443compose_file_dir: /opt/app/nificonf_dir: /etc/nifilog_dir: /data/nifi/logsdata_dir: /data/nifi/datanifi_heap_size: "1g"NIFI_ZK_CONNECT_STRING: "10.0.0.13:2181,10.0.0.14:2181,10.0.0.15:2181"SINGLE_USER_CREDENTIALS_PASSWORD: NifiAdmin123456tasks:- name: Remove old nifi container if existscommand: docker rm -f nifiignore_errors: true- name: Recreate nifi directoriesshell: |rm -rf {{ conf_dir }} {{ data_dir }} {{ log_dir }} {{ compose_file_dir }}mkdir -p {{ compose_file_dir }} \{{ conf_dir }}/certs \{{ conf_dir }}/conf \{{ data_dir }}/flowfile_repository \{{ data_dir }}/content_repository \{{ data_dir }}/provenance_repository \{{ data_dir }}/status_repository \{{ log_dir }}chown -R {{ nifi_owner }}:{{ nifi_group }} \{{ conf_dir }} \{{ data_dir }} \{{ log_dir }} \{{ compose_file_dir }}- name: Generate config fileshell: |rm -rf {{ conf_dir }}mkdir -p {{ conf_dir }}chown {{ nifi_owner }}:{{ nifi_group }} {{ conf_dir }}docker rm -f tmp-nifidocker run --name tmp-nifi --rm -d {{ nifi_image }}delegate_to: localhostrun_once: true- name: Wait until tmp-nifi container is fully startedshell: "docker logs tmp-nifi 2>&1 | grep -q 'NiFi has started'"register: nifi_statusuntil: nifi_status.rc == 0retries: 30delay: 10changed_when: falsefailed_when: nifi_status.rc != 0delegate_to: localhostrun_once: true- name: Copy nifi config fileshell: |docker cp tmp-nifi:/opt/nifi/nifi-current/conf {{ conf_dir }}docker rm -f tmp-nifichown -R {{ nifi_owner }}:{{ nifi_group }} {{ conf_dir }}delegate_to: localhostrun_once: true- name: TLS configshell: |rm -rf {{ conf_dir }}/certsmkdir -p {{ conf_dir }}/certschown -R {{ nifi_owner }}:{{ nifi_group }} {{ conf_dir }}docker run --rm \-v {{ conf_dir }}/certs:/certs \{{ nifi_toolkit_image }} \tls-toolkit standalone \--clientCertDn 'CN=NIFI, OU=NIFI' \--hostnames {{ ansible_play_hosts | join(',') }} \--keyAlgorithm RSA \--keySize 2048 \--days 36500 \--keyPassword keyPassword@123456 \--keyStorePassword keyStorePassword@123456 \--trustStorePassword trustStorePassword@123456 \--outputDirectory /certschown -R {{ nifi_owner }}:{{ nifi_group }} {{ conf_dir }}delegate_to: localhostrun_once: true- name: Copy config from control node to all nifi nodescopy:src: "{{ conf_dir }}/conf"dest: "{{ conf_dir }}"owner: "{{ nifi_owner }}"group: "{{ nifi_group }}"mode: preserve- name: Copy TLS file from control node to all nifi nodescopy:src: "{{ conf_dir }}/certs"dest: "{{ conf_dir }}"owner: "{{ nifi_owner }}"group: "{{ nifi_group }}"mode: preserve- name: Copy certs and keysshell: |cp {{ conf_dir }}/certs/{{ inventory_hostname }}/* {{ conf_dir }}/conf/cp {{ conf_dir }}/certs/*.p12 {{ conf_dir }}/conf/cp {{ conf_dir }}/certs/*.password {{ conf_dir }}/conf/cp {{ conf_dir }}/certs/*.pem {{ conf_dir }}/conf/cp {{ conf_dir }}/certs/*.key {{ conf_dir }}/conf/chown -R {{ nifi_owner }}:{{ nifi_group }} {{ conf_dir }}- name: JVM Configshell: |sed -i -e 's|^java.arg.2=.*|java.arg.2=-Xms{{ nifi_heap_size }}|' \-e 's|^java.arg.3=.*|java.arg.3=-Xmx{{ nifi_heap_size }}|' \{{ conf_dir }}/conf/bootstrap.conf- name: modify nifi.propertiesshell: |sed -i -e 's|^nifi.sensitive.props.key=.*|nifi.sensitive.props.key=SensitivePropsKey123456|' \{{ conf_dir }}/conf/nifi.properties- name: Deploy docker-compose.ymltemplate:src: docker-compose.yml.j2dest: "{{ compose_file_dir }}/docker-compose.yml"owner: "{{ nifi_owner }}"group: "{{ nifi_group }}"mode: '0644'- name: Start nifi containercommand: docker-compose -f {{ compose_file_dir }}/docker-compose.yml up -d
4. 部署
在 arc-pro-dc01(Ansible 管理节点) 服务器执行
$ pwd
/home/admin/ansible# 关闭 NiFi 集群服务器的交换分区
$ ansible-playbook nifi/disable_swap.yml# 优化 NiFi 集群服务器的系统配置
$ ansible-playbook nifi/sysctl.yml# 部署并启动 Nifi 集群
$ ansible-playbook nifi/start-nifi-container.yml
5. 访问页面
访问任一节点:- https://arc-pro-dc01:8443/nifi/
- https://arc-pro-dc02:8443/nifi/
- https://arc-pro-dc03:8443/nifi/
说明:账号密码为 admin/NifiAdmin123456
start-nifi-container.yml 中配置了密码:SINGLE_USER_CREDENTIALS_PASSWORD: NifiAdmin123456
登录成功后,显示“The Flow Controller is initializing the Data Flow.”这是因为,集群要进行选举,可能花费几分钟。
查看 /data/nifi/logs/nifi-app.log,可以看到:集群正在投票,选举会在 289 秒内完成(确实需要这么长时间)
2025-09-25 15:08:34,330 INFO [main]
o.a.nifi.controller.StandardFlowService Requested by cluster coordinator to retry connection in 5 seconds with explanation:
Cluster is still voting on which Flow is the correct flow for the cluster.
Election will complete in 289 seconds
选举完成后,刷新页面: