Centos-7下常见中间件环境的安装（二）

发表于 更新于 分类于 阅读次数： 本文字数： 16k 阅读时长 ≈ 26 分钟

接着上篇文章，现在开始部署RocketMQ集群，采用的部署方案是阿里推荐的两主两从异步刷盘的模式，因此准备了四台服务器来进行MQ集群的部署，以下是部署过程中的一些记录，

准备工作

集群机器信息

IP CPU、RAM 作用
10.100.1.21 16 vcpu、64G ram broker-a master
10.100.1.22 16 vcpu、64G ram broker-b master
10.100.1.23 16 vcpu、64G ram nameserver、broker-a slave
10.100.1.24 16 vcpu、64G ram nameserver、broker-b-slave

运行环境

这次采用的是最新版本4.2.-0-SNAPSHOT，所以jdk需要采用 jdk8 ，按照上一篇文章将jdk全部安装好。

Maven 安装

Apache现在并没有提供RocketMQ编译好的版本，所以只能自己下载源码进行编译，由于rocketmq是通过maven进行包管理的，所以要先安装maven。

  • 下载
1
2
3
jobshen@JobShen-PC:/usr/local$ curl -O http://mirrors.hust.edu.cn/apache/maven/maven-3/3.5.2/binaries/apache-maven-3.5.2-bin.tar.gz
jobshen@JobShen-PC:/usr/local$ tar -zxf apache-maven-3.5.2-bin.tar.gz
jobshen@JobShen-PC:/usr/local$ vim apache-maven-3.5.2/conf/settings.xml
  • 修改mirror 镜像源，改为阿里云的镜像：
1
2
3
4
5
6
7
 <mirrors>   
<mirror>
     <id>nexus</id>
     <mirrorOf>external:*,!spring-milestones</mirrorOf>
     <url>http://maven.aliyun.com/nexus/content/groups/public</url>
   </mirror>
 </mirrors>
  • 设置环境变量
1
2
3
4
5
6
jobshen@JobShen-PC:/usr/local$ vim /etc/profile
// 在末尾添加以下内容
export M2_HOME=/usr/local/apache-maven-3.5.2/bin
export PATH=$M2_HOME/bin:$PATH
// 保存 使其生效
jobshen@JobShen-PC:/usr/local$ source /etc/profile

RocketMQ 编译

  • 克隆代码编译
1
2
3
4
5
6
7
jobshen@JobShen-PC:/usr/local$ git clone -b develop https://github.com/apache/rocketmq.git
jobshen@JobShen-PC:/usr/local$ cd rocketmq
jobshen@JobShen-PC:/usr/local/rocketmq$ mvn -Prelease-all -DskipTests clean install -U
jobshen@JobShen-PC:/usr/local/rocketmq$ cd distribution/target
jobshen@JobShen-PC:/usr/local/rocketmq$ cp apache-rocketmq.tar.gz /usr/local/apache-rocketmq.tar.gz
jobshen@JobShen-PC:/usr/local/rocketmq$ cd /usr/local
jobshen@JobShen-PC:/usr/local$ tar -zxf apache-rocketmq.tar.gz
  • 将压缩包拷贝到集群中所有机器，并解压

RocketMQ 部署

修改目录配置信息

  • log 日志根目录 store 存储根目录
1
jobshen@JobShen-PC:/usr/local$ mkdir -p rocketmq_data/logs rocketmq_data/store
  • 修改日志配置文件中路径

将logback_broker.xml 、logback_filtersrv.xml 、logback_broker.xml、logback_namesrv.xml、logback_tools.xml中的${user.home} 修改为 /usr/local/rocketmq_data

部署NameServer

  • 在10.100.1.23、10.100.1.24两台机器上添加 nameserver 配置文件：nameserver.properties：
1
2
jobshen@JobShen-PC:/usr/local$ mkdir apache-rocketmq/conf/product
jobshen@JobShen-PC:/usr/local$ vim apache-rocketmq/conf/product/nameserver.properties
1
2
3
4
5
6
7
8
9
10
11
12
rocketmqHome=/usr/local/apache-rocketmq
kvConfigPath=/usr/local/rocketmq_data/store/namesrv/kvConfig.json
listenPort=9876
serverWorkerThreads=16
serverCallbackExecutorThreads=0
serverSelectorThreads=6
serverOnewaySemaphoreValue=512
serverAsyncSemaphoreValue=128
serverChannelMaxIdleTimeSeconds=240
serverSocketSndBufSize=4096
serverSocketRcvBufSize=2048
serverPooledByteBufAllocatorEnable=false
  • 修改jvm启动参数：
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ vim bin/runserver.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
#!/bin/sh

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
    echo "ERROR: $1 !!"
    exit 1
}

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms16g -Xmx16g -Xmn4g -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=1024m -XX:+AlwaysPreTouch -XX:-UseBiasedLocking"
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=50m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30"
JAVA_OPT="${JAVA_OPT} -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m -verbose:gc -Xloggc:/dev/shm/rmq_srv_gc.log -XX:+PrintGCDetails"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT}  -XX:-UseLargePages"
JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"

$JAVA ${JAVA_OPT} $@
  • 制作systemd脚本：namesrv.service
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ vim /lib/systemd/system/namesrv.service
1
2
3
4
5
6
7
8
9
10
[Unit]
Description=RocketMQ-Nameserver
After=network.target

[Service]
ExecStart=/usr/local/apache-rocketmq/bin/mqnamesrv -c /usr/local/apache-rocketmq/conf/product/namesrv.properties
ExecStop=/usr/local/apache-rocketmq/bin/mqshutdown namesrv

[Install]
WantedBy=multi-user.target
1
2
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ ln -s /etc/systemd/system/namesrv.service /lib/systemd/system/namesrv.service
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ systemctl daemon-reload
  • 启动nameserver
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ systemctl start namesrv

部署Broker

部署broker-a

Master
  • 在10.100.1.21上新建broker-a-master.properties文件：
1
2
jobshen@JobShen-PC:/usr/local$ mkdir apache-rocketmq/conf/product
jobshen@JobShen-PC:/usr/local$ vim apache-rocketmq/conf/product/broker-a-master.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
rocketmqHome=/usr/local/apache-rocketmq
namesrvAddr=10.100.1.23:9876;10.100.1.24:9876
brokerIP1=10.100.1.21
brokerIP2=10.100.1.21
brokerName=mq_a
brokerClusterName=ProductCluster
brokerId=0
defaultTopicQueueNums=16
autoCreateTopicEnable=false
clusterTopicEnable=true
brokerTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
storePathRootDir=/usr/local/rocketmq_data/store
storePathCommitLog=/usr/local/rocketmq_data/store/commitlog/
storePathConsumerQueue=/usr/local/rocketmq_data/store/consumequeue/
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=6000000
deleteWhen=04
diskMaxUsedSpaceRatio=75
fileReserverdTime=72
haListenPort=10912
haSendHeartbeatInterval=5000
haHousekeepingInterval=20000
haTransferBatchSize=32768
haMasterAddress=
haSlaveFallbehindMax=268435456
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
syncFlushTimeout=5000
messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
flushDelayOffsetInterval=10000
cleanFileForciblyEnable=true
  • 修改jvm启动参数：
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ vim bin/runbroker.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#!/bin/sh

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
    echo "ERROR: $1 !!"
    exit 1
}

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms48g -Xmx48g -Xmn24g"
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=64m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8"
JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy"
JAVA_OPT="${JAVA_OPT} -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT} -XX:+AlwaysPreTouch"
JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=20g"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages -XX:-UseBiasedLocking"
JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"

numactl --interleave=all pwd > /dev/null 2>&1
if [ $? -eq 0 ]
then
	if [ -z "$RMQ_NUMA_NODE" ] ; then
		numactl --interleave=all $JAVA ${JAVA_OPT} $@
	else
		numactl --cpunodebind=$RMQ_NUMA_NODE --membind=$RMQ_NUMA_NODE $JAVA ${JAVA_OPT} $@
	fi
else
	$JAVA ${JAVA_OPT} $@
fi
  • 制作systemd脚本：broker.service
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ vim /lib/systemd/system/broker.service
1
2
3
4
5
6
7
8
9
10
[Unit]
Description=RocketMQ-Broker
After=namesrv.service

[Service]
ExecStart=/usr/local/apache-rocketmq/bin/mqbroker -c /usr/local/apache-rocketmq/conf/product/broker-a-master.properties
ExecStop=/usr/local/apache-rocketmq/bin/mqshutdown broker

[Install]
WantedBy=multi-user.target
1
2
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ ln -s /etc/systemd/system/broker.service /lib/systemd/system/broker.service
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ systemctl daemon-reload
  • 启动broker
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ systemctl start broker
Slave
  • 在10.100.1.23上新建broker-a-slave.properties文件：
1
2
jobshen@JobShen-PC:/usr/local$ mkdir apache-rocketmq/conf/product
jobshen@JobShen-PC:/usr/local$ vim apache-rocketmq/conf/product/broker-a-slave.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
rocketmqHome=/usr/local/apache-rocketmq
namesrvAddr=10.100.1.23:9876;10.100.1.24:9876
brokerIP1=10.100.1.23
brokerIP2=10.100.1.23
brokerName=mq_a
brokerClusterName=ProductCluster
brokerId=1
defaultTopicQueueNums=16
autoCreateTopicEnable=false
clusterTopicEnable=true
brokerTopicEnable=true
autoCreateSubscriptionGroup=true
listenPort=10911
storePathRootDir=/usr/local/rocketmq_data/store
storePathCommitLog=/usr/local/rocketmq_data/store/commitlog/
storePathConsumerQueue=/usr/local/rocketmq_data/store/consumequeue/
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=6000000
deleteWhen=04
diskMaxUsedSpaceRatio=75
fileReserverdTime=72
haListenPort=10912
haSendHeartbeatInterval=5000
haHousekeepingInterval=20000
haTransferBatchSize=32768
haMasterAddress=
haSlaveFallbehindMax=268435456
brokerRole=ASYNC_MASTER
flushDiskType=ASYNC_FLUSH
syncFlushTimeout=5000
messageDelayLevel=1s 5s 10s 30s 1m 2m 3m 4m 5m 6m 7m 8m 9m 10m 20m 30m 1h 2h
flushDelayOffsetInterval=10000
cleanFileForciblyEnable=true
  • 修改jvm启动参数：
1
jobshen@JobShen-PC:/usr/local/apache-rocketmq$ vim bin/runbroker.sh
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
#!/bin/sh

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

#===========================================================================================
# Java Environment Setting
#===========================================================================================
error_exit ()
{
    echo "ERROR: $1 !!"
    exit 1
}

[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=$HOME/jdk/java
[ ! -e "$JAVA_HOME/bin/java" ] && JAVA_HOME=/usr/java
[ ! -e "$JAVA_HOME/bin/java" ] && error_exit "Please set the JAVA_HOME variable in your environment, We need java(x64)!"

export JAVA_HOME
export JAVA="$JAVA_HOME/bin/java"
export BASE_DIR=$(dirname $0)/..
export CLASSPATH=.:${BASE_DIR}/conf:${CLASSPATH}

#===========================================================================================
# JVM Configuration
#===========================================================================================
JAVA_OPT="${JAVA_OPT} -server -Xms32g -Xmx32g -Xmn16g"
JAVA_OPT="${JAVA_OPT} -XX:+UseG1GC -XX:G1HeapRegionSize=32m -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:SurvivorRatio=8"
JAVA_OPT="${JAVA_OPT} -verbose:gc -Xloggc:/dev/shm/mq_gc_%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintAdaptiveSizePolicy"
JAVA_OPT="${JAVA_OPT} -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=30m"
JAVA_OPT="${JAVA_OPT} -XX:-OmitStackTraceInFastThrow"
JAVA_OPT="${JAVA_OPT} -XX:+AlwaysPreTouch"
JAVA_OPT="${JAVA_OPT} -XX:MaxDirectMemorySize=15g"
JAVA_OPT="${JAVA_OPT} -XX:-UseLargePages -XX:-UseBiasedLocking"
JAVA_OPT="${JAVA_OPT} -Djava.ext.dirs=${JAVA_HOME}/jre/lib/ext:${BASE_DIR}/lib"
#JAVA_OPT="${JAVA_OPT} -Xdebug -Xrunjdwp:transport=dt_socket,address=9555,server=y,suspend=n"
JAVA_OPT="${JAVA_OPT} ${JAVA_OPT_EXT}"
JAVA_OPT="${JAVA_OPT} -cp ${CLASSPATH}"

numactl --interleave=all pwd > /dev/null 2>&1
if [ $? -eq 0 ]
then
	if [ -z "$RMQ_NUMA_NODE" ] ; then
		numactl --interleave=all $JAVA ${JAVA_OPT} $@
	else
		numactl --cpunodebind=$RMQ_NUMA_NODE --membind=$RMQ_NUMA_NODE $JAVA ${JAVA_OPT} $@
	fi
else
	$JAVA ${JAVA_OPT} $@
fi
  • 和master一样制作systemd 服务并启动

部署broker-b

在10.100.1.22上部署master，在10.100.1.24上部署slave，方式和broker-a一样，只需要注意修改ip地址就可以了。

这样，到这里为止，MQ的部署就完成了。