在入侵检测业务中，针对文件内容型的告警详情，需要匹配命中内容的上下若干行。
第一种实现
先来看一种实现：
/**
* Created by zhangli on 19-12-18.
* 高亮文本工具类
*/
public class HighLightUtils {
private static final Integer LINE_NUM = 10;
private static final int MAX_REGEX_NUM = 10;
/**
* @param content 文本内容
* @param keywords 关键字列表
* @return 高亮内容段落集
*/
public static List<MatchedContent> highlight(String content, List<String> keywords) {
if (StringUtils.isEmpty(content) || CollectionUtils.isEmpty(keywords)) {
return Collections.emptyList();
}
List<MatchedContent> partContentList = Lists.newArrayList();
for (String keyword : keywords) {
if (!content.contains(keyword)) {
continue;
}
partContentList.addAll(highlight(content, escapeRegexSpecialWord(keyword)));
}
return partContentList;
}
/**
* @param content 文本内容
* @param regex 正则表达式
* @return 高亮内容段落集
*/
public static List<MatchedContent> highlight(String content, String regex) {
return highlight(content, regex, MAX_REGEX_NUM, LINE_NUM);
}
public static List<MatchedContent> highlight(String content, String regex, int maxMatchNum, int lineNum) {
if (StringUtils.isEmpty(content) || StringUtils.isEmpty(regex)) {
return Collections.emptyList();
}
content = content.replaceAll("\\r\\n", "\n");
Pattern pattern = Pattern.compile(regex);
Matcher m = pattern.matcher(content);
List<MatchedContent> partContentList = Lists.newArrayList();
int maxNum = maxMatchNum;
while (m.find()) {
RegexMatchPoint regexMatchPoint = new RegexMatchPoint(m.start(), m.end());
partContentList.add(getPartContentMap(content, regexMatchPoint, lineNum));
if (--maxNum == 0) {
break;
}
}
return partContentList;
}
/**
* 根据正则匹配获取高亮内容及起始行
*/
private static MatchedContent getPartContentMap(String content, RegexMatchPoint m, int lineNum) {
// 获取匹配内容在文件中的行数
int startMatchLine = content.substring(0, m.getStart()).split("\\n").length;
int endMatchLine = content.substring(0, m.getEnd()).split("\\n").length;
// 高亮文件匹配内容
String highlightContent = highlightOneRegexContent(content, m);
// 截取匹配内容前后共20行(若匹配内容跨行，且大于10行，则从匹配的地方开始截取)
String partContent = getPartContent(highlightContent, startMatchLine, endMatchLine);
// 获取截取内容首行的行号
int startLine = endMatchLine - lineNum + 1;
//如果匹配的内容大于10行，则从最初匹配行开始，而不是固定的10行
if (startMatchLine < startLine) {
startLine = startMatchLine;
}
return MatchedContent.builder()
.startLine(startLine < 1 ? 1 : startLine)
.partContent(partContent)
.build();
}
/**
* 获取高亮行前后部分内容
*/
private static String getPartContent(String content, Integer startMatchLine, Integer endMatchLine) {
int start = StringUtils.ordinalIndexOf(content, "\n", endMatchLine - LINE_NUM);
if (endMatchLine - startMatchLine > LINE_NUM) {
start = StringUtils.ordinalIndexOf(content, "\n", startMatchLine - 1);
}
start = start < 0 ? 0 : start + 1;
int end = StringUtils.ordinalIndexOf(content, "\n", endMatchLine + LINE_NUM);
end = end < 0 ? content.length() : end;
return content.substring(start, end);
}
/**
* 高亮单个匹配的内容
*/
private static String highlightOneRegexContent(String content, RegexMatchPoint point) {
int start = 0;
StringBuffer highlightContentSb = new StringBuffer();
highlightContentSb.append(content.substring(start, point.getStart())).append(CommonValues.HIGH_LIGHT_START)
.append(content.substring(point.getStart(), point.getEnd())).append(CommonValues.HIGH_LIGHT_END)
.append(content.substring(point.getEnd()));
return highlightContentSb.toString();
}
private static String escapeRegexSpecialWord(String keyword) {
if (keyword != "") {
String[] fbsArr = { "\\", "$", "(", ")", "*", "+", ".", "[", "]", "?", "^", "{", "}", "|" };
for (String key : fbsArr) {
if (keyword.contains(key)) {
keyword = keyword.replace(key, "\\" + key);
}
}
}
return keyword;
}
@Setter
@Getter
@ToString
public static class RegexMatchPoint implements Comparable<RegexMatchPoint> {
private Integer start;
private Integer end;
public RegexMatchPoint(Integer start, Integer end) {
this.start = start;
this.end = end;
}
//按开始位置排序
@Override
public int compareTo(RegexMatchPoint o) {
if (start.compareTo(o.getStart()) == 0) {
return end.compareTo(o.getEnd());
} else {
return start.compareTo(o.getStart());
}
}
public RegexMatchPoint copy() {
return new RegexMatchPoint(start, end);
}
}
}
这个实现还是不错的，至少给人很好的启发，是一个很好的改进基础。
那么，它的问题在哪里呢？
- 按单个匹配高亮，如果是行内有多个匹配的话，就很难合并；
- 缺乏行号的记录，所有跟行有关的东西，都是通过 split("\n") 和 substring(start, end)来实现的；
- 合并命中内容比较困难。
构造与使用分离
何为构造与使用分离呢？ 是指构造的时候，就提取出足够的必要信息； 而在使用时则运用这些信息去处理，而不是“边构建边使用”。就像编译器做代码编译和自动生成一样，应该不会是边编译边生成代码。
边构建边使用的实现，会将构建与处理耦合在一起，一旦有需要改动，就会比较困难。
很显然，如果要构造与使用分离，那么我们需要首先拿到什么内容？ （命中内容的行号、起始位置、结束位置；所有文件行及行号） 应该先把这些必要信息提取出来。一旦我们确定了求解问题需要的必要信息，想出一个清晰的算法就比较自然了。
高亮内容合并展示的算法简述
步骤一：获取所有行及行号【行号，行内容】；
步骤二: 先找到所有匹配正则的字符串的行号及起始结束点 regexMatchPoints =(lineNo, start, end)；
步骤三：将 regexMatchPoints 按行分组；因为行内的多个匹配合并很麻烦；
步骤四：所有匹配行号，按匹配行号排序，方便最终按行号序展示；
步骤五：按行生成高亮内容展示[行号，高亮行内容]；
步骤六：按匹配行号计算起始行和结束行行号，如果已经在这个区间的行号则可以过滤（合并实现）；
步骤七：根据所有起始行号和结束行号获取对应的行内容。
实现代码
/**
* 高亮文本展示工具类
* Created by qinshu on 2021/12/31
*/
public class HighLightUtil {
private static final Logger LOG = LogUtils.getLogger(HighLightUtil.class);
/** 高亮展示前后的行数 */
private static final Integer HIGHLIGHT_LINE_NUM = 5;
/** 最大匹配多少次 */
private static final int MAX_REGEX_NUM = 10;
/**
* @param content 文本内容
* @param regex 正则表达式
* @return 高亮内容段落集
*/
public static List<MatchedFileContent> highlight(String content, String regex) {
return highlight(content, regex, MAX_REGEX_NUM, HIGHLIGHT_LINE_NUM);
}
/**
* @param base64Content 文本内容（base64编码后的文本）
* @param regex 正则表达式
* @return 高亮内容段落集
*/
public static List<MatchedFileContent> highlightBase64(String base64Content, String regex) {
if (StringUtils.isEmpty(base64Content)) {
return Collections.emptyList();
}
return highlight(Base64Utils.decodeContent(base64Content), regex);
}
public static List<MatchedFileContent> highlight(String content, String regex, int maxMatchNum, int highlightLineNum) {
if (StringUtils.isEmpty(content) || StringUtils.isEmpty(regex)) {
return Collections.emptyList();
}
content = content.replaceAll("\\r\\n", "\n");
List<String> allLines = Arrays.asList(content.split("\n"));
Pattern pattern = Pattern.compile(regex);
List<RegexMatchPoint> regexMatchPoints = findAllRegexMatches(allLines, pattern);
// 按行号分组，匹配高亮展示，因为单行多个匹配的高亮需要单行展示，分开后合并比较麻烦
Map<Integer, List<RegexMatchPoint>> regexMatchPointMap = regexMatchPoints.stream().collect(Collectors.groupingBy(RegexMatchPoint::getLineNo));
// highLightLineMap: [行号，高亮行]
Map<Integer, String> highLightLineMap = new HashMap<>();
regexMatchPointMap.forEach((lineNo, matchPointsOfLine) -> {
highLightLineMap.put(lineNo, highlightOneLineContent(allLines.get(lineNo), matchPointsOfLine));
} );
List<MatchedFileContent> partContentList = merge(highLightLineMap, allLines, highlightLineNum);
return partContentList.subList(0, Math.min(partContentList.size(), maxMatchNum));
}
private static List<MatchedFileContent> merge(Map<Integer, String> highLightLineMap, List<String> allLines, int highlightLineNum) {
// 按行号排序
List<Integer> highLightLineNos = Lists.newArrayList(highLightLineMap.keySet());
Collections.sort(highLightLineNos);
// 计算需要展示的行号
List<MatchedFileLine> matchedFileLines = Lists.newArrayList();
for (Integer highLineNo: highLightLineNos) {
if (!exist(matchedFileLines, highLineNo)) {
int startLine = highLineNo - highlightLineNum;
int endLine = 0;
if (startLine < 0) {
startLine = 0;
endLine = highLineNo + highlightLineNum;
}
else {
startLine = highLineNo - highlightLineNum + 1;
endLine = highLineNo + highlightLineNum;
}
matchedFileLines.add(new MatchedFileLine(startLine, endLine));
}
}
return matchedFileLines.stream()
.map(fileLine -> getMatchedFileContent(highLightLineMap, allLines, fileLine)).collect(Collectors.toList());
}
/**
* 获取指定行号的行内容
*/
private static String getLine(Map<Integer, String> highLightLineMap, List<String> allLines, Integer lineNo) {
String highLightLine = highLightLineMap.get(lineNo);
return highLightLine != null ? highLightLine : allLines.get(lineNo);
}
private static boolean exist(List<MatchedFileLine> matchedFileLines, Integer lineNo) {
return matchedFileLines.stream().anyMatch(fileLine -> exist(fileLine, lineNo));
}
private static boolean exist(MatchedFileLine matchedFileLine, Integer lineNo) {
return lineNo >= matchedFileLine.getStartLine() && lineNo < matchedFileLine.getEndLine();
}
/**
* 根据起始行号获取
* @param highLightLineMap 高亮行
* @param allLines 文件所有行
* @param fileLine 匹配内容上下文行号
* @return 匹配内容上下文及起始行号
*/
private static MatchedFileContent getMatchedFileContent(Map<Integer, String> highLightLineMap, List<String> allLines, MatchedFileLine fileLine) {
StringBuilder partContentBuilder = new StringBuilder();
for (int i = fileLine.getStartLine(); i < fileLine.getEndLine() && i < allLines.size(); i++) {
partContentBuilder.append(getLine(highLightLineMap, allLines, i) + "\n");
}
return new MatchedFileContent(fileLine.getStartLine() + 1, partContentBuilder.toString());
}
/**
* 获取所有正则匹配点
* @param allLines 文件内容的所有行
* @param pattern 正则匹配编译表达式
* @return 所有匹配正则表达式的字符串的位置
*/
private static List<RegexMatchPoint> findAllRegexMatches(List<String> allLines, Pattern pattern) {
// 先拿到所有的正则匹配点，行号从 0 开始
List<RegexMatchPoint> regexMatchPoints = Lists.newArrayList();
for (int i=0; i < allLines.size(); i++) {
String line = allLines.get(i);
Matcher m = pattern.matcher(line);
while (m.find()) {
RegexMatchPoint regexMatchPoint = new RegexMatchPoint(i, m.start(), m.end());
regexMatchPoints.add(regexMatchPoint);
}
}
return regexMatchPoints;
}
/**
* 高亮文本内容
*/
public static String highlightContent(String content, List<String> match) {
if (CollectionUtils.isEmpty(match)) {
return content;
}
try {
for (String matchContent : match) {
String highlightContent = String.format("%s%s%s", CommonValues.HIGH_LIGHT_START, matchContent, CommonValues.HIGH_LIGHT_END);
content = content.replaceAll(ExprUtils.escapeExprSpecialWord(matchContent), highlightContent);
}
} catch (Exception e) {
LOG.error("highlight content error, content:{}, match:{}", content, match);
}
return content;
}
/**
* 高亮一行的展示
*/
public static String highlightOneLineContent(String content, List<RegexMatchPoint> points) {
int start = 0;
int lastMatchEnd = 0;
StringBuilder sb = new StringBuilder();
for (RegexMatchPoint point: points) {
sb.append(content, start, point.getStart()).append(CommonValues.HIGH_LIGHT_START)
.append(content, point.getStart(), point.getEnd()).append(CommonValues.HIGH_LIGHT_END);
start = point.getEnd();
lastMatchEnd = point.getEnd();
}
sb.append(content.substring(lastMatchEnd));
return sb.toString();
}
@Setter
@Getter
@ToString
public static class RegexMatchPoint implements Comparable<RegexMatchPoint> {
private Integer lineNo;
private Integer start;
private Integer end;
public RegexMatchPoint(Integer lineNo, Integer start, Integer end) {
this.lineNo = lineNo;
this.start = start;
this.end = end;
}
public RegexMatchPoint copy() {
return new RegexMatchPoint(lineNo, start, end);
}
}
@Setter
@Getter
public static class MatchedFileLine {
private Integer startLine;
private Integer endLine;
public MatchedFileLine(Integer startLine, Integer endLine) {
this.startLine = startLine;
this.endLine = endLine;
}
}
}
自测
/**
* 高亮展示
* Created by qinshu on 2021/12/31
*/
public class HighlightUtilTest {
String content = "dependencies {\n" +
" testCompile group: 'junit', name: 'junit'\n" +
"\n" +
" compile project(\":detect-lib\")\n" +
" compile project(\":connect-cli\")\n" +
" compile project(\":wisteria-client\")\n" +
" compile project(\":upload-cli\")\n" +
" compile project(\":scan-client\")\n" +
" compile(\"com.qt.qt-common:config-loader\")\n" +
" compile project(\":switches-lib\")\n" +
" compile project(\":bizevent-lib\")\n" +
" compile project(\":user-client\")\n" +
" compile project(\":notif-client\")\n" +
" compile project(\":detect-client\")\n" +
" compile project(\":job-cli\")\n" +
" compile('com.qt.qt-common:redis-lib')\n" +
" compile('com.qt.qt-common:rabbitmq-lib')\n" +
" compile('com.qt.qt-common:encrypt-property-lib')\n" +
" compile project(\":leader-latch-lib\")\n" +
" compile(\"com.qt.qt-common:eventflow-lib:1.0.0-SNAPSHOT\")\n" +
" compile(\"com.qt.qt-common:intrusion-detect-lib:1.0.1\")\n" +
" compile('com.qt.qt-common:mysql-lib')\n" +
" compile('com.qt.qt-common:rule-crypto')\n" +
" compile project(\":rule-lib\")\n" +
" compile project(\":api-auth-lib\")\n" +
"\n" +
" // Spring Cloud\n" +
" // 配置中心\n" +
" compile ('org.springframework.cloud:spring-cloud-starter-zookeeper-config')\n" +
" // 服务发现\n" +
" compile ('org.springframework.cloud:spring-cloud-starter-zookeeper-discovery')\n" +
" compile ('com.netflix.hystrix:hystrix-javanica')\n" +
"\n" +
" // Spring Boot\n" +
" compile('org.springframework.boot:spring-boot-starter-web')\n" +
" compile('org.springframework.boot:spring-boot-starter-aop')\n" +
" compile('org.springframework.boot:spring-boot-starter-data-redis')\n" +
"\n" +
" // Spring\n" +
" compile('org.springframework:spring-orm')\n" +
" compile('org.springframework:spring-jdbc')\n" +
" compile('org.springframework:spring-aop')\n" +
"\n" +
" // mongodb\n" +
" compile('org.springframework.data:spring-data-mongodb:1.10.23.RELEASE')\n" +
"\n" +
" // Mysql\n" +
" runtime('mysql:mysql-connector-java')\n" +
" compile('com.zaxxer:HikariCP')\n" +
" compile('org.mybatis.spring.boot:mybatis-spring-boot-starter')\n" +
" compile('com.github.pagehelper:pagehelper-spring-boot-starter')\n" +
"\n" +
" //redisson\n" +
" compile('io.projectreactor:reactor-core:3.2.8.RELEASE')\n" +
"\n" +
" // Jackson\n" +
" compile('com.fasterxml.jackson.core:jackson-core')\n" +
" compile('com.fasterxml.jackson.core:jackson-annotations')\n" +
" compile('com.fasterxml.jackson.core:jackson-databind')\n" +
" compile('org.codehaus.jackson:jackson-core-asl')\n" +
"\n" +
" compile('joda-time:joda-time')\n" +
" compile('commons-io:commons-io:2.5')\n" +
" compile('org.apache.commons:commons-lang3:3.5')\n" +
" compile('org.apache.commons:commons-collections4:4.1')\n" +
" compile('cglib:cglib:3.2.5')\n" +
" compile('net.java.dev.jna:jna:5.8.0')\n" +
" compile('org.apache.calcite:calcite-core:1.26.0')\n" +
"\n" +
" // Test\n" +
" testCompile('org.mockito:mockito-core:2.13.0')\n" +
" testCompile('org.springframework:spring-test')\n" +
" testCompile('org.springframework.boot:spring-boot-starter-test')\n" +
"\n" +
" // string-similarity\n" +
" compile('info.debatty:java-string-similarity:0.24')\n" +
"\n" +
" compile('com.jayway.jsonpath:json-path')\n" +
"\n" +
" compile('com.qt.qt-common:cron-lib:1.0.0')\n" +
"\n" +
"}";
@Test
public void tsetHighlight() {
String regex = "org\\.apache";
List<MatchedFileContent> matched = HighLightUtil.highlight(content, regex);
Assert.assertTrue(matched.size() > 0);
}
@Test
public void testHighlightBase64() {
String content = "MG1laW5hMiAxbWVpbmEyCjBtZWluYTIgMW1laW5hMgo=";
String regex = "meina2";
List<MatchedFileContent> matchedFileContents = HighLightUtil.highlightBase64(content, regex);
Assert.assertEquals(1, matchedFileContents.size());
Assert.assertEquals("[MatchedFileContent(startLine=1, partContent=0<qthighlight--meina2--qthighlight> 1<qthighlight--meina2--qthighlight>\n" +
"0<qthighlight--meina2--qthighlight> 1<qthighlight--meina2--qthighlight>\n" +
")]", matchedFileContents.toString());
}
@Test
public void testHighLight2() {
String content = "customdir2 1\n" +
"customdir2 2\n" +
"customdir2 3\n" +
"customdir2 4\n" +
"customdir2 5\n" +
"customdir2 6\n" +
"customdir2 7 customdir2 7 customdir2 7 customdir2 7\n" +
"customdir2 8 customdir2 8 customdir2 8 customdir2 8";
String regex = "customdir2";
List<MatchedFileContent> matchedFileContents = HighLightUtil.highlight(content, regex);
Assert.assertEquals(2, matchedFileContents.size());
}
@Test
public void testHighLight3() {
String content = "customdir2 1\n" +
"customdir2 2\n" +
"customdir3 3\n" +
"customdir5 4\n" +
"customdir6 5\n" +
"customdir9 6\n" +
"customdir10 7 customdir8 7 customdird 7 customdiro 7\n" +
"customdir2 8 customdir2 8 customdir2 8 customdir2 8";
String regex = "customdir2";
List<MatchedFileContent> matchedFileContents = HighLightUtil.highlight(content, regex);
Assert.assertEquals(2, matchedFileContents.size());
}
}
小结
本文讲解了如何运用“构造与使用分离”的思想，来重构和改进高亮展示命中文本内容的算法实现。 构造与使用分离，即是在构造的时候抽取所需的必要信息，而在使用的时候去构建所需要功能，而不是边构建边使用，将构建与使用耦合在一起，后续如果有需求变更，改动就会比较麻烦。