-
-
-
- 
+ 
+
+
+
+QQ Group: 832039395
+
+email: opensca@anpro-tech.com
+
+## Contributing
+
+OpenSCA is an open source project, we appreciate your help!
+
+To contribute, please read our [Contributing Guideline](https://github.com/XmirrorSecurity/OpenSCA-cli/blob/master/docs/Contributing%20Guideline-en%20v1.0.md).
+
+*For the Chinese version of this document, please check [README](https://github.com/XmirrorSecurity/OpenSCA-cli/blob/master/README.md).
diff --git a/CHANGELOG.md b/CHANGELOG.md
new file mode 100644
index 0000000000000000000000000000000000000000..7bef2485eefc83873fb9c37b2c5a03fcb7593da6
--- /dev/null
+++ b/CHANGELOG.md
@@ -0,0 +1,23 @@
+## v2.0.0
+
+* BUG修复：使用本地漏洞库时报空指针错误
+* 功能优化：依赖节点为空也会同步到SaaS平台上
+* 文件更新：英文版README等文件更新
+* 功能优化：新增内部使用的插件参数
+* 文件更新：README等文件更新
+* 功能优化：优化java mvn的直接依赖设置
+* 功能优化：优化对于java pom的解析，过滤scope为test时的依赖
+* 功能优化：html报告添加分页；优化html报告中检测时长为0.x秒时的精度(之前0.x秒显示为空)
+* 功能优化：检测目录时过滤.git等特殊目录的检测
+* 代码优化：日志里尽量避免记录token
+* 文件新增：新增CHANGELOG.md
+* 代码优化：优化saas新接口传递的包名格式为：`[基础路径名]包名`
+* 代码优化：python解析setup.py错误时增加日志输出
+* BUG修复：fix 错误修复[解析python requirement时windows和linux打开文件的权限差异问题]
+* README更新：新增参数说明，增加maven私服库的使用说明等
+* 功能优化：优化python setup.py的解析，不在临时目录中解析，直接进入项目目录进行解析
+* BUG修复：修复解析python setup.py中的依赖时，~符号没考虑到导致包名和版本号错误的bug；修复依赖去重时导致路径为空的bug
+* 功能新增：完成大部分包管理器的特定文件中提取包名和版本号
+* 功能新增：对接saas新接口。请求响应格式变动，请求会传递项目依赖结构、项目哈希、项目名称、项目版本号等
+* BUG修复：修复提取copyright信息时可能报错的bug
+* BUG修复：针对log模块中空指针问题进行修改
\ No newline at end of file
diff --git a/README.md b/README.md
index 15837f6d948a474430646a89a1e4523c5f4e5155..1f938e866cb8f5b9f3007db82067d809ac399594 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,5 @@
- 
+ 
@@ -14,6 +14,7 @@
---
## 检测能力
+
`OpenSCA`现已支持以下编程语言相关的配置文件解析及对应的包管理器，后续会逐步支持更多的编程语言，丰富相关配置文件的解析。
| 支持语言 | 包管理器 | 解析文件 |
@@ -26,16 +27,16 @@
| `Golang` | `gomod` | `go.mod` `go.sum` |
| `Rust` | `cargo` | `Cargo.lock` |
| `Erlang` | `Rebar` | `rebar.lock` |
-| `Python` | `Pip` | `Pipfile` `Pipfile.lock` `setup.py` `requirements.txt` `requirements.in` （后两者的解析需要具备pipenv环境，需要联网。） |
+| `Python` | `Pip` | `Pipfile` `Pipfile.lock` `setup.py` `requirements.txt` `requirements.in` （有的解析会对python版本有要求，后两者的解析需要具备pipenv环境，需要联网。） |
## 下载安装
-1. 从 [releases](https://github.com/XmirrorSecurity/OpenSCA-cli/releases) 下载对应系统架构的可执行文件压缩包
+1. 从 [releases](https://gitee.com/XmirrorSecurity/OpenSCA-cli/releases) 下载对应系统架构的可执行文件压缩包
2. 或者下载源码编译(需要 `go 1.18` 及以上版本)
```shell
- git clone https://github.com/XmirrorSecurity/OpenSCA-cli.git opensca
+ git clone https://gitee.com/XmirrorSecurity/OpenSCA-cli.git opensca
cd opensca
go work init cli analyzer util
go build -o opensca-cli cli/main.go
@@ -58,16 +59,22 @@
opensca-cli -path ${project_path}
```
-连接云平台
+使用本地漏洞库
+
+```shell
+opensca-cli -db db.json -path ${project_path}
+```
+
+仅使用云漏洞库服务
```shell
opensca-cli -url ${url} -token ${token} -path ${project_path}
```
-或使用本地漏洞库
+使用v2.0.0及以上版本连接SaaS服务，检测漏洞信息、获取资产清单、生成数据看板并进行项目管理
```shell
-opensca-cli -db db.json -path ${project_path}
+opensca-cli -url ${url} -token ${token} -v2 -path ${project_path}
```
## 参数说明
@@ -80,15 +87,31 @@ opensca-cli -db db.json -path ${project_path}
| `path` | `string` | 指定要检测的文件或目录路径 | `-path ./foo` |
| `url` | `string` | 从云漏洞库查询漏洞，指定要连接云服务的地址，与 `token` 参数一起使用 | `-url https://opensca.xmirror.cn` |
| `token` | `string` | 云服务验证 `token`，需要在云服务平台申请，与 `url` 参数一起使用 | `-token xxxxxxx` |
+| `v2` | `bool` | 是否使用新的v2版本的云服务接口 | `-v2` |
| `cache` | `bool` | 建议开启，缓存下载的文件(例如 `.pom` 文件)，重复检测相同组件时会节省时间，下载的文件会保存到工具所在目录的.cache 目录下 | `-cache` |
| `vuln` | `bool` | 结果仅保留有漏洞信息的组件，使用该参数将不会保留组件层级结构 | `-vuln` |
-| `out` | `string` | 将检测结果保存到指定文件，根据后缀生成不同格式的文件，默认为 `json` 格式；支持以`spdx`格式展示`sbom`清单，只需更换相应输出文件后缀即可 | `-out output.json` |
+| `out` | `string` | 将检测结果保存到指定文件，根据后缀生成不同格式的文件，默认为 `json` 格式；支持以`spdx`格式展示`sbom`清单，只需更换相应输出文件后缀即可 | `-out output.json` |
| `db` | `string` | 指定本地漏洞库文件，希望使用自己漏洞库时可用，漏洞库文件为 `json` 格式，具体格式会在之后给出;若同时使用云端漏洞库与本地漏洞库，漏洞查询结果取并集 | `-db db.json` |
| `progress` | `bool` | 显示进度条 | `-progress` |
| `dedup` | `bool` | 相同组件去重 | `-dedup` |
+| `version` | `bool` | 显示客户端版本 | `-version` |
---
+如果要配置maven私服库，需要在配置文件里进行配置，格式如下：
+
+```json
+{
+ "maven": [
+ {
+ "repo": "url",
+ "user": "user",
+ "password": "password"
+ }
+ ]
+}
+```
+
### 漏洞库文件格式
```json
@@ -139,12 +162,24 @@ opensca-cli -db db.json -path ${project_path}
| `exploit_level_id` | 漏洞利用评级(0:不可利用,1:可利用) | 否 |
## 贡献者
+
- 张涛
- 张弛
- 陈钟
- 刘恩炙
- 宁戈
+## 问题反馈&联系我们
+
+微信技术交流群：（扫码添加小助手邀您入群）
+
+
+
+QQ技术交流群：832039395
+
+官方邮箱：opensca@anpro-tech.com
+
+
## 向我们贡献
**OpenSCA** 是一款开源的软件成分分析工具，项目成员期待您的贡献。
diff --git a/analyzer/engine/archive.go b/analyzer/engine/archive.go
index d4882cadaefef5d422a5419a2be8f00662d41110..6b4d98b57e9c1d4239b6f6756f49bc8dbb266f78 100644
--- a/analyzer/engine/archive.go
+++ b/analyzer/engine/archive.go
@@ -25,6 +25,9 @@ import (
"github.com/pkg/errors"
)
+// IgnoreDirMap 忽略的目录，用map方便判断
+var IgnoreDirMap = map[string]bool{".git": true, ".idea": true, ".vscode": true, ".svn": true}
+
// checkFile 检测是否为可检测的文件
func (e Engine) checkFile(filename string) bool {
for _, analyzer := range e.Analyzers {
@@ -130,8 +133,10 @@ func (e Engine) opendir(dirpath string) (dir *model.DirTree) {
filename := file.Name()
filepath := path.Join(dirpath, filename)
if file.IsDir() {
- dir.DirList = append(dir.DirList, filename)
- dir.SubDir[filename] = e.opendir(filepath)
+ if _, ok := IgnoreDirMap[filename]; !ok {
+ dir.DirList = append(dir.DirList, filename)
+ dir.SubDir[filename] = e.opendir(filepath)
+ }
} else {
if filter.AllPkg(filename) {
dir.DirList = append(dir.DirList, filename)
diff --git a/analyzer/engine/engine.go b/analyzer/engine/engine.go
index 4f6c7d48bb8546be0528c29cdb9068df59d51b85..dcd88605b8b19ab7fafb73a093010ab954f43ba7 100644
--- a/analyzer/engine/engine.go
+++ b/analyzer/engine/engine.go
@@ -12,6 +12,7 @@ import (
"strings"
"time"
"util/args"
+ "util/client"
"util/filter"
"util/logs"
"util/model"
@@ -57,10 +58,12 @@ func (e Engine) ParseFile(filepath string) (depRoot *model.DepTree, taskInfo rep
dirRoot := model.NewDirTree()
depRoot = model.NewDepTree(nil)
filepath = strings.ReplaceAll(filepath, `\`, `/`)
+ client.PackageBasePath = path.Base(filepath)
taskInfo = report.TaskInfo{
- AppName: strings.TrimSuffix(path.Base(filepath), path.Ext(path.Base(filepath))),
+ AppName: strings.TrimSuffix(client.PackageBasePath, path.Ext(client.PackageBasePath)),
StartTime: time.Now().Format("2006-01-02 15:04:05"),
}
+ client.PackageName = taskInfo.AppName
s := time.Now()
defer func() {
taskInfo.CostTime = time.Since(s).Seconds()
@@ -72,6 +75,9 @@ func (e Engine) ParseFile(filepath string) (depRoot *model.DepTree, taskInfo rep
return depRoot, taskInfo
} else {
if f.IsDir() {
+ //如果是目录则不用去除后缀
+ taskInfo.AppName = client.PackageBasePath
+ client.PackageName = taskInfo.AppName
// 目录
dirRoot = e.opendir(filepath)
// 尝试解析mvn依赖
diff --git a/analyzer/engine/parse.go b/analyzer/engine/parse.go
index 1f6fca1e4a4554dfaec2740370bb6e8263670a52..dc760cfbd9f9d46ef6ec418185d762553d153939 100644
--- a/analyzer/engine/parse.go
+++ b/analyzer/engine/parse.go
@@ -6,10 +6,20 @@
package engine
import (
+ "analyzer/java"
+ "crypto/md5"
+ "encoding/base64"
+ "encoding/json"
+ "fmt"
+ "io/ioutil"
+ "os"
"path"
"regexp"
"strings"
+ "util/client"
+ "util/enum/language"
"util/filter"
+ "util/logs"
"util/model"
)
@@ -20,12 +30,28 @@ const (
high
)
+// PackageVersionFile 需要解析包版本的文件数据
+type PackageVersionFile struct {
+ Language language.Type
+ FileData *model.FileInfo
+}
+
+// PackageBase package.json、composer.json提取需要的字段
+type PackageBase struct {
+ PackageName string `json:"name"`
+ PackageVersion string `json:"version"`
+}
+
// parseDependency 解析依赖
func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree) *model.DepTree {
if depRoot == nil {
depRoot = model.NewDepTree(nil)
}
var copyrightMess = make(map[string]string)
+ var h = md5.New()
+ var toHash = false
+ var packageVersionFiles []PackageVersionFile
+ var packageVersionFileCount = 0
for _, analyzer := range e.Analyzers {
// 遍历目录树获取要检测的文件
files := []*model.FileInfo{}
@@ -39,6 +65,76 @@ func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree)
for _, f := range n.Files {
if analyzer.CheckFile(f.Name) {
files = append(files, f)
+ //计算匹配到的文件hash。文件顺序会影响hash值，比如文件重命名后顺序改变
+ h.Write(f.Data)
+ if !toHash {
+ toHash = true
+ }
+ //只保存根目录下可以版本提取的文件
+ if strings.HasSuffix(f.Name, client.PackageBasePath+"/"+path.Base(f.Name)) {
+ packageVersionFileCount = packageVersionFileCount + 1
+ toParseVersion := false
+ //如果不满足条件则可能不是单一语言的包，不提取版本
+ switch analyzer.GetLanguage() {
+ //java最多只有1个
+ case language.Java:
+ if packageVersionFileCount == 1 {
+ toParseVersion = true
+ }
+ //php、ruby、go最多有两个
+ case language.Php, language.Ruby, language.Golang, language.Rust:
+ if packageVersionFileCount <= 2 {
+ toParseVersion = true
+ }
+ //js最多有3个
+ case language.JavaScript:
+ if packageVersionFileCount <= 3 {
+ toParseVersion = true
+ }
+ //python最多有5个
+ case language.Python:
+ if packageVersionFileCount <= 5 {
+ toParseVersion = true
+ }
+ }
+ if toParseVersion {
+ if analyzer.GetLanguage() == language.JavaScript {
+ //js只处理package.json
+ if !filter.JavaScriptPackage(f.Name) {
+ continue
+ }
+ } else if analyzer.GetLanguage() == language.Php {
+ //php只处理composer.json
+ if !filter.PhpComposer(f.Name) {
+ continue
+ }
+ } else if analyzer.GetLanguage() == language.Python {
+ //python只处理setup.py、pyproject.toml
+ if !filter.PythonSetup(f.Name) && !filter.PythonPyproject(f.Name) {
+ continue
+ }
+ } else if analyzer.GetLanguage() == language.Ruby {
+ //ruby只处理Gemfile
+ if !filter.RubyGemfile(f.Name) {
+ continue
+ }
+ } else if analyzer.GetLanguage() == language.Golang {
+ //go只处理go.mod
+ if !filter.GoMod(f.Name) {
+ continue
+ }
+ } else if analyzer.GetLanguage() == language.Rust {
+ //rust只处理Cargo.toml
+ if !filter.RustCargoToml(f.Name) {
+ continue
+ }
+ }
+ packageVersionFile := PackageVersionFile{Language: analyzer.GetLanguage(), FileData: f}
+ packageVersionFiles = append(packageVersionFiles, packageVersionFile)
+ } else {
+ packageVersionFiles = nil
+ }
+ }
} else if filter.CheckLicense(f.Name) {
if _, ok := copyrightMess[path.Dir(f.Name)]; !ok {
// 记录解析到的copyrigh信息
@@ -47,6 +143,7 @@ func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree)
}
}
}
+
// 从文件中解析依赖树
for _, d := range analyzer.ParseFiles(files) {
p := path.Dir(d.Path)
@@ -89,6 +186,12 @@ func (e Engine) parseDependency(dirRoot *model.DirTree, depRoot *model.DepTree)
}
}
}
+ if toHash {
+ client.PackageHash = base64.StdEncoding.EncodeToString(h.Sum(nil))
+ }
+ if client.PackageVersion == "" && len(packageVersionFiles) >= 1 {
+ parsePackageVersion(packageVersionFiles)
+ }
// 删除依赖树空节点
q := []*model.DepTree{depRoot}
for len(q) > 0 {
@@ -142,7 +245,7 @@ func parseCopyright(f *model.FileInfo) string {
if len(tks) == 0 {
continue
}
- if strings.EqualFold("copyright", tks[0]) {
+ if strings.EqualFold("copyright", tks[0]) && len(tks) > 1 {
if re.MatchString(tks[1]) {
matchLevel[high] = line
}
@@ -163,3 +266,159 @@ func parseCopyright(f *model.FileInfo) string {
}
return ""
}
+
+// 从特定文件中提取包名、版本信息
+func parsePackageVersion(packageVersionFiles []PackageVersionFile) {
+ for _, packageVersionFile := range packageVersionFiles {
+ switch packageVersionFile.Language {
+ case language.Java:
+ if filter.JavaPom(packageVersionFile.FileData.Name) {
+ p := java.ReadPom(packageVersionFile.FileData.Data)
+ client.PackageVersion = p.Version
+ if p.ArtifactId != "" {
+ client.PackageName = p.ArtifactId
+ }
+ } else if strings.HasSuffix(packageVersionFile.FileData.Name, ".gradle") {
+ //基本没有声明包名和版本号？ 暂不解析
+ }
+ case language.JavaScript, language.Php:
+ packageBase := PackageBase{}
+ err := json.Unmarshal(packageVersionFile.FileData.Data, &packageBase)
+ if err != nil {
+ logs.Warn(err)
+ } else {
+ if packageBase.PackageName != "" {
+ client.PackageName = packageBase.PackageName
+ }
+ if packageBase.PackageVersion != "" {
+ client.PackageVersion = packageBase.PackageVersion
+ }
+ }
+ case language.Ruby:
+ //根路径是否存在gemspec
+ gemspecPath := ""
+ rootDir := path.Dir(packageVersionFile.FileData.Name)
+ files, err := ioutil.ReadDir(rootDir)
+ if err != nil {
+ continue
+ }
+ for _, file := range files {
+ if !file.IsDir() {
+ if strings.HasSuffix(file.Name(), "gemspec") {
+ gemspecPath = path.Join(rootDir, file.Name())
+ if data, err := ioutil.ReadFile(gemspecPath); err == nil {
+ pkgMatch := regexp.MustCompile(`\s*spec\.name\s*=\s*["'](.*)["']\r?\n`)
+ r := pkgMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageName = string(r[1])
+ }
+ }
+ pkgVerMatch := regexp.MustCompile(`\s*spec\.version\s*=\s*["'](.*)["']\r?\n`)
+ r = pkgVerMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageVersion = string(r[1])
+ }
+ } else {
+ pkgVerPathMatch := regexp.MustCompile(`require\s*["'](.*version)["']\r?\n`)
+ r = pkgVerPathMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ //一般在lib目录
+ pkgVersionPath := fmt.Sprintf("%s/lib/%s.rb", rootDir, string(r[1]))
+ if f, err := os.Stat(pkgVersionPath); err == nil {
+ if !f.IsDir() {
+ if data, err := ioutil.ReadFile(pkgVersionPath); err == nil {
+ pkgVerMatch = regexp.MustCompile(`\s*VERSION\s*=\s*["'](.*)["']\r?\n`)
+ r = pkgVerMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageVersion = string(r[1])
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ return
+ }
+ }
+ }
+ case language.Golang:
+ //只能提取包名
+ pkgMatch := regexp.MustCompile(`module\s*(.*)\r?\n`)
+ r := pkgMatch.FindSubmatch(packageVersionFile.FileData.Data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageName = string(r[1])
+ }
+ }
+ case language.Rust:
+ pkgNameMatch := regexp.MustCompile(`\nname\s*=\s*['"](.*)['"]\r?\n`)
+ pkgVerMatch := regexp.MustCompile(`\nversion\s*=\s*['"](.*)['"]\r?\n`)
+ r := pkgNameMatch.FindSubmatch(packageVersionFile.FileData.Data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageName = string(r[1])
+ }
+ }
+ r = pkgVerMatch.FindSubmatch(packageVersionFile.FileData.Data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageVersion = string(r[1])
+ return
+ }
+ }
+ case language.Python:
+ //先从特定文件中尝试提取包名和版本
+ for _, filepath := range []string{
+ path.Dir(packageVersionFile.FileData.Name) + "/PKG-INFO",
+ path.Dir(packageVersionFile.FileData.Name) + "/METADATA",
+ } {
+ if f, err := os.Stat(filepath); err == nil {
+ if !f.IsDir() {
+ if data, err := ioutil.ReadFile(filepath); err == nil {
+ pkgNameMatch := regexp.MustCompile(`\nName: (.*)\r?\n`)
+ r := pkgNameMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageName = string(r[1])
+ }
+ }
+ pkgVerMatch := regexp.MustCompile(`\nVersion: (.*)\r?\n`)
+ r = pkgVerMatch.FindSubmatch(data)
+ if len(r) == 2 {
+ if string(r[1]) != "" {
+ client.PackageVersion = string(r[1])
+ return
+ }
+ }
+ }
+ }
+ }
+ }
+ pkgVerMatch := ®exp.Regexp{}
+ //可能存在name和version不是相邻的，暂时不考虑
+ if filter.PythonSetup(packageVersionFile.FileData.Name) {
+ pkgVerMatch = regexp.MustCompile(`setup\(\s*name\s*=\s*["'](.*)["'],\s*version\s*=\s*["'](.*)["'],`)
+ } else if filter.PythonPyproject(packageVersionFile.FileData.Name) {
+ pkgVerMatch = regexp.MustCompile(`\[tool.poetry\]\s*name\s*=\s*"(.*)"\s*version\s*=\s*"(.*)"\r?\n`)
+ }
+ if pkgVerMatch.String() != "" {
+ r := pkgVerMatch.FindSubmatch(packageVersionFile.FileData.Data)
+ if len(r) == 3 {
+ if string(r[1]) != "" {
+ client.PackageName = string(r[1])
+ }
+ if string(r[2]) != "" {
+ client.PackageVersion = string(r[2])
+ }
+ }
+ }
+ }
+ }
+}
diff --git a/analyzer/java/ext.go b/analyzer/java/ext.go
index 9a82e3879395a3bbe2df4bfa7dd870b1a118ffd6..0782760344ab46ad2146adcbc75dd6bfb2fce8b9 100644
--- a/analyzer/java/ext.go
+++ b/analyzer/java/ext.go
@@ -39,9 +39,16 @@ func MvnDepTree(path string, root *model.DepTree) {
out = bytes.ReplaceAll(out, []byte("\r"), []byte("\n"))
// 获取mvn解析内容
lines := strings.Split(string(out), "\n")
- for i := range lines {
- lines[i] = strings.TrimPrefix(lines[i], "[INFO] ")
+ i := 0
+ for _, v := range lines {
+ tags := strings.Split(v, ":")
+ // 去除scope为test的依赖
+ if len(tags) < 5 || tags[4] != "test" {
+ lines[i] = strings.TrimPrefix(v, "[INFO] ")
+ i++
+ }
}
+ lines = lines[:i]
// 捕获依赖树起始位置
title := regexp.MustCompile(`--- [^\n]+ ---`)
// 记录依赖树起始位置行号
@@ -61,6 +68,10 @@ func MvnDepTree(path string, root *model.DepTree) {
buildMvnDepTree(root, lines[start+1:i])
for _, c := range root.Children {
c.Direct = true
+ // 第二层才是直接依赖？因为顶层为空
+ for _, cc := range c.Children {
+ cc.Direct = true
+ }
}
continue
}
diff --git a/analyzer/java/mvn.go b/analyzer/java/mvn.go
index b87af0dcf05754cdb9e61c7fa89c211540885834..ede3c8238500e79d7436bbdc22d10cd5e4470689 100644
--- a/analyzer/java/mvn.go
+++ b/analyzer/java/mvn.go
@@ -200,10 +200,10 @@ func (m *Mvn) MvnSimulation() []*Pom {
}
for i := range p.Dependencies {
dp := &p.Dependencies[i]
- if dp.Scope == "provided" {
+ if dp.Scope == "provided" || dp.Scope == "test" {
continue
}
- if (dp.Scope == "test" || dp.Optional == "true") && p.Deep() > 0 {
+ if dp.Optional == "true" && p.Deep() > 0 {
continue
}
p.Complete(dp)
diff --git a/analyzer/python/analyzer.go b/analyzer/python/analyzer.go
index 807dacbb3331dc1d3ce5c2d235b7a27e018a349a..5a2d6a0f431fc9db2ded522aa40a000716b4a917 100644
--- a/analyzer/python/analyzer.go
+++ b/analyzer/python/analyzer.go
@@ -24,7 +24,8 @@ func (Analyzer) CheckFile(filename string) bool {
filter.PythonPipfile(filename) ||
filter.PythonPipfileLock(filename) ||
filter.PythonRequirementsTxt(filename) ||
- filter.PythonRequirementsIn(filename)
+ filter.PythonRequirementsIn(filename) ||
+ filter.PythonPyproject(filename)
}
// ParseFiles parse dependency from file
@@ -34,6 +35,7 @@ func (Analyzer) ParseFiles(files []*model.FileInfo) []*model.DepTree {
dep := model.NewDepTree(nil)
dep.Path = f.Name
if filter.PythonSetup(f.Name) {
+ //parseSetupTmp(dep, f)
parseSetup(dep, f)
} else if filter.PythonPipfile(f.Name) {
parsePipfile(dep, f)
@@ -41,6 +43,9 @@ func (Analyzer) ParseFiles(files []*model.FileInfo) []*model.DepTree {
parsePipfileLock(dep, f)
} else if filter.PythonRequirementsTxt(f.Name) || filter.PythonRequirementsIn(f.Name) {
parseRequirementsin(dep, f)
+ } else if filter.PythonPyproject(f.Name) {
+ //暂未支持该文件的依赖解析
+ //parsePythonPyproject(dep, f)
}
deps = append(deps, dep)
}
diff --git a/analyzer/python/oss.py b/analyzer/python/oss.py
index 4179e6681c8db2fcc7c87363b758658ac73f98ed..d7f28cf2b64ce55171a4afeda805f104547fcc50 100644
--- a/analyzer/python/oss.py
+++ b/analyzer/python/oss.py
@@ -1,3 +1,4 @@
+#coding:utf-8
import re
import sys
import json
diff --git a/analyzer/python/req.go b/analyzer/python/req.go
index f87e2eb125eb460cfbbc4421f5bbcba98c9a63b7..9ff6a1faf134801ae599e2b087d4190d590760a3 100644
--- a/analyzer/python/req.go
+++ b/analyzer/python/req.go
@@ -21,7 +21,7 @@ var replacer *strings.Replacer
func init() {
reg1 = regexp.MustCompile(`^\w`)
regGit = regexp.MustCompile(`\/([\w-]+)\.git`)
- replacer = strings.NewReplacer("# via","","\r",""," ","","#","")
+ replacer = strings.NewReplacer("# via", "", "\r", "", " ", "", "#", "")
}
func parseRequirementsin(root *model.DepTree, file *model.FileInfo) {
@@ -32,7 +32,7 @@ func parseRequirementsin(root *model.DepTree, file *model.FileInfo) {
strArry := []string{}
temp.DoInTempDir(func(tempdir string) {
// 安装piptools
- if _, err := ex.Do(ex.PipinstallPiptoos, tempdir); err != nil {
+ if _, err := ex.Do(ex.PipinstallPiptools, tempdir); err != nil {
logs.Error(err)
return
}
@@ -89,7 +89,7 @@ func parseOutData(root *model.DepTree, strs []string) {
depMap[cur.Name] = cur
nodes = append(nodes, cur.Name)
for _, name := range nodes {
- if _,ok := depMap[name]; !ok {
+ if _, ok := depMap[name]; !ok {
continue
}
parNames := parentsMap[name]
@@ -99,13 +99,13 @@ func parseOutData(root *model.DepTree, strs []string) {
directMap[dep.Name] = dep
}
}
- if _,ok := depMap[parName]; !ok {
+ if _, ok := depMap[parName]; !ok {
continue
}
parent := depMap[parName]
dep := depMap[name]
- if m,ok := childMap[dep]; ok {
- if _,ok := m[dep.Name];ok {
+ if m, ok := childMap[dep]; ok {
+ if _, ok := m[dep.Name]; ok {
continue
}
m[dep.Name] = struct{}{}
@@ -115,11 +115,11 @@ func parseOutData(root *model.DepTree, strs []string) {
}
}
}
- withRoot(root,directMap)
+ withRoot(root, directMap)
}
// 所有直接依赖连接至root
-func withRoot(root *model.DepTree,directMap map[string]*model.DepTree) {
+func withRoot(root *model.DepTree, directMap map[string]*model.DepTree) {
direct := []*model.DepTree{}
for _, n := range directMap {
direct = append(direct, n)
@@ -181,4 +181,4 @@ func getSingleModStr(reqpath string, elem string) string {
} else {
return str
}
-}
\ No newline at end of file
+}
diff --git a/analyzer/python/setup.go b/analyzer/python/setup.go
index e48dc73f5d921a48a06ad3413dd597b7d17abf71..d83955b616005766074009b6ce5a4c9f85bb9031 100644
--- a/analyzer/python/setup.go
+++ b/analyzer/python/setup.go
@@ -6,7 +6,9 @@ import (
"os"
"os/exec"
"path"
+ "path/filepath"
"strings"
+ "util/client"
"util/logs"
"util/model"
"util/temp"
@@ -25,8 +27,8 @@ type setupDep struct {
Requires []string `json:"requires"`
}
-// parseSetup 解析 setup.py 文件
-func parseSetup(root *model.DepTree, file *model.FileInfo) {
+// parseSetupTmp 在临时目录中解析setup.py文件 有的项目不止需要setup.py
+func parseSetupTmp(root *model.DepTree, file *model.FileInfo) {
temp.DoInTempDir(func(tempdir string) {
ossfile := path.Join(tempdir, "oss.py")
setupfile := path.Join(tempdir, "setup.py")
@@ -41,8 +43,11 @@ func parseSetup(root *model.DepTree, file *model.FileInfo) {
return
}
// 解析 setup.py
+ pwd := temp.GetPwd()
+ os.Chdir(tempdir)
cmd := exec.Command("python", ossfile, setupfile)
out, _ := cmd.CombinedOutput()
+ os.Chdir(pwd)
startTag, endTag := `oss_start<<`, `>>oss_end`
startIndex, endIndex := strings.Index(string(out), startTag), strings.Index(string(out), endTag)
if startIndex == -1 || endIndex == -1 {
@@ -72,3 +77,65 @@ func parseSetup(root *model.DepTree, file *model.FileInfo) {
}
})
}
+
+// parseSetup 解析setup.py文件
+func parseSetup(root *model.DepTree, file *model.FileInfo) {
+ setupfile, err := filepath.Abs(file.Name)
+ if err != nil {
+ logs.Warn(err)
+ return
+ }
+ setupfile = strings.ReplaceAll(setupfile, `\`, `/`)
+ workdir := path.Dir(setupfile)
+ //名称尽量特殊点，避免原项目存在同名文件
+ ossfile := path.Join(workdir, "oss-xm1rr0r.py")
+ // 创建 oss.py
+ if err := os.WriteFile(ossfile, ossPy, 0444); err != nil {
+ logs.Warn(err)
+ return
+ }
+ defer os.Remove(ossfile)
+ // 解析 setup.py
+ pwd := temp.GetPwd()
+ os.Chdir(workdir)
+ cmd := exec.Command("python", ossfile, setupfile)
+ out, _ := cmd.CombinedOutput()
+ os.Chdir(pwd)
+ startTag, endTag := `oss_start<<`, `>>oss_end`
+ startIndex, endIndex := strings.Index(string(out), startTag), strings.Index(string(out), endTag)
+ if startIndex == -1 || endIndex == -1 {
+ logs.Debug(string(out))
+ return
+ } else {
+ out = out[startIndex+len(startTag) : endIndex]
+ }
+ // 获取解析结果
+ var dep setupDep
+ if err := json.Unmarshal(out, &dep); err != nil {
+ logs.Warn(err)
+ }
+ // 运行时提取的包名和版本号，只提取根目录的
+ if strings.HasSuffix(file.Name, client.PackageBasePath+"/"+path.Base(file.Name)) {
+ if dep.Name != "" {
+ client.PackageName = dep.Name
+ }
+ if dep.Version != "" {
+ client.PackageVersion = dep.Version
+ }
+ }
+ root.Name = dep.Name
+ root.Version = model.NewVersion(dep.Version)
+ root.Licenses = append(root.Licenses, dep.License)
+ for _, pkg := range [][]string{dep.Packages, dep.InstallRequires, dep.Requires} {
+ for _, p := range pkg {
+ index := strings.IndexAny(p, "~=<>")
+ sub := model.NewDepTree(root)
+ if index > -1 {
+ sub.Name = p[:index]
+ sub.Version = model.NewVersion(p[index:])
+ } else {
+ sub.Name = p
+ }
+ }
+ }
+}
diff --git a/analyzer/ruby/analyzer.go b/analyzer/ruby/analyzer.go
index 6343f8032a4e5e054f1d8b31f9c6c691a809a965..8b0e61b9cecaf13062fe7c8036c7b6df8e839581 100644
--- a/analyzer/ruby/analyzer.go
+++ b/analyzer/ruby/analyzer.go
@@ -25,7 +25,7 @@ func (a Analyzer) GetLanguage() language.Type {
// CheckFile Check if it is a parsable file
func (a Analyzer) CheckFile(filename string) bool {
- return filter.RubyGemfileLock(filename)
+ return filter.RubyGemfileLock(filename) || filter.RubyGemfile(filename)
}
// ParseFiles Parse the file
@@ -36,6 +36,8 @@ func (a Analyzer) ParseFiles(files []*model.FileInfo) (deps []*model.DepTree) {
dep.Path = f.Name
if filter.RubyGemfileLock(f.Name) {
parseGemfileLock(dep, f)
+ } else if filter.RubyGemfile(f.Name) {
+ //暂不支持
}
deps = append(deps, dep)
}
diff --git a/analyzer/rust/analyzer.go b/analyzer/rust/analyzer.go
index 1fb3bb1964af65fb7bd3b2246f3404529d30e9b8..c794c5cf2da348a613cfcd7dd7c1296739144555 100644
--- a/analyzer/rust/analyzer.go
+++ b/analyzer/rust/analyzer.go
@@ -19,7 +19,7 @@ func (a Analyzer) GetLanguage() language.Type {
// CheckFile Check if it is a parsable file
func (a Analyzer) CheckFile(filename string) bool {
- return filter.RustCargoLock(filename)
+ return filter.RustCargoLock(filename) || filter.RustCargoToml(filename)
}
// ParseFiles Parse the file
diff --git a/cli/main.go b/cli/main.go
index ace61e3869f46dbc54ef66bee7284f4c80a0c0a8..5a9d6e764e7bf1a35ec2ebceeb4d49cb224c1905 100644
--- a/cli/main.go
+++ b/cli/main.go
@@ -16,11 +16,13 @@ import (
"util/report"
)
-var version string
+const VERSION = "v2.0.0"
func main() {
args.Parse()
- if len(args.Config.Path) > 0 {
+ if args.ShowVersion {
+ fmt.Println(VERSION)
+ } else if len(args.Config.Path) > 0 {
output(engine.NewEngine().ParseFile(args.Config.Path))
} else {
flag.PrintDefaults()
@@ -29,7 +31,7 @@ func main() {
// output 输出结果
func output(depRoot *model.DepTree, taskInfo report.TaskInfo) {
- taskInfo.ToolVersion = version
+ taskInfo.ToolVersion = VERSION
// 记录依赖
logs.Debug("\n" + depRoot.String())
// 输出结果
diff --git a/util/args/args.go b/util/args/args.go
index 1348a64acada629b4fd2bd76114f7d3db1cc0bc1..bd5eab64dd56a341b45501b7f85c877ab8e93837 100644
--- a/util/args/args.go
+++ b/util/args/args.go
@@ -16,8 +16,10 @@ import (
)
var (
- ConfigPath string
- Config = struct {
+ ConfigPath string
+ ShowVersion bool
+ PluginName string
+ Config = struct {
// detect option
Path string `json:"path"`
Out string `json:"out"`
@@ -28,6 +30,8 @@ var (
// remote vuldb
Url string `json:"url"`
Token string `json:"token"`
+ // remote v2
+ V2 bool `json:"v2"`
// local vuldb
VulnDB string `json:"db"`
// prvate repository
@@ -46,12 +50,15 @@ func init() {
flag.StringVar(&Config.Path, "path", Config.Path, "(必须) 指定要检测的文件或目录路径,例: -path ./foo 或 -path ./foo.zip")
flag.StringVar(&Config.Url, "url", Config.Url, "(可选,与token需一起使用) 从云漏洞库查询漏洞,指定要连接云服务的地址,例:-url https://opensca.xmirror.cn")
flag.StringVar(&Config.Token, "token", Config.Token, "(可选,与url需一起使用) 云服务验证token,需要在云服务平台申请")
+ flag.BoolVar(&Config.V2, "v2", Config.V2, "(可选,与url、token需一起使用) 是否使用新的v2版本的云服务接口")
flag.BoolVar(&Config.Cache, "cache", Config.Cache, "(可选,建议开启) 缓存下载的文件(例如pom文件),重复检测相同组件时会节省时间,下载的文件会保存到工具所在目录的.cache目录下")
flag.BoolVar(&Config.OnlyVuln, "vuln", Config.OnlyVuln, "(可选) 结果仅保留有漏洞信息的组件,使用该参数不会保留组件层级结构")
flag.StringVar(&Config.Out, "out", Config.Out, "(可选) 将检测结果保存到指定文件,根据后缀生成不同格式的文件,默认为json格式,例: -out output.json")
flag.StringVar(&Config.VulnDB, "db", Config.VulnDB, "(可选) 指定本地漏洞库文件,希望使用自己漏洞库时可用,漏洞库文件为json格式,具体格式会在开源项目文档中给出;若同时使用云端漏洞库与本地漏洞库,漏洞查询结果取并集,例: -db db.json")
flag.BoolVar(&Config.Bar, "progress", Config.Bar, "(可选) 显示进度条")
flag.BoolVar(&Config.Dedup, "dedup", Config.Dedup, "(可选) 相同组件去重")
+ flag.BoolVar(&ShowVersion, "version", false, "显示客户端版本")
+ flag.StringVar(&PluginName, "plugin", "", "内部使用，用于控制插件类型")
}
func Parse() {
diff --git a/util/client/client.go b/util/client/client.go
index c5a766078cb05bb62a7022cbcd127ca54d72d3c3..efb07047d5fe90f82ed898589331609f775038d6 100644
--- a/util/client/client.go
+++ b/util/client/client.go
@@ -18,14 +18,27 @@ import (
"os"
"path"
"regexp"
+ "strings"
"util/args"
"util/logs"
+ "util/model"
"util/temp"
"github.com/pkg/errors"
)
-// 消息响应格式
+const CliType = 10 //新的saas端定义的类型，来自cli的type固定为10
+var PluginType = map[string]int{
+ "idea": 3, //新的saas端定义的类型，来自idea的type固定为3
+}
+var (
+ PackageBasePath string
+ PackageName string
+ PackageVersion string
+ PackageHash string
+)
+
+// SaasReponse 消息响应格式
type SaasReponse struct {
// 错误消息
Message string `json:"message"`
@@ -35,7 +48,7 @@ type SaasReponse struct {
Data interface{} `json:"data"`
}
-// 检测结果响应格式
+// DetectReponse 检测结果响应格式
type DetectReponse struct {
// 加密后的消息
Message string `json:"aesMessage"`
@@ -43,7 +56,7 @@ type DetectReponse struct {
Nonce string `json:"aesNonce"`
}
-// 检测任务请求格式
+// DetectRequst 检测任务请求格式
type DetectRequst struct {
// 16位byte base64编码
Tag string `json:"aesTag"`
@@ -57,6 +70,22 @@ type DetectRequst struct {
ClientId string `json:"clientId"`
}
+// DetectRequstV2 检测任务请求格式 v2
+type DetectRequstV2 struct {
+ // 16位 大写字母
+ //ClientId string `json:"clientId"`
+ // 在saas注册
+ Token string `json:"token"`
+ ComponentInfoAddDTOList []*model.CompTree `json:"componentInfoAddDTOList"`
+ Type int `json:"type"`
+ PackageName string `json:"packageName"`
+ PackageVersion string `json:"packageVersion"`
+ PackageHash string `json:"packageHash"`
+ // 是否需要saas端进行依赖检测，只传直接依赖时才检测，DeepLimit是依赖层数限制
+ //Check bool `json:"check"`
+ //DeepLimit int `json:"deepLimit"`
+}
+
// GetClientId 获取客户端id
func GetClientId() string {
// 默认id
@@ -163,6 +192,72 @@ func Detect(reqbody []byte) (repbody []byte, err error) {
}
}
+// DetectV2 发送任务解析请求 v2版本
+func DetectV2(root *model.DepTree) (repbody []byte, err error) {
+ repbody = []byte{}
+ // 转为新版本的组件格式
+ dep := root.ToDetectComponents()
+ if len(dep.Children) == 0 {
+ logs.Debug("依赖节点为空")
+ }
+ // 构建请求
+ url := args.Config.Url + "/oss-saas/api-v1/component/task/cli/add"
+ // 添加参数
+ param := DetectRequstV2{}
+ if _, ok := PluginType[args.PluginName]; ok {
+ param.Type = PluginType[args.PluginName]
+ } else {
+ param.Type = CliType
+ }
+ //param.ClientId = GetClientId()
+ param.Token = args.Config.Token
+ param.PackageName = fmt.Sprintf("[%s]%s", PackageBasePath, PackageName)
+ param.PackageVersion = PackageVersion
+ param.PackageHash = PackageHash
+ param.ComponentInfoAddDTOList = dep.Children
+ data, err := json.Marshal(param)
+ if err != nil {
+ return repbody, err
+ }
+ // 发送数据
+ rep, err := http.Post(url, "application/json", bytes.NewReader(data))
+ if err != nil {
+ logs.Error(err)
+ return repbody, err
+ }
+ defer rep.Body.Close()
+ if rep.StatusCode == 200 {
+ repbody, err = ioutil.ReadAll(rep.Body)
+ if err != nil {
+ logs.Error(err)
+ return
+ } else {
+ // 解析响应
+ saasrep := SaasReponse{}
+ err = json.Unmarshal(repbody, &saasrep)
+ if err != nil {
+ logs.Error(err)
+ }
+ if saasrep.Code != 0 {
+ // 出现错误
+ logs.Warn(fmt.Sprintf("url:%s code:%d message: %s", url, saasrep.Code, saasrep.Message))
+ err = errors.New(saasrep.Message)
+ return
+ } else {
+ data, err = json.Marshal(saasrep.Data)
+ if err != nil {
+ logs.Error(err)
+ } else {
+ repbody = data
+ }
+ return
+ }
+ }
+ } else {
+ return repbody, fmt.Errorf("%s status code: %d", url, rep.StatusCode)
+ }
+}
+
// getAesKey 获取aes-key
func getAesKey() (key []byte, err error) {
u, err := url.Parse(args.Config.Url + "/oss-saas/api-v1/open-sca-client/aes-key")
@@ -174,20 +269,24 @@ func getAesKey() (key []byte, err error) {
param.Set("clientId", GetClientId())
param.Set("ossToken", args.Config.Token)
u.RawQuery = param.Encode()
+ // 日志里尽量避免记录token
+ logUrl := strings.ReplaceAll(u.String(), args.Config.Token, "[TOKEN]")
// 发送请求
rep, err := http.Get(u.String())
if err != nil {
+ err = fmt.Errorf("%s", strings.ReplaceAll(err.Error(), args.Config.Token, "[TOKEN]"))
logs.Error(err)
return
}
if rep.StatusCode != 200 {
- err = fmt.Errorf("url: %s,status code:%d", u.String(), rep.StatusCode)
+ err = fmt.Errorf("url: %s,status code:%d", logUrl, rep.StatusCode)
logs.Error(err)
return
} else {
defer rep.Body.Close()
data, err := ioutil.ReadAll(rep.Body)
if err != nil {
+ err = fmt.Errorf("%s", strings.ReplaceAll(err.Error(), args.Config.Token, "[TOKEN]"))
logs.Error(err)
return key, err
}
@@ -196,7 +295,7 @@ func getAesKey() (key []byte, err error) {
json.Unmarshal(data, &saasrep)
if saasrep.Code != 0 {
// 出现错误
- logs.Warn(fmt.Sprintf("url:%s code:%d message: %s", u.String(), saasrep.Code, saasrep.Message))
+ logs.Warn(fmt.Sprintf("url:%s code:%d message: %s", logUrl, saasrep.Code, saasrep.Message))
err = errors.New(saasrep.Message)
return key, err
} else {
diff --git a/util/ex/python.go b/util/ex/python.go
index 797a9b4dee9ce8f20273948e8b9f19c381ede3c3..81620d9b242abf412d391cdb5b85d8eec73b294b 100644
--- a/util/ex/python.go
+++ b/util/ex/python.go
@@ -9,12 +9,12 @@ import (
)
const (
- Python string = "python"
- PipinstallPiptoos string = "pipenv install pip-tools --skip-lock"
- PipCompilein string = "pipenv run pip-compile requirements.in"
- PipCompileCfg string = "pipenv run pip-compile setup.cfg -o temp.txt"
- PipcompileSetup string = "pipenv run pip-compile setup.py"
- RemoveVirtualCmd string = "pipenv --rm"
+ Python string = "python"
+ PipinstallPiptools string = "pipenv install pip-tools --skip-lock"
+ PipCompilein string = "pipenv run pip-compile requirements.in"
+ PipCompileCfg string = "pipenv run pip-compile setup.cfg -o temp.txt"
+ PipcompileSetup string = "pipenv run pip-compile setup.py"
+ RemoveVirtualCmd string = "pipenv --rm"
)
type CmdOpts struct {
diff --git a/util/filter/file.go b/util/filter/file.go
index 51608361d1ef758bb2267419ba9e2283f8d5171b..a956d46b8ab98633b745fea23437ed70ede2c855 100644
--- a/util/filter/file.go
+++ b/util/filter/file.go
@@ -64,6 +64,7 @@ var (
// ruby相关
var (
+ RubyGemfile = filterFunc(strings.HasSuffix, "Gemfile")
RubyGemfileLock = filterFunc(strings.HasSuffix, "Gemfile.lock", "gems.locked")
)
@@ -76,6 +77,7 @@ var (
// rust
var (
RustCargoLock = filterFunc(strings.HasSuffix, "Cargo.lock")
+ RustCargoToml = filterFunc(strings.HasSuffix, "Cargo.toml")
)
// erlang
@@ -92,11 +94,14 @@ var (
// python
var (
PythonSetup = filterFunc(strings.HasSuffix, "setup.py")
+ PythonPyproject = filterFunc(strings.HasSuffix, "pyproject.toml")
PythonPipfile = filterFunc(strings.HasSuffix, "Pipfile")
PythonPipfileLock = filterFunc(strings.HasSuffix, "Pipfile.lock")
PythonRequirementsTxt = func(filename string) bool {
return filterFunc(strings.HasSuffix, ".txt")(filename) &&
- filterFunc(strings.Contains, "requirements")(path.Base(filename)) && !filterFunc(strings.Contains, "test")(path.Base(filename))
+ filterFunc(strings.Contains, "requirements")(path.Base(filename)) &&
+ !filterFunc(strings.Contains, "test")(path.Base(filename)) &&
+ !filterFunc(strings.Contains, "dev")(path.Base(filename))
}
PythonRequirementsIn = filterFunc(strings.HasSuffix, "requirements.in")
// PythonSetupCfg = filterFunc(strings.HasSuffix, "setup.cfg")
diff --git a/util/model/component.go b/util/model/component.go
new file mode 100644
index 0000000000000000000000000000000000000000..72001e90f744a4d0cbf29fa32978d825a339b26c
--- /dev/null
+++ b/util/model/component.go
@@ -0,0 +1,99 @@
+/*
+ * @Descripation: 依赖相关数据结构
+ * @Date: 2022-11-16 10:41:37
+ */
+
+package model
+
+import (
+ "fmt"
+ "util/enum/language"
+)
+
+// 参考dependency做了细微改动
+
+// component 组件依赖
+type component struct {
+ Vendor string `json:"vendor,omitempty"`
+ Name string `json:"name,omitempty"`
+ Version *Version `json:"ver,omitempty"`
+ Language language.Type `json:"lan,omitempty"`
+
+ // 仅在v2请求时赋值
+ LanguageStr string `json:"language"`
+ //ComponentId int64 `json:"componentId"`
+ ComponentAuthor string `json:"componentAuthor"`
+ ComponentName string `json:"componentName"`
+ ComponentVersion string `json:"componentVersion"`
+ FilePath string `json:"filePath"`
+}
+
+// NewComponent 创建Component
+func NewComponent() component {
+ dep := component{
+ Vendor: "",
+ Name: "",
+ Version: NewVersion(""),
+ Language: language.None,
+ }
+ return dep
+}
+
+// String 获取用于展示的Component字符串
+func (dep component) String() string {
+ if len(dep.Vendor) == 0 {
+ return fmt.Sprintf("[%s:%s]", dep.Name, dep.Version.Org)
+ } else {
+ return fmt.Sprintf("[%s:%s:%s]", dep.Vendor, dep.Name, dep.Version.Org)
+ }
+}
+
+// CompTree 依赖树
+type CompTree struct {
+ component
+ // 是否为直接依赖
+ Direct bool `json:"direct"`
+ // 依赖路径
+ Path string `json:"path,omitempty"`
+ Paths []string `json:"paths,omitempty"`
+ // 唯一的组件id，用来标识不同组件
+ ID int64 `json:"id,omitempty"`
+ // 父组件
+ Parent *CompTree `json:"-"`
+ Vulnerabilities []*Vuln `json:"vulnerabilities,omitempty"`
+ IndirectVulnerabilities int `json:"indirect_vulnerabilities,omitempty"`
+ // 许可证列表
+ Licenses []string `json:"licenses,omitempty"`
+ // spdx相关字段
+ CopyrightText string `json:"copyrightText,omitempty"`
+ // 子组件
+ Children []*CompTree `json:"children"`
+}
+
+// NewCompTree 创建CompTree
+func NewCompTree(parent *CompTree) *CompTree {
+ dep := &CompTree{
+ ID: getId(),
+ component: NewComponent(),
+ Path: "",
+ Paths: nil,
+ Parent: parent,
+ Children: []*CompTree{},
+ }
+ if parent != nil {
+ parent.Children = append(parent.Children, dep)
+ }
+ return dep
+}
+
+// Delete [移位法]删除子节点中指定索引的节点
+func (comp *CompTree) Delete(diMap map[int]int) {
+ j := 0
+ for i, v := range comp.Children {
+ if _, ok := diMap[i]; !ok {
+ comp.Children[j] = v
+ j++
+ }
+ }
+ comp.Children = comp.Children[:j]
+}
diff --git a/util/model/dependency.go b/util/model/dependency.go
index e80df434522622cf1b7e5dcf8f94646c9f139e93..ee209162ba3c0e1f483f3db7a0bb21998791e389 100644
--- a/util/model/dependency.go
+++ b/util/model/dependency.go
@@ -6,11 +6,14 @@
package model
import (
+ "encoding/json"
"fmt"
"strings"
"sync"
"time"
"util/enum/language"
+ "util/filter"
+ "util/logs"
)
// 用于id生成
@@ -75,10 +78,10 @@ type DepTree struct {
// 是否为直接依赖
Direct bool `json:"direct"`
// 依赖路径
- Path string `json:"-"`
+ Path string `json:"path,omitempty"`
Paths []string `json:"paths,omitempty"`
// 唯一的组件id，用来标识不同组件
- ID int64 `json:"-"`
+ ID int64 `json:"id,omitempty"`
// 父组件
Parent *DepTree `json:"-"`
Vulnerabilities []*Vuln `json:"vulnerabilities,omitempty"`
@@ -189,3 +192,115 @@ func (root *DepTree) String() string {
}
return res
}
+
+// ToDetectComponents 转为检测依赖组件的格式
+func (root *DepTree) ToDetectComponents() (comp *CompTree) {
+ // 利用json.Unmarshal来实现深拷贝
+ comp = NewCompTree(nil)
+ if data, err := json.Marshal(root); err != nil {
+ logs.Error(err)
+ } else {
+ err = json.Unmarshal(data, &comp)
+ if err != nil {
+ logs.Error(err)
+ } else {
+ // 参考format，改了下顺序，先去重再转换
+ // 恢复转换后缺失的父节点
+ for _, c := range comp.Children {
+ c.Parent = comp
+ }
+ // 去重
+ q := []*CompTree{comp}
+ dm := map[string]*CompTree{}
+ for len(q) > 0 {
+ n := q[0]
+ q = append(q[1:], n.Children...)
+ // 去重
+ k := fmt.Sprintf("%s:%s@%s#%s", n.Vendor, n.Name, n.Version.Org, strings.ToLower(n.Language.String()))
+ if d, ok := dm[k]; !ok {
+ dm[k] = n
+ } else {
+ // 已存在相同组件，但是某些字段可能不一样
+
+ // 当d.Path为空或者等于n.Path时才进行合并处理
+ if d.Path == "" {
+ if n.Path != "" {
+ d.Path = n.Path
+ }
+ if n.Direct {
+ d.Direct = n.Direct
+ }
+ } else if d.Path != n.Path {
+ continue
+ }
+ // 从父组件中移除当前组件
+ if n.Parent != nil {
+ for i, c := range n.Parent.Children {
+ if c.ID == n.ID {
+ n.Parent.Children = append(n.Parent.Children[:i], n.Parent.Children[i+1:]...)
+ break
+ }
+ }
+ }
+ // 将当前组件的子组件转移到已存在组件的子依赖中
+ d.Children = append(d.Children, n.Children...)
+ for _, c := range n.Children {
+ c.Parent = d
+ }
+ }
+ }
+
+ //应该只有pom的解析存在不需要一级节点这种情况。
+ //先将二级子节点复制到一级，并记录需要删除的索引
+ deleteIndex := make(map[int]int)
+ for i, c := range comp.Children {
+ if filter.JavaPom(strings.TrimSuffix(c.Path, "/"+c.String())) {
+ deleteIndex[i] = i
+ c.Parent.Children = append(c.Parent.Children, c.Children...)
+ }
+ }
+ //再删除对应的一级节点
+ if len(deleteIndex) > 0 {
+ comp.Delete(deleteIndex)
+ }
+
+ // 保留要导出的数据
+ q = []*CompTree{comp}
+ for len(q) > 0 {
+ n := q[0]
+ q = append(q[1:], n.Children...)
+ if n.Language != language.None {
+ n.LanguageStr = strings.ToLower(n.Language.String())
+ n.Language = language.None
+ }
+ if n.Version != nil {
+ n.ComponentVersion = n.Version.Org
+ n.Version = nil
+ }
+ if n.Vendor != "" {
+ n.ComponentAuthor = n.Vendor
+ n.Vendor = ""
+ }
+ if n.Name != "" {
+ n.ComponentName = n.Name
+ n.Name = ""
+ }
+ if n.Path != "" {
+ n.FilePath = n.Path
+ n.Path = ""
+ }
+ //n.ComponentId = n.ID
+ // 不展示的字段置空
+ //n.ID = 0
+ n.Paths = nil
+ n.Licenses = nil
+ n.CopyrightText = ""
+ n.Vulnerabilities = nil
+ n.IndirectVulnerabilities = 0
+ }
+
+ }
+ }
+
+ return comp
+}
diff --git a/util/report/format.go b/util/report/format.go
index b23d439bc302bfae6a161fe1ec5eb5c5b8a0de16..a8b7d56a0cd78ae1885b0368fe27dc3839001b04 100644
--- a/util/report/format.go
+++ b/util/report/format.go
@@ -23,42 +23,30 @@ type TaskInfo struct {
// format 按照输出内容格式化(不可逆)
func format(dep *model.DepTree) {
- q := []*model.DepTree{dep}
- // 保留要导出的数据
- for len(q) > 0 {
- n := q[0]
- q = append(q[1:], n.Children...)
- if n.Language != language.None {
- n.LanguageStr = n.Language.String()
- }
- if n.Version != nil {
- n.VersionStr = n.Version.Org
- }
- if n.Path != "" {
- n.Paths = []string{n.Path}
- }
- n.Language = language.None
- n.Version = nil
- }
// 去重
if args.Config.Dedup {
- q = []*model.DepTree{dep}
+ q := []*model.DepTree{dep}
dm := map[string]*model.DepTree{}
for len(q) > 0 {
n := q[0]
q = append(q[1:], n.Children...)
// 去重
- k := fmt.Sprintf("%s:%s@%s#%s", n.Vendor, n.Name, n.VersionStr, n.LanguageStr)
+ k := fmt.Sprintf("%s:%s@%s#%s", n.Vendor, n.Name, n.Version.Org, n.Language.String())
if d, ok := dm[k]; !ok {
dm[k] = n
} else {
+ // 已存在相同组件，但是某些字段可能不一样
+
// 临时解决部分组件homepage字段不显示问题
// 因为去重时刚好把解析到homepage字段的组件去掉了
// 其他字段可能也需要类似操作
if n.HomePage != "" {
d.HomePage = n.HomePage
}
- // 已存在相同组件
+ // 是否有直接依赖
+ if n.Direct {
+ d.Direct = n.Direct
+ }
d.Paths = append(d.Paths, n.Path)
// 从父组件中移除当前组件
if n.Parent != nil {
@@ -77,6 +65,29 @@ func format(dep *model.DepTree) {
}
}
}
+
+ // 保留要导出的数据
+ q := []*model.DepTree{dep}
+ for len(q) > 0 {
+ n := q[0]
+ q = append(q[1:], n.Children...)
+ if n.Language != language.None {
+ n.LanguageStr = n.Language.String()
+ }
+ if n.Version != nil {
+ n.VersionStr = n.Version.Org
+ }
+ if len(n.Paths) == 0 {
+ if n.Path != "" {
+ n.Paths = []string{n.Path}
+ }
+ }
+ //不展示的字段置空
+ n.Path = ""
+ n.Language = language.None
+ n.Version = nil
+ n.ID = 0
+ }
}
// Save 保存结果文件
diff --git a/util/report/html.go b/util/report/html.go
index 527e079adc272960ea1ed39902301fc9208d9d18..7d101273a4e1420b057c8caeba9f3128de306fb0 100644
--- a/util/report/html.go
+++ b/util/report/html.go
@@ -76,6 +76,6 @@ func Html(dep *model.DepTree, taskInfo TaskInfo) []byte {
return []byte{}
} else {
// 替换模板数据
- return bytes.Replace(index, []byte("N$}"), append(data, '}'), 1)
+ return bytes.Replace(index, []byte("DATA_REPLACE_HERE"), data, 1)
}
}
diff --git a/util/report/html_tpl b/util/report/html_tpl
index 90a4ce35d3cfe8c292db6f479056be6349538d8f..7b8033f24fc22188e42df1d998f55f42a79608a1 100644
--- a/util/report/html_tpl
+++ b/util/report/html_tpl
@@ -1,2 +1,2 @@
-