GuardDutyのFinding typesからThreatPurposeなどの各要素をいい感じに抽出する正規表現書いてみた

こんにちは、臼田です。

みなさん、GuardDutyと戯れてますか？(挨拶

今日はクラスメソッドの創業記念日です。というわけでいつも通りのGuardDutyの話をしていきますよ←

Finding typesから要素を抽出したい

まずは課題から。

GuardDutyで脅威を検出すると、Findingsが出力されます。この中のFinding typesには、その脅威の種類を示す情報が含まれています。例えばCryptoCurrency:EC2/BitcoinTool.B!DNSのようなものですね。

現在このFinding typesは164個のアクティブな種類があり、こちらのユーザーガイドにある形式で記述されています。ここから、先頭のCryptoCurrencyなどの一通りの要素を取り出したりしたい、というのが今回の要件です。

Finding typesの形式

まず先程のガイドにある形式を解説します。ざっと図にするとこんな感じです。

5つの要素がありますので簡単に説明します。

• ThreatPurpose: 攻撃や脅威の主な目的。MITRE ATT&CKと関連性がある。
• ResourceTypeAffected: 攻撃対象となるAWSリソース
• ThreatFamilyName: 脅威や悪意のあるアクティビティの説明
• DetectionMechanism: GuardDutyが脅威を検出した方法
• Artifact: 悪意のあるアクティビティで使用されたツールのリソース

図に書いてありますが、現状確認できる限り、前方3つは必須で、後半2つはオプションであるように見受けられます。(正確な仕様は不明)

このオプション部分の2つが曲者で、順番としてはかならずDetectionMechanism、Artifactの順ですが、Finding typesによって、DetectionMechanismはあるけどArtifactはない場合もあれば(Impact:S3/AnomalousBehavior.Permissionなど)、DetectionMechanismは無いけどArtifactはあるという場合もあり(Trojan:EC2/BlackholeTraffic!DNSなど)、この値を処理するときは気をつける必要があります。

正規表現での取得の仕方

以下のような正規表現を使うと、グループの配置を保ったまま、オプションの値も考慮しつつ適切に抽出ができます。

^(.+?):(.+?)/([^.!]+?)(?:|\.([^!]+?))(?:|!(.+))$

これはGoogle SpreadsheetとPythonで動作確認済みです。主にspreadsheetで活用するために作成しました。

どのようになっているかをざっくり説明した図がこちらです。(バックスラッシュが¥になっていますがご容赦を)

要素の数が可変になる抽出なので、どうしてもIFを使ってスクリプト側で処理したりしたくなりますが、Google Spreadsheetで使いたいところもあって一発で適切に要素がとれて、必要以上にグループが取られることをなくしました。

この正規表現のコツを説明します。

(?:re)という表現は非キャプチャグループで、通常()でくくるとグループとして扱われるところを、対象とせず処理してくれる仕組みです。パターンが色々あるため|(パイプ)を利用したいので、必然的に()が必要になりますが、そうすると無駄にグループが増えてしまいます。なので(?:|[値がある場合の正規表現])とすることでうまいこと値の有無どちらも扱いつつ、グループがずれないようにできます。

ただ私の思いつきベースで作ったものなので、もっといいやり方があったら教えてください。ちなみにGoogle Spreadsheetで利用できるのはRE2 正規表現でこのガイドで使えるもの使えないものなどの説明があります。

動かしてみた

Google Spreadsheetで動かす

Google Spreadsheetで抽出する場合にはREGEXEXTRACT関数を利用すれば、1列で5列分の要素を抽出できます。

指定方法は=REGEXEXTRACT(A2,"^(.+?):(.+?)/([^.!]+?)(?:|\.([^!]+?))(?:|!(.+))$")で実際にやるとこんな感じになります。これは1列だけ関数を入れています。

ついでに、分割したものを=TEXTJOIN("!",TRUE,TEXTJOIN(".",TRUE,E2&":"&F2&"/"&G2,H2),I2)で自分で結合して、正しいかどうかを=J2=A2としてチェックしています。TEXTJOIN関数を利用すると、オプションの値が空の時は結合しないのでいい感じに処理できます。現状ではチェックした結果ですべてTRUEが返ってくるので、この処理で問題なさそうです。

Pythonで動かす

とりあえず正規表現とその正確性を確認するスクリプトとして以下を作成しました。インタラクティブモードで確認します。

import re
gdre = r'^(.+?):(.+?)/([^.!]+?)(?:|\.([^!]+?))(?:|!(.+))$'
retest = re.compile(gdre)
findings = ["Backdoor:Lambda/C&CActivity.B","CryptoCurrency:Lambda/BitcoinTool.B","Trojan:Lambda/BlackholeTraffic","Trojan:Lambda/DropPoint","UnauthorizedAccess:Lambda/MaliciousIPCaller.Custom","UnauthorizedAccess:Lambda/TorClient","UnauthorizedAccess:Lambda/TorRelay","Backdoor:Runtime/C&CActivity.B","Backdoor:Runtime/C&CActivity.B!DNS","CryptoCurrency:Runtime/BitcoinTool.B","CryptoCurrency:Runtime/BitcoinTool.B!DNS","DefenseEvasion:Runtime/FilelessExecution","Execution:Runtime/NewBinaryExecuted","Execution:Runtime/NewLibraryLoaded","Execution:Runtime/ReverseShell","Impact:Runtime/AbusedDomainRequest.Reputation","Impact:Runtime/BitcoinDomainRequest.Reputation","Impact:Runtime/CryptoMinerExecuted","Impact:Runtime/MaliciousDomainRequest.Reputation","Impact:Runtime/SuspiciousDomainRequest.Reputation","PrivilegeEscalation:Runtime/CGroupsReleaseAgentModified","PrivilegeEscalation:Runtime/ContainerMountsHostDirectory","PrivilegeEscalation:Runtime/DockerSocketAccessed","PrivilegeEscalation:Runtime/RuncContainerEscape","PrivilegeEscalation:Runtime/UserfaultfdUsage","Trojan:Runtime/BlackholeTraffic","Trojan:Runtime/BlackholeTraffic!DNS","Trojan:Runtime/DGADomainRequest.C!DNS","Trojan:Runtime/DriveBySourceTraffic!DNS","Trojan:Runtime/DropPoint","Trojan:Runtime/DropPoint!DNS","Trojan:Runtime/PhishingDomainRequest!DNS","UnauthorizedAccess:Runtime/MetadataDNSRebind","UnauthorizedAccess:Runtime/TorClient","UnauthorizedAccess:Runtime/TorRelay","CredentialAccess:RDS/AnomalousBehavior.FailedLogin","CredentialAccess:RDS/AnomalousBehavior.SuccessfulBruteForce","CredentialAccess:RDS/AnomalousBehavior.SuccessfulLogin","CredentialAccess:RDS/MaliciousIPCaller.FailedLogin","CredentialAccess:RDS/MaliciousIPCaller.SuccessfulLogin","CredentialAccess:RDS/TorIPCaller.FailedLogin","CredentialAccess:RDS/TorIPCaller.SuccessfulLogin","Discovery:RDS/MaliciousIPCaller","Discovery:RDS/TorIPCaller","Backdoor:EC2/C&CActivity.B","Backdoor:EC2/C&CActivity.B!DNS","Backdoor:EC2/DenialOfService.Dns","Backdoor:EC2/DenialOfService.Tcp","Backdoor:EC2/DenialOfService.Udp","Backdoor:EC2/DenialOfService.UdpOnTcpPorts","Backdoor:EC2/DenialOfService.UnusualProtocol","Backdoor:EC2/Spambot","Behavior:EC2/NetworkPortUnusual","Behavior:EC2/TrafficVolumeUnusual","CryptoCurrency:EC2/BitcoinTool.B","CryptoCurrency:EC2/BitcoinTool.B!DNS","DefenseEvasion:EC2/UnusualDNSResolver","DefenseEvasion:EC2/UnusualDoHActivity","DefenseEvasion:EC2/UnusualDoTActivity","Impact:EC2/AbusedDomainRequest.Reputation","Impact:EC2/BitcoinDomainRequest.Reputation","Impact:EC2/MaliciousDomainRequest.Reputation","Impact:EC2/PortSweep","Impact:EC2/SuspiciousDomainRequest.Reputation","Impact:EC2/WinRMBruteForce","Recon:EC2/PortProbeEMRUnprotectedPort","Recon:EC2/PortProbeUnprotectedPort","Recon:EC2/Portscan","Trojan:EC2/BlackholeTraffic","Trojan:EC2/BlackholeTraffic!DNS","Trojan:EC2/DGADomainRequest.B","Trojan:EC2/DGADomainRequest.C!DNS","Trojan:EC2/DNSDataExfiltration","Trojan:EC2/DriveBySourceTraffic!DNS","Trojan:EC2/DropPoint","Trojan:EC2/DropPoint!DNS","Trojan:EC2/PhishingDomainRequest!DNS","UnauthorizedAccess:EC2/MaliciousIPCaller.Custom","UnauthorizedAccess:EC2/MetadataDNSRebind","UnauthorizedAccess:EC2/RDPBruteForce","UnauthorizedAccess:EC2/SSHBruteForce","UnauthorizedAccess:EC2/TorClient","UnauthorizedAccess:EC2/TorRelay","CredentialAccess:IAMUser/AnomalousBehavior","DefenseEvasion:IAMUser/AnomalousBehavior","Discovery:IAMUser/AnomalousBehavior","Exfiltration:IAMUser/AnomalousBehavior","Impact:IAMUser/AnomalousBehavior","InitialAccess:IAMUser/AnomalousBehavior","PenTest:IAMUser/KaliLinux","PenTest:IAMUser/ParrotLinux","PenTest:IAMUser/PentooLinux","Persistence:IAMUser/AnomalousBehavior","Policy:IAMUser/RootCredentialUsage","PrivilegeEscalation:IAMUser/AnomalousBehavior","Recon:IAMUser/MaliciousIPCaller","Recon:IAMUser/MaliciousIPCaller.Custom","Recon:IAMUser/TorIPCaller","Stealth:IAMUser/CloudTrailLoggingDisabled","Stealth:IAMUser/PasswordPolicyChange","UnauthorizedAccess:IAMUser/ConsoleLoginSuccess.B","UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS","UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS","UnauthorizedAccess:IAMUser/MaliciousIPCaller","UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom","UnauthorizedAccess:IAMUser/TorIPCaller","Discovery:S3/AnomalousBehavior","Discovery:S3/MaliciousIPCaller","Discovery:S3/MaliciousIPCaller.Custom","Discovery:S3/TorIPCaller","Exfiltration:S3/AnomalousBehavior","Exfiltration:S3/MaliciousIPCaller","Impact:S3/AnomalousBehavior.Delete","Impact:S3/AnomalousBehavior.Permission","Impact:S3/AnomalousBehavior.Write","Impact:S3/MaliciousIPCaller","PenTest:S3/KaliLinux","PenTest:S3/ParrotLinux","PenTest:S3/PentooLinux","Policy:S3/AccountBlockPublicAccessDisabled","Policy:S3/BucketAnonymousAccessGranted","Policy:S3/BucketBlockPublicAccessDisabled","Policy:S3/BucketPublicAccessGranted","Stealth:S3/ServerAccessLoggingDisabled","UnauthorizedAccess:S3/MaliciousIPCaller.Custom","UnauthorizedAccess:S3/TorIPCaller","CredentialAccess:Kubernetes/MaliciousIPCaller","CredentialAccess:Kubernetes/MaliciousIPCaller.Custom","CredentialAccess:Kubernetes/SuccessfulAnonymousAccess","CredentialAccess:Kubernetes/TorIPCaller","DefenseEvasion:Kubernetes/MaliciousIPCaller","DefenseEvasion:Kubernetes/MaliciousIPCaller.Custom","DefenseEvasion:Kubernetes/SuccessfulAnonymousAccess","DefenseEvasion:Kubernetes/TorIPCaller","Discovery:Kubernetes/MaliciousIPCaller","Discovery:Kubernetes/MaliciousIPCaller.Custom","Discovery:Kubernetes/SuccessfulAnonymousAccess","Discovery:Kubernetes/TorIPCaller","Execution:Kubernetes/ExecInKubeSystemPod","Impact:Kubernetes/MaliciousIPCaller","Impact:Kubernetes/MaliciousIPCaller.Custom","Impact:Kubernetes/SuccessfulAnonymousAccess","Impact:Kubernetes/TorIPCaller","Persistence:Kubernetes/ContainerWithSensitiveMount","Persistence:Kubernetes/MaliciousIPCaller","Persistence:Kubernetes/MaliciousIPCaller.Custom","Persistence:Kubernetes/SuccessfulAnonymousAccess","Persistence:Kubernetes/TorIPCaller","Policy:Kubernetes/AdminAccessToDefaultServiceAccount","Policy:Kubernetes/AnonymousAccessGranted","Policy:Kubernetes/ExposedDashboard","Policy:Kubernetes/KubeflowDashboardExposed","PrivilegeEscalation:Kubernetes/PrivilegedContainer","Execution:Container/MaliciousFile","Execution:Container/SuspiciousFile","Execution:EC2/MaliciousFile","Execution:EC2/SuspiciousFile","Execution:ECS/MaliciousFile","Execution:ECS/SuspiciousFile","Execution:Kubernetes/MaliciousFile","Execution:Kubernetes/SuspiciousFile","Execution:ECS/SuspiciousFile","Execution:Kubernetes/MaliciousFile","Execution:Kubernetes/SuspiciousFile"]
len(findings)
for f in findings:
    m = retest.match(f)
    join_finding = "!".join(filter(None,[".".join(filter(None,["{}:{}/{}".format(*m.group(1,2,3)),m.group(4)])),m.group(5)]))
    if f != join_finding: print(f, join_finding)

% python3
Python 3.9.6 (default, May  7 2023, 23:32:44) 
[Clang 14.0.3 (clang-1403.0.22.14.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import re
>>> gdre = r'^(.+?):(.+?)/([^.!]+?)(?:|\.([^!]+?))(?:|!(.+))$'
>>> retest = re.compile(gdre)
>>> findings = ["Backdoor:Lambda/C&CActivity.B","CryptoCurrency:Lambda/BitcoinTool.B","Trojan:Lambda/BlackholeTraffic","Trojan:Lambda/DropPoint","UnauthorizedAccess:Lambda/MaliciousIPCaller.Custom","UnauthorizedAccess:Lambda/TorClient","UnauthorizedAccess:Lambda/TorRelay","Backdoor:Runtime/C&CActivity.B","Backdoor:Runtime/C&CActivity.B!DNS","CryptoCurrency:Runtime/BitcoinTool.B","CryptoCurrency:Runtime/BitcoinTool.B!DNS","DefenseEvasion:Runtime/FilelessExecution","Execution:Runtime/NewBinaryExecuted","Execution:Runtime/NewLibraryLoaded","Execution:Runtime/ReverseShell","Impact:Runtime/AbusedDomainRequest.Reputation","Impact:Runtime/BitcoinDomainRequest.Reputation","Impact:Runtime/CryptoMinerExecuted","Impact:Runtime/MaliciousDomainRequest.Reputation","Impact:Runtime/SuspiciousDomainRequest.Reputation","PrivilegeEscalation:Runtime/CGroupsReleaseAgentModified","PrivilegeEscalation:Runtime/ContainerMountsHostDirectory","PrivilegeEscalation:Runtime/DockerSocketAccessed","PrivilegeEscalation:Runtime/RuncContainerEscape","PrivilegeEscalation:Runtime/UserfaultfdUsage","Trojan:Runtime/BlackholeTraffic","Trojan:Runtime/BlackholeTraffic!DNS","Trojan:Runtime/DGADomainRequest.C!DNS","Trojan:Runtime/DriveBySourceTraffic!DNS","Trojan:Runtime/DropPoint","Trojan:Runtime/DropPoint!DNS","Trojan:Runtime/PhishingDomainRequest!DNS","UnauthorizedAccess:Runtime/MetadataDNSRebind","UnauthorizedAccess:Runtime/TorClient","UnauthorizedAccess:Runtime/TorRelay","CredentialAccess:RDS/AnomalousBehavior.FailedLogin","CredentialAccess:RDS/AnomalousBehavior.SuccessfulBruteForce","CredentialAccess:RDS/AnomalousBehavior.SuccessfulLogin","CredentialAccess:RDS/MaliciousIPCaller.FailedLogin","CredentialAccess:RDS/MaliciousIPCaller.SuccessfulLogin","CredentialAccess:RDS/TorIPCaller.FailedLogin","CredentialAccess:RDS/TorIPCaller.SuccessfulLogin","Discovery:RDS/MaliciousIPCaller","Discovery:RDS/TorIPCaller","Backdoor:EC2/C&CActivity.B","Backdoor:EC2/C&CActivity.B!DNS","Backdoor:EC2/DenialOfService.Dns","Backdoor:EC2/DenialOfService.Tcp","Backdoor:EC2/DenialOfService.Udp","Backdoor:EC2/DenialOfService.UdpOnTcpPorts","Backdoor:EC2/DenialOfService.UnusualProtocol","Backdoor:EC2/Spambot","Behavior:EC2/NetworkPortUnusual","Behavior:EC2/TrafficVolumeUnusual","CryptoCurrency:EC2/BitcoinTool.B","CryptoCurrency:EC2/BitcoinTool.B!DNS","DefenseEvasion:EC2/UnusualDNSResolver","DefenseEvasion:EC2/UnusualDoHActivity","DefenseEvasion:EC2/UnusualDoTActivity","Impact:EC2/AbusedDomainRequest.Reputation","Impact:EC2/BitcoinDomainRequest.Reputation","Impact:EC2/MaliciousDomainRequest.Reputation","Impact:EC2/PortSweep","Impact:EC2/SuspiciousDomainRequest.Reputation","Impact:EC2/WinRMBruteForce","Recon:EC2/PortProbeEMRUnprotectedPort","Recon:EC2/PortProbeUnprotectedPort","Recon:EC2/Portscan","Trojan:EC2/BlackholeTraffic","Trojan:EC2/BlackholeTraffic!DNS","Trojan:EC2/DGADomainRequest.B","Trojan:EC2/DGADomainRequest.C!DNS","Trojan:EC2/DNSDataExfiltration","Trojan:EC2/DriveBySourceTraffic!DNS","Trojan:EC2/DropPoint","Trojan:EC2/DropPoint!DNS","Trojan:EC2/PhishingDomainRequest!DNS","UnauthorizedAccess:EC2/MaliciousIPCaller.Custom","UnauthorizedAccess:EC2/MetadataDNSRebind","UnauthorizedAccess:EC2/RDPBruteForce","UnauthorizedAccess:EC2/SSHBruteForce","UnauthorizedAccess:EC2/TorClient","UnauthorizedAccess:EC2/TorRelay","CredentialAccess:IAMUser/AnomalousBehavior","DefenseEvasion:IAMUser/AnomalousBehavior","Discovery:IAMUser/AnomalousBehavior","Exfiltration:IAMUser/AnomalousBehavior","Impact:IAMUser/AnomalousBehavior","InitialAccess:IAMUser/AnomalousBehavior","PenTest:IAMUser/KaliLinux","PenTest:IAMUser/ParrotLinux","PenTest:IAMUser/PentooLinux","Persistence:IAMUser/AnomalousBehavior","Policy:IAMUser/RootCredentialUsage","PrivilegeEscalation:IAMUser/AnomalousBehavior","Recon:IAMUser/MaliciousIPCaller","Recon:IAMUser/MaliciousIPCaller.Custom","Recon:IAMUser/TorIPCaller","Stealth:IAMUser/CloudTrailLoggingDisabled","Stealth:IAMUser/PasswordPolicyChange","UnauthorizedAccess:IAMUser/ConsoleLoginSuccess.B","UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.InsideAWS","UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS","UnauthorizedAccess:IAMUser/MaliciousIPCaller","UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom","UnauthorizedAccess:IAMUser/TorIPCaller","Discovery:S3/AnomalousBehavior","Discovery:S3/MaliciousIPCaller","Discovery:S3/MaliciousIPCaller.Custom","Discovery:S3/TorIPCaller","Exfiltration:S3/AnomalousBehavior","Exfiltration:S3/MaliciousIPCaller","Impact:S3/AnomalousBehavior.Delete","Impact:S3/AnomalousBehavior.Permission","Impact:S3/AnomalousBehavior.Write","Impact:S3/MaliciousIPCaller","PenTest:S3/KaliLinux","PenTest:S3/ParrotLinux","PenTest:S3/PentooLinux","Policy:S3/AccountBlockPublicAccessDisabled","Policy:S3/BucketAnonymousAccessGranted","Policy:S3/BucketBlockPublicAccessDisabled","Policy:S3/BucketPublicAccessGranted","Stealth:S3/ServerAccessLoggingDisabled","UnauthorizedAccess:S3/MaliciousIPCaller.Custom","UnauthorizedAccess:S3/TorIPCaller","CredentialAccess:Kubernetes/MaliciousIPCaller","CredentialAccess:Kubernetes/MaliciousIPCaller.Custom","CredentialAccess:Kubernetes/SuccessfulAnonymousAccess","CredentialAccess:Kubernetes/TorIPCaller","DefenseEvasion:Kubernetes/MaliciousIPCaller","DefenseEvasion:Kubernetes/MaliciousIPCaller.Custom","DefenseEvasion:Kubernetes/SuccessfulAnonymousAccess","DefenseEvasion:Kubernetes/TorIPCaller","Discovery:Kubernetes/MaliciousIPCaller","Discovery:Kubernetes/MaliciousIPCaller.Custom","Discovery:Kubernetes/SuccessfulAnonymousAccess","Discovery:Kubernetes/TorIPCaller","Execution:Kubernetes/ExecInKubeSystemPod","Impact:Kubernetes/MaliciousIPCaller","Impact:Kubernetes/MaliciousIPCaller.Custom","Impact:Kubernetes/SuccessfulAnonymousAccess","Impact:Kubernetes/TorIPCaller","Persistence:Kubernetes/ContainerWithSensitiveMount","Persistence:Kubernetes/MaliciousIPCaller","Persistence:Kubernetes/MaliciousIPCaller.Custom","Persistence:Kubernetes/SuccessfulAnonymousAccess","Persistence:Kubernetes/TorIPCaller","Policy:Kubernetes/AdminAccessToDefaultServiceAccount","Policy:Kubernetes/AnonymousAccessGranted","Policy:Kubernetes/ExposedDashboard","Policy:Kubernetes/KubeflowDashboardExposed","PrivilegeEscalation:Kubernetes/PrivilegedContainer","Execution:Container/MaliciousFile","Execution:Container/SuspiciousFile","Execution:EC2/MaliciousFile","Execution:EC2/SuspiciousFile","Execution:ECS/MaliciousFile","Execution:ECS/SuspiciousFile","Execution:Kubernetes/MaliciousFile","Execution:Kubernetes/SuspiciousFile","Execution:ECS/SuspiciousFile","Execution:Kubernetes/MaliciousFile","Execution:Kubernetes/SuspiciousFile"]
>>> len(findings)
164
>>> for f in findings:
...     m = retest.match(f)
...     join_finding = "!".join(filter(None,[".".join(filter(None,["{}:{}/{}".format(*m.group(1,2,3)),m.group(4)])),m.group(5)]))
...     if f != join_finding: print(f, join_finding)
... 
>>> 

というわけでちゃんと使えています。

まとめ

GuardDuty FindingsのFinding typesを正規表現でいい感じに要素を抽出してみました。

是非活用してください。