Added Elastic Renderer and necessary arguments. #563
Conversation
|
Why not ingest the json with filebeats instead? vol can do vol things and elastic can do elastic things.Just wondering about the process here of bloating volatility for a single platform as good a concept as this is.V/r,MutedmouseSent from my Verizon, Samsung Galaxy smartphone
-------- Original message --------From: maurermj08 <notifications@github.com> Date: 11/6/18 17:08 (GMT-05:00) To: volatilityfoundation/volatility <volatility@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: [volatilityfoundation/volatility] Added Elastic Renderer and necessary arguments. (#563) Added an Elastic renderer. This renderer uses the optional user specified Elastic Search URL (elastic-url) and index name (index). It requires Elastic Search python client, "pip install elasticsearch". I believe this code would be useful for others when threat hunting a large number of RAM images.
Example: "vol.py --profile WinXPSP3x86 -f memory.img --output=elastic --elastic-url="http://192.168.10.100:9200" --index myindex timeliner"
You can view, comment on, or merge this pull request online at:
#563
Commit Summary
Added Elastic Renderer and necessary arguments.
File Changes
M
volatility/commands.py
(17)
A
volatility/renderers/elastic.py
(70)
Patch Links:
https://github.com/volatilityfoundation/volatility/pull/563.patch
https://github.com/volatilityfoundation/volatility/pull/563.diff
—You are receiving this because you are subscribed to this thread.Reply to this email directly, view it on GitHub, or mute the thread.
{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/volatilityfoundation/volatility","title":"volatilityfoundation/volatility","subtitle":"GitHub repository","main_image_url":"https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/volatilityfoundation/volatility"}},"updates":{"snippets":[{"icon":"DESCRIPTION","message":"Added Elastic Renderer and necessary arguments. (#563)"}],"action":{"name":"View Pull Request","url":"#563"}}}
[
{
"@context": "http://schema.org",
"@type": "EmailMessage",
"potentialAction": {
"@type": "ViewAction",
"target": "#563",
"url": "#563",
"name": "View Pull Request"
},
"description": "View this Pull Request on GitHub",
"publisher": {
"@type": "Organization",
"name": "GitHub",
"url": "https://github.com"
}
},
{
"@type": "MessageCard",
"@context": "http://schema.org/extensions",
"hideOriginalBody": "false",
"originator": "AF6C5A86-E920-430C-9C59-A73278B5EFEB",
"title": "Added Elastic Renderer and necessary arguments. (#563)",
"sections": [
{
"text": "",
"activityTitle": "**maurermj08**",
"activityImage": "https://assets-cdn.github.com/images/email/message_cards/avatar.png",
"activitySubtitle": "@maurermj08",
"facts": [
]
},
{
"title": "Commit Summary",
"facts": [
{
"name": "f3a6f07",
"value": "Added Elastic Renderer and necessary arguments."
}
]
},
{
"title": "File Changes",
"facts": [
{
"name": "Modified",
"value": "[volatility/commands.py](https://github.com/volatilityfoundation/volatility/pull/563/files#diff-0) (17 changes)"
},
{
"name": "Added",
"value": "[volatility/renderers/elastic.py](https://github.com/volatilityfoundation/volatility/pull/563/files#diff-1) (70 changes)"
}
]
}
],
"potentialAction": [
{
"name": "Add a comment",
"@type": "ActionCard",
"inputs": [
{
"isMultiLine": true,
"@type": "TextInput",
"id": "IssueComment",
"isRequired": false
}
],
"actions": [
{
"name": "Comment",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"IssueComment\",\n\"repositoryFullName\": \"volatilityfoundation/volatility\",\n\"issueId\": 563,\n\"IssueComment\": \"{{IssueComment.value}}\"\n}"
}
]
},
{
"name": "Close pull request",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"PullRequestClose\",\n\"repositoryFullName\": \"volatilityfoundation/volatility\",\n\"pullRequestId\": 563\n}"
},
{
"targets": [
{
"os": "default",
"uri": "#563"
}
],
"@type": "OpenUri",
"name": "View on GitHub"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/volatilityfoundation/volatility/pull/563.patch"
}
],
"@type": "OpenUri",
"name": "View patch"
},
{
"targets": [
{
"os": "default",
"uri": "https://github.com/volatilityfoundation/volatility/pull/563.diff"
}
],
"@type": "OpenUri",
"name": "View diff"
},
{
"name": "Unsubscribe",
"@type": "HttpPOST",
"target": "https://api.github.com",
"body": "{\n\"commandName\": \"MuteNotification\",\n\"threadId\": 407206217\n}"
}
],
"themeColor": "26292E"
}
]
|
|
I understand not wanting to bloat volatility core. I believe this renderer would ideally be under the volatility community, but it is my understanding that is not an option at this time. I originally tried going straight from the JSON output to Elastic Search, but the column/row style of the JSON output does not match the format expected by Elastic Search. This renderer appears to be the simplest and fastest method of exporting to Elastic Search. There are multiple options including writing a custom script that takes in SQLite or JSON, Filebeats, or Logstash. However, these require configuration or writing the data to disk. I remember similar discussions being had about Log2timeline's output module. I originally created a logstash parser. However, others suggested using an output module, which I ultimately found more ideal. |
![https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open](https://assets-cdn.github.com/images/email/message_cards/header.png","avatar_image_url":"https://assets-cdn.github.com/images/email/message_cards/avatar.png","action":{"name":"Open){kind=link}
|
I really like the idea of adding this as a direct rendrer/output @maurermj08 ! I've earlier been thinking of doing such integration with Elastic myself. Although I might have chosen the easier way of inputting sqlite or json through logstash or filebeat. But I do agree with the arguments you have of doing the direct integration to the elastic-node. I tried to run some Volatility modules and output them using your script above. Although I got some errors from Elastic. I've tried it with ElasticSearch 5.5 and 6.5. What version of ElasticSearch have you used writing the rendrer? |
|
I did most of my testing on ElasticSearch/Kibana 6.4.2. It uses the official Elasticsearch Python package, https://pypi.org/project/elasticsearch/, so it is easy to switch the version. It should work out of the box on version 6.x. To use a different version of Elasticsearch simply switch the Elasticsearch package to match the appropriate version using PIP: |
Added an Elastic renderer. This renderer uses the optional user specified Elastic Search URL (elastic-url) and index name (index). It requires Elastic Search python client, "pip install elasticsearch". I believe this code would be useful for others when threat hunting a large number of RAM images.
Example: "vol.py --profile WinXPSP3x86 -f memory.img --output=elastic --elastic-url="http://192.168.10.100:9200" --index myindex timeliner"