Earlier this month, the Kusto team announced graph semantics feature in Kusto.
This Kusto extension makes it possible to contextualize data in graphs which consists of Nodes and Edges that can be connected. These connections can visualize the relationships between the Nodes.
To describe graph semantics in a none-tech scenario, the best way is to look at social connections where people have connections, one to many, many to many and many to one
let Users = datatable (UserId:string , name:string)[
"1","Mattias",
"2","XboxController",
"3","Stefan",
"4","XBOX"
]; // nodes
let Knows = datatable (FirstUser:string , SecondUser:string)[
"1","2",//Mattias knows xbox controller
"2","4",//Xbox controller knows xbox
"1","3"//Mattias knows Stefan
]; // edges
Knows
| make-graph FirstUser --> SecondUser with Users on UserId
| graph-match (user)-->(middle_man)-->(friendOfAFriend)
project SecLabs_person = user.name, middle_man = middle_man.name, kusto_friend_of_friend = friendOfAFriend.name
What can it use it for?
Many-to-many relations
Hierarchical data
Networked relationships
Social Networks
Recommendation systems
Connected assets
Knowledge graphs
From a hunting perspective, we can connect systems in a unspecified amounts of times. Since we can’t to recursive join, we can use graph to connect unknown number of systems.
Time-aware
In a scenario where we have lateral movement, we could connect all devices involved and at what time (of course depending on the data source, but if we have data from Microsoft Defender XDR data with network data from the endpoint sensor and know about activities involving certain ports, 3389 for example) we could draw all show how and when the threat actor moved laterally.
Establishing a Proactive Hunting program is something which is useful and necessary today.
From working with proactive threat hunting for a long time, from when data was not available at your fingertips, things has become a lot easier in the era of SIEM and over the last years, EDR and XDR.
The technical part is the easy one. The process of establishing your hunting and connect it with existing processes is usually what’s difficult.
What is proactive Hunting?
First, what is threat hunting?
To dive in to this topic I want to point out activities that are somewhat related
The custom detections or scheduled rules is pretty clear what it is. but Tasks is something which sits in between Proactive Threat Hunting and Custom Detections. Tasks are things discovered in Hunting, and data of interest but cannot be set as a custom detection as of yet.
The reasons could be many, but for example to noisy data and no good correlation values or near-time events that can be used to reduce False-Positives. Even though we have finalized our hypothesis from threat hunting we might want to follow-up on the events, maybe on a daily basis, but we cant have it scheduled since it will give an incident fatigue.
To summarize tasks, it’s queries that might make it to a detection but for now we run it automatically or manual and do manual review on the result on a daily schedule.
To iterate back to threat hunting before we take a look at the hunts feature in Sentinel
Threat Hunting can be divided into 2 main pillars; Proactive and Reactive hunting
Proactive Threat Hunting is when you don’t know something is going on, like playing hide-n-seek, except for that you don’t know anyone is playing with you
Reactive Threat Hunting is post breach/post incident and you use threat hunting to find the outer boundaries of the incident, which could be other devices communicating with a specific IP, list all process communicating, find the same processes on other devices, and then which IP’s they are communicating with.
In the EDR era or now, the XDR era, threat hunting becomes easier on a technical level. The data collection happens automatically for many workloads and not only collected, it’s streamed.
With the power of Kusto Query Language, you can do advanced aggregations, anomaly calculations and visualizations to truly crunch and bend your data.
In Proactive Threat hunting, you will start your assignment by defining your Hypothesis, which could be something like “I would like to see if any local users have been added outside the process”
Then you check what ever data you need to discover such activities (DeviceEvents/DeviceProcessEvents hint hint) and you continue to develop the query for is and document your result.
Here is what makes Hunts feature so great. It actually allows for process integration, like we have in Microsoft 365 Defender where we can create incidents based on the results and have it handled by the incident process.
Hunts in Sentinel
Common use cases:
Proactively hunt based on specific MITRE techniques, potentially malicious activity, recent threats, or your own custom hypothesis.
Use security-researcher-generated hunting queries or custom hunting queries to investigate malicious behavior.
Conduct your hunts using multiple persisted-query tabs that enable you to keep context over time.
Collect evidence, investigate UEBA sources, and annotate your findings using hunt specific bookmarks.
Collaborate and document your findings with comments.
Act on results by creating new analytic rules, new incidents, new threat indicators, and running playbooks.
Keep track of your new, active, and closed hunts in one place.
View metrics based on validated hypotheses and tangible results.
In Sentinel go to Hunting and Hunts
Here is the list of previous hunts.
If you select New Hunt we can create a new
You can add the Hypothesis and choose if it’s Validated or not (which can be set later in the process by another Threat hunter)
When we have our new Hunting campaign, we can add queries
Adding Tactics and techniques and map entities
It’s possible to create incidents from the results to map to the incident process, and we can also start automated playbooks (entity-based) from the entity pane.
Summary
There are so many details on this feature and it has some many capabilities. The best part is that it has the process of hunting as its core. Now it’s easier to deliver threat hunting, as a service, as a consultant or if you work internally for an organization.
It rarely happens, but if it happens, you have a solution…
One of the best response actions in Microsoft Defender for Endpoint (A part of Microsoft 365 Defender) is isolate device. This locks the device in the network stack and will connect any threat actor immediately from the device, or stop the user from doing what they are doing.
This action do have some extra great features. For instance, it will allow connection to the Defender backend which allow SecOps to continue to monitor, run live response (another action which gives SecOps shell access to the endpoint) to further analyze any suspicious behavior.
So what’s force release from isolation?
force release from Isolation is a batch script which will add some registry values to the endpoint to force it to release from isolation. This could be used if something happens on the network side where the endpoint is connected or if there is any other error that could break the release from isolation function from the portal.
Even though it’s very rarely necessary, it’s great to have such feature if something happens.
Downloading script
Go to device page and click on the more actions menu
Both Stefan and Mattias from the Sec-Labs team, have conducted numerous Defender for Endpoint health checks. Apart from other configuration errors, three settings are frequently misconfigured, affecting overall security.
When we discussed this, we realized we should probably write a full configuration recommendation. But let’s start with these 3 settings that are related to each other
Number one – Cloud Protection turned off or not correct cloud block protection
Cloud protection extends the threat detection capabilities in the antimalware engine and is used to enhance real-time protection with the strength of cloud. It’s a common observation in the health checks that this is turned off or not configured to block at the correct level.
Specialized ML models, which include file type-specific models, feature-specific models, and adversary-hardened monotonic models, analyze a featurized description of suspicious files sent by the client. Stacked ensemble classifiers combine results from these models to make a real-time verdict to allow or block files pre-execution.
Behavior-based ML engine
Suspicious behavior sequences and advanced attack techniques are monitored on the client as triggers to analyze the process tree behavior using real-time cloud ML models. Monitored attack techniques span the attack chain, from exploits, elevation, and persistence all the way through to lateral movement and exfiltration.
AMSI-paired ML engine
Pairs of client-side and cloud-side models perform advanced analysis of scripting behavior pre- and post-execution to catch advanced threats like fileless and in-memory attacks. These models include a pair of models for each of the scripting engines covered, including PowerShell, JavaScript, VBScript, and Office VBA macros. Integrations include both dynamic content calls and/or behavior instrumentation on the scripting engines.
File classification ML engine
Multi-class, deep neural network classifiers examine full file contents, provides an additional layer of defense against attacks that require additional analysis. Suspicious files are held from running and submitted to the cloud protection service for classification. Within seconds, full-content deep learning models produce a classification and reply to the client to allow or block the file.
Detonation-based ML engine
Suspicious files are detonated in a sandbox. Deep learning classifiers analyze the observed behaviors to block attacks.
Reputation ML engine
Domain-expert reputation sources and models from across Microsoft are queried to block threats that are linked to malicious or suspicious URLs, domains, emails, and files. Sources include Windows Defender SmartScreen for URL reputation models and Office 365 ATP for email attachment expert knowledge, among other Microsoft services through the Microsoft Intelligent Security Graph.
Smart rules engine
Expert-written smart rules identify threats based on researcher expertise and collective knowledge of threats.
Client-side
ML engine
A set of light-weight machine learning models make a verdict within milliseconds. These include specialized models and features that are built for specific file types commonly abused by attackers. Examples include models built for portable executable (PE) files, PowerShell, Office macros, JavaScript, PDF files, and more.
Behavior monitoring engine
The behavior monitoring engine monitors for potential attacks post-execution. It observes process behaviors, including behavior sequence at runtime, to identify and block certain types of activities based on predetermined rules.
Memory scanning engine
This engine scans the memory space used by a running process to expose malicious behavior that may be hiding through code obfuscation.
AMSI integration engine
Deep in-app integration engine enables detection of fileless and in-memory attacks through Antimalware Scan Interface (AMSI), defeating code obfuscation. This integration blocks malicious behavior of scripts client-side.
Heuristics engine
Heuristic rules identify file characteristics that have similarities with known malicious characteristics to catch new threats or modified versions of known threats.
Emulation engine
The emulation engine dynamically unpacks malware and examines how they would behave at runtime. The dynamic emulation of the content and scanning both the behavior during emulation and the memory content at the end of emulation defeat malware packers and expose the behavior of polymorphic malware.
Network engine
Network activities are inspected to identify and stop malicious activities from threats.
When it’s configured (which is normally default, but should be verified that no policy is actually disabling it) the cloud block level must be configured.
The following blocking levels are available:
Setting
Details
Default blocking level
provides strong detection without increasing the risk of detecting legitimate files
Moderate blocking level
provides moderate only for high confidence detections
High blocking level
applies a strong level of detection while optimizing client performance (but can also give you a greater chance of false positives)
High + blocking level
applies extra protection measures (might affect client performance and increase your chance of false positives)
Zero tolerance
blocking level blocks all unknown executables
It is recommended to at least be on the High blocking level
If a suspicious is detected and depending on setting, a sample is sent to the cloud service for analysis while Microsoft Defender Antivirus blocks the file. As soon as a determination is made, the file is either released or blocked by the AV.
Client-based machine-learning models, blocking new and unknown malware.
Local behavioral analysis, analyze and stopping file-based and file-less attacks.
Antivirus, detecting common malware through generic and heuristic techniques.
Cloud-based protection is provided for cases when Antivirus running on the endpoint needs more intelligence to verify the intent of a suspicious file
file metadata is sent to the cloud protection service. Often within milliseconds, the cloud protection service can determine based on the metadata as to whether the file is malicious or not a threat.
The cloud query of file metadata can be a result of behavior, mark of the web, or other characteristics where a clear verdict isn’t determined.
A small metadata payload is sent, with the goal of reaching a verdict of malware or not a threat. The metadata doesn’t include personally identifiable information (PII). Information such as filenames, are hashed.
Can be synchronous or asynchronous. For synchronous, the file won’t open until the cloud renders a verdict. For asynchronous, the file opens while cloud protection performs its analysis.
Metadata can include PE attributes, static file attributes, dynamic and contextual attributes, and more
For the settings…
The Sample submission has 4 different settings. In the health checks, a common observation is that it is disabled (do not send) or always prompt.
For cloud protection to work properly, it must be configured to Send all Samples automatically or (at least) Send safe samples automatically.
To stress further, you will not have full protection if this is configured in the wrong way!
Se below for details.
After examining the metadata, if Microsoft Defender Antivirus cloud protection can’t reach a conclusive verdict, it can request a sample of the file for further inspection. This request honors the settings configuration for sample submission:
Send safe samples automatically
Safe samples are samples considered to not commonly contain PII data like: .bat, .scr, .dll, .exe.
If file is likely to contain PII, the user gets a request to allow file sample submission.
This option is the default on Windows, macOS, and Linux.
Always Prompt
If configured, the user is always prompted for consent before file submission
This setting isn’t available in macOS and Linux cloud protection
Send all samples automatically
If configured, all samples are sent automatically
If you would like sample submission to include macros embedded in Word docs, you must choose “Send all samples automatically”
This setting isn’t available on macOS cloud protection
Do not send
Prevents “block at first sight” based on file sample analysis
“Don’t send” is the equivalent to the “Disabled” setting in macOS policy and “None” setting in Linux policy.
Metadata is sent for detections even when sample submission is disabled
After files are submitted to cloud protection, the submitted files can be scanned, detonated, and processed through big data analysismachine-learning models to reach a verdict. Turning off cloud-delivered protection limits analysis to only what the client can provide through local machine-learning models, and similar functions.
Number 3 – Extended cloud block timeout
The last feature in this post, is the extended cloud block timeout.
If you read the whole post, you have probably noticed that all these 3 features should be configured together since they depend on each other to better protect the endpoint.
While the cloud protection is analyzing the file that was found suspicious, antivirus can block the file from running. As default, it will prevent the file from running for 10 seconds but this can be configured to up to 50 seconds (plus the 10 default seconds) to allow cloud protection to do more analysis like detonation before the file is released.
This happens on suspicious files, and should not have impact on end users!
Block at first sight and its prerequisites must be enabled before you can specify an extended timeout period.
Stefan and I have talked about starting a podcast for a long time and now we have, together with Pierre Thoor, released episode 1 of the Security Dojo.
Our first episode is about the podcast itself and we’re also touching previous and upcoming events.
The plan is to invite guests and talk about news and things we find interesting. We appreciate feedback
This post will be a part of a multiple posts to cover data parsing in Microsoft Sentinel.
Intro
Kusto is a powerfull query language and easy to adopt.
Even if Kusto is very powerfull, working with custom log sources is, sometimes, a mess. Some parsers requires more effort and some are very simple.
In general, when it’s possible to use operators like “parse” (link) function or “parse-kv” (link) it’s very welcome. However, the reality has a different challenge for us.
In this post we want to share a quick pro tip to solve the mystic of hidden whitespaces
The challenge of whitespaces
Whitespaces ” ” exists everywhere, the challenge is how it’s presented in log analytics.
Log analytics does a lot for the user in terms of nicely present data. It actually removes duplicate whitespaces, as well as leading and trailing whitespace.
This could result in problems like failing parsers, regex and string operators like “==”, “startswith”, “endswith” etc will fail. Especially if it’s not consistent.
Marking the string in the output view does not show the extra whitespaces
Copying the text and paste into a text-editor will not show it either like in below example where we copied the output into VS Code (we can only see one dot to show one whitespace between foo and bar)
However, the double whitespaces are interpreted during execution, and it’s only in the presentation view the extra is removed. As in below example, we used split on ” ” to show the existence of the double whitespace.
When working with multiple log sources you don’t want to search and see if they exist (which may change during the log source life cycle), you rather want a way to always make the log to look good in your parser.
Solution
To properly address this (if there aren’t any good ways to change the audit settings of the system sending the logs)
To handle the duplicate white spaces we use the replace_regex function (link here) and use the whitespace “\s” with the quantifier “+” which means one or multiple times and replace it with a space ” “.
This will search for spaces (one or more) and replace it with a single, because we don’t want to remove single spaces. And by using the same column name “SyslogMessage” we will actually reuse the same column for our clean output.
Please note that this will not change the message in the database, only during execution.
Doing this gives us the following output.
The next step is that we want to remove the leading and trailing whitespaces. If we for instance expect the first character to be a value, the leading whitespace could make our parser to fail or an analytic rule.
We have seen occasions where this happens from time to time and not all messages in a log source.
To fix the leading and trailing whitespaces we use another regex to look for start of string and end of string. But this time we want to replace with “nothing/null” which is why we can’t use this regex in the first cleaning.
In the second run we use the same column name again to cleanup the SyslogMessage. There is a best practice to always keep the original message, however, this is to solve an error from the log source and not to alter the SyslogMessage.
The regex starts with an anchor “^” to define the start of the string and followed by a whitespace “\s” since we cleaned all double whitespaces we don’t need to use the quantifier. To handle the trailing whitespace we use the OR operand “|” and check for a whitespace “\s” followed by the anchor “$” to determine the end of the string. If we get any hits it will be replaced with null and we have a clean string.
By adding these 2 lines of code to the parser, we will avoid running into strange issues which could take some time to troubleshoot.
Automated attack disruption of Ransomware and BEC (Business email compromise uses high-confidence Extended Detection and Response (XDR) signals across all workloads; endpoints, identities, email, and SaaS apps, to contain the threat quickly and effectively, to stop further impact.
These 2 scenarios are common attacks and it’s really great that they are supported by the feature in Microsoft 365 Defender.
Business Email Compromise, BEC
Threat actors are impersonating executives to trick, for example, Economic department to transfer money by impersonating the CFO or the CEO.
Automatic attack disruption can help to detect these attacks and remove the access to the accounts by disabling the compromised account, limiting their ability to send fraudulent email
Human-operated ransomware, HumOR
This attack, commonly used today, is devastating for an organization. The threat actors has full control of the environment and have usually controlled the environment for some time.
The challenge from a SecOps perspective is to be fast enough to respond to the incidents and mitigate accounts and the devices fast enough.
When the threat actors has gained privileged access, things move very quick and automatic attack disruption will contain the spreader device and disable the compromised user account
Automatic attack disruption operates in 3 key stages
Detect malicious activity and establish high confidence
Classification of scenarios and identification of assets controlled by the attacker
Trigger automatic response actions using the Microsoft 365 Defender protection stack to contain the active attack
First the detection will happen, which is achieved by AI, research-information etc.., to establish a high level of confidence in accurately detecting ransomware spread and encryption activity. The XDR-level capability correlates insights across endpoints, identities, email and SaaS apps to establish high-fidelity alerts.
A second stage will aggregate automatic analyze the activities like tampering, backup deletion, credential theft, mass lateral movement and many more to flag the assets included in the chain and trace the activities back to the remote execution TTP
Distrupting the attack
Response actions against the entities which are identified as compromised and in the public preview these two are the main actions:
If you want to find threats earlier it is now possible to use NRT rules in Defender.
Before this, we had the options to use 24h, 12h, 3h, and 1h as the schedule. This gives Defenders the possibility to detect and respond to threat much earlier.
Tables that support Continuous (NRT) frequency
AlertEvidence
DeviceEvents
DeviceFileCertificateInfo
DeviceFileEvents
DeviceImageLoadEvents
DeviceLogonEvents
DeviceNetworkEvents
DeviceNetworkInfo
DeviceInfo
DeviceProcessEvents
DeviceRegistryEvents
EmailAttachmentInfo
EmailEvents
EmailPostDeliveryEvents
EmailUrlInfo
UrlClickEvents
The NRT rules does not support externaldata operator and you can only query one table
Configuring NRT (Continuous) Rule
From the Advanced Hunting, develop your query and click and configure the Alert Details
Click Next and select impacted entities (in this case we are using an email table and therefore the impacted entities will be mailbox)Click Next and configure the actions. It’s important to think about what the actions means and make sure your query will detect exactly what you want.
Be cautious with the Isolate device when querying Device tables. If you have an error in your detection you may isolate all machines by mistake
It’s now completed!
Don’t forget that you can use the hunting if you want to take response actions on multiple entities very quickly.
From the Result of your hunting query, select the rows where you want to take action and click Take Actions
This brings your the Actions pane and you can choose which actions you need.
Depending on your query (which tables and output) you get different options for your actions. ‘
Live response is designed to enhance investigations by enabling your security operations team to collect forensic data, run scripts, send suspicious entities for analysis, remediate threats, and proactively hunt for emerging threats.
Capabilities
Run basic and advanced commands to do investigative work on a device.
Download files such as malware samples and outcomes of PowerShell scripts.
Download files in the background (new!).
Upload a PowerShell script or executable to the library and run it on a device from a tenant level.
Take or undo remediation actions.
Requirements
macOS – Minimum required version: 101.43.84. Supported for Intel-based and ARM-based macOS devices.
We use cookies on our website to give you the most relevant experience by remembering your preferences and repeat visits. By clicking “Accept”, you consent to the use of ALL the cookies.
This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.
Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.
Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.
