-
Notifications
You must be signed in to change notification settings - Fork 513
[macOS][unified_log] Initial release of macOS Unified logs integration. #15794
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
muskan-agarwal26
wants to merge
5
commits into
elastic:main
Choose a base branch
from
muskan-agarwal26:macos-0.1.0
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+7,238
−0
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
bca8c75
Initial release of macOS Unified logs integration.
muskan-crest 14c5b60
Added codeowners entry and updated changelog entry
muskan-crest 0e6ee35
Made suggested changes in dashboard
muskan-crest 1c6773a
Run build cmd to update readme
muskan-crest 90e08d4
Resolved comments provided by @btrieger
muskan-crest File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| dependencies: | ||
| ecs: | ||
| reference: git@v8.17.0 | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we probably want to use 9.2.0 since is the latest |
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| # macOS Integration for Elastic | ||
|
|
||
| ## Overview | ||
|
|
||
| The macOS integration for Elastic allows you to collect and analyze unified logs from macOS systems. This integration leverages macOS's unified logging system to provide comprehensive visibility into system activities, security events, and application behaviors on macOS endpoints. | ||
|
|
||
| macOS unified logging is Apple's centralized logging system that captures log messages from the kernel, system processes, and applications. This integration enables security teams to monitor macOS endpoints for suspicious activities, troubleshoot system issues, and maintain compliance with security policies. | ||
|
|
||
| ### Compatibility | ||
|
|
||
| The macOS integration is compatible with macOS systems that support unified logging (macOS 10.12 Sierra and later). | ||
|
|
||
| ### How it works | ||
|
|
||
| This integration uses the `unifiedlogs` input to collect log data from the macOS unified logging system. It can collect logs in real-time or from archived log files, with configurable filtering based on predicates, processes, and log levels. | ||
|
|
||
| ## What data does this integration collect? | ||
|
|
||
| This integration collects unified log messages from macOS systems using configurable predicates to filter specific event types, including: | ||
|
|
||
| - **Authentication logs**: User login/logout events, authentication failures, and credential-related activities | ||
| ``` | ||
| - 'process contains "sudo" OR composedMessage CONTAINS "sudo" OR process contains "su"' | ||
| - 'process contains "loginwindow" and composedMessage CONTAINS "sessionDidLogin"' | ||
| - 'process == "sshd"' | ||
| ``` | ||
| - **User & Account management**: User account creation, modification, and deletion events | ||
| ``` | ||
| - 'process == "sysadminctl" AND composedMessage CONTAINS "Creating user"' | ||
| - 'process == "dscl" AND composedMessage CONTAINS "create"' | ||
| - 'process == "sysadminctl" AND composedMessage CONTAINS "Deleting user"' | ||
| - 'process == "dscl" AND composedMessage CONTAINS "delete"' | ||
| - '(process == "dscl" OR process == "opendirectoryd") AND composedMessage CONTAINS "admin"' | ||
| ``` | ||
| - **Process execution monitoring**: Process creation, termination, and execution details | ||
| ``` | ||
| - 'eventMessage CONTAINS[c] "exec" OR eventMessage CONTAINS[c] "fork" OR eventMessage CONTAINS[c] "exited" OR eventMessage CONTAINS[c] "terminated"' | ||
| - 'subsystem == "com.apple.securityd" AND (composedMessage CONTAINS "code signing" OR composedMessage CONTAINS "not valid")' | ||
| - 'composedMessage CONTAINS "com.apple.quarantine"' | ||
| ``` | ||
| - **Network activity**: Network connections, DNS queries, and network-related events | ||
| ``` | ||
| - 'composedMessage CONTAINS "connect" AND (composedMessage CONTAINS "TCP" OR composedMessage CONTAINS "UDP")' | ||
| - 'composedMessage CONTAINS "disconnect" OR composedMessage CONTAINS "closed connection"' | ||
| - 'subsystem == "com.apple.necp" AND composedMessage CONTAINS "new connection"' | ||
| - 'eventMessage CONTAINS[c] "listening" AND eventMessage CONTAINS[c] "service"' | ||
| ``` | ||
| - **File reads/writes**: File system access, modifications, and permission changes | ||
| ``` | ||
| - '(eventMessage CONTAINS "open" OR eventMessage CONTAINS "write" OR eventMessage CONTAINS "unlink" OR eventMessage CONTAINS "rename") AND ((processImagePath BEGINSWITH "/System") OR (processImagePath BEGINSWITH "/bin") OR (processImagePath BEGINSWITH "/sbin") OR (processImagePath BEGINSWITH "/usr" AND NOT processImagePath BEGINSWITH "/usr/local") OR (processImagePath BEGINSWITH "/etc"))' | ||
| - 'subsystem == "com.apple.quarantine" OR eventMessage CONTAINS "com.apple.quarantine"' | ||
| ``` | ||
| - **System changes**: System configuration changes, software installations, and updates | ||
| ``` | ||
| - 'subsystem == "com.apple.security" OR subsystem == "com.apple.systempolicy" OR subsystem == "com.apple.installer" OR process == "Installer" OR process == "softwareupdated" OR eventMessage CONTAINS[c] "removed package" OR eventMessage CONTAINS[c] "forget package"' | ||
| ``` | ||
| - **Advanced monitoring**: Detailed system and application behavior logs | ||
| ``` | ||
| - '(composedMessage CONTAINS ".plist" AND (composedMessage CONTAINS "write" OR composedMessage CONTAINS "modified")) OR (composedMessage CONTAINS ".ssh" AND (composedMessage CONTAINS "write" OR composedMessage CONTAINS "modified")) OR (process == "kernel" AND composedMessage CONTAINS "boot") OR (process == "launchd" AND (composedMessage CONTAINS "started" OR composedMessage CONTAINS "listening")) OR (process == "loginwindow" AND composedMessage CONTAINS "sessionDidLogin") OR (composedMessage CONTAINS "posix_spawn" OR composedMessage CONTAINS "exec") OR (subsystem == "com.apple.securityd" AND (composedMessage CONTAINS "code signing" OR composedMessage CONTAINS "not valid"))' | ||
| ``` | ||
|
|
||
| ### Supported use cases | ||
| The macOS integration in Elastic enables comprehensive monitoring and analysis of system activities, network traffic, and application behavior across macOS devices. It supports use cases such as detecting security incidents, tracking network usage, auditing system events, and analyzing performance trends. By collecting and visualizing unified logs, it helps security and IT teams gain real-time visibility, identify anomalies, ensure compliance, and enhance overall endpoint security within macOS environments. | ||
|
|
||
| ## What do I need to use this integration? | ||
|
|
||
| ### From Elastic | ||
|
|
||
| - Elastic Agent must be installed on the macOS system you want to monitor | ||
| - Appropriate permissions to read system logs on the macOS system | ||
|
|
||
| ### From macOS | ||
|
|
||
| The integration requires: | ||
|
|
||
| - macOS 10.12 Sierra or later (for unified logging support) | ||
| - Appropriate system permissions to access unified logs | ||
| - For some log categories, administrative privileges may be required | ||
|
|
||
| ## How do I deploy this integration? | ||
|
|
||
| This integration requires Elastic Agent to be installed on the macOS systems you want to monitor. | ||
|
|
||
| ### Agent-based installation | ||
|
|
||
| Elastic Agent must be installed. For more details, check the Elastic Agent [installation instructions](docs-content://reference/fleet/install-elastic-agents.md). You can install only one Elastic Agent per host. | ||
|
|
||
| ## Setup | ||
|
|
||
| 1. In the top search bar in Kibana, search for **Integrations**. | ||
| 2. In the search bar, type **macOS**. | ||
| 3. Select the **macOS** integration from the search results. | ||
| 4. Select **Add macOS** to add the integration. | ||
| 5. Enable and configure only the collection methods which you will use. | ||
|
|
||
| **Basic Configuration:** | ||
| - Enable the log categories you want to collect: | ||
| - Authentication | ||
| - User & Account management | ||
| - Process execution monitoring | ||
| - Network activity | ||
| - File reads/writes | ||
| - System changes | ||
| - Advanced monitoring | ||
|
|
||
| **Advanced Configuration (Optional):** | ||
| - **Predicate**: Use NSPredicate-based filtering to collect specific log messages | ||
| - **Process**: Specify particular processes to monitor (by PID or name) | ||
| - **Start/End dates**: Define time ranges for historical log collection | ||
| - **Log levels**: Configure which log levels to include (info, debug, backtrace, signpost) | ||
| - **Archive/Trace files**: Specify log archive or trace files to process | ||
|
|
||
| 6. Select **Save and continue** to save the integration. | ||
|
|
||
| ### Validation | ||
|
|
||
| #### Dashboards populated | ||
|
|
||
| 1. In the top search bar in Kibana, search for **Dashboards**. | ||
| 2. In the search bar, type **macOS**. | ||
| 3. Select a dashboard for the dataset you are collecting, and verify the dashboard information is populated. | ||
|
|
||
| ## Performance and scaling | ||
|
|
||
| - Unified log collection can generate significant data volume, especially with debug-level logging enabled | ||
| - Consider using predicates to filter logs and reduce data volume | ||
| - Monitor system performance impact when collecting high-volume log categories | ||
|
|
||
| For more information on architectures that can be used for scaling this integration, check the [Ingest Architectures](https://www.elastic.co/docs/manage-data/ingest/ingest-reference-architectures) documentation. | ||
|
|
||
| ## Reference | ||
|
|
||
| ### ECS field reference | ||
|
|
||
| #### Unified logs | ||
|
|
||
| {{fields "unified_log"}} | ||
|
|
||
| ### Inputs used | ||
|
|
||
| These inputs can be used in this integration: | ||
|
|
||
| - [Unified Logs](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-unifiedlogs) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| # newer versions go on top | ||
| - version: 0.1.0 | ||
| changes: | ||
| - description: Initial release. | ||
| type: enhancement | ||
| link: https://github.com/elastic/integrations/pull/15794 |
3 changes: 3 additions & 0 deletions
3
packages/macos/data_stream/unified_log/_dev/test/pipeline/test-common-config.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| fields: | ||
| tags: | ||
| - preserve_duplicate_custom_fields |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be owned by @nfritts's team, the input is owned by their team and also system unified logs. Any reason we are linked here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch @narph agreed, + @marc-gr as he has been assisting with development here