Palo Alto Networks - Cortex Data Lake

Posted Oct 18, 2020 Updated Dec 15, 2023

By Grace L

8 min read

[toc]

Cortex Data Lake

# data lake > log forwarding > query empty

# Data Lake Terabyte?

overview

Palo Alto Networks® Cortex Data Lake

provides cloud-based, centralized log storage and aggregation
secure, resilient, and fault-tolerant
ensures logging data is up-to-date and available when need it.
provides a scalable logging infrastructure that alleviates the need for to plan and deploy Log Collectors to meet log retention needs.
If already have on premise Log Collectors, the new Cortex Data Lake can easily complement existing setup. can augment existing log collection infrastructure with the cloud-based Cortex Data Lake to expand operational capacity as business grows, or to meet the capacity needs for new locations.
With this service, Palo Alto Networks takes care of the ongoing maintenance and monitoring of the logging infrastructure so that can focus on business.

Required Ports and FQDNs Required for Cortex Data Lake

If using a Palo Alto Networks firewall to secure traffic between Panorama, the firewalls, and Cortex Data Lake,

use the App-ID paloalto-logging-service in a Security policy rule to
- allow Panorama and the firewalls to connect to Cortex Data Lake
- and forward logs on TCP 444 and 3978 (default ports for the application)
If firewall has an Applications and Threats content version earlier than 8290, must also allow the panorama app-id in a security policy rule.

If using another vendor’s firewall

use the following table to identify the fully qualified domain names (FQDNs) and ports that must allow traffic to ensure that Panorama and the firewalls can successfully connect to Cortex Data Lake.

link

host Cortex Data Lake in the following regions:

Americas (US)
Europe (Netherlands)
UK
Singapore
Canada
Japan

Cortex Data Lake Log Sources

Here are the products and services that can send logs to Cortex Data Lake:

sources	log type	Note
Palo Alto Networks Firewalls link	firewalls log	onboard individual firewalls directly to Cortex Data Lake. Use the `Explore app` to view all log records that the firewalls forward to Cortex Data Lake.
Panorama-Managed Firewalls link	firewalls log	onboard firewalls to Cortex Data Lake at scale, instead of onboarding each individual firewall. All Cortex Data Lake logs are visible directly in Panorama.
Prisma Access	the logs, ACC, and reports from Panorama for remote network and mobile user traffic	Prisma Access deploys and manages the security infrastructure globally to secure remote networks and mobile users. Prisma Access logs directly to Cortex Data Lake. - To enable logging for Prisma Access, must purchase a Cortex Data Lake license. - Log traffic does not use the licensed bandwidth purchased for Prisma Access.
Cortex XDR	Cortex XDR alerts are automatically written to Cortex Data Lake as log records	other apps can read and respond to alerts. - These log records are not visible in `Explore`; - can use `Log Forwarding app` to forward XDR alerts to the email or Syslog destination and configure email alert notifications within XDR.

Cortex Data Lake Log Types

link

In the Cortex Data Lake app, can set how much of overall log storage would like to allocate to the following log types:

Log Type | Description —|— . | Cortex XDR Logs alert | Information for all alerts raised in Cortex XDR. xdr | (Cortex XDR Pro per Endpoint only) All EDR data collected on the endpoint. . | Common Logs | config / Configuration logs | entries for changes to the firewall configuration. system / System logs | entries for each system event on the firewall. . | Firewall Logs | auth / Authentication logs | information about authentication events that occur when end users try to access network resources for which access is controlled by Authentication Policy rules. eal Enhanced application logs | data that increases visibility into network activity for Palo Alto Networks apps and services, like Cortex XDR. extpcap Extended packet capture | packet captures in a proprietary Palo Alto Networks format. The firewall only collects these if enable extended capture in Vulnerability Protection or Anti-Spyware profiles. file_data Data filering logs | entries for the security rules that help prevent sensitive information such as credit card numbers from leaving the area that the firewall protects. globalprotect | GlobalProtect system logs
LSVPN/satellite events
GlobalProtect portal and gateway logs
Clientless VPN logs hipmatch HIP Match logs | the security status of the end devices accessing network. iptag IP-Tag logs | how and when a source IP address is registered or unregistered on the firewall and what tag the firewall applied to the address. sctp Stream Control Transmission Protol logs | events and associations based on logs generated by the firewall while it performs stateful inspection, protocol validation, and filtering of SCTP traffic. threat Threat logs | entries generated when traffic matches one of the Security Profiles attached to a security rule on the firewall. traffic Traffic logs | entries for the start and end of each session. tunnel Tunnel Inspection logs | entries of non-encrypted tunnel sessions. url URL Filtering logs | entries for traffic that matches the URL Filtering profile attached to a security policy rule. userid User-ID logs | information about IP address-to-username mappings and Authentication Timestamps, such as the sources of the mapping information and the times when users authenticated.

Cortex Data Lake Apps

Cortex Data Lake includes the following apps to manage and view logs.

Cortex Data Lake:
1. After activating Cortex Data Lake, the Cortex Data Lake app is listed on the hub.
2. If have multiple instances of Cortex Data Lake, choose which instance of the app want to open.
3. Use the Cortex Data Lake app to configure log storage quota and to check the status of a Cortex Data Lake instance.
Explore:
1. Use Explore to search, filter, and export log data.
2. This app offers critical visibility into enterprise’s network activities by allowing to easily examine network log data.
Log Forwarding app:
1. Use this app to archive the logs send to Cortex Data Lake for long-term storage, SOC, or internal audit directly from the Cortex Data Lake.
2. forward Cortex Data Lake logs to an external destination such as a Syslog server or an email server. (Or, continue to forward logs directly from the firewalls to Syslog receiver).

Cortex Data Lake License activation

Cortex Data Lake collects log data from next-generation firewalls, Prisma Access, and Cortex XDR.

When purchase Cortex Data Lake, all firewalls registered to support account receive a Cortex Data Lake license.
also receive an auth code to activate Cortex Data Lake instance.

activation

Use the hub to activate Cortex Data Lake.
Use the hub to activate Cortex XDR.
- To activate the Cortex XDR app, you must be assigned a required role and locate activation email containing a link to begin activation in the hub.
- Activating Cortex XDR automatically includes activation of Cortex Data Lake.
  1. Begin activation.
  2. Provide details about the Cortex XDR app to activate
  3. Review the end user license agreement and Agree & Activate.
  4. Manage Apps > current status of apps.
  5. log in to Cortex XDR app to confirm access for the Cortex XDR app interface.
  6. Allocate Log Storage for Cortex XDR
    1. Allocate Storage Based on Log Type
    2. storage allocation for Cortex Data Lake and adjust the quota as needed.
      1. Select Cortex Data Lake instance.
      2. Select Configuration > logging storage settings.
      3. The Cortex Data Lake depicts storage allocation graphically. As you adjust storage allocation, the graphic updates to display the changes to storage policy.
      4. The Cortex Data Lake storage policy specifies the distribution of total storage allocated to each app or service and the minimum retention warning (not supported with Cortex XDR).
      5. Allocate quota for Cortex XDR:
        If purchased quota for firewall logs, allocate quota to the Firewall log type.
        To use the same Cortex Data Lake instance for both firewall logs and Cortex XDR logs, ust first associate Panorama with the Cortex Data Lake instance before you allocate quota for firewall logs.
        Review storage allocation for Cortex XDR according to the formula:
        1TB for every 200 Cortex XDR Pro endpoints for 30 days
        By default,
        80% available storage is assigned to logs and data,
        and 20% is assigned to alerts.
        It is recommended to review the status of Cortex Data Lake instance after about two weeks of data collection and make adjustments as needed but to use the default allocations as a starting point.
        Apply changes.
        Monitor data retention.
        From Cortex XDR > Cortex XDR License > Endpoint XDR Data Retention:
        Current number of days your data has been stored in Cortex XDR Data Lake.
        The count begins the as soon as you activate Cortex XDR.
        Number of retention days permitted according to the quota you allocated.
    3. You must be an assigned an Instance Administrator or higher role to for Cortex Data Lake to manage logging storage.

Start Sending Logs to Cortex Data Lake

Before you start sending logs to Cortex™ Data Lake, you must:

Configure the XDR Agent
- Enable the Cortex XDR agent to Monitor and Collect Enhanced Endpoint Data link
Activate your Cortex Data Lake instance
- Connect Firewalls to Cortex Data Lake
Sending log data to Cortex Data Lake from
- Cortex Data Lake License
- Connect the firewall to Cortex Data Lake.
- other sources
- Panorama-managed firewalls Forward Logs to Cortex Data Lake (Panorama-Managed)
- Prisma™ Access Configure the Service Infrastructure
- Cortex XDR Prevent
- Cortex XDR Pro per Endpoint
  - Begin activation.
  - Provide details about the Cortex XDR app activating.
  - Review the end user license agreement and Agree & Activate.
  - Manage Apps to view the current status of your apps.
  - When your app is available, log in to your Cortex XDR app to confirm that you can successfully access the Cortex XDR app interface.
  - Allocate Log Storage for Cortex XDR.
  - Assign roles to additional administrators, if needed.
  - Complete your configuration.
- Cortex XDR Pro per TB