Google Cloud Monitoring

 

• Cloud Monitoring collects metrics, events, and metadata, hosted uptime probes, and application instrumentation to gain visibility into the performance, availability, and health of your applications and infrastructure.

Features

• Collect metrics from multicloud and hybrid infrastructure in real time.
• Metrics, events, and metadata are displayed with rich query language that helps identify issues and uncover significant patterns.
• Reduces time spent navigating between systems with one integrated service for metrics, uptime monitoring, dashboards, and alerts.

Workspaces

• Cloud Monitoring utilizes workspaces to organize and manage its information.
• A Workspace can manage the monitoring data for a single Google Cloud project, or it can manage the data for multiple Google Cloud projects and AWS accounts.
• But, a Google Cloud project or an AWS account can only be associated with one Workspace at a time.
• You must have at least one of the following IAM role name for the Google Cloud project to create a Workspace:
  • Monitoring Editor
  • Monitoring Admin
  • Project Owner

Cloud Monitoring Agent

• The Cloud Monitoring agent is a collectd-based daemon that collects application and system metrics from virtual machine (VM) instances.
• The Monitoring agent collects disk, network, CPU, and process metrics by default.
• You can configure the Monitoring agent to monitor third-party applications.

Pricing

• Monitoring charges only for the volume of ingested metric data and Cloud Monitoring API read calls that exceed the free monthly allotment.
• Non-chargeable metrics and Cloud Monitoring API write calls don’t count towards the allotment limit.

Google Cloud Logging

 

• An exabyte-scale, fully managed service for real-time log management.
• Helps you to securely store, search, analyze, and alert on all of your log data and events.

Features

• Write any custom log, from any source, into Cloud Logging using the public write APIs.
• You can search, sort, and query logs through query statements, along with rich histogram visualizations, simple field explorers, and the ability to save the queries.
• Integrates with Cloud Monitoring to set alerts on the logs events and logs-based metrics you have defined.
• You can export data in real-time to BigQuery to perform advanced analytics and SQL-like query tasks.
• Cloud Logging helps you see the problems with your mountain of data using Error Reporting. It helps you automatically analyze your logs for exceptions and intelligently aggregate them into meaningful error groups.

Cloud Audit Logs

Cloud Audit Logs maintains audit logs for each Cloud project, folder, and organization. There are four types of logs you can use:

1. Admin Activity audit logs

• Contains log entries for API calls or other administrative actions that modify the configuration or metadata of resources.
• You must have the IAM role Logging/Logs Viewer or Project/Viewer to view these logs.
• Admin Activity audit logs are always written and you can’t configure or disable them in any way.

2. Data Access audit logs

• Contains API calls that read the configuration or metadata of resources, including user-driven API calls that create, modify, or read user-provided resource data.
• You must have the IAM roles Logging/Private Logs Viewer or Project/Owner to view these logs.
• You must explicitly enable Data Access audit logs to be written. They are disabled by default because they are large.

3. System Event audit logs

• Contains log entries for administrative actions taken by Google Cloud that modify the configuration of resources.
• You must have the IAM role Logging/Logs Viewer or Project/Viewer to view these logs.
• System Event audit logs are always written so you can’t configure or disable them.
• There is no additional charge for your System Event audit logs.

4. Policy Denied audit logs

• Contains logs when a Google Cloud service denies access to a user or service account triggered by a security policy violation.
• You must have the IAM role Logging/Logs Viewer or Project/Viewer to view these logs.
• Policy Denied audit logs are generated by default. Your cloud project is charged for the logs storage.

Exporting Audit Logs

• Log entries received by Logging can be exported to Cloud Storage buckets, BigQuery datasets, and Pub/Sub topics.
• To export audit log entries outside of Logging:
  • Create a logs sink.
  • Give the sink a query that specifies the audit log types you want to export.
• If you want to export audit log entries for a Google Cloud organization, folder, or billing account, review Aggregated sinks.

Pricing

• All features of Cloud Logging are free to use, and the charge is only applicable for ingested log volume over the free allotment. Free usage allotments do not come with upfront fees or commitments.

Google Cloud Billing

 

• You can configure billing on Google Cloud in a variety of ways to meet different needs.
• To use Google Cloud services, you must have a valid Cloud Billing account,

Features

• If you have a project that is not linked to a Cloud Billing account, you will have limited use of products and services available for your project.

Cloud Billing Account & Payments Profile

• Cloud Billing Account
  • It is set up in Google Cloud and is used to define who pays for a given set of Google Cloud resources and Google Maps Platform APIs.
  • Access control to a Cloud Billing account is established by IAM roles.
  • A Cloud Billing account is connected to a Google payments profile.
• Google Payments Profile
  • Stores your payment instrument like credit cards and debit cards, to which costs are charged.
  • Stores information about who is responsible for the profile.
  • This serves as a document center where you can view invoices and payment history.

Cloud Billing Reports

• The Cloud Billing Reports page allows you to view your Google Cloud usage costs at a glance and discover and analyze trends.
• It shows a chart that plots usage costs for all projects linked to a Cloud Billing account.
• You can select a date range, specify a time range, configure the chart filters, and group by project, service, SKU, or location to filter how you view your report.
• Moreover, you can also forecast future costs using the Cloud Billing Reports to check out how much you are projected to spend, up to 12 months in the future.

Cloud Billing Budgets

• You can define the scope of the budget to apply in:
  • Entire Cloud Billing account
  • One or more projects
  • One or more products
  • Other budget filters applicable to your Cloud Billing account.
• You can specify the budget amount to your requirement, or base the budget amount on the previous month’s spend.
• Moreover, you can also specify email alerts and declare the recipients in the following ways:
  • Using the role-based option (default), where you can send email alerts to billing admins and users on the Cloud Billing account.
  • Using Cloud Monitoring, where you can enlist other people in your organization (for example, project managers) to receive budget alert emails.
  • You can also use Pub/Sub for a more programmatic notification approach.

Overview of Cloud Billing roles in IAM

The following predefined Cloud Billing IAM roles are designed to allow you to use access control to enforce separation of duties in managing your billing:

• Billing Account Creator (roles/billing.creator)
  • Create new self-serve (online) billing accounts.
  • Assigned at organization Level
  • Use this role for initial billing setup or to allow the creation of additional billing accounts. Users must have this role to sign up for Google Cloud with a credit card using their corporate identity.
• Billing Account Administrator (roles/billing.admin)
  • Manage billing accounts (but not create them).
  • Can be assigned at the organization level or billing account.
  • This role is an owner role for a billing account. Use it to manage payment instruments, configure billing exports, view cost information, link and unlink projects, and manage other user roles on the billing account.
• Billing Account User (roles/billing.user)
  • Link projects to billing accounts.
  • Can be assigned at the organization level or billing account.
  • This role has very restricted permissions, so you can grant it broadly, typically in combination with Project Creator. These two roles allow a user to create new projects linked to the billing account on which the role is granted.
• Billing Account Viewer
  • View billing account cost information and transactions.
  • Can be assigned at the organization level or billing account.
  • Billing Account Viewer access would usually be granted to finance teams. It provides access to spend information but does not confer the right to link or unlink projects or otherwise manage the properties of the billing account.
• Project Billing Manager (roles/billing.projectManager)
  • Link/unlink the project to/from a billing account.
  • Can be assigned at the organization level or billing account.
  • This role allows a user to attach the project to the billing account, but does not grant any rights over resources. Project Owners can use this role to allow someone else to manage the billing for the project without granting them resource access.

Google Cloud Console

• Google Cloud Console is a web admin interface to manage your Google cloud infrastructure.

Features

• You can create projects on Google Cloud Console.
• With Cloud Console, you can quickly find and check the health of all your cloud resources in one place, including virtual machines, network settings, and data storage.
• Logging
  • Manage and audit user access to project resources.
  • Track down production issues quickly by viewing logs.
• You can explore the Google Cloud Marketplace and launch cloud solutions with just a few clicks.
• Billing
  • View a detailed billing breakdown of your bills.
  • Set spending budgets to avoid unexpected surprises
• Cloud Console enables you to connect to your virtual machines via Cloud Shell. You can quickly handle admin tasks using this instant-on Linux machine equipped with your favorite tools including Google Cloud SDK preconfigured and authenticated.

Pricing

• Cloud Console is available at no cost to Google Cloud Platform customers.

 

Google Cloud Dataproc

 

• Build fully managed Apache Spark, Apache Hadoop, Presto, and other OSS clusters on the Google Cloud Platform using Cloud Dataproc.

Features

• You can spin up resizable clusters quickly with various virtual machine types, disk sizes, number of nodes, and networking options on Cloud Dataproc.
• Dataproc provides autoscaling features to help you automatically manage the addition and removal of cluster workers.
• Cloud Dataproc has built-in integration with the following Google Cloud services for a more complete and robust platform.
  • Cloud Storage
  • BigQuery
  • Cloud Bigtable
  • Cloud Logging
  • Cloud Monitoring
  • AI Hub
• It is capable of image versioning. This will allow you to switch between different versions of the tools you want to use.
• To avoid charges for inactive clusters, you can utilize Dataproc’s scheduled deletion.
• You can manage your clusters via
  • Cloud Console Web UI
  • Cloud SDK
  • RESTful APIs
  • SSH access.
• Dataproc can be provisioned with custom images according to your needs.
• Workflow templates provide a flexible and simple mechanism for managing and executing workflows.

Pricing

• Only pay for the resources you use and lower the total cost of ownership of OSS
• Dataproc pricing is based on the number of vCPUs and the duration that they run.

Google Cloud Dataflow

 

• Cloud Dataflow is a fully managed data processing service for executing a wide variety of data processing patterns.

Features

• Dataflow templates allow you to easily share your pipelines with team members and across your organization.
• You can also take advantage of Google-provided templates to implement useful but simple data processing tasks.
• Autoscaling lets the Dataflow automatically choose the appropriate number of worker instances required to run your job.
• You can build a batch or streaming pipeline protected with customer-managed encryption key (CMEK) or access CMEK-protected data in sources and sinks.
• Dataflow is integrated with VPC Service Controls to provide additional security on data processing environments by improving the ability to mitigate the risk of data exfiltration.

Pricing

• Dataflow jobs are billed per second, based on the actual use of Dataflow batch or streaming workers. Additional resources, such as Cloud Storage or Pub/Sub, are each billed per that service’s pricing.

Google Cloud Dataprep

 

• Cloud Dataprep by Trifacta is an intelligent data service for visually exploring, cleaning, and preparing structured and unstructured data for analysis, reporting, and machine learning.

Features

• You can transform structured or unstructured datasets of any size — megabytes to petabytes — with equal ease and simplicity.
• Cloud Dataproc can transform datasets stored in CSV, JSON, or relational table formats.
• You can process data stored in Cloud Storage, BigQuery, or from your desktop, then export the refined data to BigQuery or Cloud Storage for storage, analysis, visualization, or machine learning.
• Uses a proprietary algorithm that interprets the data transformation intent of a user’s data selection.
• You can leverage hundreds of transformation functions readily available to turn your data into the asset you want.
• Cloud Dataprep enables users to collaborate on similar flow objects in real-time or to create copies for other team members to use for independent tasks.
• Explore your data through interactive visual distributions to assist in your discovery, cleansing, and transformation process.
• Cloud Dataprep automatically generates one or more samples of the data for display and manipulation in the client application to achieve performance optimization.

Pricing

• Pricing is split across two variables;
  • Design – is priced on a per-project basis for an unlimited number of users.
  • Execution – consists of the Dataflow usage for running jobs in Dataprep.