Chapter 3: End-to-end protection of our IT environment
In the playbook for IT operations management ebook, we discussed how ManageEngine evolved its IT infrastructure over the years. As a result of this evolution, we noticed an increase in the attack surface, creating multiple entry points for cyberattacks. We incorporated more stringent security measures to reduce the attack surface, fortified the cornerstones of IT security (the CIA triad), and implemented protective measures across our IT environment.
The CIA triad:
The cornerstone of IT security
The CIA triad, which stands for Confidentiality, Integrity, and Availability, is the core underpinning of our data security operations.
The CIA triad encompasses the possible scenarios where your information could be at risk, making it the foundation of information security. Consider the following scenarios: compromise of personal data, network hacking, or a DoS attack on a website. Each of them signifies a violation of one or more principles of the CIA triad and the risk that the data owner faces. Therefore, the CIA triad serves as a starting point for companies to improve their data security. Companies that implement measures to uphold each principle of the CIA triad have a strong foundation for data security. At ManageEngine, we use the triad approach to strengthen our basics despite having a dedicated command center and stringent controls for IT security.
Confidentiality
Confidentiality, the first principle of the CIA triad, ensures that we restrict access to and prevent unauthorized disclosure of data critical to ManageEngine. Confidentiality entails keeping our customers’ and employees’ data private and ensuring that only authorized personnel can access such information.
For instance, only authorized support personnel get access to a customer’s critical data, given that they need to troubleshoot an issue for the customer. This access has additional limitations so that the support personnel can only perform specific tasks related to troubleshooting. In the unfortunate event that data is exposed inadvertently, it is encrypted at rest so that those attempting to misuse it would be unsuccessful.
We use methods like encryption, multi-factor authentication, and biometric access for confidentiality.
Encryption
Humans traditionally used encryption to share a secret message with another party. Encryption involves transforming a message into a jumble of unrecognizable data that only the intended recipient can understand. Furthermore, the intended recipient can decrypt the unrecognizable data only through a specific decoding process. Therefore, this method acts as a formidable shield, preventing data theft by rendering it incomprehensible to unauthorized individuals. We use encryption for the same purpose: to secure data at rest and in transit.
We use the AES 256 algorithm, a standard encryption known to all. However, the strength of encryption lies in the secrecy of the key used to encrypt and decrypt the data. Accessing the encrypted data without the specific key is futile, leaving intruders with meaningless cipher text. Encryption helps us keep the data in our IT environment safe from prying eyes.
Multi-factor authentication
In an ever-evolving threat landscape, passwords are no longer adequate to safeguard our IT environment. A massive surge in cybersecurity attacks in the past decade has made it evident that relying solely on usernames and passwords leaves our systems vulnerable to compromise and unauthorized access. To reinforce our defense, we use multi-factor authentication (MFA) as a crucial element in our access security strategy. By implementing MFA, we add a layer of protection, significantly enhancing the security posture of our IT environment.
At ManageEngine, we use our in-house MFA application, which requires users to verify their identity using a combination of factors beyond the regular username and password. Using face ID, badges, or device identification, our users can establish various authentication modes and select their preferred sign-in method. In the event of failure of a preferred authentication mode, users can authenticate using alternate verification methods, including push notifications, one-time passwords (OTP), or QR codes. This multi-layered approach ensures enhanced security and protects our IT environment from potential breaches.
Our users also have the option to use our in-house OTP authenticator, which generates time-based OTPs that refresh every few seconds. Our MFA application prompts users to enter the app-generated OTP whenever they sign in, ensuring an added layer of security.
Biometric access
We exercise strict access controls to protected spaces such as data centers (DCs) and network operations centers. Apart from MFA, we also require employees to pass through biometric access. Furthermore, we regularly audit biometric access records to spot anomalies.
Integrity
Integrity refers to data not being tampered with. It’s lost if the data is tampered with or compromised. At ManageEngine, we take preventive measures like access control, backups, and strict permissions to maintain data integrity. Additionally, encryption also plays a crucial role in integrity.
File permissions
Our workspace management system gives users efficient control over file management and preserves data integrity. Teams create workspaces to share files and assign specific permissions to ensure secure access for users. For instance, some users may be given privileges only to view files within the workspace, and others may have privileges such as viewing, editing, and sharing.
Additionally, the system reserves file deletion authority exclusively for the workspace owner, ensuring file integrity. Furthermore, we employ version control and restoration features to access previous versions through the version history of a file. The file owner holds the authority to delete versions, safeguarding shared files from accidental or unauthorized deletions by other users.
We regularly perform internal audits to scrutinize file permissions and management, swiftly detecting any deviations and ensuring strict adherence to our security standards by our employees.
Secure access for users
We use a single sign-on (SSO) system to allow users to access our cloud services using the same sign-in page and authentication credentials. By default, access to our cloud happens only through our integrated Identity and Access Management (IAM) system. We ensure that users who use a different identity provider adhere to the Security Assertion Markup Language (SAML) standard.
Through our SSO system, we secure the login process, ensure compliance with security standards, exercise effective access control and reporting, and reduce the risk of password fatigue and weak passwords.
Furthermore, we employ technical access controls and internal policies to prohibit our employees from arbitrarily accessing user data. We adhere to the principles of least privilege and role-based permissions to minimize the risk of data exposure.
We have a central directory to maintain access to production environments. We mark such access requests as critical and authenticate them using a combination of strong passwords, two-factor authentication, and passphrase-protected SSH keys. We also facilitate such access through a separate network with stricter rules and hardened devices. Additionally, we log all the operations and audit them periodically.
Data backups
At ManageEngine, we recognize our responsibility to safeguard customer data, even in the face of unforeseen circumstances or potential compromise. While we strongly encourage customers to create their backups for convenience, we prioritize data integrity by maintaining our backups.
To ensure data integrity, we execute daily incremental backups and weekly full backups of our databases using our unified data center management console. These backups are securely stored with the original data and undergo encryption with the robust AES-256-bit algorithm.
However, the right balance between data retention and risk mitigation is critical to overall data security. Hence, we retain all backed-up data for three months. Should a customer require data recovery within this retention period, we swiftly restore their data and provide secure access to facilitate their operations.
To further fortify the safety of the backed-up data, we employ a redundant array of independent disks (RAIDs) within our backup servers. Our advanced data management console automatically verifies the integrity and validation of the backups, triggering a rerun in the event of any failure.
We prioritize the resilience and security of our customer's valuable data, allowing them to focus on their core business while entrusting us with data protection and recovery.
Availability
Availability means providing timely and reliable access to our resources, including applications, systems, and data, to our customers. If our customers can't access these resources, the resources would be of no use—as in the case of a denial-of-service attack.
We take preventive measures such as regular upgrades, backups, and business continuity and disaster recovery (BCDR) strategies to ensure high availability to our customers.
Business continuity and disaster recovery
We focus heavily on providing uninterrupted services and maintaining high availability, which instills trust in our customers. A key component of achieving high availability is effective disaster recovery.
We employ resilient storage that replicates across multiple data centers to guarantee data resilience. Real-time replication ensures that data from the primary DC is mirrored promptly in the secondary DC. During the unfortunate event of a primary DC failure, operations transition seamlessly to the secondary DC, minimizing downtime. Both primary and secondary DCs are equipped with redundant internet service providers, further enhancing reliability.
We shield the system and services from the effects of possible server failures by using a distributed grid architecture. This architecture enables our users to carry on with their duties, even without noticing failures in the servers, if any. Furthermore, we ensure device-level redundancy and prevent single-point failures using multiple switches, routers, and security gateways.
In addition to redundancy, we prioritize business continuity through various physical measures. Our facilities feature power backup systems, temperature control mechanisms, and cutting-edge fire prevention systems, ensuring uninterrupted operations. Furthermore, we have comprehensive business continuity plans for critical functions like support and infrastructure management.
To learn more about how we conduct our overall business conductivity and disaster recovery, check out our e-book here.
System upgrades
We prioritize the seamless functioning of our systems, for which upgrades are crucial. We achieve this using a centralized solution that manages endpoints, covering deployment, patching, and overall management. We also conduct regular audits to assess the status of our systems, software, and operating systems, allowing us to upgrade whenever necessary.
We ensure periodic updates using our efficient network configuration management tool for our network infrastructure. This tool empowers our administrators to review network device configurations regularly and push necessary updates. Additionally, we schedule backups of network device configurations to ensure their integrity and availability.
Continuous protection for our IT environment
Overview
At ManageEngine, we prioritize the protection of our IT environment at every stage. The image below illustrates how we protect our IT environment, starting from the developer's endpoint to the user's environment, encompassing our entire IT infrastructure.
Principles
Secure by design
In today's evolving threat landscape, relying solely on a separate security function is no longer sufficient. At ManageEngine, we embrace a security-by-design approach, embedding security consciousness into every development phase. By making security an integral part of our processes, we assure our customers that all of our offerings are inherently secure.
To implement security by design, we adhere to a comprehensive change management policy that governs every change in our development pipeline, including introducing new features. This policy ensures that all changes are authorized, validated, and thoroughly assessed for potential security implications before implementing them into production.
Our software development life cycle (SDLC) enforces strict adherence to secure coding guidelines. We employ advanced code analysis tools, vulnerability scanners, and rigorous manual review processes to screen code changes for potential security issues. This approach helps us identify and mitigate vulnerabilities early in the development process.
Our security engineering team builds and maintains a robust security framework aligned with OWASP standards within the application layer. This framework incorporates functionalities to mitigate common threats like SQL injection, cross-site scripting, and application layer denial-of-service (DOS) attacks.
Secure by default
At ManageEngine, we prioritize security from the moment a user begins using our offerings. Our products are secure by default, requiring minimal configuration changes. Our customers trust us to protect their data from the beginning with our default configurations, settings, and features.
For instance, our MFA application ensures secure access by default to our offerings, providing an added layer of protection for your account. Likewise, we automatically provision SSL certificates on our network, enabling HTTPS connections for your domain and subdomains without any additional setup from your end.
Comprehensive asset inventory
At ManageEngine, our IT leaders emphasize the importance of understanding what we're securing in order to protect our entire IT environment. In the early days of AdventNet, managing a handful of assets was as simple as recording them on a spreadsheet. However, with our global presence and more than 15,000 employees, maintaining an accurate asset inventory is critical to our security program.
So, how do we gather information about our extensive range of assets across the IT landscape?
We rely on our integrated asset management solution that provides us with comprehensive visibility. This solution enables us to track assets in various stages, from development to deployment, including those in the cloud, firewalls, and other components.
We establish a solid foundation for our IT security program by maintaining a centralized repository of all our assets.
Secure endpoints
From Figure 4, we can infer that our IT environment begins with our software engineers and the machines (laptops, computers, phones, etc.) they use, and ends with the customer accessing our offerings. Securing a software engineer's endpoint (laptop or desktop) is the first step toward securing our entire IT environment.
During the early days of the internet, endpoint security was straightforward with antivirus solutions, which were predominantly database-oriented. The antivirus provider maintains an elaborate database of signatures, file extensions, and other attributes that mark malware. If something in our system matches that database, the solution recognizes it as a virus, and we take necessary action.
However, with the ever-expanding threat landscape and new modes of attacks, we can no longer depend on antivirus alone for the complete security of our endpoints. Therefore, we established an endpoint detection and response (EDR) system to protect our endpoints completely from multiple modes of cyberattack.
Here's how our EDR system functions:
We have an EDR agent running in our endpoints that collects data using multiple parameters—everything from machine start to user login, user activity, process activity, and telemetry data. Telemetry data is mainly user history, application history, network history, and process history. The agent also collects network history: which connections are going and coming, which interface is present, etc.
This data moves to the EDR engine, which correlates the data with predetermined security rules. Each threat, such as ransomware or malware, has specific characteristics. We use an ML-based threat intelligence engine that correlates the machine data and threat rules. The engine has an internal scoring system that provides scores for the events (using corresponding data) fed into it. If the score crosses a threshold for a particular type of threat, the engine categorizes it as a threat. If the engine spots a threat, it sends out alerts automatically to our IT team.
Each alert sent to the IT team becomes a ticket, which we manage using our enterprise service management solution. The IT team reviews the report from the EDR engine to understand its legitimacy. The EDR engine has stringent security rules to enhance protection. Therefore, some alerts could be false positives, which are usual events that the engine categorizes as threats, just to be cautious. Once the IT team eliminates the false positive threats, the genuine threats get escalated to the concerned technician.
The EDR engine provides a detailed report on the threat it detects. The technician reviews that report and chooses the best course of action. For instance, if the technician finds out that a particular endpoint is compromised, they contact the owner of the endpoint and initiate cleanup.
This process ensures we monitor our endpoints for malicious activity and neutralize any threats before they harm our IT environment.
Apart from EDR, we run vulnerability scanning for each endpoint regularly. Based on the results from these scans, we periodically patch them. There is a distinction between the default system upgrade patching and vulnerability patching. We do the former, using our endpoint management solution. The user can choose not to upgrade. However, vulnerability patching takes a higher priority, because it is mandatory and performed by the central security team.
Our asset inventory serves as the input for vulnerability scanning, and the results from vulnerability scanning serve as the input for patching.
Integrity verification
We perform integrity verification at the following stages in our SDLC, as depicted in Figure 4:
- When code moves from the repository to the build.
- When code moves from the build to the deployment.
- When code moves from deployment to the cloud environment.
Here's how we perform integrity verification at multiple stages:
Integrity verification is a vital security control that prevents supply chain attacks. When a piece of code moves from one stage to another, we apply integrity verification to ensure the code passes on as it should and hasn't been tampered with.
The software engineer generates a "sign" at the repository. A sign here is a checksum or a bunch of text we can compile from a fragment of deployable code. It's important to note that sign is not related to the individual who generates it but rather to the contents of the code.
Therefore, as depicted in Figure 6, every time a code moves from one stage of the SDLC to another, the party that sends the code generates a sign, and so does the party that receives the code. For instance, the sign the software engineer generates at the repository should be the same as the sign the deployment team generates. If the sign doesn't match, it implies that the code has been tampered with, and the event becomes an incident.
The integrity verification is complete only when all signs generated at each stage of the SDLC match each other.
Fortified code repository
Security checks
We follow security by design as a key principle in our SDLC to embed security at the early stages of development. We employ crucial security checks at the early stages of our repository, as depicted in Figure 4. We have a dedicated configuration management (CM) team that ensures our products are in line with the requirements they set out to achieve. A crucial component of the CM's function is to validate our code for security attributes.
Once our software engineers push a code into the repository, the CM team performs the following code checks:
- Localization checks: Language-based checks suitable to the location
- Static variable check: If static variables are declared locally
- Empty check: If a block of code is empty
They also perform other checks specific to the product requirements. If the CM team does not approve a fragment of code, our CM tool triggers a mail to the concerned engineer with the required changes. After making those changes, the engineer checks into the CM tool, and the process continues until the CM team approves the code.
Static application security testing (SAST)
Our in-house SAST tool scans identifies and reports security vulnerabilities in a piece of deployable code. The SAST tool promotes secure software development by helping software engineers identify the attack surfaces in their applications, and it also provides solutions to minimize and secure those attack surfaces.
The SAST tool takes in a fragment of executable code and applies a set of rules to the code. Based on the adherence to those rules, the tool generates a security report, analyzing if and where the scanned code violates those rules. Depending on the severity of the violations, the SAST tool blocks deployment temporarily, prompting the software engineer to fix these violations.
Protected cloud
We protect our IT environment at the application layer using a web application firewall (WAF). According to industry standards, a WAF is a firewall component that protects an application even before a user can enter the application. It’s supposed to function as a layer of protection before an application. However, ManageEngine improvises it and provides WAF as an built-in library in the application itself. The WAF protects an application at the entry and functions as a security library in the application code, becoming inseparable from the code. Improvising WAF in such a manner helps us enforce our principle of security by default even more firmly.
WAF is a set of rules governing actions on our applications at its core. If a user requests entry into an application, the application must respond to it based on our WAF.
For instance, if the user wants to access a URL, the WAF checks a set of parameters before giving them access. Here, it functions as a system of controls. Likewise, before fulfilling an application request, the WAF mandates implementing a preliminary set of security controls first. The WAF library has an elaborate list of such security controls built into the application. For example, authentication, while being an operation, is also defined by a set of security controls in our library.
WAF is built and maintained by our security engineering team. The security operations team ensures that the application teams use this library in their codes. While the WAF is configurable, the application teams must configure these libraries in their code. The library also has provisions to shield the application from other attacks like injection, DoS, and identity-based attacks. For instance, to minimize the attack surface area, the library whitelists specific URL parameters. We fulfill the application request only if these parameters fall under these specific categories. Otherwise, we get an alert for a URL injection attack. Information generated from a user’s browser, such as URL, cookie, and header, comes under scrutiny and must satisfy the whitelist before it can proceed.
The WAF library provides baseline security for the application because it defines entry points into an application and the responses from the application. For example, WAF ensures that the other applications running on a user’s browser cannot read or access any information from our applications.