Zr Ware™ Redundancy/Security System

Infrastructure

• Zr Ware™ is hosted and run in Amazon Web Services (AWS) as Software as a service (Saas). The main benefit we get from running in the cloud is the ability to easily configure monitoring events such as CPU utilization, disk usage, and memory usage and be alerted when some threshold is crossed. If we find that the load on the system is increasing, we have the ability to quickly scale up (or down) the amount of hardware to handle the load required on the system. It is easy to configure firewalls and security rules, security certificates to only allow the traffic we want in to the system.
• The database runs in Amazon Relational Database Services (RDS) using a PostgreSQL database
• The back end code is written in Java and is built on top of the Spring Boot Framework
• The front end/javascript is mostly on top of Angular 1 JS. We are looking to rewrite the front-end in ReactJS as Angular 1 JS is not actively supported used in the open source community. Components like the Online Scheduler and the User Invoice Link/Waiver Link are currently written in ReactJS.

Security

• Access to any resource in AWS requires Multi-Factor Authentication (MFA).
• The database only accepts incoming requests for a fixed range of IP Addresses defined by the VPC in AWS.
• Port forward access to the database for read/write operations requires that a user first be authorized to SSH into an appropriate environment from which they can then communicate with the database.

Database

• Every night at 8 AM UTC a full copy is created of the database and we store a copy of that database for 1 year in Amazon S3 cloud storage that can only be referenced by user's that have permission to view/download that backup.

What if there is a Major Outage?

For this article, we'll define a "major outage" as you hit Zrware.com and you don't get any response back.

We utilize Amazon Cloud Watch monitoring and set up health checks that AWS will hit every 10 seconds to help us know if the endpoint is healthy. In the event that a major outage occurs, a text message is sent to the team informing them that there is a problem.

Here are several reasons the system could be down:

• The CPU on the computers running the software is too high and they cannot respond to any more requests
  • Solution: Add more computers to handle the load or upgrade the hardware on the existing computers. This can all be done with a few clicks in the AWS Console or through the command line. We have scripts to help us do this very quickly.
  • Mitigation: Setup alerts that if we start to hit 60, 70, 80% CPU usage, make sure we add more hardware before we have an outage.
• We run out of disk space on the computer
  • Solution: Free up more disk space
  • Mitigation: Make sure we have processes in each computer to make sure that temp files are being cleaned up on a regular basis and that log files do not get too large overtime (i.e. log rotation). Setup alerts that if disk usage is getting too high alert someone to manually take action to free up disk space.
• Database encounters a deadlock situation and the java database connection sessions need to be restarted
  • Solution: Restart the java process on each computer to create a new database session. Fix the code that is creating a deadlock situation
  • Mitigation: Code reviews at the engineering level
• Database CPU is too high
  • Solution: Scale up the amount of hardware on the database and restart the database and the java processes.
  • Mitigation: Leverage monitoring on the CPU and review CPU trends overtime to ensure we have the proper amount of hardware for the load we are processing.
• Database runs out of space
  • Solution: Add more space to the database
  • Mitigation: Leverage AWS Cloud watch to monitor the amount of free space on the database level.

Schedule Backups

Every night as part of a nightly process that runs, Zr Ware™ will send an email of what your schedule looked like at the start of the day in the event that you were to have some kind of outage during the day that took more than just a few minutes to resolve. If you are not receiving this email, please reach out to support.

What can I do if there is a major outage?

In theory, the monitoring we have setup should help us know as soon as something is wrong. But please reach out to support as quickly as possible if you feel that something is wrong. The Zr Ware™ team will then work as quickly as possible to address the problem in AWS to restore service.

Credit Cards

In order to be PCI compliant, any web forms that accept credit card payments now leverage javascript SDKs from the Credit Card Payment gateways that insert an iframe where the Credit Card number can be securely tokenized. That token, meaningless by itself, is now used to securely apply payments to invoices.

PCI compliance is currently reviewed annually.