ETC achieves 300% performance improvement and increased efficiency after migrating to OCI

The authors would like to thank Austin Moseley, senior manager of IT Services, and Brett Coffee, director of back-office solutions at ETC, for their insights and contributions to this article.

ETC is a leading U.S. intelligent transportation systems and smart mobility provider, developing and delivering best-in-class solutions for tolling, congestion management, urban mobility, and multimodal transportation needs. ETC’s passionate and innovative team has been driving the future of mobility since 1999 with many industry firsts, including all-electronic tolling (AET), dynamic pricing, agency interoperability, and hosted mobility solutions.

For over two decades, ETC has delivered sophisticated solutions to many of the largest toll authorities in the U.S., including statewide programs, county networks and tolling-specific authorities. ETC has served top mobility authorities in the United States, including the Washington State Department of Transportation (WSDOT), Harris County Toll Road Authority (HCTRA), Illinois Tollway, North Texas Tollway Authority (NTTA) and State Road and Tollway Authority (SRTA). Today, ETC is one of a few providers to offer “cloud-first” solutions for the industry.

Figure 1: An ETC-equipped toll road

ETC's solutions process over two billion transactions annually, totaling over three billion in revenues for our customers. They incorporate the latest in evergreen open source and software-as-a-service (SaaS) technologies and Big Data architecture through its innovative riteSuite products.

ETC’s riteSuite provides a full range of tolling and mobility functions, from the roadside data collection to backend processing and payment. The integrated subsystems of riteSuite consistently score the highest technical marks within the industry.

This paper discusses the following cloud-based products:

• riteHorizon is a vertically integrated financial enterprise resource planning (ERP) and customer relationship management (CRM) back-office platform.
• ETC’s Interoperability Hub complies with U.S. National Interoperability Protocol (NIOP) and enables transaction processing, reporting, and reconciliation between multistate tolling authorities and different regional hubs.
• The Mobility Commerce Platform (MCP) is ETC’s forward-looking solution to serve as the vital bridge between the tolling systems and third-party emerging technologies, such as smartphone-based services, connected and autonomous vehicles, vehicle-to-Infrastructure (V2I) systems, parking, payments, curb management, and multimodal transportation.

Figure 2: End-to-end mobility: ETC products and services.  

The company’s objectives

ETC identified the following goals for its cloud migration project:

• Reduce time to market, as compared to traditional on-premises infrastructure. Make infrastructure provisioning operationally easy, so that new projects can be completed quickly and cost-effectively.
• Reduce maintenance burden by switching to a model that uses infrastructure-as-a-service (IaaS) and platform-as-a-service (PaaS). On-premise maintenance and deployment work, such as hygiene of infrastructure, maintenance of WAN connectivity, and buildout of data centers, took up valuable engineering and development time that could otherwise be dedicated to new value-added features for our customers.. Removing this complexity sharpens ETC’s focus on delivery of new business capabilities with the assurance of secure and efficient IT operations.
• Reduce capital expenditures for deployment of IT infrastructure by shifting to a consumption-based pricing (CBP) model.
• Reduce downtime because of hardware or systems issues by utilizing the inherently greater reliability of engineered cloud solutions.
• Set up a cloud-based disaster recovery environment that satisfies the enterprise’s business continuity requirements.

The solution: Oracle Cloud Infrastructure

After careful evaluation of available options, ETC recognized Oracle as the cloud provider that best fit ETC’s technical and organizational needs. Because ETC’s riteHorizon commercial back-office  system uses Oracle Database, Oracle Exadata Cloud Service, with its high I/O storage and high-bandwidth networking, was a perfect fit for cloud migration. The following Oracle products serve as the vital backbone for ETC’s back-office solutions:

• Exadata Cloud Service: Oracle Exadata Cloud Service is the best place for customers to run Oracle Database workloads in the cloud. Dedicated X8M infrastructure is isolated from other users, allowing database teams to improve security, performance, and uptime for customer databases.
• Block Storage: Oracle Cloud Infrastructure (OCI) Block Volumes provide customers reliable, high-performance block storage designed to work with a range of virtual machines and bare metal instances.
• Object Storage: OCI Object Storage enables customers to securely store any type of data in its native format.
• Data Transfer Appliance: Data Transfer Appliance securely moves terabytes or petabytes data between on-premises data centers and the cloud.
• Compute: OCI Compute provides fast, flexible, and affordable compute capacity to fit any workload need from performant bare metal servers and VMs to lightweight containers.
• Dynamic routing gateway (DRG): A virtual router that provides a path for traffic between on-premises networks and OCI virtual cloud networks (VCNs) and can also be used to route traffic between VCNs.
• Compartments: A collection or logical grouping of related cloud resources.
• Network Visualizer: Network Visualizer provides a diagram of the implemented topology of all VCNs in a selected region and tenancy.
• Site-to-site VPN: Site-to-site IPSec connection between the on-premises network and the VCN. The IPSec protocol suite encrypts IP traffic before the packets are transferred from the source to the destination and decrypts the traffic when it arrives.
• Local peering gateway: Devices used to support local VCN peering, the process of connecting two VCNs in the same region so that their resources can communicate using private IP addresses without routing the traffic over the internet or through your on-premises network.
• Remote peering gateway: Devices used to support remote VCN peering, the process of connecting two VCNs in different regions (but the same tenancy).
• Network security groups (NSGs): NSGs act as a virtual firewall for your Compute instances and other kinds of resources.
• Notifications: OCI Notifications is a highly available, low latency publish/subscribe service that sends alerts and messages to Oracle Functions, email, SMS, and message delivery partners, including Slack, PagerDuty, and ServiceNow.
• Monitoring: OCI Monitoring helps organizations optimize the resource utilization and uptime of their infrastructure and applications.
• Application Performance Monitoring (APM): OCI APM provides deep visibility into the performance of applications and enables DevOps professionals to diagnose issues quickly to deliver a consistent level of service.
• Logging Analytics: OCI Logging Analytics is a machine learning-based cloud service that monitors, aggregates, indexes, and analyzes all log data from on-premises and multicloud environments.
• Logging: OCI Logging lets DevOps customers easily review log data, diagnose issues, and use the rules engine to trigger serverless functions or alerts.
• Oracle Marketplace: The Marketplace is where integrators can purchase and implement third-party software that is commonly used on-premises to help with risk mitigation, staff knowledge, or unique features required by solutions.

ETC’s ERP and CRM solution using OCI

ETC’s ERP and CRM solution uses OCI to provide a highly available, cost-effective, and easily managed platform to build and run the tolling solution. The solution utilizes OCI’s Exadata Cloud Service, IaaS, and PaaS to quickly deploy and operate environments that support major toll revenue collection systems.

As mentioned, riteHorizon is a vertically integrated financial ERP and CRM platform. It has supported some of the largest toll agencies in the nation and has processed over two billion transactions per year, managing millions of customer accounts and billions of dollars in revenue. ETC’s riteHorizon provides high-volume transaction processing, an intuitive CRM system for customer service, an online self-service portal, and a fully reconcilable general ledger system that balances to the penny.

Figure 3: riteHorizon interface

The Central United States Interoperability Hub (CUSIOP)

Among projects that utilize riteHorizon is the Central U.S. Interoperability Hub Project, which provides interoperability between various member agencies in Texas, Oklahoma, and Kansas. In the context of toll systems, interoperability refers to an arrangement between different public and private road operators in one of more states, where transponders issued by one entity are accepted at facilities belonging to other entities.

Figure 4: CUSIOP fact sheet and interoperability across the United States. CUSIOP was the first interoperable tolling network to implement the national interoperability protocol.

CUSIOP is the nation’s first interoperability hub designed to national interoperability (NIOP) standards, which became operational in 2017. ETC designed, developed, tested, and implemented this system in compliance with U.S. national interoperability standards and facilitated transaction processing, reporting, and reconciliation for the seven participating agencies in the CUSIOP. CUSIOP processes over 600 million tolls a year and is a vital means of ensuring mobility in the central region of the U.S.

CUSIOP project implementation

ETC moved the CUSIOP hub’s operations to OCI, including test, development, user assistance, integration, production, and disaster recovery. Each environment was deployed as standby for non-production workloads and then switched over, preserving all workstreams for development teams while keeping downtime to hours.

Production was handled the same way: ETC engineers used OCI regions as disaster recovery sites and existing disaster recovery plans to drive production cutovers. The actual production cutover occurred on a normal workday and was seamless to the CUSIOP agencies.

The following architecture depicts the overall workflow at ETC. Agencies, such as Central Texas Regional Mobility Authority (CTRMA) and Texas Department of Transportation (TxDOT), upload transaction files (referred to as tag validation lists or TVLs) through SFTP to network attached storage (NAS). The transaction files are processed by submission process (SP) for validation and sent to the IO Processing (IOP) Hub. The IOP Hub sends an acknowledgment to the agencies with a return code to indicate whether a new or a duplicate validated tag has been processed. The IOP Hub then clears the transaction and reconciles its records. A performance report displays the transaction lists submitted and acknowledgments received, along with information on the transactions.

The architecture includes a report server used to query the database and generate reports. Scheduled reports can be generated, published to the NAS server, and then accessed by agencies securely through SFTP. Large monthly reports are generated directly by the database, which writes directly to the NAS server.  

Figure 5: Overall workflow of transactions

For the CUSIOP implementation on OCI, ETC chose two regions to host their six environments: Phoenix (PHX) and Ashburn (ASH). The Phoenix region hosts the CUSIOP’s test, development, user acceptance testing, and disaster recovery environments. Each of these environments has their own subnets, Compute instances, and storage servers. All environments share the Exadata Cloud Service infrastructure. They’re isolated by multiple Oracle home directories and access controls at the database level.

ETC’s architecture includes the following features of OCI:

• Compartments to separate environments across testing, development, user acceptance testing, and disaster recovery
• Flexible instances based on AMD E4 platform for the capability to scale cores and memory independently
• Block storage over file system storage for their NAS server to reduce cost
• Various security tools like the NSGs, Palo Alto Firewall from OCI Marketplace, duo multi-factor authentication (MFA), IBM QRadar, and private subnets
• Oracle Database on the latest Exadata Cloud Service X8M to run the high volume of data transactions
• Oracle Data Transfer Appliance to transfer data from on-premises to OCI. The appliance enabled the transfer over 20 TB of database backups and 3 TB of images.

The following diagram shows the reference deployment architecture on OCI:

Figure 6: ETC’s multi-region architecture on OCI

Results

ETC has completed several major migrations. OCI has replaced approximately 90% of legacy infrastructure for ETC’s back office clients, including clients who maintain their own infrastructure. They plan to replace the remaining 10% as the infrastructure reaches end of life. OCI is now the standard infrastructure for solution deployments that involve riteHorizon.

Performance has doubled by moving to OCI. Oracle’s high-speed storage and networking greatly improved latency on key processes and jobs. For riteHorizon, ETC achieved a lower total cost and ownership and speed to market that enabled it to win new business. Furthermore, the company realized significant efficiencies in environment build labor and long-term maintenance, increasing scalability, reliability, disaster recovery, and burst capacity.

Further, the time spent managing infrastructure with OCI is next to zero because most infrastructure and applications have been migrated to the cloud. With OCI, ETC has saved an estimated 90% in time required to manage infrastructure. Before OCI, ETC’s engineers had to regularly patch the infrastructure and conduct vulnerability management reviews. Often, the security model for the system became outdated quickly and more controls were needed. OCI simplified external vulnerability management and infrastructure vulnerability management.

By moving to OCI, ETC also applied its affinity for Oracle-based solutions. OCI wasn’t only the most cost-effective cloud for the ETC back office, but it also provided an estimated 200–300% performance improvement since migration.

“The Oracle sales, customer success, and architecture teams were very supportive,” said Brett Coffee, director of back-office solutions at ETC. “The best part of the process has been embedded architectural assistance provided as part of Oracle Cloud Lift services. Partnering with Oracle enabled us to meet our objectives and further the adoption and confidence in cloud computing within our industry.”

Figure 7: CUSIOP cloud migration success in numbers.

Next steps

As a next step, ETC is evaluating cloud native technologies, such as OCI Kubernetes Engine (OKE), OCI Functions, and OCI Streaming. Platform services to be adopted include Oracle Autonomous Database and Oracle Analytics.  

ETC’s technology team is migrating to Oracle Analytics as a comprehensive solution for building reports and conducting predictive analytics for targeted revenue streams. They plan to use Oracle Analytics to replace the more static report packages from the past. By modernizing the analytics, ETC is enabling its agency customers to use state-of-the-art dashboarding, artificial intelligence (AI), and machine learning (ML) to increase customer service, predictive analytics, and ensuring scalability for future road and user base growth. ETC also plans to deploy new back-office customers and its software development workspaces on OCI to support revenue collection for several toll agencies throughout the United States.

To learn more about Oracle Cloud Infrastructure and see how we can help you with cloud adoption, see the following resources:

• Explore more technical case studies on the OCI Technical Case Studies portal
• Get cloud resources at the OCI Cloud Adoption Framework portal
• Watch the Built & Deployed video on ETC (10:58)
• Try Oracle Cloud Free Tier