This is first part of series of articles about “Complete Set of Tactics for Reducing Tech Debt: Lessons Learned from 14 Years of Building Java Enterprise Systems”.

• Part 2: “Comparative Technological Audit“
• Part 3: “New Beginning with Microservices Architecture“.

In the beginning of 2010, a client approached Avra (at that time under the ‘Makrologic’ brand), the software development agency I worked for, with a big vision for a new enterprise system. The system was to be a B2B platform for employers to provide additional benefits to their employees, such as sport cards or free lunches, which employees could purchase using the system or special cards at local restaurants. The client was not familiar with the technology, so my team had the freedom to choose the best-known technologies at the time. The team consisted of Bartek, CEO of Avra, Wiktor – business analysts and senior backend Java engineers Wojciech, Piotr, and Kuba, as well as frontend engineers.

We followed waterfall methodology and started the project with the MVC approach. For that we applied our Custom System Solutions service. We used jsp and jQuery to develop the system, which was the popular choice at the time. We applied appropriate design and architecture patterns and the best-known practices of the time, such as DRY rules, and continuous integration. We used Spring Framework and Hibernate as our go-to frameworks, and the JVM tech stack. However, we were lacking all the good things that are now available, such as containers and clouds.

The client was a startup, who fought for its first customers and place on the market. It was obvious that they were pushing us to deliver more and more without focusing on the tech debt that was growing. We kept the client updated about our progress in an iterative way, and Bartek’s presence in the team gave the client confidence that the CEO was keeping an eye on the project.

Despite following the best-known practices at the time, we weren’t monitoring the performance of the system sufficiently. The client was pushing to get all their features ready for their first big client.

Little did we know that this approach would lead to a big failure in the next phase of the project.

It was the day our client had been eagerly waiting for: the launch of the new enterprise system we had been developing for almost a year. However, the excitement quickly turned into a nightmare when our client couldn’t onboard their first big client, with more than 2000 employees, due to the system constantly crashing with only 20 users online. Why did this happen? due to lack of communication between business and technology teams we were not aware about the potential rump-up of the users.

“Houston we have a performance problem”. The client was devastated, and so were we. We knew we had to act fast to fix the problem and regain their trust. We started by launching JProfiler to diagnose the issue and found that there were a lot of inefficiencies on the Hibernate level. We also had not adjusted our load tests after adding new features, which had caused more stress on the system than we’ve expected.

I remember working tirelessly for two days straight, starting on Tuesday morning and leaving the office only on Wednesday evening, to fix the issue. We adjusted caching layers and removed bottlenecks diagnosed by JProfiler, and finally, the system started to work properly again.

After the crisis was over, we reflected on what went wrong and what we could have done differently. The urgency to ship new features blindfolded us on non-functional requirements of the projects.

We were surprised how early our freshly developed system required a service that provides continuous auditing of enterprise systems to ensure optimal performance and prevent critical system failures. This project and all next ones enabled us to create our “Continuous Performance Auditing” service, which offers regular checks and performance profiling to detect potential issues before they become catastrophic. By utilizing this service, CTOs and technical teams can:

This service can be a valuable partner to any enterprise system development project, allowing for continuous improvement and optimization without interrupting the development cycle.

With an external partner working hand in hand with the team, detect and address issues earlier.

Monitoring the system from the beginning to test the performance of the system can save a lot of work because by finding issues earlier you can improve the system gradually.

Despite the setback, we were able to regain the client’s trust and to continue the project. But we knew that we had to be more vigilant in the future and keep a close eye on the system’s performance to avoid another disaster.

Over the next three years, our development team working on the project grew to more than 12 members, and we worked on various parts of the system still following our Custom System Solutions service approach. However, our biggest challenge came when the sales department promised a whole new module to a huge end-client, assuming it was very similar to something that already existed in our system, and would be easy for us to develop. This assumption was one of the root causes of the technical debt that started to grow rapidly.

Since we couldn’t deliver the promised module as a separate new one because it was too expensive, and because it was very similar to the existing one, we were forced to reuse the same module for completely two different domains. The modules looked similar, unfortunately, the business logic was different. This decision caused the technical debt to accumulate even faster, and we started to feel the consequences.

The client hired a new project manager who was very aggressive and pushed us to the limits of our mental capabilities. It was a real battle. We had to break our technical backbone and our rules in developing the project, otherwise, we would lose the client. So, we sacrificed some of our technical best practices to meet promises that none consulted with us and enforced on us.

Moreover, the collaboration model from the very beginning was fixed-price, so we had to estimate every change before we delivered. Because of the broken backbone of our technical best practices, our estimations grew to such big sizes that the client started to question whether we were a good software agency at all. A vicious circle.

As a result, the lead time (LT) and time to repair (TTR) were growing, and it was challenging to convince the client to invest in architecture improvements.

Eventually, the LT and TTR grew so much that the client decided to audit our code by an external company, which we’ll describe in the next chapter. Unfortunately, the client lost trust in our team and wanted to ask the external company whether we were a good software agency or not (spoiler alert: we were a good one 🙂 )

 In the beginning, it’s crucial to choose the right technologies and follow best practices. However, monitoring system performance and addressing non-functional requirements should never be neglected. It’s also important to be cautious when making promises to clients, as overlooking technical debt and breaking technical best practices can lead to severe consequences. Continuous auditing and performance profiling services can play a vital role in maintaining system optimization and preventing critical failures.

Managing technical debt requires a proactive approach. By prioritizing architecture improvements, monitoring system performance, and addressing technical debt early on, organizations can avoid accumulating debt that hampers development speed and quality. Collaboration between business and technology teams is essential to ensure realistic estimations and prevent the erosion of technical best practices. Continuous improvement and optimization, along with external audits, can help mitigate and manage technical debt effectively

In Java development, mitigating technical debt involves adhering to best practices, such as applying appropriate design and architecture patterns, following DRY rules, and leveraging frameworks like Spring and Hibernate. Continuous monitoring of system performance, load testing, and utilizing tools like JProfiler can help identify and address inefficiencies. Additionally, investing in continuous auditing and performance profiling services can aid in detecting and resolving potential issues before they escalate. Prioritizing architectural improvements and maintaining a balance between feature development and debt management are crucial in mitigating technical debt in Java projects.

Read more:

• Part 2: “Comparative Technological Audit“
• Part 3: “New Beginning with Microservices Architecture“.