Trending News
Home / Technology / AI & IT / Agile and SCRUM have become dominant software project development methods

Agile and SCRUM have become dominant software project development methods

Project management is a booming trend in all organizations. All the business people want to find a way to manage their projects effectively to boost the business’s productivity, and the only solution is to use the methodologies of project management.


In the middle of the 1990s, the common way of developing software was a heavyweight software development methodology with a sequential life cycle approach called “the Waterfall,”  comprising of writing complete requirements, documentation of design and architecture, followed by coding, and then the whole product was tested, based on a detailed test plan.


Someone, usually the business analyst, first wrote a business requirements document that captured everything the business needed in the application. These business requirement documents were long, detailing everything: overall strategy, comprehensive functional specifications, and visual user interface designs. Technologists took the business requirement document and developed their own technical requirements document. This document defined the application’s architecture, data structures, object-oriented functional designs, user interfaces, and other non-functional requirements.


This waterfall software development process would finally kick off coding, then integration, and finally testing before an application was deemed production-ready. The whole process could easily take a couple of years.  It was the general notion that this was the only way of keeping projects on schedule and out of trouble, and if projects were facing problems, the belief was that all would be well if the methodology was strictly followed.


However, due to the rigid nature of the Waterfall approach, time estimates were often unreliable, as they did not accommodate for any unexpected events or changes. The customers rarely or never used over 60% of system features and functions. This can be explained by the fact that requirements were based on the initial customer needs and were not changed throughout the process. Lack of opportunity for customers to provide feedback on the system. Good ideas that came up along the way were not utilized and the process was so strict and set in stone, that these ideas were seen as a threat to the project instead of an opportunity to improve the product. Extremely hard to respond to changes, When iterating a phase the iteration takes considerable effort for rework.


When the system is put to use the customer discovers problems of early phases very late and system does not reflect current requirements. This approach also put great emphasis on documentation. This resulted in overly detailed requirement documents that would not be read, and when they were, they often resulted in a misinterpretation of the information. High effort and costs for writing and approving documents for each development phase. The waterfall model increases lead-time due to that large chunks of software artifacts have to be approved at each gate. Big-bang integration and test of the whole system in the end of the project can lead to unexpected quality problems, high costs, and schedule overrun. The final result was products that did not reach their full potential and projects that finished late, due to unrealistic time estimates or changes in the project that could not be avoided.


Innovation and product development is the cornerstone to organizational growth in the market environment today. But for new product development the stakes are high, requirements increasing and there is a demand of delivering faster in order to beat the competition to the market. New product development is a complex endeavour, which can often be difficult to handle and challenging to see in advance what the end result will be. There is great uncertainty and unexpected things happen along the way, which affects the scope and direction of a product development project. Therefore these projects can often be difficult to plan, and plans become obsolete soon after their creation.

Agile Software Development

New ways of managing development emerged within the software industry in the beginning of the 21st century, which revolutionized the way of developing software. Creators of alternative methods in software development came together and formed the Agile Manifesto in 2001. They stressed collaboration over documentation, self- organization rather than rigid management practices, and the ability to manage to constant change rather than lock yourself to a rigid waterfall development process. From those principles was born the agile methodology for software development.


Agile Software Development became a collective term for the new methods, which are all based on iterative and incremental development of self-organizing cross-functional teams, rapidly responding to change Agile Software Development is based on an iterative life cycle. This means that in one iteration cycle, an increment of the software is analysed, designed, built and tested. By the end of the iteration, the software increment is fully usable.  These increments build up over time to form the final product.


This method allows for quick and economic reaction to changes and errors that come up throughout the process. The incremental deliveries of a “finished” product ensure that there is always a potentially useable version of functioning product available. Halfway through the Waterfall process there is no usable code or product ready, while the Agile methods would have 50% of the final product finished and usable. This is related directly to when testing is performed in the process. It emphasizes the importance of testing continually throughout the process, as it allows for proactive measures when it comes to errors and software bugs.


Above all, Agile is a mindset and it’s a set of values and principles. Agile is a way of thinking and acting. Agile is all about short cycles, iterative and incremental delivery, failing fast, getting feedback, delivering business value to customers early and about people, collaboration and interaction. Agile is a mindset which is all about transparency, inspection and adaptation.


Agile engineering saves companies not only time, but also money in the long run. The agile method in product development has a higher startup time and initial cost, but the cost per final product is low. Plus, the end product is shaped by market demands, ultimately yielding greater profit. Another benefit of agile engineering is that it encourages teams to fail quickly. By failing faster, teams learn and improve at a faster pace than those that do not. Learning from failure through prototypes helps companies quickly build better products. By validating assumptions and collecting data, these products are made in a more accurate, evidential way.


Agile methodology

An agile software development process always starts by defining the users and documenting a vision statement on a scope of problems, opportunities, and values to be addressed. The product owner captures this vision and works with a multidisciplinary team (or teams) to deliver on this vision. Agile processes always begin with the user or customer in mind.


The Product Owner is the person that is financially responsible of the product, someone who is required to be the voice of the customer, including any internal stakeholders. That person prioritizes items according to the needs of the product and the stakeholders. That person distills all the insights, ideas, and feedback to create a product vision. These product visions are often short and straightforward, but they nonetheless paint a picture of who the customer is, what values are being addressed, and a strategy on how to address them. His or her responsibility is to define this vision and then work with a development team to make it real.


To work with the development team, the product owner breaks down the product vision into a series of user stories that spell out in more detail who the target user is, what problem is being solved for them, why the solution is important for them, and what constraints and acceptance criteria define the solution. These user stories are prioritized by the product owner, reviewed by the team to ensure they have a shared understanding on what is being asked of them.


In agile, the development team and its members’ responsibilities differ from those in traditional software development. Teams are multidisciplinary, composed of a diverse group of people with the skills to get the job done. Because the focus is on delivering working software, the team has to complete end-to-end functioning applications. So the database, business logic, and user interface of part of the application is developed and then demoed—not the whole application. To do this, the team members have to collaborate. They meet frequently to make sure everyone is aligned on what they are building, on who is doing what, and on exactly how the software is being developed.


In addition to developers, software development teams can include quality assurance (QA) engineers, other engineers (such as for databases and back-end systems), designers, and analysts, depending on the type of software project.


Scrum, Kanban, and other agile frameworks

These methods have received good reviews from the industry and many large companies use them in their development today. Many agile frameworks that provide specifics on development processes and agile development practices, aligned to a software development life cycle. The most popular agile framework is called scrum, and is currently being used by companies such as Yahoo!, Microsoft, Google and more. Scrum is described as “an iterative, incremental framework for projects and product or application development”.

It focuses on a delivery cadence called a sprint and meeting structures that include the following:

  • Planning — where sprint priorities are identified
  • Commitment — where the team reviews a list or backlog of user stories and decides how much work can be done in the sprint’s duration
  • Daily standup meetings — so teams can communicate updates on their development status and strategies


There are three roles in Scrum, the Product or Project Owner, the Scrum Master and the Team. Together these three roles form the Scrum Team. The Scrum Master is a person with experience in Scrum and he or she supports the Product Owner and the Team in complying with the Scrum rules. The Scrum Master also protects the team from external interruption, in order for the team to be focused on their work. A Team is a group of approximately seven people that executes the prioritized tasks and develops the final product.


The development work is performed in cycles called Sprints. Each Sprint can be from 1 to 4 weeks long, and are iterated without a pause. At the start of a Sprint the team commits to what work items they will complete by the end of the upcoming Sprint. The team chooses from a list of items, called the Product Backlog that has been prioritized by the Product Owner. This prioritization is based on input from all stakeholders of the product.


When a Sprint has started, no changes can be made to the committed task list, nor can the Sprint be elongated. The Sprint will finish on a set date whether or not the Team has completed what it had committed to. The idea is that by the end of a Sprint a working product increment will be ready. That is, a part of the final product is done or finished. The definition of “done” is of high importance, as in the case of software development, the product has to be fully tested and integrated and ready for use to be considered “done.”


During a Sprint a 15-minute Daily Meetings are held. This is a way for the team members to communicate to each other what their status is. Each team member communicates the following; what they have done since the last meeting, what they are going to do now, and if there are any impediments in their way. This allows other members to offer their help, but further discussions about the details will continue after the Daily Meeting. All other technical or more detailed discussion can be taken up after the meeting.


A sprint is developed and tested in a parallel fashion. This means that testing is not a separate phase but an integral part of the development process. The main aim of the testing team is to ensure early identification of bugs, issues and defects.


An important part of Scrum is “inspect and adapt”. At the end of a Sprint, a Sprint Review is held that focuses on the product. One part of the Review is a demonstration of the finished product part to all stakeholders. However, the main purpose of the Review is an in-depth conversation about the product between the Team members and the Product Owner, on the status as well as getting advice.


Sprint end with a demo meeting where the functionality is shown to the product owner, followed by a retrospective meeting where the team discusses what went well and what needs improvement in their process.  This meeting focuses on the inspecting and adapting the process and is facilitated by the Scrum Master. There the Team can identify how the process can be improved. After reviewing the previous retrospective action items, the meeting focuses on the following questions: What went well? What did not go well? What can we improve in the next sprint? The best way to do this is to gather answers in a round robin so that everyone has a chance to express himself or herself. A final question is suggested to gauge team happiness: What is one thing that will make you happier in the next sprint? This question will often find people’s deep-seated concerns about the project.



Kanban is a workflow management method for defining, managing and improving services that deliver knowledge work. It aims to help you visualize your work, maximize efficiency, and improve continuously.


You can start building your Kanban system by setting up the most straightforward Kanban board with three basic columns – “Requested”, “In Progress” and “Done”. When constructed, managed, and functioning correctly, it serves as a real-time information repository, highlighting bottlenecks within the system and anything else that might interrupt smooth working practices.


One of the main goals when implementing a Kanban system is to create a smooth, healthy flow. Instead of micro-managing people and trying to keep them busy all the time, you should focus on managing the work processes and understanding how to get that work faster through the system. This would mean that your Kanban system is creating value more quickly. One of Kanban’s primary functions is to ensure a manageable number of active items are in progress at any one time. If there are no work-in-progress limits, you are not doing Kanban.


Agile practices

While agile frameworks define process and collaboration, agile development practices are specific to addressing software development tasks performed in alignment with an agile framework.


User Stories: User stories are widely used within Agile Software Development. Use of or their definition is not described anywhere in the core literature on the Scrum framework. Despite this they are often used to represent Backlog items. A user story describes functionality that will be valuable to either a user or purchaser of a system or software. User stories are composed of three aspects:
• A written description of the story used for planning and as a reminder
• Conversations about the story that serve to flesh out the details of the story
• Test that convey and document details and that can be used to determine when a story is complete”


Visual Management: When the sprint has started, engineers need a tool to communicate within the team about what task they are working on (in case there are dependencies), as well as their status. A dashboard is an effective way to track tasks and communicate status. You can use sticky notes on a wall, but you can also use Excel or digital tools like Trello, Axosoft and Jira. The tasks, written on post-it notes, move across the board’s columns during the Sprint. These columns are often labelled “To Do”, “Work In Progress”, and “Done”.


Planning Poker: Planning poker is not prescribed in the core literature on Scrum but is also a widely used technique for relative estimating of stories or work items, using points as a relative unit, Some teams adopt pair programming, where two developers code together to drive higher quality code and to enable more senior developers to mentor junior ones. More advanced teams adopt test-driven development and automation to ensure the underlying functionality delivers the expected results. Many teams also adopt technical standards so that the developer’s interpretation of a user story doesn’t lead to just the functionality desired but also meets security, code quality, naming conventions, and other technical standards.


Lean Project Management

Lean project management is the application of lean manufacturing principles to the practice of project management. The goal of lean project management is to maximize value while minimizing waste. The Project Management Institute sums it up: “To be Lean is to provide what is needed, when it is needed, with the minimum amount of materials, equipment, labor, and space.”


Organizations that use lean project management can expect: Reduced lead times, Lower inventory and storage costs, Decreased overall costs, Improved productivity and efficiency, Greater quality, and Higher customer satisfaction.


First published in 1996, the book Lean Thinking by James P. Womack and Daniel T. Jones introduced five key principles that can be used to apply the lean concept to project management.

  • Specify value: What is the project’s value in the mind of the customer?
  • Map the value stream: A “value stream” map shows the entire process for creating the product or project. Once this process is mapped, it can be analyzed for waste, such as unnecessary steps that tax resources or compromise quality.
  • Make value flow by eliminating waste: Creating an improvement plan will eliminate the waste identified in the value stream. This plan represents a “future state” for the project’s process.
  • Make value flow at the customer’s demand: The ideal scenario is to move the project forward or create the product when requested by the customer. Get as close to this as possible to reduce inventory and save resources.
  • Embrace continuous improvement in pursuit of perfection: Regularly reassess the project process to eliminate waste and maximizing productivity and efficiency.


Success Factors in Deployment and Sustaining of Agile Methods

Pikkarainen et al. (2012) presented a study on successful Agile deployment focusing on strengths and barriers. They stated the following in regards to management support and tailored process models: “The analysis revealed the importance of management providing the necessary goals and support for agile development. It also indicated the significance of defining a tailored process model and giving developers the freedom to improve their own agile development process continuously during agile deployment.”

They further gave four recommendations to managers the following that should be taken into account when planning an Agile deployment, which were:
1) Make sure that the management is committed and gives continuous support for agile deployment
2) Make sure that both team, management and all stakeholders has a clear vision, understanding and awareness of agile methods
3) Give the needed freedom to the teams to tailor the agile methods to make [them] better fit to their specific needs
4) Make the continuous tailoring of the agile based process model also in the organizational level”


In regards to sustaining usage of Agile Methods, Senapathi and Srinivasan  identified and presented nine critical factors that have impact: Management Support, Attitude, Motivation, Team Composition, Training, Agile Mind-set, Technical Competence and Expertise, Agile Engineering Practices, and Methodology Champion. They conclude that their findings indicate the following: “Our findings also indicate that the right balance and combination of various factors with an emphasis on continuous improvement will be crucial for achieving true agile sustainability in organizations.”


Although scrum dominates, there are other agile frameworks:

Kanban works as a fan-in and fan-out process where the team pulls user stories from an intake board and funnels them through a staged development process until they are completed. Some organizations adopt a hybrid agile and waterfall approach, using agile processes for new applications and waterfall for legacy ones. There are also several frameworks to enable organizations to scale the practice to multiple teams.



One model that makes waterfall and agile get along is the Water-scrum-fall model. Business analysis and release management teams follow the traditional waterfall methods, while the development and testing team scrum methods in a limited way. Water-scrum-fall method employs the traditional waterfall approach for planning, requirements gathering, budgeting and documenting the project’s progress. When there are enough details to begin development, the team switches to a timeboxed, iterative version of Scrum for product development. This method uses agile principles and scrum communication techniques in day-to-day activities related product development.


Organizations use water-scrum-fall model when they want details in the planning phase so they can make accurate estimations of the budget. If a project initial phase is carried out in a plan-driven way, it is more likely to convince management about the idea and they will feel secure when allocating funds. Another reason for adopting water-scrum-fall model is the tendency of developers and testers to instinctively turn to agile practices during development. This happens because agile practices empower them and give them opportunities to collaborate as required by the limitations of the project.



Agifall approach was first presented at Vancouver Digital Project Managers Meetup Group. It combines the best of waterfall and agile by injecting the agile into a loose waterfall process. The aim of Agifall is to increase the speed, decrease the cost and improve the quality. Agifall approaches planning in a user-centric manner and use quick prototype tools. It carries the planning and requirements activities of waterfall in an agile manner by breaking them into user stories and prioritizing them in the sprint. In the Agifall method, you don’t wait for one phase to complete before starting the next phase; rather you begin the next phase as soon as you can. This means that you can begin independent development of some modules or components while the planning phase is still in progress. The development phase follows the usual agile principles. Agifall model suggests graphic designing and testing in parallel with the development phase.


Hybrid can go right or wrong:

The agile-waterfall hybrid model is far from perfect compromise. The elaborate documentation and completion of each phase required by waterfall methodology would be rather a burden to complete during a single agile sprint. The evolution of techniques such as backlog management instead of comprehensive documentation is one of the best examples of successful adoption of hybrid model. The hybrid model is best suited for projects which demand the team to deliver constantly changing requirements within a limited time frame. When a test team manager or lead has to adopt a particular method during the planning phase, the best way to proceed is to select the method that best suits the project needs. Moreover, the team needs to have a clear understanding of the hybrid model and implementation methodology, otherwise there’s a real possibility that it will make a mess of the project, and no benefits would be reaped.


AI in Project Management

Artificial Intelligence (AI) and machine learning have found a wide range of business applications. The most important capabilities of AI, that we already know, are processing large amounts of data quickly, finding patterns in data, learning from it and making predictions. Due to its unique ability to monitor specific patterns and forecasting project scenarios and outcomes, AI is and will continue to be an emerging trend in the project management area.


AI-based tools assist project managers in handling different tasks during each phase of the project planning process. It also enables project managers to process complex project data and uncover patterns that may affect project delivery. AI also automates most redundant tasks, thereby enhancing employee engagement and productivity. As per Gartner, AI will eliminate 80% of today’s manual project management tasks by 2030. AI machines will take over everything from planning to data collection, tracking to reporting, and so on.


AI can improve the accuracy of the project planning and supports the project manager to monitor the project’s progression. This is especially valuable when dealing with large and complex projects. AI-enhanced project management tools can help in making the right decision on the best allocation of resources for your project. Machine learning algorithms can be used to provide estimates of duration, resource and budget requirements for project activities based on historical information from previous projects.


AI can provide real-time data and project status updates through data visualization. This can help the team and management to discuss the status of a project and allows informed decision-making on the project duration, cost, and strategy The data-interpretation capability of AI can provide real-time insights into project metrics. It can enable project managers to make data-driven decisions based on past experience.


AI software enhances visibility for projects across the spectrum, which enables detection of risks early on so they can be handled before they pose a threat to the completion or quality of the project. For instance, the combination of machine learning with Monte Carlo simulation can help project managers to improve the evaluation and simulation of risks and opportunities of the overall project or specific tasks.



References and Resources also include:

About Rajesh Uppal

Check Also

Cloud Computing technology and trends driven by AI & Machine learning, IoT and 5G

Cloud computing is the on-demand delivery of IT resources over the internet with pay-as-you-go pricing. …

Leave a Reply

Your email address will not be published. Required fields are marked *

error: Content is protected !!