Long ago I had a dream to create an agile management tool so fluid and powerful that it would make your agile projects fun, make your team successful, and you would love it.

Today that dream is a reality.

Agile teams, Exia is for you.

At Exia we’ve been working on a site for a customer that requires email relay through the hosting servers. That means we have to look up forum support information. Why not capture a snapshot of the correct information while we’re there and annotate it?

Of course we can also do a quick copy/paste of the info into the workitem details so that it’s searchable:

You can do either one or both. The great thing is that we’ve worked hard to get the process down to just a few seconds, so that screen capture is fun.

At Exia we are now using Exia every day to manage our own work. Capturing screen shots as we go has become an indispensable tool that helps us work faster and more accurately. It’s hard to imagine now what it was like working before we could grab images of everything we’re doing so easily and have them automatically shared among the team.

I hope you enjoy this feature as much as we do!

Nigel

A new Exia feature makes it quick and easy to filter work items by the user to whom they’re assigned. Just open the Filter panel by clicking the Filter button at the top of any work item list. In the Assigned row of radio buttons, click Select. The drop-down list of users is enabled and you can select any user on your team.

If you want to see each user’s list in succession, just click the drop-down so that it’s the focused control, then user the up/down arrows to change the drop-down selection. The work item list will instantly adjust to show the items for the selected user. It’s a great way to get a bird’s eye view if you’re a manager.

Clicking on any of the numbers takes you right to the relevant list.

This feature makes it quick and easy to navigate work item lists and focus on your own active tasks.

Nigel

Because Exia compliments TFS and Visual Studio, Microsoft has a strong interest in Exia, so we were invited to do a major presentation to Microsoft. The audience included Aaron Bjork, Tony Goodhew, Anthony Cangialosi, Tom Lindeman, Peter Larsen, Buck Hodges and Siddharth Bhatia. Exia crashed every three minutes. Nobody was impressed and the only consolation was that they realized it could possibly be a Microsoft related issue. But nobody could pin it down.

The bug was so elusive and difficult to reproduce that we were really stuck. No product. No chance for sales. No potential for growth. Nothing. Nada.

Then suddenly, thanks to the diligent work of a Microsoft expert named Miha Markic in solving a similar bug and posting the simple solution here, and the blind luck that I happened to hit that page in relation to something completely unrelated, our bug was solved.

Thanks to Miha for saving Exia. We are back in development again. For everyone who’s been watching our progress toward release, we apologize for the 9 month delay. We expect a stable and usable Exia early next quarter and look forward to getting successive trial versions in your hands via the Exia site.

Nigel

I encourage you to download the trial, or take advantage of the special price and buy your license now.

I’m the chief architect and designer of The Exia Process, and I’ve really taken great care to try to make it as user friendly as possible. Here’s a video that introduces you to all the main features: http://youtu.be/WQESOY52vPo.

Check it out and please give me feedback. My team and I are always looking for ways to make it better.

Here are a couple of screen shots…

Why do I care?

Because our flagship product is a desktop app, while our competitors’ are all web apps, and when our customers think they need a web app, we lose sales.

So I was glad to see what Steve Jobs didn’t say about web apps vs desktop apps, in his keynote presentation at the 2011 Apple WorldWide Developer Conference. What he didn’t say was this:

“For applications that require intensive user interaction, such as incremental search, image display, drag and drop and interactive functionality, such as iTunes, a desktop solution is better than a web solution. In fact, a web solution sort of sucks.”

How didn’t he say that?

Toward the end of the presentation, Jobs was comparing iCloud to the equivalent Amazon and Google offerings. At the top of his feature chart was the platform comparison. iCloud uses iTunes. The others are web apps. The Apple crowd clapped their approval.

On the Mac platform then, it seems that having a desktop app is a clear advantage.

Why is this less so on the Windows platform? That’s obvious. Because Mac users love their operating system, while Windows users tend to hate theirs. They want to leave their OS behind. They think web apps will help them do that.

That’s not going to happen. Windows is getting better, and Windows apps are getting better. We think, for instance, that our flagship product is a great Windows app. Most important, we think it’s better than our competitors’ web apps.

I think Steve Jobs might agree, and if you’re looking for a fluid software process management tool, I hope you agree too.

July 2009

This research paper considers the alarming failure rate of software projects and the failure of project management tools to stem the tide.

We consider what kind of tool might help to increase the success rate of software projects. We believe that while the features of the tool are important, the underlying characteristics of the tool are even more important. We identify five such characteristics—flow, communicative transparency, process guidance, visualizations and integration—and argue why these characteristics are important in software-specific project management.

Finally we consider what a tool might look like that implements these characteristics.

Overview

Recent trends in the failure rates of software project management are alarming. The most recent Standish Group report on software success (April 2009) shows a marked increase in the project failure rates, with only 32% of projects succeeding.

Through Exia’s experience in software project management, we have identified five critical success factors for software project management tools; flow, communicative transparency, process guidance, visualizations and integration. Our PLAY project, and the resulting Exia Play software product, was conceived to create tools that incorporate these success factors. The product is currently under development in our research labs.

Success will require innovation in application lifecycle tools well beyond current standards, and will require development of novel and creative ways of integrating those tools with existing Microsoft technologies and tools. The result will be an innovative foundation to significantly improve flow, communicative transparency, process guidance, visualization and integration in software projects. Our hope is that the result will be a significant contribution to the science of software engineering.

Project Background

This section describes the background of the project, starting with the need—the failure of software project management—and continuing to an overview of project management tools and methodologies. Finally we analyze factors we have identified—flow, communicative transparency, process guidance, visualizations and integration—that influence the success of project management tools.

Software Project Management Failure

The chronic failure of the software engineering process is well known. The just-released 2009 Standish Group CHAOS Report (Figure 1), which shows a sharp downward trend in software delivery capability, reinforces the severity of the problem. The report shows a significant downturn in project success rates and an upturn in failure rates, with just 32% of software projects ending in success.

Figure 1: The latest CHAOS report by the Standish Group points to a downturn in software success rates.

In the past, much of the failure of software projects was attributable to technical software issues. Software often simply failed to work as it was supposed to. Today this is less the case. Software platforms are becoming mature and predictable in a number of domains. In the web and business application domain, both Java and .NET are rapidly developing into solid, engineering quality tools. The days of cottage industry software development are being left behind and the days of software engineering are here.

As the CHAOS report shows, however, high quality technical platforms do not necessarily lead directly to higher quality software projects.

Project Management Tools

Fixing the problem of software mismanagement is a global challenge that has not gone unnoticed by project management vendors. Dozens of companies offer many different flavours of software project management tools. Figure 2 through Figure 6 show typical systems. Features typically include release management, customer management, users, defect management, feature management, scrum management, feature backlog management, and so on.

Figure 2: AxoSoft OnTime software project management

Figure 3: Microsoft Project

Some systems, such as AxoSoft OnTime and Microsoft Project (Figure 2, Figure 3), are client-server based, requiring installation on the corporate premises. Others are web based and require only an account and a browser to access projects.

Figure 4: Journyx project management

Figure 5: Daptive Scrum

Figure 6: Mingle agile project manager

Of the available tools, Microsoft Project is the most popular tool for software project management because of its familiar GANTT interface, integration with Microsoft Office, and widespread adoption as a general purpose project management tool.

Methodologies

Software lifecycle methodologies attempt to create planning frameworks on which good plans can be built. They do this by encapsulating best practices in the form of tools and guidance.

The first software methodologies advocated a linear, waterfall progression of tasks from analysis to requirements to construction, testing, sign-off and so on. Later methodologies recognized that software development has elements of discovery and feedback, and introduced the notion of cyclical development. The best known of the cyclical methodologies is the Rational Unified Process, or RUP (Figure 7).

Figure 7: The Rational Unified Process is the best known early proponent of a cyclical approach to software development

Modern methodologies take this further, advocating agility, leanness, de-emphasis of written plans, emphasis of working software, and strong communication. Collectively, these modern, lightweight and flexible methodologies are termed Agile methodologies.

One of the leading proponents of Agile is Microsoft, who provide deep support for the Agile process with tools that are tightly integrated into the application development tool suite, Visual Studio (Figure 8). Agile development seeks to increase the value of software development by increasing the feedback available to customers and developers. Some of the feedback comes from more frequent releases to production, some from more frequent testing, some from more frequent software builds, and some from social structures that encourage conversation and dialog.

Figure 8: Microsoft is a strong proponent of Agile, with deep support integrated into Visual Studio

Success Factors

In spite of a plethora of software process management tools, backed by methodologies that are formulated by the world’s leading process experts, software projects continue to fail. Why is this so?

Through our experience with dozens of tools on thousands of man hours of projects, Exia has identified five factors critical for the success of software intended for the management of software projects. These factors are flow, communicative transparency, process guidance, visualizations and integration. It is our belief that available tools consistently fail in on or more of these factors.

We elaborate on these five factors below.

Flow

Current thinking around lightweight, Agile processes rest on principles that contradict those found in much software development. Principles such as flow, communicative transparency, responsibility for quality and accountability are based on the acknowledgement that software is written by people. In contrast to the deterministic approach which proposes that if you could only enforce the perfect process, you would succeed no matter how talented or experienced the development team, Agile emphasizes the novelty of much of the software development process (1).

Modern software process nearly universally embraces the notion of shorter release cycles. The principle of flow states that, all other things being equal, it is more valuable to deliver smaller batches of functionality more frequently than bigger batches of functionality less frequently(1). While tools have evolved to support the individual activities of software development, these tools are clumsy, complex, and unable to efficiently support frequent transition between activities dictated by an emphasis on flow rather than large batch development.

Flow is valuable for all software development. As agile software development pushes the principle of flow further and faster, annual releases become quarterly releases, then monthly, weekly and even daily.

One consequence of increased flow is an increase in the proportion of time spent transitioning between activities. Even if the proportion of time spent on, for instance, analysis versus coding stays the same, the proportion of time spent switching contexts rises because the number of transitions rises from tens to hundreds and even thousands.

To enable efficiency in such a process, the priority of tools must shift from supporting the efficiency of a given activity to the degree of efficiency in switching between activities (1).

Consider the following example. Using cards on a wall for planning has many disadvantages—the cards are isolated in a single location, they are impermanent, they can be lost or damaged. However on thing well supported by a card wall is transitions. A pair can be in the middle of coding(/analyzing/designing/testing), discover a new requirement out of scope for the current cycle, jot it on a card, and get back to coding without interrupting their development flow—elapsed time ten seconds.

Figure 9: As many disadvantages as a card wall has, it has the great advantage of fast transitions

Software that is to be successful for managing Agile projects must keep the speed of transitions in mind. This is a primary reason why web based applications such as Mingle™ don’t work for modern software project management. They fail to recognize that a three second latency for actions is simply too much. Three seconds times a thousand actions a day represents nearly an hour in lost time. Worse, a three second latency is enough for a developer to lose concentration.

In summary, successful applications for agile process management must have highly responsive interfaces that support very rapid context switching, thus ensuring that project flow is unimpeded.

Communicative Transparency

Every qualitative change of an order of magnitude creates a qualitative change. Thus, the change from the 10 KPH of a horse to the 100 KPH of a car didn’t just result in faster transportation, it (eventually) changed peoples’ attitudes towards transportation, and the role mobility played in their lives.

In going from annual deployments to monthly deployments to daily builds we encounter two orders of magnitude of quantitative change. Agile software development, then is going to be qualitatively different. Programmers need to accept primary responsibility for the quality of their work. To do this efficiently requires a way for everyone to find out what everyone else is doing. However updating systems to keep everyone informed all the time requires an inordinate amount of input. At some point everyone would theoretically be spending all their time reporting the progress they would be making if they weren’t spending all their time reporting progress.

Stated another way, as more people are added to a software project, the number of possible pathways of communication increases exponentially. Keeping everyone updated becomes a significant impediment to progress Figure 10

Figure 10: Productivity is not linearly related to the number of developers but instead increases and then decreases, because the communication challenge increases exponentially.

This effect was first postulated by Fred Brooks in The Mythical Man Month (2), and has come to be known as Brooks’ Law.

Successful tools for modern software engineering need to focus carefully on the communication challenge. They need to be able to convey information about status, work, progress and activities without manual input, or where manual input is required, that input must be very quick and easy. They need to make this information available in a format that is digestible at a glance, and which enables people viewing the information to quickly achieve a level of confidence in their cognizance of the information presented, such that they can make a decision with confidence, and move on to their next task. We call this quality communicative transparency.

A key factor in communicative transparency is narrowing the scope of information to be conveyed. Instead of conveying too much information, the software needs to scope the information as narrowly as possible. For instance a decision about the next iteration is easier to make than a decision about all the iterations in the project.

Process Guidance

The modern automobile is a wonder of hidden complexity. It performs an intricate array of functions at high speed on demand. Yet its primary interface is exceedingly simple: two pedals and one wheel.

By contrast, a single screen from a typical software project management system has 30 buttons and menus (see Figure 2 for example).

The automobile has other dials and buttons, but the grouping and size of controls makes it clear they are secondary to the primary ones. As soon as you sit in a car, it’s abundantly clear that the 18″ steering wheel is more important than the radio control.

Software project management systems in general do a poor job of providing an interface that manages, hides, and supports interaction with, the underlying complexity of the tools. The result is that decisions about what to do on a given screen or form take too long. This slows down the ability to rapidly context switch and results in frustration and confusion.

Project management systems need to incorporate guidance. That is, they need to adapt in response to where the user is in the software development workflow by encouraging some actions over others. To follow our car analogy, it used to be possible to put a car into reverse when moving forward. Nowadays most cars prevent it. Cars are harder to turn sharply when moving fast than they are when moving slowly. After a car has been moving for a minute, a seatbelt light encourages the user to buckle up. These are three different types of guidance which are each applicable to the particular need. Software for project management needs to be made to work in a similar way, preventing what must be prevented, discouraging actions that are less valuable and encouraging actions that are more valuable.

Figure 11: Software that tries to convey too much information only causes confusion and creates more of a communication gap (A), since the project manager cannot speak the language of the developers. Instead software should distill information into meaningful process guidance (B).

Figure 12 shows a fragment of an Agile process workflow. Following this example, a process management application could be aware of the position of the project in the workflow, and show different options to the user when the workflow is in the Develop Overall Model stage than when the workflow is in the Build Feature List stage, rather than confusing the user with all possible options at all times.

Figure 12: Agile workflow fragment

In summary, successful tools for software project management need to incorporate various levels and types of guidance that lead the project manager and users along the path of best practice while shielding them from unnecessary complexity.

Visualizations

Project management systems are useless unless they’re understood. Since software project management is inherently complex, understanding the management process is not easy. The vast majority of software process management systems fail to provide intuitive and easy to understand interfaces to their functionality.

Software process management systems need to incorporate familiar interfaces to reduce learning curves, increase usability, and promote adoption.

Figure 13: Software needs to model reality (A), not present unfamiliar interfaces (B)

Looking at the example of Figure 14, for instance, it’s clear that someone who’s used to using manual sticky notes on a board will be more comfortable with the software solution on the left below than the software on the right below, even though the functionality may be equivalent. The solution on the left presents a simple, easier to understand, and more intuitive interface.

Figure 14: Equal functionality software systems can present radically different levels of familiarity

Figure 15 illustrates a visualization of a single agile iteration. Iteration days are numbered sequentially next to the calendar dates. The phases of the iteration are shown as block arrows and each is colored separately. The overlap of phases is indicated with sloped lines. The inputs and outputs are clearly shown. The iteration work is shown at the bottom. Overall the graphic uses familiar metaphors of calendars, arrows folders and sticky notes. Event though this particular graphic is relatively complex and conveys a lot of information, it uses the user still has a very clear idea of what has to be done in the iteration and when, because of the use of clear metaphors, colors and graphics.

Figure 15: Visualization of a single iteration

Software project management systems therefore need to incorporate intuitive graphical user interfaces that enable the users to grasp and utilize the full functionality available, without becoming confused.

Integration

Microsoft is investing heavily in development of powerful tools for software process management. These tools support automated testing, automated build and deployment, source control, bug tracking and server based work item management.

While these tools are a necessary part of the software development lifecycle, they focus heavily on the development and testing side of the equation. There is little strategy to bridge the integration gap between the project management team and the development team.

In fact, the array of new tools and techniques for developers and testers coming out of the Microsoft camp, while very useful for developers, only serves to widen the gap of understanding between the development team and management, by presenting management with new and unfamiliar decision points, each of which results in a to-do item that takes time to resolve, and whose resolution requires specialized technical knowledge that is often held only by the technical team.

A project manager is understandably reluctant to adopt new tools which load up his in-basket and leave his decision making capability subject to information supplied by the technical lead (Figure 16).

Figure 16: Project managers are reluctant to adopt new tools which flood their in-box with confusing decisions (A). Software management tools need to provide sufficient integration that these decisions become done items automatically.

What management needs is tools that provide deep integration points with the development cycle, and that process the incoming information against the process guidance to translate the flood of incoming decisions into completed actions. This will have the effect of reducing the load on the project manager, rather than increasing it.

Software Development Objectives

Exia has identified five factors critical to the success of software process management tools, which we described in the section Success Factors on page 8. Our experience leads us to believe there is no tool or set of tools currently on the market that satisfies these factors. The continued failure rate of software projects, the rapid growth of the project management tools market, and our knowledge of our own customer base convinces us that there would be a significant demand for a tool that satisfies these success factors.

Our objective is to build and market such a tool.

Tool Characteristics

The tool will possess characteristics that satisfy each of the five critical success factors. These characteristics are outlined below:

Tool Prototype

We have already begun work on a PLAY tool prototype. Preliminary screen shots are illustrated below.

The requirements screen (Figure 17) captures user requirements in a hierarchical format. The prototype already demonstrates some of the key capabilities, such as the ability for the user to just click and type, and have changes saved automatically (Figure 18). This ease of use promotes project flow.

Figure 17: Requirements screen prototype

Figure 18: The system allows the user to just click and type. Changes are saved automatically, encouraging project flow

Figure 19 shows a prototype features screen. It mirrors a real whiteboard format, and when complete will allow drag and drop and touch operations analogous to real whiteboard operations with real sticky notes.

To edit sticky notes, the user just clicks and types. There are no interruptions required to click save buttons or to start editing. If the text gets long, the text boxes automatically expand to accommodate. If the text is very long, the note itself expands (Figure 20). These types of usability details ensure project flow by enabling rapid context switching, ensuring there is no loss of concentration, and promoting rapid learning curve.

Figure 19: Features screen prototype

Figure 20: Text blocks automatically accommodate the text typed into them. The user just types.

Figure 21 shows an alternative view of the requirements. This view presents requirements as a tree, and can be expanded and collapsed at any node. The entire requirements can be scrolled and viewed must as it will look when printed. Unique to this view is that it still enables full editing (Figure 22). The user can therefore edit in whatever view is most comfortable. The preference of edit view will often change depending on the user’s mind set, and it is important to accommodate this by offering different ways to accomplish the task.

Figure 21: Requirements alternate view prototype

Figure 22: The user can click and edit just like in the other views. Having a choice of ways to accomplish the same task is important for promoting flow and usability.

Figure 23 shows the print preview prototype. Print preview is available with a single click, making it quick and easy for the user to review the final output.

Figure 23: Requirements print preview prototype

Research and Development work plan

We have described the background of the proposed project in terms of the need and the failure of process management tools to meet that need. We have outlined the five success factors critical for successful tooling of software process management. We have described our objectives in terms of building a tool that incorporates features enabling these critical factors and have shown prototype screens of the tool that is currently under way in our labs.

Development of this tool will require substantial supporting R&D (Figure 24). This section describes the underlying research and development required to develop the tool features.

Figure 24: PLAY product development comprises the five success factors. Development of the required features requires supporting R&D.

The subsections follow a consistent format. Each section has a title for the technology area of research. Following this is a brief description of the technology. Finally a list of the features that are supported by the R&D is presented, and for each feature, the required R&D is described.

Windows Presentation Foundation (WPF) and XML Application Markup Language (XAML)

The Windows Presentation Foundation (or WPF) is a graphical subsystem for rendering user interfaces in Windows-based applications. WPF, initially released as part of .NET Framework 3.0, is another step in Microsoft’s evolving rich client strategy. Designed to remove dependencies on the aging GDI subsystem, WPF is built on DirectX, which provides hardware acceleration and enables modern UI features like transparency, gradients and transforms. WPF provides a consistent programming model for building applications and provides a clear separation between the user interface and the business logic.(5)

WPF also offers a new markup language, known as XAML, which is an alternate means for defining UI elements and relationships with other UI elements. A WPF application can be deployed on the desktop or hosted in a web browser. It also enables rich control, design, and development of the visual aspects of Windows programs. It aims to unify a number of application services: user interface, 2D and 3D drawing, fixed and adaptive documents, advanced typography, vector graphics, raster graphics, animation, data binding, audio and video.

Enabling Flow—Very rapid response, latency consistently below 0.3 seconds.

R&D: Research is required to find ways to utilize graphics capability to ensure fast response is achieved. We need to understand how to avoid overloading the system, for instance with large graphics or saved images. We will research which UI elements provide fast response and which don’t, and research how to customize slower UI elements. As an example, the treeview control must implement lazy loading, wherein only visible tree elements are shown and invisible ones are not shown until the treeview node is expanded (Figure 25) (6).

Figure 25: Treeview lazy loading and caching for performance

Research is required to investigate how to avoid duplication of overhead when implementing multiple graphics, such as a visual “sticky note” element (Figure 26)

Figure 26: Avoiding graphic overhead duplication

Research is required for multi-threading in WPF, specifically how to use it to update behind the scenes so that all saves happen automatically. The user just clicks and types and does not have to click a “save” or “cancel” button (7).

Research into “undo/redo” frameworks is required, resulting in implementation of an undo framework (Figure 27). An undo/redo framework is required if we are going to implement automatic saving of changes, otherwise automatic saving will be unacceptable. Research is required to investigate alternatives such as versioning of data structures (8).

Figure 27: Undo/Redo Framework

Enabling Process Guidance—Built-in workflow capability.

R&D: Research is required to find ways to link Microsoft Windows Workflow engine to WPF GUI, in order to present an easy to understand, intuitive view of the current workflow status to the user. Research is required to determine how to ensure that it is immediately obvious from looking at the workflow designer screens exactly how the application behaves in any given situation. Research is required to determine how to replace all switch and spaghetti logic with workflow patterns, since the application will require a great deal of workflow logic as it matures, and spaghetti logic techniques will result in application degradation and maintenance problems over time. Research is required to determine how to ensure agility so that new states, transitions and conditional logic can be added as the application grows without having to re-plumb the models or views. Research is required to determine how to save and restore workflow state to/from a remote server for multiple thousands of users, with minimal latency (9).

Enabling Visualizations

R&D: Research is required to implement WPF graphics, animation and media capabilities and support for multimedia, vector graphics, animation and content composition. It will be necessary to investigate ways to utilize these for optimal presentation of graphics that represent real world entities. (10). Research is required on how to use WPF styles to achieve consistent styling across the application. Research is required to determine the best ways to achieve and benefit from the scaling capabilities of WPF to achieve resolution independence thereby ensuring the application doesn’t get smaller and smaller as screen resolutions continue to increase, since we anticipate the application having a long lifespan. Research is required on how to use data and content templates to consistently apply object information to UI elements. Research is required on how to use animation triggers to design the UI to respond to actions with minimal code. Research is required to understand and develop appropriate drag and drop frameworks to ensure as much interactivity with the mouse as possible (11).

Cloud Services

A cloud service is a runtime environment in the internet “cloud”, hosted and maintained by a third party provider such as Microsoft or Amazon, that delivers on-demand compute, storage and automated systems management.

The deployment model for the PLAY product will use cloud services for hosting the data. PLAY will be a Smart Client that will download to the user’s desktop and install from a web site link. Once installed, the application will access data over the web using web services over HTTP. The data will reside on a cloud service, such as the Microsoft Azure cloud service.

Enabling Flow—Anytime/anywhere access, easy to try, easy to use, easy to subscribe and unsubscribe

R&D: Use of a cloud service will ensure 100% uptime, scalability, performance, data security and loss prevention. The user will be able to download the client application, install and connect to their data from anywhere in the world. Calls to retrieve and save data will be transmitted via the web service using specific data transport protocols. Research will be required to determine the best cloud service to use and how to develop a client side library to encapsulate cloud service API calls. If the Microsoft Azure service is used, research will be required to determine how best to integrate with it, since the Azure service is a different type of cloud service than conventional hosted server providers such as Amazon Web Services. With conventional cloud services, infrastructure is required which must be installed by the consumer (Figure 28). With the Azure service, access to data is direct via HTTP.

Figure 28: Amazon vs. Azure–choosing the correct cloud service

The Azure environment will eventually be a full operating platform, like Windows and gaming, will have all the usual elements – a programming model, service architectures, language environments, an SDK— plus a couple of pieces unique to the cloud, including a desktop simulation environment and a Web service gate to the cloud(12).

Given the extent of Microsoft’s vision for Azure, we believe there is significant potential to simplify the programming for advanced features of PLAY that promote flow through the use of Azure, and our cloud R&D will identify and exploit this potential.

Enabling Process Guidance—Workflow capability, access control and user rights, information filtering

R&D: Research will be required to determine how to make the best use of hosted workflow capabilities on the cloud service. (For more on Workflow see the next section.) By hosting workflow on the cloud we can potentially gain a simpler API, the benefit of not having to set up and configure the service, 100% uptime, performance and scalability.

Research will be required to understand how best to leverage the cloud hosted access control in Azure. We will need to investigate how to use REST and ATOM to implement management rules, how to implement the security token service (STS), and which of the protocols we can best leverage (CardSpace, certificate, Live, etc.) (13).

Research will be required to understand how best to filter information based on user rights, workflow state and activity.

Workflow

R&D: Research will be required to understand how best to integrate with and leverage Microsoft Windows Workflow service. This service, combined with the Azure cloud service, provides a set of activities optimized for sending, receiving, and manipulating HTTP and Service Bus messages, a set of hosted tools to deploy, manage and track the execution of workflow instances, and a set of management API’s. It is unclear what is the current state of the tools and APIs and how much work will be required to efficiently implement workflow, to provide appropriate hooks and extension points, how to version workflow when the application version changes, and how to integrate workflow visually with the application graphics.

Team System Integration

Microsoft Team System is Microsoft’s flagship application development software suite for the .NET platform. It incorporates all versions of the integrated development environment (IDE) Visual Studio, and a server based team collaboration platform that combines source control, work item tracking, build management, process guidance and business intelligence.

R&D: Integrating with Team System will be a key success factor for the PLAY tool. Because Team System is vast and complex, research and development will be required to determine the best way to integrate.

Research will be required to investigate existing Team System front end process initiatives and understand how they work. From this we will gain ideas, insights, concepts that will help form our general guidance as to starting points and direction. Once such example is Apprise, a comprehensive solution for software development process and project management, created by HP on top of Team System (14). With this in mind, we will begin the key focus of our Team System integration R&D, which will be to find ways to distill complex information into formats that can be understood and actioned easily, but without losing vital detail (Figure 29).

Figure 29: The main focus of the Team System integration R&D will be how to distill complex information into formats that can be understood and actioned easily, but without losing vital detail

Research will be required to determine the best ways to exchange information, whether and how much of the information exchange will be one way or two way (Figure 30).

Figure 30: Research will be required to determine the best ways to exchange information, and whether the information flow should be one or two way

Research will be required to achieve two way integration. On the one hand a change to an item in the PLAY tool should have a rippling effect, either triggering a notice of a change required to Team System, or executing the change directly. On the other hand a change in Team System should trigger either a direct change in the PLAY tool, or provide a notice of change. The nature of these rippling effects is not known and will requires investigation and trial and error.

Over the past five years Exia has developed a suite of process management tools using RoboHelp and Visio. These tools, some of which are illustrated in Figure 31, Figure 33 and Figure 34, are not integrated with the software development process in any automated way. Hence they require constant manual updating, which is time consuming, prone to error, and requires extensive training. We plan to investigate if and how Team System can be extended to support integration of these tools using the PLAY front end.

The Process Flow diagram (Figure 31) illustrates the flow of document and work items through Team System. Research is required to determine how best to integrate this diagram into PLAY. For instance, clicking an icon might cause the PLAY application to jump to the corresponding location where the assets are managed in PLAY (Figure 32). This concept might enable project managers who are not very familiar with how Team System works to better understand its flow, and how the items in Team System related to items in PLAY. Another possibility might be to highlight, color, or otherwise place indicators at strategic points on the diagram to indicate underlying status. For instance if Risks have not been adequately addressed, the Risks icon might turn orange or red. Investigation, trial and error and user feedback will help determine the usefulness of these ideas.

Figure 31: Process flow diagram illustrates the flow of document and work items through Team System.

Figure 32: Clicking an icon might take the user to the area where the associated assets are managed in PLAY

Figure 33 shows a fragment of the Scenarios Mapping diagram. This diagram represents an alternative view of the scenarios, or features, that comprise the system functionality. The benefit of this view is that it shows all features at a glance, and enables features to be assigned to personas, or users. This diagram could theoretically be generated on the fly in the PLAY tool from scenario information. Investigation, trial and error is required to determine the variability and usefulness of such a feature.

Figure 33: Scenarios mapping diagram

The iteration diagram shown in Figure 34 has been proven very useful in managing iterative development, as it helps keep developers on track and focused on the task at hand. This diagram could be generated from the iteration planning information in the PLAY tool. Research and development is needed to determine whether this would be useful, how and which parts to generate, and whether it should have any level of interactivity after it is generated.

Figure 34: Iteration diagram

Process Evolution

A solid software engineering tool need to be backed by a well constructed process. Yet the Agile process is still evolving, and a key part of Agile is that it is just that—agile—and therefore to a certain extent it defies being constrained by rigid steps. Instead of a linear, waterfall type process, Agile is defined by cycles and iterations (Figure 35). Governance focuses on achieving and maintaining levels of performance in five areas.

Figure 35: Agile is defined by cycles and iterations, not rigid steps. Governance focuses on achieving and maintaining levels of performance in five areas.

The problem is mapping this iterative structure to a system in such a way that progress can be tracked. Research and development will be required to derive the appropriate process, and to house the process in Windows Workflow Foundation. As the process is being derived, the appropriate structures for storing process information will need to be derived as well. The derivation of the process will proceed largely by trial and error with feedback.

Figure 36: Process derivation

A key strategy of the PLAY tool is to deliver guidance. The MSF for Agile Software Development Guidance offers guidance of a sort, but falls short of the type of guidance we envision in PLAY.

For example, a project manager using the MSF guidance might land on the How to: Get started with MSF page. This page instructs him to start with his role page. He goes to the role page for Project Manager and finds a step by step. There are three headings: Guide Iteration, Plan an Iteration and Guide Project. He clicks Guide Project and lands on the Guide Project activities page (Figure 37).

Figure 37: Guide Project activities

This looks promising. From here he clicks the Review Objectives page, and lands on the Review Objectives detail page shown in Figure 38.

Figure 38: Review Objectives activity

Here is where the usefulness of this guidance starts to break down. The project manager reads the entry criteria “The scenario list is up to date and prioritized”, and “the quality of service requirement list is up to date and prioritized. From here he has little idea what to do. The following questions arise:

• Can I start this task if those prerequisites are not completed?
• What if the prerequisites are partially completed?
• I know this is an iterative process, should I not just have those prerequisites completed for an iteration, or for a couple of iterations? Or does this mean for the whole project?
• How do I create the scenario list and the quality of service requirements list? How do I prioritize them?

The project manager then reads: “Evaluate the market or user expectations for the product. Consider competitors in the market, previous systems, or business needs.”

What should he do? Is a report required or expected? How should this information be conveyed to the team? How important is it? What format should it be in?

Detailed process research is required to determine how to solve each of these problems. Consider the improved screen in Figure 39.

Figure 39: Improved “Review Objectives” workflow screen

This screen illustrates the following improvements:

1. It tells the user where he is in the current project.
2. It tells the user how far out he should be thinking for the current review, in this case out to iteration 3. This scopes the thinking and lets the user focus on the next few weeks, rather than having to worry about the entire project scope at once.
3. It shows at a glance with checkmarks whether the work is done or not. If the checkmarks are all checked, it’s done.
4. It provides buttons to perform the activities that are required in order to complete the tasks.

This screen goes a long way to solving the problem of what the user should be doing on this page, allowing him to focus quickly on what needs to be done and access the tools to complete it.

Similar R&D, trial and error is required to elaborate in detail the portions of the Agile process that are to be included in the PLAY release, and to determine the best way to provide guidance, help and automation to ensure project flow.

Summary

This section has elaborated in detail on the R&D effort required to support the PLAY project, the areas of primary R&D focus being WPF/XAML, cloud services, workflow, team system integration and process evolution.

Improvements Sought

Our goal in this project is to create the basis for a revolutionary software tool that offers the promise of a significant leap forward in functionality and usefulness over existing tools. The area in which we seek improvement are those identified in the Success Factors section on page 8; flow, transparency, process guidance, visualizations and automation.

Flow

We determined that successful applications for agile process management must have highly responsive interfaces that support very rapid context switching, thus ensuring that project flow is unimpeded.

We seek the following improvements related to the flow of the application:

• An average improvement of 50% reduction in the time it takes for a user to context switch while using the PLAY application over traditional applications.
• A decrease in 50% in the time taken to enter the average piece of information. Information that before took ten seconds should take five seconds or less.
• A decrease in 30% in the time taken to grasp the meaning of a screen, form, or subsection, and to act on it.

Communicative Transparency

We seek the following improvements related to the ability of the application to convey information and in doing so contribute to transparency of the development effort and improvement in project velocity:

• Reduction by 40% of the time taken updating statuses, yet with the same or greater status information reported to other team members.
• Reduction by 20% of the average time taken to absorb the information on a screen and have enough confidence in that information and the underlying principles that the information is conveying that action can be confidently taken.
• Reduction by 20% of the amount of communication churn, that is repeating the same communication over and over again. Churn will be reduced by action being taken rather than being postponed due to information loss or confusion.
• Increase of 10% in average project velocity.

Process Guidance

We seek to improve the process guidance component of software engineering so that the following improvements are realized:

• Recognizable ability of the software to influence correct decisions over incorrect ones, for instance warning when too much time is allocated for software development and not enough for testing.
• 20% improvement in the success rate of software projects due to improved guidance.

Visualizations

We seek the following qualitative and quantitative improvements from visualizations in the PLAY system:

• Discernable improvement in the ability of users to quickly grasp and understand information presented.
• 20% reduction in the amount of time required to understand a screen or form and take action, compared to similar systems.
• Ability to run large format presentations using touch screen technology.
• A subjective “hands-on” feel that makes users feel as though they are manipulating real world objects directly, more than when using comparative systems.

Integration

We seek the following improvements related to integration of the PLAY system with other systems, in particular with Microsoft Team System:

• Subjectively measured “high” level of improvement in a project manager’s ability to view, comprehend and take action on information generated in Team System, when compared to the project manager using the Team System reports and output directly.
• 20% reduction in the time spent updating Team System information.

Domain of Science

Software Engineering

The domain of science in which the improvements are sought is software engineering. Software engineering is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software(15). It applies both computer science and engineering principles and practices to the creation, operation, and maintenance of software systems. Software Engineering requires expertise in data management, design and algorithm paradigms, programming languages, and human-computer interfaces, and it requires knowledge of digital hardware systems. It also demands an understanding of and appreciation for systematic design processes, non-functional system properties, and large integrated systems.

History of Software Engineering

The term software engineering first appeared in the late 1950s and early 1960s. Programmers have always known about civil, electrical, and computer engineering and debated what engineering might mean for software.

The NATO Science Committee sponsored two conferences[2] on software engineering in 1968 and 1969, which gave the field its initial boost. Many believe these conferences marked the official start of the profession of software engineering(16).

From 1965 to 1985 the science of software engineering advanced rapidly, spurred by the “software crisis” of the 60s, 70s and 80s, during which many of the challenges of software development were initially identified. Many software projects ran over budget and behind schedule. Others caused more severe damage such as property damage or loss of life.

It is interesting to note that the start of the software crisis is coincidental with the inception of the software industry itself. It seems that software development has always been in crisis, and this is an indication of the difficulty of the task.

The 1980s were marked by the search for the “silver bullet” that would solve the software crisis. Among the approaches explored were tools, discipline, formal methods, process and professionalism. The failure of any one of these to significantly improve software success rates was captured most eloquently by Fred Brooks in his 1987 paper No Silver Bullet:

“I believe the hard part of building software to be the specification, design, and testing of this conceptual construct, not the labor of representing it and testing the fidelity of the representation. We still make syntax errors, to be sure; but they are fuzz compared with the conceptual errors in most systems. If this is true, building software will always be hard. There is inherently no silver bullet.” (17)

Brooks’ major contribution to software engineering, however, was the book “The Mythical Man-Month”, which he postulated what has since become known as Brooks Law; Adding manpower to a software project which is late makes it later. (2)

In the 1990s the rapid rise of the Internet resulted in great demand for new and more complex software, which strained the industry’s ability to deliver it and intensified the search for improved software engineering methods. Methodologies that advocated iterative and incremental development became popular, with the most successful of these being the Rational Unified Process (RUP).

Current trends in software engineering are toward progressively more lightweight, flexible methodologies collectively termed Agile. These methodologies are based on iterative development and promote a self-disciplined project management process, teamwork and accountability. Other directions in which software engineering is developing include:

• better frameworks and tools resulting in greater predictability in the construction process;
• better integration of modeling tools with the software itself, reducing time spent updating code from models and vice versa; and
• software factories which encapsulate best practices for software product line and enable development of software products from archetypes by drag and drop and customization.

Software engineering is undergoing an effort to define its boundary and content, the seminal example being IEEE’s Software Engineering Body of Knowledge (18).

Summary

This white paper has attempted to shed some light on problems with current software process management tools and to suggest lines of thinking which might result in improved tools. We have outlined the project background in terms of the need for improved tools, the current state of software project management, current tools and success factors that we and others have identified to try to address the shortfalls. We have outlined out planned R&D work plan for the development of a tool that can help manage the software development process. We have enumerated the improvements that we hope to achieve.

These are exciting times for the software industry. The level of quality of software manufacturing tools has reached the point that quality repeatability is possible. Thus software is now entering what we term the age of engineering. Now it is only a matter of time and innovation before the challenge of managing software projects with engineering quality and discipline is finally met.

It is our hope that our Exia’s experience in software engineering, combined with our knowledge of current tools such as Microsoft Team System, positions us to lead the charge in the creation of unique and leading edge tools that can play a significant part in meeting this challenge.

We invite interested contributors to contact us and participate in this research and development effort. Please visit our web site at www.exiaprocess.com

Bibliography

1. Beck, Kent et al. Manifesto for Agile Software Development. Agile Manifesto. [Online] Jan 20, 2001. [Cited: June 05, 2008.] http://www.agilemanifesto.org.

2. Beck, Kent. Microsoft Download Center. Microsoft. [Online] 6 27, 2008. [Cited: June 06, 2008.] http://www.microsoft.com/downloads/details.aspx?FamilyId=AE7E07E8-0872-47C4-B1E7-2C1DE7FACF96&displaylang=en.

3. Brooks, Frederick P.
The Mythical Man-Month: Essays on Software Engineering, Anniversary Edition (2nd Edition). s.l. : Addison-Wesley, 1975. ISBN: 0-201-00650-2 (1975 ed.), 0-201-83595-9 (1995 ed.).

4. Bhuta, Falguni. The Business of Technology. Red Herring. [Online] August 31, 2007. [Cited: April 30, 2009.] http://www.redherring.com/Home/22701.

5. Waxer, Cindy. Project management comes of age. Small Business. [Online] CNN Money, April 06, 2009. [Cited: March 10, 2009.] http://money.cnn.com/2009/04/03/smallbusiness/task_masters.fsb/.

6. Wikipedia. Windows Presentation Foundation. Wikipedia. [Online] July 02, 2009. [Cited: July 05, 2009.] http://en.wikipedia.org/wiki/Windows_Presentation_Foundation.

7. Smith, Josh. Simplifying the WPF TreeView by Using the ViewModel Pattern. CodeProject. [Online] May 22, 2009. [Cited: July 01, 2009.] http://www.codeproject.com/KB/WPF/TreeViewWithViewModel.aspx.

8. Podila, Pavan. WPF Multithreading with BackgroundWorker . [Online] August 01, 2007. [Cited: July 05, 2009.] http://pavanpodila.spaces.live.com/blog/cns!9C9E888164859398!246.entry.

9. Osenkov, Kirill. A Simple Undo/Redo Framework. [Online] June 29, 2009. [Cited: July 06, 2009.] http://blogs.msdn.com/kirillosenkov/archive/2009/06/29/new-codeplex-project-a-simple-undo-redo-framework.aspx.

10. Hoffman, Kevin. Orchestrating a WPF App using MVC and Windows Workflow Foundation. [Online] April 03, 2007. [Cited: July 05, 2009.] http://dotnetaddict.dotnetdevelopersjournal.com/wforchestration.htm.

11. Microsoft Corporation. WPF Graphics, Animation, and Media Overview. [Online] 03 01, 2009. [Cited: 07 05, 2009.] http://msdn.microsoft.com/en-us/library/ms742562.aspx.

12. Dunant, Ron. WPF Drag and Drop Smorgasbord. [Online] June 10, 2009. [Cited: July 06, 2009.] http://www.codeproject.com/KB/WPF/WpfDragAndDropSmorgasbord.aspx.

13. Foley, Mary Jo. Will Azure Evolve Into Microsoft’s Fifth Major Platform? [Online] April 01, 2009. [Cited: July 07, 2009.] http://redmondmag.com/articles/2009/04/01/will-azure-evolve-into-microsofts-fifth-major-platform.aspx.

14. Microsoft Corporation. Access Control Service. [Online] Jan 10, 2009. [Cited: March 15, 2009.] http://msdn.microsoft.com/en-us/library/dd582744.aspx.

15. —. HP Builds Comprehensive Solution for Software Development Project and Process Management . [Online] October 10, 2005. [Cited: Dec 12, 2008.] http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?CaseStudyID=49127.

16. Wikipedia. Software engineering. Wikipedia. [Online] March 20, 2009. [Cited: July 06, 2009.] http://en.wikipedia.org/wiki/Software_engineering.

17. —. History of Software Engineering. Wikipedia. [Online] Wikipedia, Jan 20, 2009. [Cited: July 08, 2009.] http://en.wikipedia.org/wiki/History_of_software_engineering.

18. Brooks, Frederick P. No Silver Bullet: Essence and Accidents of Software Engineering. Computer. 1987, Vol. XX, 4.

19. IEEE. SWEBOK – Guide to the Software Engineering Body of Knowledge. IEEE Computer Society. [Online] August 23, 2004. [Cited: 12 01, 2008.] http://www2.computer.org/portal/web/swebok/html/contentsch1#ch1.

This blog briefly previews the Document Library feature of the Exia Process, which is nearing readiness for release.

The idea behind the project library is first that it can store any type of project documentation, (high level design, detailed design, user doc, architecture etc. ) but second (and more important) that it can tie together details from all the work items. You can now add detail to any work item in a predictable way, and as you write the detail, it’s all gathered together for you in the library.

Example 1: Objective – Ensure that High Level Design is captured for all work.

• Create a library book called High Level Design.

  

• When writing a user story, click Add to add documentation.
• Select “High Level Design” from the drop-down. The title defaults to [book name] – [user story name], in this case “High Level Design – Usability Details”

  

• Double-click the documentation line you just added, and edit the document page.

  

• The document page you just created is added to the High Level Design book. Later on you can go to this book and arrange the pages, and add new pages. You will always be able to access this page from the user story where it was created. Vice versa you will be able to navigate from the book page to the user story.
• You can continue to add High Level Design pages to any work item. As work evolves, these will get added while the developers or project managers are thinking about them. Later this will make it much easier to pull together a complete high level design document.

Example 2. Objective – Capture Use Cases

The team has decided that user stories are not sufficiently in-depth. User stories will be used, but each user story must also be expanded into a formal Use Case.

1. Create a Use Cases book in the library.
2. Create a User Story.
3. Add Documentation.
4. Select the Use Cases book.

   

5. Double-click the documentation line to edit the use case for the user story.

   

The use case is automatically added to the Use Cases book. All use cases can be created this way, and at any time they can be organized in the Use Cases book, and supporting documentation and other use cases can be added to the book as well.

Of course multiple use cases can be added per work item.

Example 3. Objective – Capture security considerations for a highly sensitive application.

The team is working on a highly sensitive application. Security considerations are a primary concern.

1. Create a Security book in the library.
2. Mandate that every user story, bug, task and issue must have at least one documentation entry added for the Security book.
3. Later, the application could check for this condition automatically.