On bullets and beans

I'm not a logistician. I don't even play one on TV. But I did stay at a Holiday Inn Express last night, so here goes...

With thanks to my Chief Ottawa Correspondent for pointing me in the right direction, I had a chance to read Chapter 2 of the Auditor General's report to the House of Commons, which deals specifically with "Support for Overseas Deployments - National Defence (pdf)." As usual with these reports, I found it insightful, plainspoken, and to the point.

In most organizations - especially big ones with very specialized sub-units like, say, an Auditor General - if you're a hammer, every problem coincidentally turns out to be a nail. So it was refreshing to see a watchdog organization say, in essence, that while there are issues with operational support to the Afghan mission, overall it seems to be working: for the most part, the pointy end is getting the in-theatre support it needs.

National Defence has been able to deliver to troops its equipment and supplies that they need to do the job in Afghanistan. While we did note that commanders have expressed concerns over some supply chain shortcomings, we found no reports of supply chain problems that had significantly affected operations. This is largely because the high level of dedication and hard work of Canadian Forces personnel enabled them to deliver the needed support.

For those who don't speak governmental bureaucratese, that's pretty high praise for the men and women on the ground doing the grunt work of getting people and materiel into the hands of the troops at the tip of the spear. In fact, the report gets even more blunt later:

While there is little information available to quantifiably assess the supply chain’s performance, our observation is that results are often achieved more by military personnel’s concerted efforts than by the system’s design.

In other words, the supply folks are doing whatever it takes to make things happen. Good on them for that.

Of course, that doesn't mean there aren't problems. And of course, the AG lays out just what those problems are. I won't go into each in detail, as you might as well read the whole document at that point. In fact, if you're interested in this sort of stuff, I'd recommend you do just that.

But for the rest of you, I'll just touch on some highlights. First, the design of our supply system isn't the best at supporting deployed operations. The realities on the ground don't match the theoretical model.

I'll give you an example. The base at Kandahar runs on the same stocking system as bases back in Canada: hold 30 days of stock on everything you need at your own warehouses. The system is set to automatically reorder certain stock when levels dip below a trigger point. For some stock, however, the reordering is a manual affair, with priority codes inputted by the folks in theatre.

The first problem is that it often takes more than 30 days to deliver stock to KAF.

That's partly because of transport shortages: during the audit period, which as far as I can gather was roughly 2006-2007, we chartered two to three commercial aircraft per week to move materiel, while we used our own aircraft only once per week. It's partly because we haven't always got the right equipment to properly load the cargo for transport:

The proportion of some types of materiel-handling equipment that is in working order has been unacceptably low. It was reported that some materiel-handling equipment deteriorated so badly that it needed immediate replacement. Logistic reports from Kandahar showed that the shortage and lack of working equipment was affecting the ability to support day-to-day operations. Logistic reports also stated that the unavailability of working materiel-handling equipment at CFB Trenton, used to load and unload supply planes, delayed the return of various types of equipment shipped by these planes for repair.

It's partly because of lack of availability of some supplies from the supplier, like spare parts for the Nyalas. Interesting passage:

...shortages of spare parts from the manufacturer contributed to the armoured wheeled vehicle known as the Nyala being sent back to Canada. The three Nyala in Kandahar were out of service for months and subsequently returned to Canada as the parts could no longer be acquired, due in part to obsolescence.

That's like getting a much needed role player at the trade deadline, and then having him go down with a career-ending injury two games into the schedule with his new team. Ugh.

It's partly because our purchasing models aren't always spot on with their projections, like how the CF underestimated the need for spare parts for our heavy logistics trucks, which were going through parts far quicker than expected because of the rough operating conditions and the tempo of operations. And it's partly because ordering more supplies isn't always easy, as the example cited below illustrates:

Soldiers in Afghanistan are protected by body armour, helmets, and goggles, which has saved lives by protecting vital areas. Consequently, the hospital treats a large number of serious injuries to arms and legs. Between February and July 2007, 281 orthopaedic surgeries were performed, representing half of all the surgeries during that time. Surgeons needed large numbers of orthopaedic surgical pins to fix and correct the bone fractures on wounded soldiers and civilians.

Orthopaedic pins are expensive and since orders for pins cost more than the $5,000 procurement approval limit, any purchases of more pins had to go through the government contracting process. In April 2006, National Defence recognized the high demand for these items and initiated negotiations for a standing offer of agreement to ensure a quick supply. By October 2006, an agreement was signed with a vendor, with a purchase limit of $40,000. However, by this time the Canadian Forces medical system needed over $400,000 in surgical items from the vendor and it was not until November 2006, seven months after the need was identified, that stocks were shipped to Kandahar.

In other words, there are a whole pile of reasons why the base at KAF should be stocking more than 30 days worth of supplies.

The second problem is that we can't always keep track of the supplies once they arrive:

2.29 We found that in many cases when supplies seemed to be arriving late, the goods had already been received but the mission in Afghanistan was unaware. The mission has a database that lets users know if items are in transit. It cannot, however, provide information on when supplies are likely to arrive. The Department is shipping 85 tonnes of goods weekly by contracted airlift to the mission, and we found that it is able to track goods while in transit, but can lose sight of supplies once they arrive at their destination. In Kandahar, it can be difficult for supply technicians who run the supply warehouses that receive shipments to know what has arrived or where to find it.

2.30 National Defence has an established system of stock numbers, package tracking numbers, and waybills to know what the requested items are and where they are in the supply chain, but technicians receiving the planeloads of supplies are required to deal with shipments manually. Supply technicians must physically find the goods, check their condition and quantity, and write down that they have been received. Therefore, supply technicians in Kandahar may not actually know that some supplies have arrived until they find the boxes and put them on the shelves. As a result, some items may be reordered or even forgotten, which can result in surplus stocks, unnecessary delays, or wasted shipments. National Defence regularly reviews its holdings in Afghanistan to maintain control and sends in rotation support assistance teams to conduct inventory counts. During the last review of its inventory holdings, the team found that the Department had lost track of a significant amount of inventory — over $7 million of items could not be located — but it found another $6.6 million of items that were not listed as part of the mission’s holdings. The mission may have been unaware that it had the items, which included spare parts to repair equipment. [my emphasis]

That's almost $14M of lost stuff. Now, I suspect that those numbers, while sounding huge, represent a very small percentage of the overall materiel shipped to KAF. But it's still astounding, especially for an organization that is a stickler for accountability (Pretty much everyone I know in uniform has said some variation of this: "Hey, be careful with that - I signed for it!").

We lose stuff partly because we're understaffed, partly because we're way behind on the technology we're using, and partly because we're not even using the technology we have to our best advantage:

2.32 The Department has recognized that it has a shortage of supply personnel to deal with the volume of goods arriving in Kandahar and, from time to time, it sends in technical assistance teams to help clear backlogs. It also increased the number of contract personnel it uses to perform supply functions. However, the technicians are using a bar coding system in a manual way, not in the electronic way they are trained to do in Canada. This slows down the receipt of goods in Afghanistan. [again, my emphasis]

How badly was the support end of the mission understaffed at the beginning? Check out this graphic:


From May 2006 to July 2007, the support staff were tripled. And we're still a bit short, as I understand it (It would be interesting to know the number of TAV's required on the support side in the last twelve months. But I'm not going to eat up somebody's time at CEFCOM asking for a compilation of that data. Of course, if anyone reading has the info at their fingertips...).

So, given the realities of the supply system leading into KAF, how do the soldiers, sailors, and airmen respond? They do whatever it takes, of course. They pull out the gun tape and baling wire, and get 'er done. Just look at this fleet reliability graphic:


That's just plain impressive, especially once you know about the supply chain difficulties. How do they do it?

Well, for one thing, they cannibalize equipment when necessary:

2.44 Although undesirable, maintenance personnel are permitted, when necessary, to borrow parts from one piece of equipment in order to make timely repairs to another. Our audit found that borrowing was necessary on some critical fleets in order to keep enough equipment available to meet mission needs.

...

2.52 In 2003, before beginning operations in Kandahar, the Department introduced tactical uninhabited aerial vehicles into operations at Kabul. About 85 flights were flown until June 2004 when the vehicles were no longer sustainable due to crashes and failures. In February 2006, the vehicles were reintroduced into operations at Kandahar, resulting in a number of challenges, including crashes, frequent flight cancellations due to equipment problems, and shortages of spare parts, with long lead times to reorder. By September 2006, the Department recognized that the vehicles’ sustainability was again at risk. A number of actions have been taken, including extensive borrowing of parts from other aircraft to keep the aircraft operating. Our review of flight data from February to August 2007 shows that Canadian Forces operators, maintenance personnel, and supply personnel managed to keep the fleet operational but serviceability and parts availability issues persisted. [Babbler's note: I suspect that if you included the cost of crashes, the Sperwers would be among the most expensive aircraft the CF owns, based on cost per hour of flight time]

...

2.54 The Canadian Forces in Kandahar keeps a reserve fleet of equipment, known as operational stock, to be used when the number of available vehicles declines. For example, a LAV III light armoured vehicle damaged beyond local repair could be replaced by a vehicle from the reserve if one is available. This reserve stock has also been used in Kandahar as a pool through which equipment is rotated in order to undergo upgrades, such as the installation of additional armour and protection, without affecting the number of pieces of equipment available for operations.[my emphasis]

For another, they game the system to some degree. That is to say, they figured out how to manipulate the system so that the really important stuff got where it needed to be. Here's what the AG saw:

2.25 We observed that requests are identified in the system with both required delivery date and priority. Some 47 percent of requests for items from the main depot in Canada are coded as either operationally critical or essential. As well, 81 percent of these orders ask for delivery within 10 days—a cycle that has been very difficult to achieve. The supply chain is expected to use the priority code in conjunction with the required delivery date to determine the most appropriate and economic mode of transportation to meet the required delivery date. For example, less urgent items can be shipped by sea and road, which is less costly than by air. However, most items were needed within a time frame that only shipment by air could accommodate.

2.26 We observed that while some supplies needed on a priority basis arrived quickly, often supplies listed as operationally critical or essential did not. We wanted to know if there was an impact on operations if supplies that were described as operationally critical or essential arrived late. We could find no reports of late supplies seriously affecting operations. However, we did review reports where delays in receiving parts reduced the number of military vehicles and equipment ready to be put into service.

2.27 We also found that the system sometimes recorded required delivery dates that were the same as the date the request was made, thus making timely delivery impossible. We found that these unrealistic required delivery dates, combined with high priority coding, resulted in special handling and increased transportation costs. The Department is currently redrafting its instructions to provide better guidance on requesting and shipping supplies and to try to minimize high priority demands.

2.28 We found that higher priority items were generally transported faster than those of lower priority. Nevertheless, we noted that when items were identified as particularly high priority by the user in-theatre, those responsible for getting them to Kandahar did not rely on the supply chain, but instead made phone calls back to Canada to ensure that items were flagged and shipped right away. [my emphasis, yet again]

If you can decode the bureaucratese, what that means is that people order everything "high-priority," knowing full well it won't arrive in the time frame they're asking for, but also knowing that if they don't, their item will get bumped for someone else's "high-priority" item. When it's really, REALLY mission-critical, phone calls get made to flag stuff informally outside the system. There's nothing like the supply-chain mafia - mostly sergeants and higher - to get what you need when you need it. Scroungers are doing what scroungers have always done. There's a reason why it's such a valued unofficial skill in the military.

And this manipulation is serious enough that the department is redrafting its guidance on what constitutes "high-priority." Which means a bunch of supply officers are going to be leaning on a bunch of senior NCO's, who will then squeeze a bunch of junior ranks to not game the system. Which will work for awhile, until a crisis hits, and they need to start gaming the system to get what they need.

Which is how it should be. That tension between following the system, and going outside it to accomplish the mission has always been with us, and will always be with us. Leaders know how to balance the demands of each. And our military is chock-full of leaders from private to general.

So, what is DND doing to address these concerns, you ask?

2.33 National Defence should review its practices for tracking materiel once the materiel has arrived to ensure the arrival and storage is accurately recorded in a timely manner. (2.29–2.32)

Agreed. The Department has initiated a project that should address this issue. The first phase of the Asset Visibility Project is to put in place a Canadian radio frequency identification capability to track consignments moving to and from Afghanistan. This capability will remove our current dependency on U.S. support and is expected to be in place by December 2008.

The second phase of the project is to develop an interim capability that will provide visibility of items in transit both within Canada and abroad. This interim capability is expected to be in place by 2009.

The third phase is to develop a comprehensive capability that will provide real time, or near-real time, visibility of assets throughout their life cycle. In other words, a system that provides visibility from the time the Department takes custody of an asset until the time we dispose of that asset.

2.36 National Defence should continue its efforts to develop the performance measurement of its supply system, including assessing whether supplies are received in a timely manner appropriate to priority and need.(2.34–2.35)

Agreed. The Department has initiated a performance measurement system for the supply chain. To date, 15 key performance indicators have been developed to monitor system performance using data from the Canadian Forces Supply System. At the conclusion of a user trial, this initiative is set to roll out to a number of supply management organizations in the fall of 2008.

The 15 key performance indicators developed so far focus on a variety of key measures within the Department’s Supply System, such as average cycle times, depot stock reactivation rates, requisition volumes, and requisition satisfaction.

Work is continuing to prioritize and further develop performance indicators based on the Department’s strategic direction and the perceived needs of the supply chain community.

2.53 National Defence should review how it establishes stock levels for the parts it needs to keep existing and new equipment operating at expected serviceability targets, with a view to obtaining and delivering parts to users in a timely way. It should take into account changes as wear and tear to equipment increases on deployments, as better information becomes available on the performance of new equipment, or as the level of support from the manufacturers changes. (2.50–2.52)

Agreed. The Department is now implementing a Distribution Resource Planning tool, complete with a modelling capability that is expected to significantly improve the ongoing identification of inventory requirements. This electronic tool will improve the Canadian Forces Supply System by addressing a significant weakness in inventory rationalization and optimization. It will also provide the necessary information to make complex decisions regarding what to repair, and what to buy, in what quantities, and where to position it. Rollout is expected to begin in the spring of 2008.

The rapid introduction of new equipment to a theatre can be mitigated by the early identification of an initial provisioning plan for spare parts. The initial provisioning plan will be entered into the electronic Distribution Resource Planning tool and will be monitored by comparing actual usage to the estimated requirements identified in the plan. Within a few months, the Distribution Resource Planning tool will identify the optimal forecast methodologies and algorithms to use for the equipment in question. At that point, the initial provisioning plan can be subsumed into the normal day-to-day inventory management of the Distribution Resource Planning tool.

Which goes to show that DND can say all the right things when required. Whether it actually does them...we'll see. I'm especially interested in finding out how the Asset Visibility Project and the Distribution Resource Planning tool work out. I hope the department is drawing on expertise from both within and outside the military community for this, since private firms like FedEx, WalMart, etc have a pretty good grip on ordering, stocking, moving, and tracking stuff. And since every efficiency they achieve makes them money, they're pretty motivated as well. Can't hurt asking.

I asked a fellow I know what he thought of the problems, the recommendations, and the proposed solutions. He has pretty extensive supply-chain experience both in uniform and on civvie-street. He was less than impressed:

...What is shocking is that they are still struggling with these issues in 2008. Read up on the CFSSU (took 15 years to produce 1970's functionality and last I heard was trying to roll out bar codes... I wonder how long it will take them to get to RFID), MASIS ($165M DRP system and still going... I think the navy might, might be using this - incredibly its an ERP that doesn't integrate with other ERP systems i.e. FMAS), or MASOP (what about performance measures?) initiatives if you are interested in more background on this. Fuck. Btw, the US learned about asset visibility in a war zone during the 80's with the invasion of Grenada. They couldn't find anything in the mountain of sea containers so just reordered everything from the US again. They went to smart bar codes first and then RFID years ago.

I can understand the frustration, especially since this is an area of expertise and he can see the potential gains the CF system is missing out on.

But here's the thing: what they're doing is working. Oh, it's not working as well as it could, and it's likely not working as cheaply as it could, but it's working. To their credit, the auditors recognized this:

We have reviewed the audit reports for the supply operations of the United States and British forces and they show problems similar to those experienced by the Canadian Forces. This suggests, given the long experience of both those countries in overseas combat missions, that some of the issues may be inevitable for military operations with long supply chains.

So the CF supply system will keep plugging along, gaining efficiencies where it can, taking two steps forward for every one step back, and in the end, getting the job done so that the operators can get on with the mission at hand.

Update: Un-freakin'-believable. A CP story at the CTV News site covers many of the same positives and negatives from the AG report that I do, but comes up with this headline:

Supply lines to Afghanistan rife with problems: AG

You know what's "rife with problems"? The state of journalism in this country, when the nation's top watchdog gives a pat on the back to DND support staff, yet the news organizations come up with a spintastic headline like that.

Boo. Hiss.