I came to this publication the month after my predecessor added Boardwalk Public Partners (NYSE: BWP) to the Conservative Portfolio, with a “safety rating” of 3.

Boardwalk didn’t earn the maximum rating of 4 at the time only because it hadn’t raised its payout in a year. Whereas it checked all the other boxes, boasting a preponderance of fixed-fee contracts, healthy distribution coverage and, one assumes, a debt load that also didn’t throw up any red flags.

Sure management had warned in the preceding annual report that a greater-than-usual proportion of its transmission contracts would be expiring in 2013, in a much less hospitable environment for gas shippers than prevailed when these were signed. But how do you adjust for that when you’re married to a binary checklist that awards either a point, or none, based on a rote percentage of fixed contracts?

You don’t. You call Boardwalk a 3 and note its ambitious growth plans, insider buying and the welcome skepticism of most analysts on Wall Street.

Over in the Growth Portfolio, Eagle Rock Energy Partners (Nasdaq: EROC) had also been deemed worthy of the second-highest safety grade. Despite its high exposure to the slumping price of natural gas and NGLs, we must assume Eagle Rock earned its own third point because it had raised its payout by a penny per unit not quite a year earlier, and rules are rules.

So that was where things stood almost exactly a year ago, and you may not need to guess how events progressed. Eagle Rock was dumped from the portfolio in May, cut its distribution by 30 percent in October and lost 47 percent of its value between the last two April Fool’s days.

Boardwalk has depreciated 54 percent over the same span, with much of the decline coming on Feb. 10 after the partnership cut its payout by more than 80 percent. Those fixed contracts expired all right, and it turned out there wasn’t nearly as much demand to ship natural gas from Texas to the south-central region now amply and cheaply supplied by the nearby Marcellus. Go figure.

We recommended selling Boardwalk in November, and it wasn’t because the safety rating told us to. It was because we had listened to the substance and the tone of management’s increasingly open frustration on the earnings conference calls, and monitored the daily market verdict of the unit price – the same inexact evidence that led us to bail on Eagle Rock six months earlier.

So in the end, the vaunted safety rating system proved no help at all, and no match for educated inference.

Yet no subscriber has asked us to cut out the gimmicky scoring and focus on the qualitative reporting that has proven more useful in the recent past. Instead, when a new publishing system started to randomly and surreptitiously award safety ratings of 6 and 8, we heard immediately from investors for whom this system remains a core part of the decision-making process.

In many ways, that’s understandable. We live in a complex and highly uncertain world, and nothing reassures like simple integer derived from demonstratively thorough due diligence. Unfortunately, that integer is just a number rather than a predictive model for the full range of possible outcomes. But if you believe otherwise, a 3 or a 4 can look almost like a promise that nothing will ever go wrong. It is a marvelous marketing aid if you’re selling peace of mind and doughy optimism.

We aim to provide more, namely sophisticated and nuanced analysis. And while we won’t abruptly abandon the safety ratings so many have relied on for so long, we’ll be looking to make them more realistic and useful, at the minimum.

If we’re going to assign proprietary numbers on top of actionable buy, sell or hold recommendations, price targets and a wealth of industry statistics, we need to make sure that our grades inform instead of oversimplifying and confusing.

So this is a request for feedback and for help with change that, it seems to me, is clearly needed.

It’s needed not just because the current scoring system failed to flag the two biggest MLP blowups of the past year. It’s also needed because the composite score is only as good as its components, which are only as good as the metrics they measure, which are only as pure as the intentions of a given general partner.

Our safety ratings rely on declarations of distributable cash flow, which permits estimates of the coverage ratio. Yet distributable cash flow can be manipulated by classifying maintenance spending as growth investments, and in other ways.

Distribution increases would be a better measure of a partnership’s health if they couldn’t be financed by debt and equity issuance. Fixed-fee contracts inevitably carry expiration dates.

Reducing all of these quibbles and variables to a single integer is not necessarily the path to greater understanding of the risks.

Defenders of the status quo might argue that, while the safety ratings may have struck out once or twice, they continue to reliably predict the steady distribution increases by most of the partnerships we recommend.

Unfortunately, this is less a point in favor of business as usual than an acknowledgement that, at the moment, the vast majority of MLPs are at little to no risk of an imminent cut in distributions. That’s why partnerships that have upset those calculations have been punished so dramatically. The more mundane risk is simply that the price action negates the return from distributions.

That being the case perhaps our future, improved safety ratings should consider such longer-term risk factors as the strategic position of the assets and the potential misalignment of the interests of general and limited partners, for MLPs paying out incentive distribution rights. Perhaps they need to be more subjective, or more descriptive. Or maybe the three portfolios (Conservative, Growth and Aggressive) already adequately differentiate degrees of risk.

We don’t have all the answers yet, but it’s past time to have this discussion. What do you think?