Scores and stars

Posted June 28, 2012 @ 2:29 p.m.
Posted By Kevin Fishbain

This is Part Three of PFW’s look inside the helmet industry as it pertains to the NFL. In Part Three, associate editor Kevin Fishbain writes about how helmets get approved by NOCSAE, and the new star-rating system created at Virginia Tech. 

In ongoing efforts to improve helmets to help protect the heads of football players, manufacturers must put their helmets through rigorous tests to get approved, and now there is a star-rating system to help consumers distinguish different helmet brands.

Let’s start with the test every helmet must pass before it is used in an NFL game, the NOCSAE (National Operating Committee on Standards for Athletic Equipment) test, which has been the standard for helmets since 1973.

The NOCSAE test uses a helmet on a headform and involves 27 impacts around the helmet through drop tests, including two at high temperatures (115 degrees), 14 impacts at high speed (17.94 feet/second) and 11 at low and medium speeds (11.34 to 16.04 ft/sec).

“We have changed our standards to make it clear the quality level we require before a helmet can be certified is essentially a non-defect standard,” said Mike Oliver, executive director of NOCSAE about how the test has evolved of late. “It’s a tightening up of the quality control. It requires more testing and more equipment.”

“The NOCSAE standard is very rigorous. If it meets it, it’s a very good helmet,” said Thad Ide, senior vice president of research and product development at Riddell. Ide said that Riddell also does its own tests on helmets.

“They are creating the standards by which we manufacture equipment,” said Glenn Beckman, director of marketing communications at Schutt Sports. “They have the scientific and medical minds able to take empirical data and create the standards by which we need to make our equipment.”

Helmets must score less than 1200 SI (severity index) after its test to pass the standard. The scores helmets receive are not correlated to their effectiveness in preventing concussions. The SI score is simply a pass/fail.

“A helmet that passes (the test) at 800 is no better than one that passes at 600. We don’t do those comparisons,” Oliver said.

Helmet manufacturers are also prohibited from using scores comparatively. One can make inferences, though, from how scores have changed over the years.

“As a generic statement, I don’t know if it means helmets are better or worse in preventing concussions, but 15, 20 years ago, helmets would have an average SI score of 800. That was considered pretty good,” Oliver said. “The trend we’ve seen with larger helmets and more padding, SI numbers have, across the board, gotten lower. That doesn’t necessarily mean they’re performing better.”

There are a few things that representatives from helmet manufacturers pointed out as ways NOCSAE’s test could continue improving to account for other intangibles that can affect the head, such as rotational acceleration (hits that are not head-on), the temperature and low-level hits.

“Even if you talk to people at NOCSAE themselves, they would say one of the major limitations, or opportunities, depending on how you want to look at it, is this concept of rotational acceleration, the spinning of the head,” said Vin Ferrara, founder of Xenith. “The machine does not test for that. The drop test is a straight, linear drop of a headform. (Rotational acceleration) appears to be important in terms of potential risk of concussion.”

Oliver said NOCSAE is working to add different accelerations to its helmet tests.

“We underwent a grant last year to refine that process more and generate those kinds of accelerations in a helmet test,” he said. “A large bulk of recent research has looked at what the threshold would be in limiting rotational accelerations to protect against concussions.”

Beckman said that Schutt has “had our differences” with NOCSAE, and one of them is accounting for temperature.

“We believe the temperature variable needs to play more of a role in verification of a helmet. The true measure of a helmet occurs on the field, not in the testing laboratory,” Beckman said.

As detailed in Part Two, Schutt’s helmets put an emphasis on temperature, especially with their TPU cushioning, which they believe fares better in extreme temperatures than other helmets’ padding.

Oliver explained how NOCSAE does factor in temperature.

“We require helmets have to be conditioned to 115 degrees and impacted at a high temperature. We can make sure there is no significant performance difference at high or low temperatures,” he said. But most of the test still takes place at a constant temperature.

“A standard, by definition, means that all helmets are tested identically. A helmet tested in New York can be compared to one tested in California? We do know that doesn’t represent the environment where they’re being used, but that’s how standards are done,” Oliver said.

Low-level hits have increasingly become a bigger part of the NOCSAE test. These are the hits that linemen may experience on every snap, and one hit may not do much, but repeated occurrences can do serious damage.

“About three years ago, we incorporated a low-level threshold for those impacts,” Oliver said. Starting in February of this year, the NOCSAE test added 24-inch drop impacts (other drops in the test occur at 60, 48 and 36 inches) with a pass/fail of 300 SI, which raised the total number of impacts for helmets to 27.

The score of 1200, though, has not proven to be a difficult score for a helmet to achieve, which leads to one of the discussions taking place in the helmet industry.

“All helmets greatly exceed the NOCSAE standard. They score well below the minimums,” Beckman said. As Oliver pointed out, NOCSAE is seeing "average scores in the 500s."

While people who spoke to PFW for this story agreed that the standard hasn’t weeded out many helmets, the consensus was that hard data is needed to support a decrease of the score.

“We can change the SI number, but you wouldn’t see a difference in injury risk. I’m pretty confident you wouldn’t see a change in concussion numbers,” Oliver said. “If we could add a pass/fail component with science behind it, that’s a change that we would be happy to make.

Though Beckman discussed some of Schutt’s concerns with the NOCSAE standard, he agreed that there hasn’t been concrete reason for a change in the minimum SI score.

“To NOCSAE’s credit, they’ve been very steadfast in their beliefs that they shouldn’t change things for the sake of changing,” he said.

There are concerns over the calls for changing the score, as it could actually prove to be dangerous.

“I worry with pressure coming from a couple congressmen pushing for a change. If we just change it in a way that’s not evidenced-based, we really run the risk of causing more harm than good,” said Kevin Guskiwiecz, Professor and Director of The University of North Carolina’s Center for the Study of Retired Athletes.

The intent of the NOCSAE standard is to make sure helmets will prevent brain injuries, such as skull fractures or brain bleeds.

“We know that standard of 1200 does a very good job. 99.9 percent of the time, that helmet’s going to do what it’s certified to do, which is prevent catastrophic brain injuries,” Guskiewiecz said.

It’s not just lowering the score, but changing the test and the score to try to better prevent concussions could cause problems as well. Guskiwiecz presented a hypothetical of sorts to explain what could happen by designing the test to be geared more toward concussion prevention.

“If we went and changed that standard, and as a result, helmets need to create a softer shell on the outside with more absorbent padding on the inside, we meet the standard. At the end of the day, the new standard reduces the concussion by 25 percent, but increases your risk of catastrophic injury to five percent,” he said. “To me, the cost-benefit ratio is not good. I’m not putting my kid in a helmet that might reduce his (chances of a) concussion by 25 percent, an injury that typically solves in 10 days, but the risk of brain bleed that results in brain damage (increases).

“That’s the issue they’re faced with. Some people say pick a new standard and apply it, and I’m thinking to myself, gosh, I don’t want my kid wearing that helmet.”

Like many of the moving parts within the helmet industry, the NOCSAE standard is always evolving to help make helmets safer, but vast changes cannot be made to the standard, nor would it necessarily be beneficial. NFLPA medical director Thom Mayer explained his view on it with the following phrase:

“You go by the book until you’ve written the book,” he said. “They are the standard against which helmet testing has been judged. There has not been another standard better than NOCSAE.”

• • • • •

Cardinals QB Kevin Kolb recently made a helmet switch — he went from the Schutt Air XP helmet to the Riddell Revolution Speed. As ESPN’s Mike Sando pointed out, using Virginia Tech’s star-rating system for helmets, Kolb went from the three-star Schutt helmet to a five-star Riddell helmet.

Dr. Stefan Duma created the star-rating system at Virginia Tech, and it is in its second year. A five-star helmet, the best possible rating, provides a reduction in concussion risk, according to Duma’s study, which is available for consumers as they search for the safest helmet.

“We wanted to develop something for consumers that gave them a very in-depth, biomechanical analysis, but was also very independent,” Duma said, pointing out that his group does not accept money from manufacturers, or the NFL.

When the first ratings came out in 2011, the Riddell VSR4 received the second-lowest rating, but it was also the most common helmet in the NFL during the ’10 season. Riddell advised teams to stop using the helmet.

In the 2012 study, the Riddell 360, Riddell Revolution Speed and Rawlings Quantum Plus received five stars. The four-star helmets included the Schutt ION 4D, Schutt DNA Pro +, Rawlings Impulse, Xenith X1, Riddell Revolution, Rawlings Quantum and Riddell Revolution IQ. The Schutt Air XP and Xenith X2 helmets received three stars, and the Schutt Air Advantage got two stars.

“We suggest and encourage people to wear four- and five-star helmets,” Duma said, adding that there are more choices in those categories following the 2012 ratings.

The idea for a helmet-rating system sparked for Duma after his group did automobile safety research on seat belts and air bags, and the star-rating system is common in the automobile industry.

The testing protocol is based on eight years of research on helmets that had accelerometers, or sensors that measure acceleration. Duma’s group used that data found on the field when testing the helmets in a laboratory, performing 120 drop tests, and then formed a STAR value, looking at the exposure multiplied by the risk.

On its website, the star-rating system includes a “Frequently Asked Questions" section, which outlines some of the limitations to the study, but some in the helmet industry are very concerned about the results that consumers now have at their disposal.

Glenn Beckman, director of marketing communications at Schutt Sports:

“The good is, very talented researchers, like Duma and his team at Virginia Tech, are bringing their talents and expertise to the problem. Our big issue is we believe that the while the data collected is great, the conclusions are premature and very possibly dangerous.

“Our position has always been that no company can fully claim any helmet can significantly reduce the risk of concussions. There are too many variables that go into the causation of concussions, and variables helmets have virtually no effect on.

“Even though our helmets are highly rated, we believe the false sense of security generated by these kinds of simplistic ratings are dangerous.”

Duma responded to the “dangerous” comment.

“The challenge for the person who says it’s dangerous — how is having better padding and reducing head acceleration dangerous? How is that dangerous? No one can prove that,” he said. “The helmet with better padding and better acceleration reduces your risk. I am very comfortable saying that.”

Beckman and Vin Ferrara, founder of Xenith helmets, may be competitors in the helmet industry, but they agree on having strong concerns regarding the Virginia Tech study.

Here is Ferrara’s take on Duma’s study:

“I have huge problems with that study on multiple levels,” he said.

“While I support the idea for new concepts for helmet testing, once you take that next leap from espousing a concept to doing press releases and star ratings and this over-simplified expression of something that’s highly complicated, I think it’s reckless and hugely misleading.

“They are ignoring the risk of repeated, low-energy impacts over time. Most people with medical credibility would tell you it likely leads up to this CTE (Chronic Traumatic Encephalopathy).

“Of all the things I’ve seen in the helmet world in eight plus years, I have the biggest issue with the Virginia Tech study. I could go on for a long, long time. I said it last year when our helmet was rated quiet well, according to (Duma). This year, he said our X2 is three stars and our X1 is four stars. I have data from the world’s leading independent certification lab comparing the two and the data is virtually identical.

“The comparison between brands is simply a speculative leap and it’s misleading to people. It’s as misleading as anything I’ve ever seen by any sales rep.

“This is going to be an evolved process, an evolution of technologies. There definitely needs to be new methods for testing helmets, I’m all for it. But what I’m really against is the leap (Duma) took.”

Duma responded to the comment about low-level hits.

“People miss that part of the story quite often. We’re the only group that includes all the low-impact hits,” he said. “Our STAR rating incorporates 1,000 impacts, a bulk of those are low — 12 inches, 24 inches — and we multiply that by a large exposure. The low-impact, repetitive hits you get a lot, we account for that very strongly.”

Virginia Tech uses a NOCSAE drop tower system when it tests helmets to determine STAR values, and, as mentioned in their FAQs, “All helmets will still be tested in accordance with NOCSAE, which includes additional impact locations, including a theoretically worst-case scenario random location.”

But NOCSAE has expressed its concerns about the Virginia Tech study as well. Mike Oliver, NOCSAE’s executive director:

“The first time they came out with the report, we were very concerned about that program for lots of reasons.

“Our biggest concern — if you read the Virginia Tech study, Stefan makes it pretty clear — the helmet comparisons are only reliable if you’re talking about an adult, large helmet worn by a starting collegiate football player. Beyond that, there’s no applicability.”

In a statement released in June, NOCSAE had stronger words regarding the Virginia Tech STAR ratings and warnings for consumers.

“NOCSAE advocates ongoing research on athletic safety in order to gain a deeper understanding of protecting athletes from concussions,” it reads. “However, NOCSAE does not recommend that parents and athletes form decisions on the safest and most effective equipment based on any single individual data point, rating, or measurement, including the Virginia Tech STAR football helmet rating system.

“As stated in the STAR FAQ on the Virginia Tech website, the STAR rating system is 'a theoretical calculation that is based on a probabilistic analysis of impact exposure and injury risk.'  This theoretical calculation relies on a single head acceleration criteria to predict the probability of a concussion, which is a complex event involving different and changing forces, linear and rotational accelerations, impact duration, player concussion history, overall health, helmet fit, and potentially even genetics.”

Riddell’s Thad Ide, senior vice president of research and product development, did not have similar criticism for Duma’s work.

“Even Dr. Duma would tell you that system of ranking helmets will evolve as more data becomes available. I think it’s a great first step,” Ide said. “It’s really a first of its kind way of evaluating helmets’ performance and protective performance. It’s not the end. He’ll continue to evolve that.”

Duma pointed out that his group posted similar results to a study by the NFL.

“The NFL did a study in January and we looked at both. The results were very consistent with our results. The helmets that did the best we found did the best,” he said.

The NFL and NFLPA sponsor a laboratory study to “assess the performance of helmets worn by NFL players, and shared those results with teams to share with players, team athletic trainers, physicians and equipment managers and the NFLPA, including their medical director, to share with players.”

A league spokesman said in regards to Duma’s study, “We support any institution’s efforts to study this issue, but we don’t have a stance on the Virginia Tech study.”

In the science world, Duma cannot be a stranger to criticism, but he said that the helmet manufacturers had opportunities to speak up before the first ratings were released in 2011.

“People are forgetting that three months before we released (the study), we invited all the manufacturers to Virginia Tech. We showed them the STAR formula, how we got it, how it was derived, the equation and what we were going to do,” he said. “We gave everyone three months to give us feedback. Not a single manufacturer gave us any feedback.

“We did not blindside anyone, quite the opposite. We wanted to work with them. We gave them three months to give us feedback. We have an open policy — any manufacturer can come to our lab and test for free.”

When asked if he was surprised by the harsh criticism from a couple helmet manufacturers, Duma responded, “It didn’t surprise me. I’m sure they wish they were five-star.”

Tomorrow: The future of the 'official' helmet

Read Part Two: The big four

Read Part One: Protecting the yolk

Follow Kevin Fishbain on Twitter